C

AWS Data Engineer

Capco
NewPosted today

LOCATION

Bengaluru · Onsite

EXPERIENCE

6 - 12 Years

SKILLS REQUIRED

AWSDatabricksApache SparkPySparkData PipelinesData QualityCI/CDTerraformData GovernanceDelta Lake

Job description

Overview

Capco is hiring an AWS Data Engineer to design, build and maintain scalable Databricks data pipelines and cloud-based data platforms for banking, financial services and energy clients.

What you'll do

  • Design, develop, test and maintain scalable data pipelines and transformation workflows using Databricks.
  • Build and optimise data processing solutions using Apache Spark, PySpark and SQL.
  • Develop batch and, where required, streaming data pipelines to support enterprise data processing requirements.
  • Implement robust data ingestion, cleansing, transformation and enrichment processes across multiple data sources.
  • Develop and maintain Delta Lake tables, applying appropriate data modelling, partitioning and optimisation strategies.
  • Implement reusable, modular and maintainable data engineering frameworks and components.
  • Contribute to the design and implementation of cloud-based data platforms and lakehouse architectures, applying Medallion Architecture (Bronze, Silver, Gold) where appropriate.
  • Optimise Spark workloads through efficient transformations, query tuning, cluster configuration and resource management.
  • Monitor and troubleshoot data processing workloads, identifying performance bottlenecks and implementing improvements.
  • Design and develop data ingestion and integration pipelines connecting structured, semi-structured and unstructured data sources.
  • Implement data orchestration workflows using Databricks Workflows, AWS or equivalent technologies.
  • Develop data validation, reconciliation and quality control processes to ensure data accuracy and consistency.
  • Implement appropriate error handling, logging, monitoring and recovery mechanisms.
  • Support data migration and modernisation initiatives, including migration from legacy data platforms to cloud-based solutions.
  • Apply software engineering best practices, including modular design, code reviews, automated testing and documentation.
  • Develop and maintain CI/CD pipelines for deploying Databricks code and configurations across environments.
  • Use Git, Azure DevOps or equivalent tools for source control, collaboration and release management.
  • Implement unit, integration and data quality testing for data engineering solutions.
  • Support automated deployment using Databricks Asset Bundles, Terraform or equivalent tools where appropriate.
  • Troubleshoot and resolve defects across development, testing and production environments.
  • Implement appropriate data access controls, security configurations and governance standards within Databricks.
  • Work with Unity Catalog or equivalent governance capabilities to support data discovery, access management and lineage.
  • Apply data quality, metadata management and data lifecycle principles.
  • Support compliance with organisational security policies and relevant financial services regulatory requirements.
  • Implement monitoring and alerting to improve pipeline reliability and operational resilience.
  • Collaborate with platform, security and architecture teams to ensure solutions meet enterprise standards.
  • Engage with client stakeholders to understand business requirements and translate them into scalable technical solutions.
  • Take ownership of complex technical deliverables or defined data engineering workstreams.
  • Contribute to solution architecture discussions, technical design decisions and implementation planning.
  • Provide technical guidance, code reviews and mentoring to junior engineers and consultants.
  • Communicate technical concepts, delivery progress, risks and dependencies to technical and non-technical stakeholders.
  • Support technical estimation, sprint planning and delivery coordination within Agile teams.

What you'll need

  • Typically 6+ years of experience in data engineering, software engineering or related technology roles, including substantial hands-on experience with Databricks.
  • Strong practical experience developing data pipelines and transformation workloads using Databricks.
  • Advanced proficiency in Python, PySpark and SQL.
  • Strong understanding of Apache Spark architecture, distributed data processing and performance optimisation.
  • Hands-on experience with Delta Lake, including data ingestion, transformation, optimisation and data management.
  • Experience designing and implementing ETL/ELT pipelines for large-scale data processing.
  • Good understanding of lakehouse architecture, data modelling and data warehousing concepts.
  • Strong knowledge and hands-on experience of AWS development in some of the following: Lambda, Glue, Step Functions, IAM roles, Lakeformation, EventBridge, SNS, SQS, EC2, Security Groups, CloudFormation, RDS, DynamoDB, Redshift.
  • Experience with data orchestration tools such as Databricks Workflows or Azure Data Factory.
  • Practical experience implementing data quality checks, error handling, logging and monitoring.
  • Experience using Git and CI/CD pipelines within a data engineering environment.
  • Understanding of data security, access controls and governance principles.
  • Strong troubleshooting and performance-tuning capabilities for Spark and Databricks workloads.
  • Experience working within Agile software delivery environments.
  • Strong analytical thinking and structured problem-solving capabilities.
  • Ability to independently own and deliver complex technical workstreams.
  • Experience contributing to technical design and solution architecture decisions.
  • Strong written and verbal communication skills, including the ability to explain technical concepts to non-technical stakeholders.
  • Ability to engage effectively with client stakeholders and multidisciplinary delivery teams.
  • Experience providing technical guidance and mentoring to junior team members.
  • Ability to identify delivery risks, manage dependencies and recommend appropriate solutions.
  • Strong organisational skills and the ability to manage competing priorities.
  • Proactive, collaborative and client-focused approach to delivery.

Nice to have

  • Experience delivering data engineering solutions within banking, capital markets, insurance or other financial services environments.
  • Databricks Certified Data Engineer Associate or Professional certification.
  • Experience with Unity Catalog, Delta Live Tables / Lakeflow Spark Declarative Pipelines and Databricks SQL.
  • Exposure to structured streaming and real-time data processing using Spark Structured Streaming or Kafka.
  • Experience with infrastructure as code using Terraform or equivalent technologies.
  • Familiarity with Microsoft Fabric, Snowflake, Synapse Analytics or other modern data platforms.
  • Experience with cloud data migration and legacy data platform modernisation programmes.
  • Knowledge of dimensional modelling, Data Vault and enterprise data architecture principles.
  • Familiarity with data observability, metadata management and data lineage solutions.
  • Exposure to cost optimisation and workload management within cloud data platforms.
  • Understanding of data governance and regulatory considerations relevant to financial services.
  • Experience supporting technical proof-of-concepts and evaluating emerging data technologies.
  • Excellent verbal communication skills.
  • Good interpersonal skills.
  • Ability to provide technical leadership for the team.

Details

  • Location: Bangalore, Pune.
  • Experience: 6-12 years.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

AWS Data Engineer