Overview
Capco is hiring an AWS Data Engineer to design, build and maintain scalable Databricks data pipelines and cloud-based data platforms for banking, financial services and energy clients.
What you'll do
- Design, develop, test and maintain scalable data pipelines and transformation workflows using Databricks.
- Build and optimise data processing solutions using Apache Spark, PySpark and SQL.
- Develop batch and, where required, streaming data pipelines to support enterprise data processing requirements.
- Implement robust data ingestion, cleansing, transformation and enrichment processes across multiple data sources.
- Develop and maintain Delta Lake tables, applying appropriate data modelling, partitioning and optimisation strategies.
- Implement reusable, modular and maintainable data engineering frameworks and components.
- Contribute to the design and implementation of cloud-based data platforms and lakehouse architectures, applying Medallion Architecture (Bronze, Silver, Gold) where appropriate.
- Optimise Spark workloads through efficient transformations, query tuning, cluster configuration and resource management.
- Monitor and troubleshoot data processing workloads, identifying performance bottlenecks and implementing improvements.
- Design and develop data ingestion and integration pipelines connecting structured, semi-structured and unstructured data sources.
- Implement data orchestration workflows using Databricks Workflows, AWS or equivalent technologies.
- Develop data validation, reconciliation and quality control processes to ensure data accuracy and consistency.
- Implement appropriate error handling, logging, monitoring and recovery mechanisms.
- Support data migration and modernisation initiatives, including migration from legacy data platforms to cloud-based solutions.
- Apply software engineering best practices, including modular design, code reviews, automated testing and documentation.
- Develop and maintain CI/CD pipelines for deploying Databricks code and configurations across environments.
- Use Git, Azure DevOps or equivalent tools for source control, collaboration and release management.
- Implement unit, integration and data quality testing for data engineering solutions.
- Support automated deployment using Databricks Asset Bundles, Terraform or equivalent tools where appropriate.
- Troubleshoot and resolve defects across development, testing and production environments.
- Implement appropriate data access controls, security configurations and governance standards within Databricks.
- Work with Unity Catalog or equivalent governance capabilities to support data discovery, access management and lineage.
- Apply data quality, metadata management and data lifecycle principles.
- Support compliance with organisational security policies and relevant financial services regulatory requirements.
- Implement monitoring and alerting to improve pipeline reliability and operational resilience.
- Collaborate with platform, security and architecture teams to ensure solutions meet enterprise standards.
- Engage with client stakeholders to understand business requirements and translate them into scalable technical solutions.
- Take ownership of complex technical deliverables or defined data engineering workstreams.
- Contribute to solution architecture discussions, technical design decisions and implementation planning.
- Provide technical guidance, code reviews and mentoring to junior engineers and consultants.
- Communicate technical concepts, delivery progress, risks and dependencies to technical and non-technical stakeholders.
- Support technical estimation, sprint planning and delivery coordination within Agile teams.
What you'll need
- Typically 6+ years of experience in data engineering, software engineering or related technology roles, including substantial hands-on experience with Databricks.
- Strong practical experience developing data pipelines and transformation workloads using Databricks.
- Advanced proficiency in Python, PySpark and SQL.
- Strong understanding of Apache Spark architecture, distributed data processing and performance optimisation.
- Hands-on experience with Delta Lake, including data ingestion, transformation, optimisation and data management.
- Experience designing and implementing ETL/ELT pipelines for large-scale data processing.
- Good understanding of lakehouse architecture, data modelling and data warehousing concepts.
- Strong knowledge and hands-on experience of AWS development in some of the following: Lambda, Glue, Step Functions, IAM roles, Lakeformation, EventBridge, SNS, SQS, EC2, Security Groups, CloudFormation, RDS, DynamoDB, Redshift.
- Experience with data orchestration tools such as Databricks Workflows or Azure Data Factory.
- Practical experience implementing data quality checks, error handling, logging and monitoring.
- Experience using Git and CI/CD pipelines within a data engineering environment.
- Understanding of data security, access controls and governance principles.
- Strong troubleshooting and performance-tuning capabilities for Spark and Databricks workloads.
- Experience working within Agile software delivery environments.
- Strong analytical thinking and structured problem-solving capabilities.
- Ability to independently own and deliver complex technical workstreams.
- Experience contributing to technical design and solution architecture decisions.
- Strong written and verbal communication skills, including the ability to explain technical concepts to non-technical stakeholders.
- Ability to engage effectively with client stakeholders and multidisciplinary delivery teams.
- Experience providing technical guidance and mentoring to junior team members.
- Ability to identify delivery risks, manage dependencies and recommend appropriate solutions.
- Strong organisational skills and the ability to manage competing priorities.
- Proactive, collaborative and client-focused approach to delivery.
Nice to have
- Experience delivering data engineering solutions within banking, capital markets, insurance or other financial services environments.
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience with Unity Catalog, Delta Live Tables / Lakeflow Spark Declarative Pipelines and Databricks SQL.
- Exposure to structured streaming and real-time data processing using Spark Structured Streaming or Kafka.
- Experience with infrastructure as code using Terraform or equivalent technologies.
- Familiarity with Microsoft Fabric, Snowflake, Synapse Analytics or other modern data platforms.
- Experience with cloud data migration and legacy data platform modernisation programmes.
- Knowledge of dimensional modelling, Data Vault and enterprise data architecture principles.
- Familiarity with data observability, metadata management and data lineage solutions.
- Exposure to cost optimisation and workload management within cloud data platforms.
- Understanding of data governance and regulatory considerations relevant to financial services.
- Experience supporting technical proof-of-concepts and evaluating emerging data technologies.
- Excellent verbal communication skills.
- Good interpersonal skills.
- Ability to provide technical leadership for the team.
Details
- Location: Bangalore, Pune.
- Experience: 6-12 years.
Read the full description and apply on the company’s own careers page.