Data Engineer - Python and Azure Data Bricks

Overview

Capco is hiring a Data Engineer with strong Python and Azure Databricks expertise to design, build, and optimize scalable data pipelines and modern data platforms.

What you'll do

• Design, develop, and maintain scalable data pipelines using Python and Azure Databricks.
• Build and optimize ETL/ELT workflows for ingesting, transforming, and processing large datasets.
• Develop data solutions using Apache Spark within the Databricks environment.
• Integrate data from multiple structured and unstructured sources.
• Ensure data quality, reliability, security, and governance across data platforms.
• Implement performance tuning and optimization for large-scale data processing workloads.
• Monitor and troubleshoot production data pipelines and resolve data-related issues.

What you'll need

• 4+ years of experience in Data Engineering.
• Strong hands-on programming experience in Python.
• Expertise in Azure Databricks.
• Expertise in Apache Spark (PySpark).
• Expertise in SQL and Delta Lake.
• Experience with Azure data services including Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), and Azure Synapse Analytics.
• Knowledge of Git, CI/CD pipelines, and DevOps practices.

Details

• Works on building batch and real-time data pipelines.
• Must be able to work in Agile development environments.
• Requires a Bachelor's or Master's degree in a relevant field.