Associate Data Engineer

Overview

As an Associate Data Engineer, you will build and maintain scalable data pipelines in Databricks to deliver reliable datasets for analytics, reporting, and machine learning.

What you'll do

• Develop, test, and maintain data pipelines using Databricks, PySpark, and Python.
• Ingest, transform, and process structured and semi-structured data from multiple sources.
• Build scalable ETL/ELT workflows for analytics, reporting, and machine learning use cases.
• Perform data cleansing, validation, and quality checks.
• Optimize Spark jobs and Databricks notebooks for performance, reliability, and cost efficiency.
• Monitor scheduled jobs and workflows to ensure timely and successful data delivery.

What you'll need

• 2-6 years of experience in a relevant field.
• Hands-on experience with Python for data processing, scripting, and automation.
• Strong working knowledge of PySpark and distributed data processing concepts.
• Proven hands-on experience using Databricks (notebooks, clusters, jobs, workflows, Delta tables, and performance optimization).
• Working knowledge of SQL for querying, transforming, and validating data.
• Experience with Delta Lake and lakehouse architecture concepts.

Nice to have

• Exposure to Delta Lake, Unity Catalog, or lakehouse architecture.
• Familiarity with cloud-based data platforms, preferably AWS, Azure, or GCP.
• Exposure to machine learning workflows using MLflow, scikit-learn, or similar tools.
• Experience with Tableau, Power BI, or similar data visualization tools.
• Experience operating in an Agile/Scrum environment.

Details

• Work in an Agile or iterative development environment.