Senior Associate – Databricks Data Engineer, Advisory

Overview

Design, build, and operate scalable, high-performance data pipelines on the Databricks Lakehouse Platform. The role involves hands-on development with Apache Spark, Databricks notebooks, Delta Lake, and cloud-native services, collaborating with analytics, AI/ML, and business teams.

What you'll do

• Design, build, and maintain end-to-end data pipelines using Databricks, PySpark, and Spark SQL.
• Implement batch and incremental data processing using Delta Lake and multi-hop architecture.
• Develop and optimize Databricks notebooks, jobs, and workflows.
• Ingest, transform, and curate large-scale structured and semi-structured datasets.
• Support analytics, reporting, and downstream data consumption use cases.
• Ensure data quality, reliability, lineage, and governance.
• Collaborate with data scientists, analysts, and architects on AI/ML workloads.
• Optimize Spark jobs for performance and cost efficiency.
• Adhere to enterprise security, access control, and compliance standards.
• Provide production support and troubleshoot data pipeline issues.
• Document technical designs, data flows, and operational runbooks.
• Mentor junior engineers and contribute to best practices.

What you'll need

• 4+ years of experience as a Data Engineer with strong Databricks expertise.
• Hands-on experience with Apache Spark, PySpark, and Spark SQL.
• Strong knowledge of Delta Lake and Lakehouse architecture.
• Advanced SQL skills.
• Experience with ETL/ELT patterns and data warehousing concepts.
• Exposure to at least one cloud platform: Azure, AWS, or GCP.
• Understanding of distributed computing concepts.
• Experience working in Agile teams.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, with 60% above.
• Required degrees/fields of study include Bachelor of Engineering and Bachelor of Technology.

Nice to have

• Experience with Unity Catalog and data governance.
• Exposure to Auto Loader or streaming frameworks.
• CI/CD for data pipelines.
• Python for data engineering and automation.
• Databricks certification at Associate or Professional level.

Details

• Location: Gurugram Downtown 4.
• Experience range stated: 4-7 years.
• Travel requirements: Not Specified.
• Work visa sponsorship is not available.
• Government clearance is not required.