Manager – Databricks Data Engineer, Advisory

Overview

Design, build and operate scalable, high-performance data pipelines on the Databricks Lakehouse Platform, with hands-on development using Apache Spark, Databricks notebooks, Delta Lake and cloud-native services, working closely with analytics, AI/ML and business teams.

What you'll do

• Design, build and maintain end-to-end data pipelines using Databricks (PySpark / Spark SQL).
• Implement batch and incremental data processing using Delta Lake and multi-hop architecture.
• Develop and optimize Databricks notebooks, jobs and workflows.
• Ingest, transform and curate large-scale structured and semi-structured datasets.
• Support analytics, reporting and downstream data consumption use cases.
• Ensure data quality, reliability, lineage and governance.
• Collaborate with data scientists, analysts and architects on AI/ML workloads.
• Optimize Spark jobs for performance and cost efficiency.
• Adhere to enterprise security, access control and compliance standards.
• Provide production support and troubleshoot data pipeline issues.
• Document technical designs, data flows and operational runbooks.
• Mentor junior engineers and contribute to best practices.

What you'll need

• 6+ years of experience as a Data Engineer with strong Databricks expertise.
• Hands-on experience with Apache Spark, PySpark and Spark SQL.
• Strong knowledge of Delta Lake and Lakehouse architecture.
• Advanced SQL skills.
• Experience with ETL/ELT patterns and data warehousing concepts.
• Exposure to at least one cloud platform (Azure / AWS / GCP).
• Understanding of distributed computing concepts.
• Experience working in Agile teams.
• Bachelor's or Master's degree in Computer Science, Engineering or related field (60% above).

Nice to have

• Experience with Unity Catalog and data governance.
• Exposure to Auto Loader or streaming frameworks.
• CI/CD for data pipelines.
• Python for data engineering and automation.
• Databricks certification (Associate / Professional).

Details

• Location: Gurugram Downtown 4.
• Years of experience required: 6-10 years.
• Visa sponsorship: No.
• Government clearance required: No.
• Job posting end date: May 17, 2026.