Manager – Databricks Data Engineer, Advisory

Overview

Manage the design, development, and operation of scalable data pipelines on the Databricks Lakehouse Platform. The role uses Spark, Delta Lake, cloud services, and collaboration with analytics and AI/ML teams.

What you'll do

• Design and maintain end-to-end Databricks data pipelines.
• Implement batch and incremental processing with Delta Lake.
• Develop and optimise Databricks notebooks, jobs, and workflows.
• Ingest, transform, and curate structured and semi-structured datasets.
• Ensure data quality, reliability, lineage, and governance.
• Optimise Spark jobs for performance and cost efficiency.
• Mentor junior engineers and provide production support.

What you'll need

• 7–12 years of experience as a Data Engineer.
• Strong Databricks, Apache Spark, PySpark, and Spark SQL experience.
• Strong knowledge of Delta Lake and Lakehouse architecture.
• Advanced SQL skills.
• Experience with ETL/ELT patterns and data warehousing.
• Exposure to Azure, AWS, or GCP.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field with 60% or above.

Nice to have

• Experience with Unity Catalog and data governance.
• Exposure to Auto Loader or streaming frameworks.
• CI/CD experience for data pipelines.
• Python experience for data engineering and automation.
• Databricks Associate or Professional certification.

Details

• Location: Bengaluru, India.
• Travel requirements: Not specified.
• Work visa sponsorship: Not available.