Lead Data Engineer

Overview

Lead Data Engineer focused on Databricks enterprise data platform performance, governance, and AI-native operations maturity.

What you'll do

• Design and develop scalable data pipelines and Lakehouse solutions on Databricks.
• Tune Databricks workloads for performance and cost, including cluster sizing, query optimization, and Delta Lake table design.
• Establish best practices for partitioning, clustering, and workload isolation.
• Operationalize Unity Catalog for governance, including access control, lineage, and security.
• Build monitoring and self-healing automation using Databricks-native AI and agentic capabilities.
• Drive CI/CD workflows for Databricks assets using DevOps deployment and release management best practices.

What you'll need

• 6+ years of data engineering experience.
• 2+ years hands-on Databricks experience in enterprise settings.
• Deep understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and Workflow orchestration.
• Advanced Python (PySpark) and SQL skills.
• Proven ability to tune Spark workloads for cost and performance at production scale.
• Working knowledge of CI/CD practices and DevOps principles applied to data workloads.
• Experience with observability tooling for Databricks.

Nice to have

• Experience with Databricks-native AI capabilities and agentic frameworks.
• Familiarity with Databricks Serverless Compute and DBSQL performance tuning.
• A Databricks Certified Professional.
• Exposure to Infrastructure-as-Code.

Details

• Location: Bangalore.