Lead Data Engineer

Overview

Lead Data Engineer, a senior individual-contributor and technical leadership role that sets design standards and guides a team's engineering practices for data platforms built at scale.

What you'll do

• Set and enforce design standards, patterns, and best practices for pipelines built on Databricks, Spark, Python, PySpark, and dbt.
• Provide technical leadership and direction to a team of engineers, owning architecture and design reviews.
• Lead the design of CI/CD (GitLab, Jenkins), DataOps, and data observability strategy for the team.
• Own Databricks platform administration standards and deployment governance using Databricks Asset Bundles.
• Drive resolution of complex technical challenges across pipelines and the platform.
• Mentor and grow engineers, establishing a strong code review and quality culture.
• Partner with product and stakeholders on roadmap, prioritization, and technical trade-offs.
• Own the deployment, migration, and upgrade strategy for the team's data platforms.

What you'll need

• Bachelor's or Master's degree in computer science or a related technical field.
• 8+ years of data engineering experience, including demonstrated technical leadership.
• Expert hands-on experience with Databricks (including administration), Spark, Python, and PySpark.
• Deep dbt/TDD and CI/CD (GitLab, Jenkins) expertise.
• Proven ability to define engineering standards and lead a team's practices.
• Strong architecture and platform ownership across observability, cost, and performance.
• Excellent communication and stakeholder management skills.

Nice to have

• Experience in banking, e-commerce, credit cards or payment processing.
• Exposure to both SaaS and premises-based architectures.
• A post-secondary degree in computer science, mathematics, or quantitative science.

Details

• Location: Pune, India.