Senior Databricks Engineer, Apache Spark and AWS - Vice President

Overview

Senior Databricks Engineer to modernize and evolve data processing on Databricks running on AWS, including transitioning from a legacy Cloudera Hadoop platform.

What you'll do

• Refactor and modernize existing Spark pipelines to Databricks native architectures.
• Eliminate legacy Hadoop dependencies and adopt AWS cloud-native patterns.
• Build and optimize Databricks-native solutions using Delta Lake, Workflows, and job clusters.
• Translate high-level architecture into detailed technical designs.
• Analyze and improve Spark job performance and simplify complex pipelines.
• Write clean, modular, testable code and contribute to shared frameworks and libraries.
• Develop unit, integration, and data validation tests and support production releases.

What you'll need

• 10+ years of experience in data engineering or distributed systems.
• Strong expertise in Apache Spark (JavaSpark / PySpark), Databricks on AWS, and Delta Lake.
• Experience using SQL.
• AWS services experience with large-scale distributed data processing.
• Strong background modernizing/refactoring legacy data platforms into cloud architectures.
• Experience with Spark performance tuning and large-scale batch optimization.
• Clear communication skills across global teams.

Details

• Location: Pune, Maharashtra, India.