Senior Data Engineer - Apache Spark and SQL - Vice President

Overview

Senior Data Engineer (Databricks/Spark) responsible for engineering and modernizing a data processing platform on Databricks (AWS), including migration from legacy Cloudera Hadoop.

What you'll do

• Refactor and modernize existing Spark pipelines to Databricks native architectures.
• Eliminate legacy Hadoop dependencies and adopt AWS cloud-native patterns.
• Build and optimize Databricks-native solutions using Delta Lake, Workflows orchestration, and job clusters/auto scaling.
• Design data models and pipeline patterns to make solutions scalable, maintainable, and production-ready.
• Analyze and improve Spark job performance and simplify complex pipelines.
• Write clean, modular, and testable Spark/Databricks code and contribute to shared frameworks.
• Collaborate with architects, platform teams, and DevOps for troubleshooting and technical guidance.

What you'll need

• 11+ years of experience in data engineering or distributed systems.
• Strong expertise in Apache Spark (JavaSpark/PySpark), Databricks on AWS, and Delta Lake.
• Hands-on experience with SQL.
• Experience with AWS services and large-scale distributed data processing.
• Ability to modernize/refactor legacy data platforms into cloud-based architectures.
• Strong background in Spark performance tuning and large-scale batch optimization.
• Ability to translate architecture into detailed technical designs.

Details

• Location: Pune, Maharashtra, India.
• Employment type: Full time.