Overview
Senior Data Engineer / Data Engineering Developer role focused on building and optimizing large-scale distributed data processing solutions using Databricks, Apache Spark, Scala, and Azure.
What you'll do
- Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Scala.
- Build ingestion, transformation, and validation frameworks for high-volume data processing.
- Develop and optimize Spark jobs for performance, scalability, and reliability.
- Implement and manage Databricks workflows, job orchestration, and scheduling.
- Troubleshoot production issues, performance bottlenecks, and pipeline failures.
- Optimize cluster utilization and Spark execution plans for processing efficiency.
- Support production environments and participate in incident resolution and root cause analysis.
What you'll need
- 10+ years of experience in data engineering.
- 6+ years of hands-on Databricks and Spark development experience.
- Strong hands-on experience with Databricks, Apache Spark, and Scala.
- Experience with Spark SQL and PySpark.
- Cloud experience with Microsoft Azure, including ADLS and ADF.
- Strong cloud-native ETL/ELT pipeline development and distributed data processing experience.
- Strong SQL development and query optimization skills.
Nice to have
- Delta Lake and Unity Catalog experience.
- CI/CD pipelines experience (Azure DevOps, Harness).
- Snowflake experience.
- Data quality and validation framework experience.
- Financial services / reference data domain knowledge.
- AI-assisted development tools experience (e.g., GitHub Copilot).
- Databricks Certification and Azure Certification experience.
- Experience with real-time streaming (Kafka, Spark Structured Streaming) and exposure to lakehouse architecture patterns.
Details
- Location: Hyderabad, India.
Read the full description and apply on the company’s own careers page.
