Data Engineering Developer - Assistant Manager

Overview

Assistant Manager role for a Data Engineer focused on large-scale data processing using Databricks, Apache Spark, and Scala on Azure.

What you'll do

• Design, develop, and maintain scalable data pipelines with Databricks, Apache Spark, and Scala.
• Build ingestion, transformation, and validation frameworks for high-volume data processing.
• Develop and optimize Spark jobs for performance, scalability, and reliability.
• Implement and manage Databricks workflows, job orchestration, and scheduling.
• Troubleshoot production issues, performance bottlenecks, and pipeline failures.
• Implement best practices for coding, testing, CI/CD, and deployment.
• Support production environments and participate in incident resolution and root cause analysis.

What you'll need

• 10+ years of experience in Data Engineering.
• 6+ years of hands-on Databricks and Spark development experience.
• Strong production-grade Scala application development experience.
• Hands-on Databricks, Apache Spark, Scala, Spark SQL, and PySpark.
• Experience with distributed data processing frameworks and ETL/ELT pipeline development.
• Cloud experience with Microsoft Azure, including ADLS and ADF.
• Strong SQL development and query optimization skills.

Nice to have

• Delta Lake and Unity Catalog.
• CI/CD pipelines (Azure DevOps, Harness).
• Snowflake exposure.
• Data quality and validation frameworks.
• Financial Services / Reference Data domain knowledge.
• AI-assisted development tools such as GitHub Copilot.
• Databricks Certification and Azure Certification.
• Real-time streaming exposure (Kafka, Spark Structured Streaming).
• Exposure to Lakehouse architecture patterns.

Details

• Location: Bangalore, India.