Senior Databricks Data Engineer - Assistant Manager

Overview

Senior Databricks Data Engineer (Assistant Manager) to design, build, and optimize high-throughput data pipelines using Databricks for near real-time processing.

What you'll do

• Design and develop configurable, high-throughput data pipelines using DataBricks.
• Build end-to-end data ingestion pipelines for streaming, events, and batch.
• Design, develop, and deploy scalable ETL pipelines using Databricks SQL and Scala/PySpark.
• Optimize and tune Databricks job performance, including query optimization and resource management.
• Troubleshoot and perform impact analysis, root cause analysis, and maintenance for data/environment issues.
• Own the software development lifecycle including design, coding, integration testing, deployment, and documentation.

What you'll need

• 7+ years of overall IT experience.
• 5+ years of Data Engineering experience handling large datasets.
• 5+ years of hands-on Databricks experience using SQL & Scala/PySpark.
• 5+ years of cloud-based development experience using Azure/AWS.
• Bachelor’s degree in a computer or IT related subject.
• Hands-on troubleshooting experience for DevOps pipelines and Azure services.

Details

• Location: Hyderabad, India.