Staff Data Engineer

Overview

Staff Data Engineer focused on building and optimizing cloud-based data ingestion, transformation, and ETL/ELT pipelines on Azure.

What you'll do

• Design, develop, and maintain high-performance ETL/ELT pipelines using PySpark and Spark SQL in Databricks.
• Build and orchestrate data workflows in Azure.
• Implement hybrid data integration between on-prem databases and Azure Databricks using tools like ADF and HVR/Fivetran.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Enforce best practices for data quality, governance, and documentation.
• Perform root cause analysis and troubleshoot issues to ensure pipeline reliability.

What you'll need

• Minimum 6+ years of hands-on experience in data engineering.
• Proficiency in PySpark, Spark SQL, and distributed processing.
• Strong knowledge of Azure services including ADF, Databricks, and ADLS.
• Experience with SQL, data modeling, and performance tuning.
• Familiarity with Git, CI/CD pipelines, and agile practices.
• Bachelor’s degree in Computer Science, Engineering, or related field.

Nice to have

• Experience with orchestration tools such as Airflow or ADF.
• Expertise with Databricks, Delta Lake, Unity Catalog, and Azure Data Services.
• Knowledge of real-time streaming tools such as Kafka and Event Hub.
• Exposure to APIs, data integrations, and cloud-native architectures.
• Familiarity with enterprise data ecosystems such as S/4 HANA and BDC.

Details

• Location: Bengaluru, KA, India.