Data Engineer II (Databricks & Pyspark)

Overview

Build and maintain scalable data pipelines and lakehouse solutions on Databricks, delivering clean, reliable data to analytics and ML teams.

What you'll do

• Design and implement ELT pipelines using PySpark and Databricks Workflows.
• Implement data lakehouse architecture and optimize data storage and retrieval for performance and cost efficiency.
• Write efficient, reusable, and scalable Python code for data processing and automation tasks.
• Orchestrate complex workflows using Azure Data Factory and integrate with other Azure services.
• Collaborate with analysts and data scientists to model and deliver trusted datasets.
• Monitor and troubleshoot data pipelines, ensuring high availability and reliability.
• Use CI/CD pipelines, DevOps practices, and version control with Git.

What you'll need

• 4+ years in data engineering.
• 3+ years with Databricks.
• Strong PySpark skills, including DataFrames, Spark SQL, window functions, and UDFs.
• Advanced SQL skills, including complex joins, CTEs, aggregations, and query optimization.
• Experience with Delta Lake, including ACID transactions, time travel, and schema evolution.
• Familiarity with Unity Catalog, data lineage, and access control.
• Cloud experience with Azure, including ADLS Gen2 and ADF, or AWS/GCP equivalent.
• Comfort with Git, notebooks-as-code, and automated testing.

Details

• Location: Chennai, India.