Databricks Data Engineer - Manager

Overview

Databricks Data Engineer Manager role focused on large-scale data engineering and implementation across cloud platforms, with deep expertise in Databricks and data governance controls.

What you'll do

• Implement large-scale data engineering solutions.
• Work with Databricks components including notebooks, clusters, jobs, and Delta Lake.
• Configure Unity Catalog and RBAC for data governance and access control.
• Build data processing and programming using Scala, Python, PySpark, and SQL.
• Optimize Apache Spark and troubleshoot/debug PySpark and SQL code.
• Use Azure data services such as Data Factory and Blob Storage with Azure Databricks.
• Set up CI/CD using Azure DevOps.

What you'll need

• Extensive experience in data engineering and large-scale implementation.
• Experience on both Azure and AWS cloud.
• Expertise in Databricks platforms (notebooks, clusters, jobs, Delta Lake).
• Experience with Unity Catalog and RBAC configuration.
• Programming/data processing skills in Scala, Python, PySpark, and SQL.
• CI/CD experience with Azure DevOps.
• Ability to debug and optimize PySpark/SQL code.

Nice to have

• Experience with AWS using Docker, Kubernetes, and Helm framework.

Details

• Work schedule: Hybrid.