Databricks Data Engineer - Senior Manager

Overview

Senior Manager role for a Databricks Data Engineer who has built large-scale data engineering solutions across clouds.

What you'll do

• Build data engineering and implementations at large scale.
• Develop and process data using Scala, Python, PySpark, and SQL.
• Configure and use Databricks platform capabilities including Notebooks, Clusters, Jobs, and Delta Lake.
• Set up Unity Catalog and configure RBAC for data access.
• Use Azure services such as Data Factory and Blob Storage together with Azure Databricks.
• Implement CI/CD using Azure DevOps.
• Debug and optimize PySpark/SQL code using performance optimization techniques.
• Apply secure coding principles, and follow data governance and compliance standards.

What you'll need

• Experience as a data engineer in data engineering and large-scale implementations.
• Expertise in Databricks platform (Notebooks, Clusters, Jobs, Delta Lake).
• Experience with Unity Catalog and RBAC configuration.
• Experience with Azure cloud and Azure services (Data Factory, Blob Storage, Azure Databricks).
• CI/CD experience with Azure DevOps.
• Programming and data processing experience using Scala, Python, PySpark, and SQL.
• Ability to debug and optimize PySpark/SQL code.

Nice to have

• Experience with AWS using Docker, Kubernetes, and Helm frameworks.

Details

• Work schedule: Hybrid.