Principal Azure Databricks Platform Engineer

Overview

Lead the engineering, administration, automation, governance, and continuous modernization of an enterprise data platform built on Microsoft Azure and Azure Databricks. This is a senior, hands-on technical leadership role focused on engineering and operating the data platform rather than developing ETL pipelines.

What you'll do

• Lead the design, implementation, administration, and optimization of enterprise Azure Databricks environments across development, staging, and production.
• Manage Databricks account-level and workspace-level configurations, including identity, permissions, networking, compute, workspace settings, and governance.
• Establish and enforce enterprise Databricks architecture, administration standards, platform security policies, and operational best practices.
• Design and manage compute policies, SQL warehouses, serverless capabilities, jobs compute, workload isolation, and resource utilization.
• Troubleshoot complex platform issues involving performance, configuration, permissions, networking, availability, and integration.
• Implement platform monitoring, audit logging, cost observability, and proactive operational controls.
• Enable multi-team and multi-data-product environments with appropriate security, isolation, and access boundaries.
• Develop and maintain reusable Infrastructure as Code using the Databricks Terraform Provider and Azure Terraform Provider.
• Automate Databricks account and workspace administration, Unity Catalog resources, permissions, compute policies, and other platform configurations.
• Implement Databricks Declarative Automation Bundles (formerly Databricks Asset Bundles / DAB) for standardized deployment and management of Databricks resources.
• Establish reusable Terraform modules, deployment templates, CI/CD pipelines, and automated environment provisioning.
• Develop automation using Databricks CLI, REST APIs, Python, PowerShell, or other scripting technologies.
• Implement Git-based deployment practices, configuration management, infrastructure drift detection, and automated policy enforcement.
• Drive consistent, repeatable, and auditable deployments across environments.
• Lead enterprise implementation and administration of Databricks Unity Catalog, including catalogs, schemas, permissions, external locations, storage credentials, and governance policies.
• Define scalable data access models using Entra ID groups, service principals, managed identities, and role-based access controls.
• Establish standards for data product isolation, least-privilege access, environment separation, ownership, and secure data sharing.
• Implement and enhance auditing, lineage, data discovery, and platform governance capabilities.
• Work with enterprise governance and security teams to integrate Databricks with broader data governance solutions such as Microsoft Purview.
• Ensure compliance with organizational security, privacy, and data management standards.
• Continuously evaluate and introduce new Databricks capabilities to improve platform functionality, developer experience, scalability, and cost efficiency.
• Lead platform enablement and establish operational patterns for Unity Gateway / AI Gateway, Lakebase, Lakeflow, Delta Sharing, and Databricks Apps and AI capabilities.
• Assess platform feature maturity, architectural fit, security implications, and operational readiness before enterprise adoption.
• Develop reference architectures, reusable platform components, and standardized onboarding patterns for engineering teams.
• Design and manage the Azure infrastructure supporting enterprise Databricks environments, including networking, storage, identity, and security integrations.
• Implement secure connectivity and integration between Databricks and enterprise Azure services.
• Design and enforce infrastructure security, high availability, backup, disaster recovery, monitoring, and cost optimization practices.
• Collaborate with enterprise cloud, security, networking, and operations teams to maintain reliable, compliant platform services.
• Serve as a technical authority for Azure Databricks platform engineering and administration.
• Lead complex technical initiatives from architecture and design through implementation and operational handover.
• Mentor engineers through design reviews, technical coaching, knowledge sharing, and hands-on problem-solving.
• Establish engineering standards, reusable architectural patterns, operational runbooks, and platform best practices.
• Conduct technical assessments, identify platform gaps, and drive continuous improvement initiatives.
• Collaborate with data engineers, solution architects, security teams, and business stakeholders to understand platform requirements and deliver scalable solutions.

What you'll need

• 8+ years of relevant experience in cloud engineering, platform engineering, infrastructure automation, or enterprise data platform technologies.
• Strong hands-on experience with Microsoft Azure, including infrastructure architecture, networking, storage, identity, and security.
• Significant hands-on production experience administering and engineering Azure Databricks platforms, preferably 4+ years.
• Proven experience implementing Databricks platform capabilities beyond basic workspace creation, cluster provisioning, or notebook execution.
• Demonstrated experience leading technical initiatives, mentoring engineers, and establishing platform engineering standards.
• Advanced Databricks administration, account and workspace configuration, compute governance, operational troubleshooting, and platform performance optimization.
• Deep understanding of Unity Catalog architecture, permissions, governance, security, and resource administration.
• Experience with Databricks SQL warehouses, serverless compute, workload management, and cost optimization.
• Strong knowledge of Databricks security models, service principals, identity management, and enterprise deployment patterns.
• Hands-on experience with the Databricks Terraform Provider, including management of Databricks-specific resources.
• Strong Terraform skills covering reusable modules, state management, environment configuration, and automation.
• Experience with Declarative Automation Bundles (Databricks Asset Bundles), Databricks CLI, APIs, and CI/CD.
• Proficiency with Git-based engineering practices and automated deployments using GitHub Actions, Azure DevOps, or similar tools.
• Scripting and automation experience using Python, PowerShell, or Bash.
• Strong knowledge of Azure networking, private endpoints, DNS, ADLS Gen2, Key Vault, and Entra ID.
• Experience securing and operating Azure-hosted enterprise data platforms.
• Familiarity with Azure monitoring, logging, infrastructure governance, and disaster recovery practices.
• Proven ability to independently own complex technical solutions and make architectural decisions.
• Experience mentoring engineers and providing technical direction to teams.
• Ability to translate enterprise requirements into practical platform engineering solutions.
• Strong troubleshooting, analytical, communication, and stakeholder management skills.

Nice to have

• Hands-on experience with newer Databricks capabilities such as Unity Gateway / AI Gateway, Lakebase, Lakeflow, Delta Sharing, Databricks Apps, or AI platform services.
• Experience implementing platform governance, automation, or reusable deployment frameworks for large enterprise Databricks environments.
• Experience with Databricks system tables, audit logs, platform observability, and FinOps.
• Experience designing multi-tenant or data-mesh-oriented enterprise data platforms.
• Familiarity with Azure-based data engineering and orchestration services such as Azure Data Factory, without necessarily being a hands-on ETL developer.
• Relevant Databricks, Terraform, or Microsoft Azure certifications.

Details

• Location: Hyderabad, India.
• Work mode: Hybrid.