Overview
Lead the design, build, governance and operation of enterprise-grade data engineering and AI-agent solutions on the Databricks Lakehouse Platform and Databricks Agent Bricks / Mosaic AI Agent Framework, and provide hands-on technical leadership to a team of data and analytics engineers.
What you'll do
- Define and govern Databricks Lakehouse and Agent Bricks architecture principles, design standards and reusable reference patterns for enterprise data and AI-agent solutions.
- Design end-to-end data engineering patterns covering ingestion, bronze/silver/gold medallion architecture, Delta Live Tables, streaming, batch and data product publishing on Unity Catalog.
- Design Agent Bricks / Mosaic AI Agent Framework patterns for analytical agents, RAG applications, Genie spaces, AI/BI dashboards and business assistants grounded on governed Lakehouse data.
- Translate approved business and AI use cases into solution design documents, data contracts, platform-fit decisions, integration patterns and production-readiness plans.
- Define model and tool selection guidance covering accuracy, latency, cost (DBU/token), privacy, security, data sensitivity and business purpose.
- Establish engineering guardrails for prompt engineering, RAG grounding, tool calling, human-in-the-loop approvals, write-back controls and auditability for Agent Bricks solutions.
- Support AI/data governance, ARB submissions, decision records, architecture exceptions and platform standards for Databricks solutions.
- Evaluate Databricks capabilities including Unity Catalog, Delta Live Tables, Workflows, SQL Warehouses, MLflow, Model Serving, Vector Search, Genie, AI/BI Dashboards and Agent Bricks for enterprise fit.
- Design DEV / QA / PROD-ready deployment patterns using Databricks Asset Bundles and CI/CD, in collaboration with cloud, infrastructure, network, security, integration and AI architecture teams.
- Define secure ingestion and integration patterns from SAP, SharePoint, ServiceNow, Coupa, Salesforce, REST APIs, file shares and event sources using Azure Data Factory, Auto Loader and partner connectors.
- Design API-led integration patterns using Azure API Management, Entra ID, OAuth, managed identity, service principals and approved enterprise gateways for Databricks-hosted agents and endpoints.
- Lead technical design reviews for data pipelines, Unity Catalog models, agent tools, vector indexes, model endpoints, evaluation workflows, monitoring and release gates.
- Build and review hands-on production-grade code in PySpark, Python and SQL; enforce coding standards, peer reviews, unit/integration testing and automated quality checks.
- Support partner and vendor technical reviews to validate delivery approach, platform compatibility, security posture and architecture compliance.
- Define operational runbooks for Databricks workspaces, Unity Catalog, jobs, DLT pipelines, Agent Bricks lifecycle, prompt/version control, evaluations, monitoring, telemetry and incident support.
- Ensure Databricks solutions follow least-privilege RBAC/ABAC, access reviews, environment segregation, audit logging, data protection, lineage and Responsible AI controls.
- Provide technical leadership, mentoring and guidance to data engineers, analytics engineers, platform admins, business analysts and implementation partners.
- Monitor and optimize solution design for DBU consumption, storage, SQL Warehouse usage, vector search cost, model endpoint usage, latency, quality, security and scalability.
- Review delivered solutions against engineering principles, Orbia governance expectations, SOX/ITGC controls and production acceptance criteria before go-live.
- Lead and mentor a small team of data/analytics engineers, including code reviews, sprint planning, capacity planning, performance feedback and skill development.
What you'll need
- Minimum 12-14 years of hands-on experience on the Databricks Lakehouse Platform including Apache Spark, PySpark, Delta Lake, Unity Catalog, Delta Live Tables (DLT), Workflows, SQL Warehouses, MLflow and Model Serving.
- Minimum 2 years of hands-on or strong architecture experience with Databricks Agent Bricks / Mosaic AI Agent Framework, Vector Search, Genie spaces, AI/BI Dashboards and RAG-based agentic applications grounded on Lakehouse data.
- Strong programming skills in Python, PySpark and advanced SQL; deep understanding of distributed computing, performance tuning, partitioning, Z-Ordering, file optimization and Delta best practices.
- Experience with CI/CD for Databricks using Databricks Asset Bundles, Azure DevOps and/or GitHub Actions, infrastructure-as-code (Terraform) and environment promotion patterns across DEV / QA / PROD.
- Experience with the Azure data ecosystem including ADLS Gen2, Azure Data Factory, Azure Key Vault, Entra ID, Microsoft Purview, Azure Monitor, Azure API Management and Azure OpenAI services.
- Experience designing and operating ingestion pipelines from SAP (ECC/S4 HANA, BW, BTP), SharePoint, ServiceNow, Coupa, Salesforce, REST APIs, files and streaming sources using Auto Loader, ADF and partner connectors.
- Experience designing RAG and agent solutions using SharePoint, enterprise documents, semantic indexes, embeddings, vector search, citations, chunking, grounding evidence and answer evaluation.
- Experience designing secure enterprise integrations using REST APIs, OAuth 2.0, managed identities, service principals, APIM, event-driven patterns, SAP BTP, ServiceNow APIs or Microsoft Graph APIs.
- Strong understanding of governance controls including Unity Catalog (catalogs/schemas/tables, row/column-level security, ABAC), RBAC, least privilege, segregation of duties, access reviews, private endpoints, audit logging, DLP, Purview, data classification, lineage and SOX / ITGC expectations.
- Experience in AI/data governance, Responsible AI, model and agent evaluation, prompt engineering, hallucination control, content safety, human approval gates and production release criteria.
- Demonstrated experience leading a team of data engineers / analytics engineers, including hands-on technical leadership, mentoring, code reviews, sprint planning and performance development.
- Experience preparing architecture review documents, solution and low-level design documents, technical standards, operating procedures, decision records and ARB-ready materials.
- Strong understanding of cost management for Databricks and AI solutions including DBU usage, SQL Warehouse sizing, storage, vector search cost, model endpoints, monitoring and FinOps guardrails.
- Experience working in an international environment at a global company with cross-functional teams and business stakeholders across multiple regions.
- Excellent communication skills in English, both verbally and in writing.
- Capabilities to build trustworthy relationships with business users, platform teams, security, architecture, vendors and implementation partners.
- Self-starter with the ability to provide technical leadership, mobilize business partners, analyze requirements, challenge solution assumptions, push back when required and remain result-oriented.
Details
Read the full description and apply on the company’s own careers page.