Orbia logo

Lead Data & AI Engineer – Databricks Data Analytics & Agent Bricks

Orbia
Posted this week

LOCATION

Pune · Hybrid

EXPERIENCE

12+ Years

SKILLS REQUIRED

DatabricksData EngineeringData ArchitectureTechnical LeadershipData GovernancePySparkPythonSQL

Job description

Overview

Lead the design, build, governance and operation of enterprise-grade data engineering and AI-agent solutions on the Databricks Lakehouse Platform and Databricks Agent Bricks / Mosaic AI Agent Framework, and provide hands-on technical leadership to a team of data and analytics engineers.

What you'll do

  • Define and govern Databricks Lakehouse and Agent Bricks architecture principles, design standards and reusable reference patterns for enterprise data and AI-agent solutions.
  • Design end-to-end data engineering patterns covering ingestion, bronze/silver/gold medallion architecture, Delta Live Tables, streaming, batch and data product publishing on Unity Catalog.
  • Design Agent Bricks / Mosaic AI Agent Framework patterns for analytical agents, RAG applications, Genie spaces, AI/BI dashboards and business assistants grounded on governed Lakehouse data.
  • Translate approved business and AI use cases into solution design documents, data contracts, platform-fit decisions, integration patterns and production-readiness plans.
  • Define model and tool selection guidance covering accuracy, latency, cost (DBU/token), privacy, security, data sensitivity and business purpose.
  • Establish engineering guardrails for prompt engineering, RAG grounding, tool calling, human-in-the-loop approvals, write-back controls and auditability for Agent Bricks solutions.
  • Support AI/data governance, ARB submissions, decision records, architecture exceptions and platform standards for Databricks solutions.
  • Evaluate Databricks capabilities including Unity Catalog, Delta Live Tables, Workflows, SQL Warehouses, MLflow, Model Serving, Vector Search, Genie, AI/BI Dashboards and Agent Bricks for enterprise fit.
  • Design DEV / QA / PROD-ready deployment patterns using Databricks Asset Bundles and CI/CD, in collaboration with cloud, infrastructure, network, security, integration and AI architecture teams.
  • Define secure ingestion and integration patterns from SAP, SharePoint, ServiceNow, Coupa, Salesforce, REST APIs, file shares and event sources using Azure Data Factory, Auto Loader and partner connectors.
  • Design API-led integration patterns using Azure API Management, Entra ID, OAuth, managed identity, service principals and approved enterprise gateways for Databricks-hosted agents and endpoints.
  • Lead technical design reviews for data pipelines, Unity Catalog models, agent tools, vector indexes, model endpoints, evaluation workflows, monitoring and release gates.
  • Build and review hands-on production-grade code in PySpark, Python and SQL; enforce coding standards, peer reviews, unit/integration testing and automated quality checks.
  • Support partner and vendor technical reviews to validate delivery approach, platform compatibility, security posture and architecture compliance.
  • Define operational runbooks for Databricks workspaces, Unity Catalog, jobs, DLT pipelines, Agent Bricks lifecycle, prompt/version control, evaluations, monitoring, telemetry and incident support.
  • Ensure Databricks solutions follow least-privilege RBAC/ABAC, access reviews, environment segregation, audit logging, data protection, lineage and Responsible AI controls.
  • Provide technical leadership, mentoring and guidance to data engineers, analytics engineers, platform admins, business analysts and implementation partners.
  • Monitor and optimize solution design for DBU consumption, storage, SQL Warehouse usage, vector search cost, model endpoint usage, latency, quality, security and scalability.
  • Review delivered solutions against engineering principles, Orbia governance expectations, SOX/ITGC controls and production acceptance criteria before go-live.
  • Lead and mentor a small team of data/analytics engineers, including code reviews, sprint planning, capacity planning, performance feedback and skill development.

What you'll need

  • Minimum 12-14 years of hands-on experience on the Databricks Lakehouse Platform including Apache Spark, PySpark, Delta Lake, Unity Catalog, Delta Live Tables (DLT), Workflows, SQL Warehouses, MLflow and Model Serving.
  • Minimum 2 years of hands-on or strong architecture experience with Databricks Agent Bricks / Mosaic AI Agent Framework, Vector Search, Genie spaces, AI/BI Dashboards and RAG-based agentic applications grounded on Lakehouse data.
  • Strong programming skills in Python, PySpark and advanced SQL; deep understanding of distributed computing, performance tuning, partitioning, Z-Ordering, file optimization and Delta best practices.
  • Experience with CI/CD for Databricks using Databricks Asset Bundles, Azure DevOps and/or GitHub Actions, infrastructure-as-code (Terraform) and environment promotion patterns across DEV / QA / PROD.
  • Experience with the Azure data ecosystem including ADLS Gen2, Azure Data Factory, Azure Key Vault, Entra ID, Microsoft Purview, Azure Monitor, Azure API Management and Azure OpenAI services.
  • Experience designing and operating ingestion pipelines from SAP (ECC/S4 HANA, BW, BTP), SharePoint, ServiceNow, Coupa, Salesforce, REST APIs, files and streaming sources using Auto Loader, ADF and partner connectors.
  • Experience designing RAG and agent solutions using SharePoint, enterprise documents, semantic indexes, embeddings, vector search, citations, chunking, grounding evidence and answer evaluation.
  • Experience designing secure enterprise integrations using REST APIs, OAuth 2.0, managed identities, service principals, APIM, event-driven patterns, SAP BTP, ServiceNow APIs or Microsoft Graph APIs.
  • Strong understanding of governance controls including Unity Catalog (catalogs/schemas/tables, row/column-level security, ABAC), RBAC, least privilege, segregation of duties, access reviews, private endpoints, audit logging, DLP, Purview, data classification, lineage and SOX / ITGC expectations.
  • Experience in AI/data governance, Responsible AI, model and agent evaluation, prompt engineering, hallucination control, content safety, human approval gates and production release criteria.
  • Demonstrated experience leading a team of data engineers / analytics engineers, including hands-on technical leadership, mentoring, code reviews, sprint planning and performance development.
  • Experience preparing architecture review documents, solution and low-level design documents, technical standards, operating procedures, decision records and ARB-ready materials.
  • Strong understanding of cost management for Databricks and AI solutions including DBU usage, SQL Warehouse sizing, storage, vector search cost, model endpoints, monitoring and FinOps guardrails.
  • Experience working in an international environment at a global company with cross-functional teams and business stakeholders across multiple regions.
  • Excellent communication skills in English, both verbally and in writing.
  • Capabilities to build trustworthy relationships with business users, platform teams, security, architecture, vendors and implementation partners.
  • Self-starter with the ability to provide technical leadership, mobilize business partners, analyze requirements, challenge solution assumptions, push back when required and remain result-oriented.

Details

  • Location: Pune, India.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Lead Data & AI Engineer – Databricks Data Analytics & Agent Bricks