AI engineer-Databricks Data Platform

Overview

Design, build, configure, integrate and operationalize enterprise Data and AI solutions on the Databricks platform, enabling data engineers, data scientists, AI engineers and business teams.

What you'll do

• Design, build, configure and maintain Databricks workspaces across development, testing and production environments.
• Implement platform standards, reusable frameworks, templates and best practices.
• Manage Unity Catalog, clusters, SQL Warehouses, compute policies, workspace configurations and access controls.
• Automate platform deployment and configuration using Infrastructure as Code (Terraform) and CI/CD pipelines.
• Build and integrate scalable data pipelines using Databricks Lakehouse architecture.
• Design ingestion frameworks for batch, streaming, API, database and file-based integrations.
• Implement Delta Lake, Structured Streaming and medallion architecture patterns.
• Integrate Databricks with enterprise data platforms such as Snowflake, SAP, Oracle, SQL Server, Azure Data Factory, Kafka and cloud storage services.
• Enable ML and Generative AI workloads on Databricks, including MLflow, model lifecycle management, feature stores and model serving.
• Support AI engineers and data scientists with scalable development environments.
• Integrate Databricks with LLMs, vector databases, AI gateways and enterprise AI platforms.
• Establish CI/CD pipelines for data and AI workloads, with automated testing, deployment, monitoring and rollback mechanisms.
• Create reusable deployment frameworks and engineering accelerators, and support release management and environment promotion.
• Implement enterprise security controls, RBAC, data masking, encryption and audit logging.
• Configure and manage Unity Catalog governance policies and ensure compliance with enterprise security, privacy and regulatory requirements.
• Partner with cybersecurity teams on platform hardening and vulnerability remediation.
• Implement platform observability, monitoring, logging, alerting and operational dashboards.
• Optimize platform performance, cost, scalability and reliability, and support production operations and incident management.
• Collaborate with architects, data engineers, AI engineers, security teams and business stakeholders.
• Provide technical guidance, participate in architecture reviews and platform roadmap planning, and mentor junior engineers.

What you'll need

• Bachelor's or master's degree in computer science, engineering, IT or a related field.
• 5+ years of experience in Data, Platform or Cloud Engineering.
• 3+ years of hands-on experience with the Databricks Lakehouse Platform.
• Strong expertise in Delta Lake, Unity Catalog, MLflow, Databricks Workflows, Structured Streaming, PySpark and Spark SQL.
• Experience working with cloud platforms such as Azure, AWS or GCP.
• Proficiency in Python and SQL.
• Strong knowledge of DevOps, CI/CD, Infrastructure as Code (IaC) and platform automation.
• Hands-on experience with Terraform, GitHub Actions, Azure DevOps or equivalent CI/CD tools.

Nice to have

• Experience with Generative AI, Agentic AI and LLM-based applications.
• Experience integrating Databricks with Snowflake and enterprise AI platforms.
• Knowledge of Kubernetes, Docker, APIs, Kafka and event-driven architectures.
• Databricks Certified Professional or Associate certifications.
• Experience supporting enterprise-scale Data and AI platforms.

Details

• Location: Bangalore, Karnataka, India.