Principal Data Engineer- Databricks & AI platform

Overview

Principal Data Engineer to own and set the technical direction for GoTo’s Databricks data platform and ML/AI-ready data infrastructure.

What you’ll do

• Own the architecture and multi-year roadmap for Databricks, including Unity Catalog, Delta Lake, and workflow orchestration.
• Set engineering standards for how data pipelines are built, governed, and scaled.
• Design and evolve a medallion (Bronze/Silver/Gold) lakehouse architecture for GenAI and ML use cases.
• Establish data quality, security, and access control standards and ensure teams follow them.
• Tune Spark jobs and cluster configurations to improve performance and cost efficiency.
• Lead migrations from legacy platforms to Databricks, including data modeling and pipeline redesign.

What you’ll need

• 12+ years in data engineering.
• At least 3+ years hands-on with Databricks in production.
• Deep expertise in Unity Catalog, Delta Lake, and Spark performance tuning at scale.
• Strong SQL and Python.
• Experience with at least one major cloud platform (AWS, Azure, or GCP) and infrastructure-as-code tooling.
• Experience with CI/CD for data pipelines, including Databricks Asset Bundles.
• Experience leading technical decisions and mentoring other engineers.

Nice to have

• Experience with orchestration tools like Airflow or dbt.
• Experience with streaming/messaging systems like Kafka.
• Exposure to GenAI, RAG, or LLM-based data applications.
• Experience leading a legacy data warehouse to Databricks migration.

Details

• Location: Bangalore, KA, IN.