Lead Data Engineer - Data Engineering 4C

Overview

Lead the design, development, optimization and production deployment of a Finance Databricks data platform, covering data modeling, pipeline engineering, migrations and production support.

What you'll do

• Design, build and maintain enterprise-grade dimensional (star schema) data models, along with source-to-target mappings (STTM) and seed data management.
• Build robust batch ingestion and processing pipelines using PySpark, Delta Lake, Lakebase and Databricks Workflows.
• Lead end-to-end production migrations, environment promotions (Dev/QA/Prod), historical data backfills, schema migrations and legacy system transitions onto the Databricks Lakehouse with minimal downtime.
• Troubleshoot and resolve complex production pipeline failures, memory leaks, Spark driver/worker OOM issues, data skewness, schema drift and performance bottlenecks.
• Perform execution plan analysis, query tuning and compute optimization across Serverless and Classic SQL Warehouses, including Delta table layout optimizations (Z-Ordering, Liquid Clustering, Compaction) and caching strategies.
• Implement and enforce enterprise-grade data security, column and row-level access controls, data masking and audit lineage using Databricks Unity Catalog.
• Implement automated deployment pipelines for Databricks assets using Git, Databricks Asset Bundles (DABs) and DevOps practices.

What you'll need

• 6 to 10 years of core Data Engineering experience with strong, hands-on expertise on the Databricks Lakehouse platform.
• Proven track record of managing production data pipelines, debugging complex distributed systems, analyzing Spark UI/DAG logs and resolving operational outages.
• Hands-on experience executing complex production migrations, data cutovers, schema evolution handling and historical data loads.
• Core technical stack: PySpark, SQL, Delta Lake internals, Databricks SQL Warehouses, Lakebase and Unity Catalog.
• Expertise in conceptual, logical and physical data modeling alongside detailed STTM creation.
• Hands-on experience configuring Unity Catalog, Access Control Lists (ACLs), role-based access control (RBAC) and data privacy policies.
• Practical knowledge of Databricks Asset Bundles (DABs), Git workflows and CI/CD pipelines.
• Qualifications: Bachelor's in Business Analytics, Computer Science or Statistics, or Master's in Data Science.
• Required skills: Agile Methodology, Banking Capital Markets, Client Relations, Cloud Computing, Collaboration Tools, Commercial Sales, Consumer Packaged Goods (CPG), Databricks Platform, Data Engineering, Data Literacy, Design Thinking, Electronic Hardware, Executive Presence, Inclusion, Manufacturing Equipment, People Leadership, Personal Effectiveness, Risk Management, Semiconductors, Snowflake (Platform), Sourcing and Procurement, Storytelling.
• Language: English, Proficient (C2).

Nice to have

• Exposure to any other data platform such as Snowflake or Microsoft Fabric.
• Certifications: Certified Data Management Professional (Riversand); Databricks Certified Associate Developer for Apache Spark 3.0 or Apache Spark (Databricks Academy); Microsoft Certified: Azure Data Engineer Associate; Oracle Database 12c Certified Implementation Specialist.

Details

• Location: Bangalore (Prestige Technology Park IV).
• Work mode: Hybrid.
• Work shift: Day Job (India).
• Job type: Regular.