Overview
Hands-on Databricks Data Engineer role at PwC's Advisory line, designing, building and optimizing scalable data pipelines on the Databricks Lakehouse.
What you'll do
- Build reliable, scalable batch and streaming pipelines using PySpark and Databricks Workflows/Jobs.
- Implement Delta Lake best practices: schema enforcement/evolution, ACID transactions, time travel, OPTIMIZE/ZORDER, and VACUUM.
- Develop Delta Live Tables (DLT) or Structured Streaming pipelines with Auto Loader for ingestion.
- Write robust modular code in Python (packaging, logging, configuration, error handling).
- Write efficient and scalable SQL queries for data extraction and reporting.
- Optimize Spark jobs (partitions, joins, bucketing, caching, AQE, broadcast hints, file sizing).
- Work with SQL Warehouses, Photon, DBR versions, cluster policies, and pools.
- Orchestrate Workflows/Jobs: tasks, dependencies, parameters, retries, alerts, schedules.
- Apply Unity Catalog: permissions, data lineage, audit, external locations, shares; catalog-first design.
- Ensure idempotent, repayable, and incremental pipelines with proper checkpoints and watermarks.
- Implement data quality checks (expectations tests in DLT/Great Expectations) and unit/integration tests for pipelines.
- Ensure data quality, integrity, and governance across the pipeline.
- Work with CI/CD tools to automate deployment and testing of data workflows.
- Work with stakeholders to define data SLAs, schemas, and interface contracts.
- Document pipelines, data sets (data dictionary, lineage), and usage guidelines.
What you'll need
- 5 to 7 years of experience.
- Bachelor of Technology (BE/BTech).
- Databricks certified Data Engineer Associate.
- Strong SQL, Python and PySpark skills.
- Understanding of Delta Lake.
- Experience with production-grade ETL/ELT and orchestration.
- Hands-on knowledge of Azure ADF and ADLS.
- Primary skills: Databricks Platform, PySpark, SQL, Data Modelling, Python (scripting, data manipulation), Azure (ADF, ADLS).
Nice to have
- Analytical mindset with strong problem-solving ability.
- Secondary skills: CI/CD (Git, Jenkins, Azure DevOps, etc.).
Details
- Location: Bengaluru Millenia.
- Management level: Senior Associate.
- Visa sponsorship: No.
- Government clearance required: No.
- Job posting end date: September 17, 2026.
Read the full description and apply on the company’s own careers page.