Software Engineer II

Overview

Software Engineer II on the Data Ingestion team, responsible for building and maintaining scalable data ingestion pipelines on Databricks for DSP, SSP, measurement, and enrichment data.

What you'll do

• Build and productionise ingestion pipelines using Databricks, PySpark, SDP, Autoloader, and Delta.
• Orchestrate pipelines through scheduling and ingestion services.
• Onboard DSPs, regions, and third-party data feeds.
• Contribute to the Unified Ingestion Service and self-serve ingestion capabilities.
• Strengthen data health, lineage, compliance, governance, and incident workflows.
• Manage releases, platform hygiene, vulnerability remediation, and production issues.
• Monitor and optimise pipeline reliability and cost.

What you'll need

• 2–3 years of experience in data engineering and data integration.
• Hands-on experience building cloud data pipelines, preferably on AWS.
• Extensive PySpark notebook development and advanced SQL experience.
• Working knowledge of Delta Lake, DLT, Autoloader, Unity Catalog, and Spark.
• Understanding of data modelling, schema evolution, data transformation, and data quality.
• Experience with APIs, S3, SFTP/FTP, email drops, event streams, and CDC ingestion patterns.
• Knowledge of SDLC, version control, testing, CI/CD, release management, and incident management.

Nice to have

• Azure or GCP cloud experience.
• Hive experience for migration work.
• Terraform or infrastructure-as-code and IAM exposure.
• Familiarity with Grafana and job-level monitoring.

Details

• Location: Bengaluru.
• Hybrid work environment.