Overview
Design, build and optimize scalable data pipelines and integration solutions on Google Cloud Platform to power enterprise Data & Analytics products, and lead the technical transition of the team's platform to Databricks.
What you'll do
- Design, develop and maintain robust, production-grade data pipelines (batch and near-real-time) on GCP using BigQuery, Cloud Composer (Airflow), Dataflow, Dataproc, Cloud Functions and Pub/Sub.
- Build and maintain inbound and outbound data integration layers, handling ingestion from diverse source systems and delivering curated datasets to downstream consumers.
- Develop and enforce data transformation logic (cleansing, enrichment, aggregation, reconciliation) to ensure data accuracy and consistency across the D&A estate.
- Implement orchestration frameworks using Talend, Aecorsoft or Cloud Composer, ensuring reliable scheduling, dependency management, error handling and retry mechanisms.
- Serve as technical lead for the planned migration from GCP-native services to Databricks, partnering with architecture and product teams to define the target-state platform.
- Conduct proof-of-concept builds on Databricks (Delta Lake, Unity Catalog, Databricks SQL, Workflows) to validate migration patterns, performance benchmarks and cost models.
- Develop migration playbooks mapping current pipelines and data objects to Databricks equivalents, and execute phased cutovers with minimal business disruption.
- Establish Databricks best practices for the team, including notebook standards, cluster policies, job configurations and lakehouse design patterns.
- Integrate data quality checks into pipeline workflows and collaborate with the DQ function (e.g. Ataccama) to operationalise rules, thresholds and alerting.
- Support data governance activities, including cataloguing, lineage documentation and metadata management, and ensure adherence to defined SLAs for governed data objects.
- Monitor data freshness, completeness and accuracy signals, and proactively surface and remediate data anomalies.
- Own L2/L3 support for data pipelines and engineering workloads within the D&A portfolio, including incident triage, root-cause analysis and post-incident corrective actions.
- Build and maintain observability (monitoring dashboards, alerting and runbooks) to reduce MTTD and MTTR for pipeline failures and data SLA misses.
- Apply infrastructure-as-code practices (Terraform) for environment provisioning, IAM, networking and resource management on GCP.
- Embed FinOps discipline by enforcing resource tagging/labelling, identifying and remediating waste, and contributing to cost-optimization initiatives.
- Work with cross-functional teams, vendor partners and business stakeholders to translate business requirements into scalable data solutions.
- Mentor and guide junior engineers, conduct code reviews and share best practices.
- Participate in vendor governance activities, supporting evaluation, onboarding and management of technology and service partners.
What you'll need
- Bachelor's Degree or PG (Education / Certifications).
- 8+ years of hands-on data engineering experience with at least 3 years on GCP.
- Strong GCP fundamentals: BigQuery, Cloud Storage, Cloud Composer (Airflow), Dataflow, Dataproc, Pub/Sub, Cloud Functions, IAM, Monitoring/Logging.
- Proficient in SQL and Python.
- ETL/ELT pipeline design: proven ability to build, optimise and troubleshoot complex data pipelines at scale.
- Orchestration tools experience with Talend, Aecorsoft, Apache Airflow or similar.
- Working knowledge of Terraform for GCP resource provisioning.
- CI/CD practices with GitHub Actions, Jenkins or equivalent; comfortable with PR-based workflows and automated testing.
- Exposure to DQ tools (Ataccama, Great Expectations or similar) and metadata/lineage management concepts.
- Observability and monitoring experience: dashboards, alerts and logging for pipeline health (Cloud Monitoring, Grafana or Datadog).
- Strong troubleshooting skills and ability to work under pressure during incidents.
- Excellent communication and stakeholder management skills, able to explain technical concepts to non-technical audiences and collaborate across global teams.
- Navigating platform migrations with minimal business disruption.
- Leading technical initiatives and mentoring junior engineers.
- Working effectively with IT outsource partners and transitioning work from partner to internal teams.
Nice to have
- Databricks experience or strong interest: familiarity with Delta Lake, Unity Catalog, Databricks SQL or Spark-on-Databricks; hands-on POC experience preferred.
- PySpark or Spark SQL experience.
- Knowledge of BI tools such as Tableau, Excel, Alteryx, R or Python.
- Experience with Kubernetes/GKE for data workloads.
- Familiarity with data lakehouse architecture patterns.
- Exposure to ML pipeline support or feature engineering workflows.
- Security tooling awareness (Dependabot, SonarQube, Wiz or similar).
- Bash scripting and Linux administration basics.
- Experience working in a global IT function with regional or global responsibilities.
Details
- Location: Mumbai, India.
- No relocation support available.
- Job type: Regular.
Read the full description and apply on the company’s own careers page.