Overview
The Senior Data Engineer will be part of the Data Engineering and Analytics team, designing, building, validating and supporting enterprise data pipelines that integrate SAP S/4HANA data into the Databricks Lakehouse through SAP Datasphere and Amazon S3. The engineer owns assigned SAP datasets end to end within approved architecture, security and change-control standards.
What you'll do
- Own end-to-end onboarding of assigned SAP objects following the standard onboarding runbook, including source analysis, CDS wrapper view creation, Datasphere replication flow setup, Bronze ingestion configuration, validation and sign-off.
- Design, build and maintain Bronze ingestion pipelines in Databricks (Auto Loader, Lakeflow Declarative Pipelines, PySpark, Delta Lake) for full, delta, CDC and watermark loads, handling restartability, deletes and schema changes within approved design patterns.
- Operate and monitor Datasphere replication flows into Amazon S3, recover failed runs, and work with the platform administration team to resolve connectivity and capacity issues.
- Run source-to-Bronze reconciliation as a continuous control; review PASS/FAIL results, investigate aggregate and record-level discrepancies, and document corrective actions.
- Plan and execute SIT/UAT scenarios covering full loads, deltas, deletes, and field additions or removals; reconcile SAP validation output against Bronze data and compile acceptance evidence.
- Monitor pipeline observability dashboards and alerts; resolve ingestion failures and schema drift, lead root cause analysis (RCA) for complex issues, and implement preventive actions.
- Promote all changes through Git and CI/CD, review peer code and configurations, and apply approved Unity Catalog access controls for assigned datasets.
- Develop automation and reusable components that improve onboarding, reconciliation, monitoring and recovery.
- Maintain the SAP object inventory (extraction method, load pattern, Bronze location, delivery wave, status and exceptions) along with per-object operational documentation, recovery procedures and knowledge-transfer material.
- Resolve Jira and ServiceNow tickets for SAP data pipelines, record hypercare incidents and resolutions, mentor junior engineers, and provide after-hours on-call support on a rotating basis.
What you'll need
- 6+ years of enterprise data engineering or data integration experience, including hands-on Databricks delivery and production SAP data integration.
- Hands-on Databricks experience with PySpark, Spark SQL, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines (DLT), Workflows, Databricks SQL and Unity Catalog.
- Advanced SQL and Python, including reconciliation queries, aggregate comparisons, record-level validation and automation scripting.
- SAP S/4HANA knowledge, including core table structures, CDS view development, ODP delta extraction and record-level verification (SE16).
- Strong understanding of full, delta, pseudo-delta and CDC loads, including watermarks, delete handling, restartability, error handling and schema drift.
- Practical experience with Git, code review, testing and CI/CD deployment across environments.
- Experience with pipeline monitoring and incident management using Databricks dashboards, Grafana, Jira and ServiceNow.
- Proven ability to troubleshoot complex production issues, lead root cause analysis and maintain clear technical documentation.
- Strong communication and collaboration skills across SAP, platform and business teams, with experience mentoring junior engineers.
- Flexibility to work occasional non-standard hours and participate in a rotating on-call schedule; international travel may be needed.
- Typically requires a minimum of 6 years of related experience with a Bachelor's degree; or 3 years and a Master's degree; or a PhD without experience; or equivalent work experience.
Nice to have
- Experience with SAP Datasphere / Business Data Cloud replication flows, or replication tools such as SLT, HVR or Fivetran.
- SIT/UAT, cutover and hypercare experience on an ERP program.
- Exposure to dbt.
- Experience using AI-enabled engineering tools, such as Cursor, Claude, ChatGPT or GitHub Copilot.
Details
- Location: Bengaluru (Manyata), India.
- Not eligible for visa sponsorship.
- Occasional non-standard hours and rotating on-call; international travel may be needed.
Read the full description and apply on the company’s own careers page.