Senior Data Engineer

Overview

Design, build and maintain scalable data pipelines and integrations within a Databricks-based data platform, delivering reliable, secure and analytics-ready datasets. Reports directly to the Senior Manager - Business Insights & Analytics.

What you'll do

• Design, develop and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL and Delta Lake.
• Design and implement a medallion architecture (Bronze, Silver and Gold layers) to deliver trusted, well-structured, analytics-ready datasets.
• Build and maintain integrations with enterprise applications, REST APIs, databases and other source systems.
• Develop reusable ingestion frameworks to accelerate onboarding of new data sources.
• Build reliable batch and near-real-time ingestion pipelines with incremental loads, CDC, schema evolution and error recovery.
• Configure and manage Databricks Workflows and implement ingestion and transformation pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines.
• Optimize Apache Spark and Delta Lake workloads using execution-plan analysis, efficient join strategies, data-skew handling, partitioning and SQL query tuning to improve performance and control compute costs.
• Implement Unity Catalog governance, including access controls, lineage, auditing, and sensitive-data protection through masking and row-level security.
• Maintain and support existing data workflows running on Amazon Redshift and Amazon EC2, collaborating with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments and operational issues.
• Follow engineering best practices, including version control, code reviews, automated testing, CI/CD and technical documentation.
• Partner with Data Analytics and business stakeholders to translate requirements into trusted, well-documented, analytics-ready datasets.
• Provide technical guidance to team members and identify opportunities to improve automation, engineering standards, platform reliability and resource utilization.

What you'll need

• 5+ years of experience in data engineering or a related technical role, including developing and supporting production data pipelines.
• Strong hands-on expertise in Databricks, including developing, troubleshooting and optimizing production workloads.
• Strong proficiency in Python/PySpark, SQL and Apache Spark for scalable data processing and transformation.
• Practical experience with Delta Lake, including table management, incremental processing and performance optimization.
• Experience with data architecture and modeling, including medallion architecture and the development of analytics-ready datasets.
• Experience with Databricks Workflows or equivalent orchestration solutions.
• Proven experience building data ingestion pipelines and integrating REST APIs, databases and enterprise applications.
• Understanding of reliable data processing patterns, including retries, idempotency, schema evolution, backfills and failure recovery.
• Working knowledge of AWS services and cloud infrastructure, particularly IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms.
• Experience with Git/GitHub, code reviews, automated testing and CI/CD practices.
• Strong problem-solving skills and the ability to independently design, build, deploy and maintain production data solutions.
• Good communication and collaboration skills to work effectively with Data Analytics, Engineering, IT/Cloud, Security and business stakeholders.

Nice to have

• Experience with Databricks Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), Auto Loader and advanced incremental ingestion patterns.
• Experience implementing Unity Catalog governance, including fine-grained access controls, data lineage, auditing and sensitive-data protection.
• Experience with Databricks Genie, Genie Agents and Genie Code.
• Experience in AI/ML, Data Science, LLMs, contextualization and Amazon Bedrock.

Details

• Location: Noida, India.
• Work mode: Hybrid.