Lead Data Engineer

Overview

Lead the development of scalable, cloud-native data platforms supporting Mastercard’s global data ecosystem. Build resilient, high-performance batch and real-time data systems across data lakes and data warehouses.

What you'll do

• Design and build scalable batch and real-time data pipelines using Spark, Kafka, and (preferred) Apache Flink.
• Develop robust ETL/ELT frameworks for structured and unstructured data.
• Build and optimize data ingestion and transformation pipelines for Data Lakes and Data Warehouses.
• Implement stream processing solutions for near real-time use cases.
• Ensure data quality, lineage, observability, and governance across pipelines.
• Optimize data jobs for performance, scalability, and cost efficiency.
• Design and operate cloud-native data platforms on AWS, Azure, or GCP.
• Leverage managed services such as S3/ADLS/GCS, EMR/Databricks, BigQuery/Redshift/Snowflake.
• Implement Infrastructure as Code using Terraform, CloudFormation, or equivalent.
• Ensure high availability, fault tolerance, and disaster recovery.
• Drive cost optimization strategies for large-scale data workloads.
• Implement secure data access controls aligned with enterprise standards.
• Build reusable data frameworks, libraries, and pipeline templates.
• Drive adoption of CI/CD, automated testing, and observability.
• Develop and enhance developer tooling and platform capabilities.
• Contribute to cloud-agnostic platform architecture and automation.
• Provide technical leadership, mentorship, and design guidance.
• Conduct code reviews, architecture reviews, and best practice enforcement.
• Collaborate with architects, product owners, and cross-functional teams.
• Act as a Subject Matter Expert (SME) for data platform initiatives.
• Promote engineering excellence through documentation, design standards, and innovation.
• Work effectively across globally distributed teams.
• Abide by Mastercard’s security policies and practices.
• Ensure the confidentiality and integrity of accessed information.
• Report any suspected information security violation or breach.
• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

What you'll need

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
• Strong proficiency in Object-Oriented Programming and Design (OOP/OOAD).
• Java (JDK 8+).
• Experience building data services and distributed systems.
• Strong understanding of multithreading, scalability, and performance tuning.
• Strong hands-on experience with AWS, Azure, or GCP.
• Experience with cloud-native data services: S3, ADLS, GCS, Databricks, EMR, or BigQuery/Redshift.
• Strong experience with Apache Spark (Core, SQL, Structured Streaming).
• Hands-on experience with Kafka or equivalent messaging platforms.
• Experience with real-time processing frameworks: Apache Flink or Spark Streaming.
• Strong understanding of ETL/ELT design patterns and pipeline architectures.
• Experience with data formats: Parquet, Avro, or ORC.
• Knowledge of data modeling, including dimensional modeling and star/snowflake schemas.
• Proficiency in Infrastructure as Code: Terraform, CloudFormation, or ARM templates.
• Experience with Docker and Kubernetes.
• Solid understanding of cloud networking, IAM, and security best practices.
• Experience with workflow orchestration tools: Airflow or equivalent.
• Strong SQL skills and experience with Data Warehouse platforms.
• Understanding of data governance, lineage, and observability frameworks.
• Experience with CI/CD tools: Jenkins, GitHub Actions, etc.
• Strong testing practices using JUnit or equivalent frameworks.
• Experience with monitoring and observability tools: Splunk, Dynatrace, Prometheus, etc.
• Familiarity with performance testing tools: JMeter, Gatling.
• Understanding of secure development practices, including PCI DSS and GDPR.
• Proven ability to lead and mentor engineering teams.
• Strong problem-solving and system design skills.
• Passion for innovation, automation, and continuous improvement.
• Ability to operate effectively in a fast-paced, global environment.

Nice to have

• Python and/or Go.
• Apache Flink.

Details

• Location: Pune, India.