Lead Data Engineer

Overview

As a Lead Data Engineer within Mastercard's Data Collection & Engineering organisation, you will design, build, and scale next-generation data platforms and analytical ecosystems. You will develop high-performance, cloud-enabled data pipelines supporting enterprise data warehouse and lakehouse environments, advanced analytics, business intelligence, regulatory reporting, and data-driven decision making.

What you'll do

• Design, develop, test, and deploy high-quality, secure, scalable, and resilient data pipelines using Apache Spark, Java/Scala across Hadoop and cloud-native object storage platforms.
• Build and maintain batch and near real-time data processing frameworks capable of supporting petabyte-scale workloads.
• Develop reusable engineering components and frameworks that accelerate data product delivery while maintaining enterprise standards.
• Design and implement a “build once, run anywhere” architecture supporting seamless deployment across on-premises and public cloud environments without code changes.
• Implement data lineage, metadata management, data cataloguing, data quality controls, and observability capabilities across the data ecosystem.
• Collaborate with architects and platform teams to establish scalable design patterns and engineering best practices.
• Contribute to migration initiatives moving legacy ETL and data warehouse workloads from on-premises environments to cloud-native architectures.
• Leverage cloud services such as Amazon S3, EMR, Glue, and related data services to improve scalability, reliability, and operational efficiency.
• Drive adoption of modern lakehouse and distributed compute architectures.
• Lead end-to-end development activities including requirement analysis, solution design, coding, testing, deployment, and production support.
• Mentor and guide junior engineers through code reviews, technical coaching, and engineering best practices.
• Partner with product owners, analysts, architects, and business stakeholders to deliver high-quality solutions within committed timelines.
• Troubleshoot complex production incidents and perform root cause analysis to identify and implement long-term remediation strategies.
• Ensure compliance with Mastercard's engineering, security, quality assurance, and operational governance standards.
• Continuously identify opportunities to improve performance, automation, monitoring, and process efficiency.
• Evaluate emerging data technologies and conduct proof-of-concept initiatives to determine their applicability within Mastercard's data ecosystem.
• Contribute to engineering innovation and continuous improvement initiatives across the organisation.
• Abide by Mastercard’s security policies and practices.
• Ensure the confidentiality and integrity of the information being accessed.
• Report any suspected information security violation or breach.
• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

What you'll need

• 10-12 years of experience delivering enterprise-scale Data Warehouse, Data Lake, or Data Lakehouse solutions.
• Proven experience implementing multiple end-to-end data engineering projects within large-scale distributed computing environments.
• Hands-on experience migrating ETL and analytics workloads from on-premises platforms to cloud-native environments.
• Strong development experience using Apache Spark, Scala or Java, Hadoop ecosystem technologies, and Object Storage platforms.
• Experience building orchestration and workflow solutions using Apache Airflow, Apache NiFi, and similar enterprise scheduling frameworks.
• Strong SQL expertise and experience with relational and NoSQL database technologies including Oracle, SQL Server, Cassandra, and DynamoDB.
• Working knowledge of cloud platforms, preferably AWS, including Amazon S3, EMR, AWS Glue, and cloud-native data services.
• Strong analytical and problem-solving capabilities.
• Experience operating within Agile delivery environments.
• Excellent written and verbal communication skills.
• Proven ability to collaborate within geographically distributed and matrix-based teams.
• Self-starter with strong ownership, accountability, and execution focus.
• Ability to learn emerging technologies quickly and apply them effectively to business challenges.

Details

• Location: Hyderabad, India.