Data Engineer II

Overview

Data Engineer II role on Mastercard's Data Platform & Orchestration team, building cloud-native data platforms that support Mastercard's global data ecosystem.

What you'll do

• Develop and maintain batch and near real-time data pipelines using Java, Python, Spark, and SQL.
• Build and support ETL/ELT pipelines for processing structured and semi-structured data.
• Contribute to scalable data processing solutions across Data Lake and Data Warehouse environments.
• Assist in developing real-time data ingestion and streaming solutions using Kafka and Spark Structured Streaming.
• Perform data validation, reconciliation, and quality checks to ensure reliable data delivery.
• Collaborate with senior engineers to troubleshoot and resolve data pipeline and production issues.
• Write efficient SQL queries and optimize data processing jobs for performance and scalability.
• Develop and deploy applications and data pipelines in cloud environments such as AWS, Azure, or GCP.
• Contribute to automation, monitoring, and observability initiatives to improve platform reliability.
• Support CI/CD activities including code integration, testing, deployment, and release processes.
• Participate in code reviews and follow engineering best practices for secure and maintainable software development.
• Build reusable components and utilities that improve team productivity and consistency.
• Collaborate effectively with product owners, architects, data engineers, and platform teams.
• Support AI and machine learning initiatives by developing, validating, and preparing data pipelines used for analytics, predictive modeling, and Generative AI applications.
• Explore modern AI-enabled data engineering capabilities, including metadata-driven pipelines, data quality automation, and intelligent operational monitoring.
• Continuously learn and adopt new technologies, tools, and engineering practices.

What you'll need

• 1-2 years of experience in software engineering, data engineering, or related technical roles.
• Knowledge of Java (JDK 8+) or Python programming.
• Understanding of object-oriented programming concepts and software development fundamentals.
• Familiarity with REST APIs and microservices concepts.
• Basic hands-on experience with Apache Spark, SQL, and data processing concepts.
• Understanding of ETL/ELT workflows and data pipeline architectures.
• Knowledge of relational databases and data modeling fundamentals.
• Basic understanding of Kafka or other messaging platforms.
• Familiarity with distributed systems concepts and data processing at scale.
• Experience or academic exposure to AWS, Azure, or GCP.
• Familiarity with cloud storage and compute services such as S3, ADLS, Databricks, or similar platforms.
• Basic understanding of containerization technologies such as Docker.
• Familiarity using Git for source code management, and CI/CD tools such as Jenkins, GitHub Actions, or similar platforms.
• Understanding of software testing concepts and unit testing frameworks.
• Basic understanding of AI/ML concepts and how data engineering supports model development and deployment.
• Strong analytical and problem-solving skills.
• Good verbal and written communication skills.
• Ability to learn quickly and adapt to new technologies.
• Ability to work effectively in a fast-paced global environment.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Nice to have

• AI exposure.
• Exposure to monitoring and logging tools.
• Familiarity with Python-based data analysis libraries such as Pandas or NumPy.
• Interest in Generative AI, Large Language Models (LLMs), or AI-powered data solutions.

Details

• Location: Pune, India.
• Corporate security responsibility: abide by Mastercard's security policies, ensure confidentiality and integrity of information accessed, report suspected security violations, and complete mandatory security training.