Senior Data Engineer-2

Overview

Senior Data Engineer for the Small and Medium Enterprise (SME) platform, owning data pipelines for an SME data mart on prem and on cloud and working with cross-functional teams to drive data-driven decisions.

What you'll do

• Independently design and implement scalable, reliable and reusable data pipelines using SQL, PySpark, Impala and other frameworks.
• Develop and maintain efficient ETL processes to handle large volumes of data (billion level) from different sources into centralized storage (data warehouse and lake house).
• Collaborate with data scientists, data analysts and software engineers in scrum teams to develop data solutions that enable access, analysis and visualization.
• Work with data architects to ensure data infrastructure aligns with data architecture and governance standards, and ensure data quality, security and compliance with industry regulations.
• Troubleshoot and resolve issues with data pipelines and integrations, and monitor and optimize performance of data systems.
• Maintain documentation for data pipelines, troubleshooting logs and data processing systems.
• Support the design and implementation of data models, algorithms and feature stores serving BI reporting and machine learning models.
• Mentor and support junior data engineers.
• Stay up to date with emerging data engineering technologies and recommend their adoption as appropriate.

What you'll need

• Demonstrated ability to independently design and implement data pipelines in large-scale data systems and distributed computing frameworks following best coding practice.
• Extensive hands-on experience with SQL and Spark.
• Experience with big data tools or technologies (Hadoop, NiFi, Kafka, Impala, Hive, Oozie, Airflow etc.).
• Knowledge of cloud-based data storage and processing systems (Azure, AWS, GCP, Databricks etc.).
• Experience with data modelling (SQL and NoSQL), data ETL and data warehousing concepts.
• Excellent communication skills, with the ability to work in a team and collaborate with data professionals and business stakeholders.
• Strong attention to detail and the ability to work under tight deadlines in a fast-paced environment.
• Working experience of using CI/CD to keep data pipelines scalable and reliable.
• Working experience of using common programming languages (Python preferred).
• BSc or higher degree in computer science or a relevant field.

Nice to have

• Experience with implementing analytical data pipelines for data science projects and experience creating feature stores.
• Databricks certifications (Data Engineer Professional, Hadoop Migration Architect).
• AWS Big Data or Data Analytics certification and/or Azure Data Engineer certification.

Details

• Location: Pune, India.
• Every person working for or on behalf of Mastercard is responsible for information security: abide by security policies, ensure confidentiality and integrity of information accessed, report suspected violations or breaches, and complete periodic mandatory security trainings.