Senior Specialist, Data Engineering

Overview

Senior Data Engineer role at MSD in Hyderabad, designing, building and maintaining scalable data pipelines and platforms, with reliability, performance and governance ensured through automation, monitoring, logging and documentation.

What you'll do

• Design, develop, and maintain data pipelines to extract data from various sources and populate a data lake and data warehouse.
• Work closely with data scientists, analysts, and business teams to understand data requirements and deliver solutions aligned with business goals.
• Build and maintain platforms that support data ingestion, transformation, and orchestration across various data sources, both internal and external.
• Use data orchestration, logging, and monitoring tools to build resilient pipelines.
• Automate data flows and pipeline monitoring to ensure scalability, performance, and resilience of the platform.
• Monitor, troubleshoot, and resolve issues related to the data integration platform, ensuring uptime and reliability.
• Maintain thorough documentation for integration processes, configurations, and code to ensure easy onboarding for new team members and future scalability.
• Develop pipelines to ingest data into cloud data warehouses.
• Create and deliver standard reports in accordance with stakeholder needs.
• Work within a matrix organizational structure, reporting to both the functional manager and the project manager.
• Participate in project planning, execution, and delivery, ensuring alignment with both functional and project goals.

What you'll need

• Bachelor's degree in Information Technology, Computer Science or any Technology stream.
• 7-11 years of experience in developing data pipelines & data infrastructure, ideally within a drug development or life sciences context.
• Strong SQL skills and solid programming experience in Python (including PySpark).
• Practical experience with AWS S3 and core AWS console concepts (IAM, regions, basic cloud architecture).
• Hands-on experience with Apache Spark (concepts and usage) and Databricks (platform use, Delta Lake, Unity Catalog).
• Experience with cloud object storage (e.g., Amazon S3) and cloud data warehouses or lakehouses (e.g., Redshift, Snowflake, Azure Synapse, Delta Lake).
• Familiarity with data modeling, ETL/ELT patterns, and data transformation best practices.
• Experience implementing monitoring, logging, and data quality checks for operational pipelines.
• Familiarity with CI/CD concepts and pipelines for data deployments, and experience with Git-based tooling (GitHub Actions, Jenkins, or Azure DevOps).
• Proven ability to work collaboratively with data scientists, analysts, and business stakeholders.
• Good interpersonal and communication skills (verbal and written).
• Product and customer-centric approach and innovative thinking.

Nice to have

• Experience with AI Assisted code development, GitHub Copilot, people management, and team lead experience.
• Prior experience in a global or matrixed organization.
• Exposure to observability and data quality platforms (Monte Carlo, Great Expectations, Deequ).
• Knowledge of orchestration and scheduling tools (Airflow, Prefect, Azure Data Factory).

Details

• Location: Hyderabad (Hitec City Raidurg), India.
• Flexible work arrangement: Hybrid.
• No travel required.
• No visa sponsorship.
• Relocation: Domestic.