Sr. Specialist, Data Engineering

Overview

Design, build and maintain scalable data pipelines and platforms, ensuring pipeline reliability, performance and governance through automation, monitoring, logging and documentation. Operates within a matrix environment, participating in project planning, delivery and cross-team collaboration.

What you'll do

• Design, develop, and maintain data pipelines to extract data from various sources and populate a data lake and data warehouse.
• Work closely with data scientists, analysts, and business teams to understand data requirements and deliver solutions aligned with business goals.
• Build and maintain platforms that support data ingestion, transformation, and orchestration across various data sources, both internal and external.
• Use data orchestration, logging, and monitoring tools to build resilient pipelines.
• Automate data flows and pipeline monitoring to ensure scalability, performance, and resilience of the platform.
• Monitor, troubleshoot, and resolve issues related to the data integration platform, ensuring uptime and reliability.
• Maintain thorough documentation for integration processes, configurations, and code to ensure easy onboarding for new team members and future scalability.
• Develop pipelines to ingest data into cloud data warehouses.
• Create and deliver standard reports in accordance with stakeholder needs.
• Work within a matrix organizational structure, reporting to both the functional manager and the project manager.
• Participate in project planning, execution, and delivery, ensuring alignment with both functional and project goals.

What you'll need

• Bachelor's degree in Information Technology, Computer Science or any Technology stream.
• 7-11 years of experience in developing data pipelines and data infrastructure, ideally within a drug development or life sciences context.
• Strong SQL skills and solid programming experience in Python (including PySpark).
• Practical experience with AWS S3 and core AWS console concepts (IAM, regions, basic cloud architecture).
• Hands-on experience with Apache Spark (concepts and usage) and Databricks (platform use, Delta Lake, Unity Catalog).
• Experience with cloud object storage (e.g., Amazon S3) and cloud data warehouses or lakehouses (e.g., Redshift, Snowflake, Azure Synapse, Delta Lake).
• Familiarity with data modeling, ETL/ELT patterns, and data transformation best practices.
• Experience implementing monitoring, logging, and data quality checks for operational pipelines.
• Familiarity with CI/CD concepts and pipelines for data deployments; experience with Git-based tooling (GitHub Actions, Jenkins, or Azure DevOps).
• Proven ability to work collaboratively with data scientists, analysts, and business stakeholders.
• Good interpersonal and communication skills (verbal and written).
• Product and customer-centric approach and innovative thinking.

Nice to have

• Experience with AI Assisted code development, GitHub Copilot, people management, and team lead experience.
• Prior experience in a global or matrixed organization.
• Exposure to observability and data quality platforms (Monte Carlo, Great Expectations, Deequ).
• Knowledge of orchestration and scheduling tools (Airflow, Prefect, Azure Data Factory).

Details

• Location: Hyderabad (Hitec City Raidurg), India.
• Flexible work arrangement: Hybrid.
• Relocation: Domestic.
• Travel: No travel required.
• Visa sponsorship: No.