Overview
Data Engineer based in BMS Hyderabad, part of the Data Platform team, delivering data and analytics capabilities for Data Platforms and the Data Engineering Community.
What you'll do
- Develop and maintain ETL/ELT pipelines for ingesting data from various sources into the data warehouse.
- Take end-to-end ownership, innovate and drive initiatives through completion.
- Optimize data storage and retrieval to ensure efficient performance and scalability.
- Collaborate with data architects, data analysts and data scientists to understand their data needs and ensure the data infrastructure supports them.
- Ensure data quality and integrity through data validation and testing.
- Implement and maintain security protocols to protect sensitive data.
- Stay up-to-date with emerging trends and technologies in data engineering and analytics.
- Closely partner with the Enterprise Data and Analytics Platform team, other functional data teams and Data Community lead to enable adoption of data and technology strategy.
- Stay knowledgeable of evolving trends in data platforms and product-based implementation.
- Work in a fast-paced environment with minimal oversight.
- Create and maintain optimal data pipeline architecture and assemble large, complex data sets that meet functional and non-functional business requirements.
- Identify, design and implement internal process improvements, including automating manual processes, optimizing data delivery and re-designing infrastructure for greater scalability.
- Establish agile and product-oriented teams that work effectively with teams in the US and other global BMS sites.
- Lead analysis of current states, deliver strong recommendations on complexity in the environment, and execute to bring complex solutions to completion.
What you'll need
- 3-5 years of hands-on experience implementing and operating data capabilities and cutting-edge data solutions, preferably in a cloud environment.
- Breadth of experience across the full life cycle of data management, including data lakehouses, master/reference data management, data quality and analytics/AI ML.
- Expertise in Databricks.
- Expertise in designing and building real-time data ingestion pipelines in AWS.
- Expertise in CloudFormation, including developing and using Custom Resources in CFTs.
- Expertise in developing GitHub Workflows and integrating them with AWS.
- Expertise using boto3 APIs for Lambda, S3, Glue, Crawlers, Data Zone (Data Zone listed as preferred).
- Expertise in building APIs using AWS services.
- In-depth knowledge and hands-on experience with AWS Glue services and the AWS data engineering ecosystem.
- Hands-on experience developing and delivering data and ETL solutions with AWS data services (Redshift, Athena, Lake Formation, etc.).
- 5+ years of experience in data engineering or software development.
- Strong programming skills in languages such as Python, R, PyTorch, PySpark, Pandas, Scala.
- Experience with SQL and database technologies such as MySQL, PostgreSQL, Presto.
- Experience with cloud-based data technologies such as AWS, Azure or Google Cloud Platform.
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration skills.
- Prior experience working in an Agile/product-based environment.
Nice to have
- Experience with AWS Data Zone (preferred, boto3 usage).
- Experience with Cloudera Data Platform and Tableau labs.
- Functional knowledge or prior experience in Life Sciences Research and Development.
Details
- Location: Hyderabad, Telangana, India.
- Work model is assigned based on core responsibilities (site-essential, site-by-design, field-based or remote-by-design).
Read the full description and apply on the company’s own careers page.