Data Engineer II

Overview

Design, build and maintain cloud data pipelines and workflows that serve reporting and analytics for ABC Fitness, working from the Hyderabad office 2-3 days a week.

What you'll do

• Design, develop, and maintain efficient data pipelines to serve reports using cloud ETL/ELT tools such as Azure Data Factory, Apache Airflow, and Databricks.
• Design, implement, and manage data workflows, notebooks, and jobs, ensuring seamless data orchestration and integration across cloud platforms and Databricks.
• Create and optimize SQL objects, including stored procedures, tables and views for great performance.
• Implement data quality checks and validation processes to ensure accuracy, completeness, and consistency of data across different stages of the pipeline.
• Monitor data pipelines and processes, troubleshoot issues, and implement solutions to prevent recurrence.
• Work on own initiative and take responsibility for delivery of high-quality solutions.
• Collaborate with stakeholders to understand reporting requirements and provide support in developing interactive dashboards using Power BI for data visualization.
• Maintain comprehensive documentation of data pipelines, workflows, and data models.
• Adhere to best practices in data engineering and ensure compliance with organizational standards.

What you'll need

• 2+ years of experience in a data engineering role.
• Bachelor's degree in computer science, Information Technology, or a related field.
• Experience in data management best practices, including demonstrated experience with data profiling, sourcing, and cleansing routines utilizing typical data quality functions involving standardization, transformation, rationalization, linking and matching.
• Proficient in SQL and Python, with the ability to translate complexity into efficient code.
• Experience working with different types of databases and data platforms, including Azure SQL DB, Azure Synapse SQL Pool, AWS Redshift, MySQL, Databricks SQL/Delta Lake, etc.
• Experience with Azure DevOps and/or GitHub.
• Experience with Azure Data Factory, Apache Airflow, and/or Databricks workflows/jobs.
• Hands-on experience with Databricks, including notebooks, clusters, Delta Lake, and performance optimization for scalable data processing.
• Effective communication skills (verbal and written) in English.
• Genuine passion about technology and solving data problems.
• Structured thinking with the ability to break down ambiguous problems and propose impactful data modeling designs.
• Ability to use data to inform decision making and drive outcomes.
• Ability to understand, document and convert business requirements into data models.
• Flexibility to work overlapping U.S. business hours to support collaboration with global teams.
• Driven and self-motivated with excellent organizational skills.
• Comfortable learning innovative technologies and systems.

Nice to have

• Experience building data models for Power BI.
• Working knowledge of Gen 2 Azure Data Lake, Storage Account, Blobs, Azure Function, Logic App.
• Working knowledge of Databricks Unity Catalog, Delta Live Tables, and medallion architecture patterns.
• Working knowledge of AWS S3, EMR.

Details

• Location: Hyderabad office, India.
• Work mode: Hybrid, working from the Hyderabad office 2-3 days a week.
• Must be able to work from the Hyderabad office 2-3x a week.
• Flexibility to work overlapping U.S. business hours.