AI/ML Data Engineer

Overview

Build and optimize scalable data engineering pipelines and Feature Store workflows for AI/ML data solutions. The role focuses on ingesting, transforming, modeling, and governing data from diverse sources.

What you'll do

• Develop and optimize data pipelines, workflows, and Feature Store solutions.
• Design and architect scalable data engineering pipelines for storage and processing.
• Implement data transformations and quality checks for accuracy, completeness, security, and consistency.
• Integrate data from diverse sources using AWS S3 and potentially Snowflake.
• Create advanced data models and schemas.
• Apply data governance and management best practices.

What you'll need

• 5+ years of data analysis and engineering experience.
• Bachelor's degree in computer science, statistics, informatics, information systems, or another quantitative field.
• Advanced web-crawling and scraping skills.
• Experience building end-to-end pipelines for semi-structured and unstructured data.
• Proficiency in Python, PySpark, SQL, RDBMS, and ETL/ELT.
• Working knowledge of APIs or stream-based extraction, including Salesforce APIs.
• Experience with SQL data warehouses such as Snowflake.

Nice to have

• Databricks experience.
• Familiarity with AWS Glue, EMR, EC2, RDS, monitoring, and IAM.
• Kafka, Spark, and Kafka Streaming experience.
• Workflow automation using GitHub Actions.
• Experience performing root-cause analysis.

Details

• Hybrid role requiring an average of 2 days per week from an HPE office.
• Location: Bengaluru, Karnataka, India.