Data Engineer

Overview

Build and operate scalable, secure, high-performance data pipelines on AWS that support data ingestion for upstream and downstream applications within the Data Ingest Batch Integration organization.

What you'll do

• Design, develop, and maintain distributed data pipelines on AWS that ingest, process, and deliver data at scale.
• Implement both batch and event-driven data processing patterns using AWS-native services.
• Build and support event-driven data solutions using asynchronous, decoupled architectures to enable near real-time processing and system scalability.
• Provide hands-on engineering using Python and AWS Glue with PySpark to process large datasets efficiently.
• Apply best practices in distributed systems, performance optimization, and fault tolerance.
• Ensure end-to-end data quality, implement validation and monitoring checks, establish data lineage, and manage orchestration workflows to ensure reliable and auditable data movement.
• Build and deploy data services using Docker containers and manage workloads on Kubernetes following cloud-native and DevOps best practices.
• Collaborate with data consumers, architects, platform teams, and business stakeholders to gather requirements, design scalable solutions, and deliver high-quality outcomes.
• Contribute to monitoring, logging, alerting, and incident response for data platforms.
• Ensure systems meet reliability, performance, and security standards.

What you'll need

• B.E in Computer Science, Engineering, or equivalent practical experience.
• 4+ years of experience in data engineering or distributed systems development.
• Strong hands-on experience with AWS cloud services.
• Advanced proficiency in Python.
• Hands-on experience with Apache Spark and AWS.
• Experience with AWS services such as Glue, Lambda, Step Functions, EventBridge, S3, Athena, and RDS Aurora PostgreSQL.
• Experience building event-driven data pipelines.

Nice to have

• Experience designing distributed data processing architectures.
• Strong knowledge of data quality, data lineage, and orchestration.
• Experience in API development using AWS Lambda, API Gateway, and Python.
• Hands-on experience with Docker and Kubernetes.

Details

• Location: Hyderabad.