Data Engineer, Specialist

Overview

Design, develop and optimize data pipelines and architectures that support scalable, efficient data processing, with technical ownership of business-critical data products and platforms, including production reliability and incident management.

What you'll do

• Develop and maintain efficient ETL (Extract, Transform, Load) pipelines that ingest, process and transform large datasets from diverse sources.
• Build and manage data storage solutions using relational and non-relational databases, ensuring high availability, performance and scalability.
• Work with APIs, third-party data sources and internal databases to create unified datasets that provide actionable insights for business stakeholders.
• Partner with business intelligence analysts, data scientists and software engineers to design and implement data-driven solutions that meet business requirements.
• Design optimized data schemas and structures that support analytical workloads and improve query performance.
• Leverage orchestration tools like Apache Airflow or Prefect to automate ETL workflows and improve operational efficiency.
• Provide technical guidance, review code and promote best practices within the data engineering team.
• Lead and mentor Data and AI Engineers, providing technical direction, coaching, and engineering guidance.
• Lead solution architecture and technical design, ensuring alignment with business objectives, product priorities, and engineering standards.
• Design and optimize scalable data pipelines, cloud data platforms, data models, and storage solutions supporting analytical and operational workloads.
• Architect reliable batch and real-time data processing solutions, incorporating data quality, governance, security, performance, and scalability.
• Partner with Product Owners, business stakeholders, Architects, Analysts, and Engineering teams to translate requirements into sustainable data solutions.
• Take end-to-end technical ownership across design, development, deployment, production reliability, and continuous improvement.
• Evaluate technical options and make informed architecture decisions considering performance, scalability, security, cost, reliability, and maintainability.
• Establish engineering and operational practices covering CI/CD, automation, testing, observability, monitoring, release readiness, and production support.
• Lead complex production incident resolution, root-cause analysis, preventive actions, and reliability improvements, while guiding engineers in troubleshooting and reducing operational risk.
• Drive engineering excellence through architecture/design/code reviews, reusable patterns, DevOps practices, and AI-assisted development.

What you'll need

• Minimum 8 years of relevant work experience, with at least 4 years focused on designing and developing scalable data pipelines, ETL frameworks and database management systems.
• Bachelor's degree (B.E./B.Tech) in Computer Science or IT, or Bachelor's in Computer Applications (BCA) from a recognized institution.
• Proficiency in SQL and database technologies such as PostgreSQL, MySQL, Oracle, or SQL Server.
• Hands-on experience with big data frameworks like Apache Spark, Hadoop, or Databricks.
• Strong programming skills in Python, Scala, or Java for data processing and automation.
• Experience with cloud platforms such as AWS, Azure, or GCP, particularly in data services like AWS Glue, Redshift, Big Query, or Snowflake.
• Deep hands-on experience architecting and delivering AWS-based data solutions using services such as S3, Glue, Lambda, Redshift, Athena, Step Functions, SNS/SQS, IAM, CloudWatch, Kinesis, and EMR.
• Ability to make architecture decisions covering scale, security, reliability, performance, and cost.
• Hands-on experience or strong working knowledge of Amazon Bedrock, including Knowledge Bases, Guardrails and Agents, along with SageMaker, Lambda, S3, OpenSearch Serverless vector capabilities, Step Functions, and infrastructure as code using CDK or CloudFormation.
• Strong analytical, problem-solving, troubleshooting, and technical decision-making skills.
• Effective communication and stakeholder management across technical, business, and product teams.
• Ability to navigate ambiguity, balance competing priorities, and proactively manage risks, dependencies, and trade-offs.

Details

• Location: Hyderabad, Telangana, India (Vanguard India). Only qualified external applicants will be considered.
• Hybrid working model for most employees.
• Willingness to provide technical leadership during critical production incidents, major releases, and business-critical situations outside regular working hours when required.