Data Engineer, Specialist

Overview

Design, develop and optimize data pipelines and architectures that support scalable data processing, with technical ownership of business-critical data products and platforms, including production reliability and incident management.

What you'll do

• Develop and maintain efficient ETL pipelines that ingest, process and transform large datasets from diverse sources.
• Build and manage relational and non-relational data storage solutions ensuring high availability, performance and scalability.
• Work with APIs, third-party data sources and internal databases to create unified datasets that provide actionable insights for business stakeholders.
• Partner with business intelligence analysts, data scientists and software engineers to design and implement data-driven solutions that meet business requirements.
• Design optimized data schemas and structures that support analytical workloads and improve query performance.
• Leverage orchestration tools like Apache Airflow or Prefect to automate ETL workflows and improve operational efficiency.
• Provide technical guidance, review code and promote best practices within the data engineering team.
• Lead and mentor Data and AI Engineers, providing technical direction, coaching and engineering guidance.
• Lead solution architecture and technical design, ensuring alignment with business objectives, product priorities and engineering standards.
• Design and optimize scalable data pipelines, cloud data platforms, data models and storage solutions supporting analytical and operational workloads.
• Architect reliable batch and real-time data processing solutions, incorporating data quality, governance, security, performance and scalability.
• Partner with Product Owners, business stakeholders, Architects, Analysts and Engineering teams to translate requirements into sustainable data solutions.
• Take end-to-end technical ownership across design, development, deployment, production reliability and continuous improvement.
• Evaluate technical options and make architecture decisions considering performance, scalability, security, cost, reliability and maintainability.
• Establish engineering and operational practices covering CI/CD, automation, testing, observability, monitoring, release readiness and production support.
• Lead complex production incident resolution, root-cause analysis, preventive actions and reliability improvements, guiding engineers in troubleshooting and reducing operational risk.
• Drive engineering excellence through architecture, design and code reviews, reusable patterns, DevOps practices and AI-assisted development.

What you'll need

• Minimum 8 years of relevant work experience, with at least 4 years focused on designing and developing scalable data pipelines, ETL frameworks and database management systems.
• Bachelor's degree (B.E./B.Tech) in Computer Science or IT, or Bachelor's in Computer Applications (BCA) from a recognized institution.
• Proficiency in SQL and database technologies such as PostgreSQL, MySQL, Oracle or SQL Server.
• Hands-on experience with big data frameworks like Apache Spark, Hadoop or Databricks.
• Strong programming skills in Python, Scala or Java for data processing and automation.
• Experience with cloud platforms such as AWS, Azure or GCP, particularly in data services like AWS Glue, Redshift, BigQuery or Snowflake.
• Deep hands-on experience architecting and delivering AWS-based data solutions using services such as S3, Glue, Lambda, Redshift, Athena, Step Functions, SNS/SQS, IAM, CloudWatch, Kinesis and EMR.
• Ability to make architecture decisions covering scale, security, reliability, performance and cost.
• Hands-on experience or strong working knowledge of Amazon Bedrock, including Knowledge Bases, Guardrails and Agents, along with SageMaker, Lambda, S3, OpenSearch Serverless vector capabilities, Step Functions, and infrastructure as code using CDK or CloudFormation.
• Strong analytical, problem-solving, troubleshooting and technical decision-making skills.
• Effective communication and stakeholder management across technical, business and product teams.
• Ability to navigate ambiguity, balance competing priorities, and proactively manage risks, dependencies and trade-offs.

Details

• Based in Hyderabad, Telangana at Vanguard India.
• Hybrid working model for most employees.
• Willingness to provide technical leadership during critical production incidents, major releases and business-critical situations outside regular working hours when required.
• Only qualified external applicants will be considered.