Senior Specialist, GSF DnA Data Engineer

Overview

Senior Data Engineer role to design and build trusted, scalable, and cost-efficient data platforms on AWS and Databricks, leading hands-on development of batch and streaming pipelines and acting as a technical mentor.

What you'll do

• Design, develop, and operate end-to-end data pipelines (ETL/ELT) to ingest data from diverse sources into an AWS-based data lakehouse and data warehouse (batch and/or streaming).
• Build curated datasets using dimensional modeling (star/snowflake), SCD patterns, and conformed dimensions to support BI and self-service analytics.
• Partner with product managers, analysts, and data scientists to understand requirements, define source-to-target mappings, and deliver accurate, discoverable, reusable datasets.
• Define and implement data quality controls (validation rules, reconciliations, anomaly checks), data contracts, and SLAs; partner with governance to maintain catalog, lineage, and metadata.
• Implement orchestration, logging, monitoring, and alerting to ensure reliable operations (data observability, pipeline health, backfills, incident triage).
• Apply engineering best practices: automated unit/integration tests, code reviews, and CI/CD to promote changes safely across environments.
• Develop transformations on Databricks using Python/PySpark and Spark SQL; optimize jobs for performance and cost.
• Write and optimize complex SQL for analysis, transformations, and warehouse/lakehouse consumption.
• Build on AWS using services such as S3, IAM, Glue, Lambda, Step Functions, EMR/ECS/Fargate, and CloudWatch; implement secure access patterns and least-privilege principles.
• Use Infrastructure as Code (Terraform) to provision and manage cloud resources; promote reusable modules and automated deployments.
• Package and run workloads using Docker where appropriate.
• Use GitHub for version control; follow branching strategies (e.g., trunk-based or GitFlow) and maintain high-quality pull requests.
• Process large datasets using PySpark and lakehouse formats (e.g., Delta/Parquet), applying best practices for reliability and scalability.
• Create reproducible prototypes and analyses using notebooks when appropriate, then productionize with packaging, testing, and documentation.
• Work in an Agile environment (Scrum/Kanban), contributing to sprint planning, estimation, demos, and continuous improvement.
• Create and maintain technical documentation (data flows, runbooks, data dictionaries) and contribute to team standards and playbooks.
• Coach and mentor other data engineers through onboarding, pairing, code/design reviews, and knowledge-sharing sessions.
• Provide technical leadership by proposing patterns/standards (lakehouse, warehousing, data quality, CI/CD), influencing the roadmap, and communicating trade-offs to stakeholders.

What you'll need

• Primary skills: AWS, Databricks, Python, PySpark.
• Secondary skills: CI/CD, SQL.
• 8+ years of experience in Data Engineering, building production-grade data platforms and pipelines.
• Strong AWS experience, including data services (S3, Glue, Lambda, Step Functions, EMR/ECS/Fargate) and operational tooling (CloudWatch).
• Proficient in Python and PySpark (Spark SQL), with strong data engineering fundamentals.
• Deep understanding of data warehousing and lakehouse concepts, including dimensional modeling (star/snowflake), SCDs, and performance patterns.
• Hands-on with Databricks Lakehouse on AWS (e.g., Delta Lake) and at least one cloud data warehouse (e.g., Redshift or similar).
• Experience working in Agile delivery teams, collaborating across roles and communicating clearly with stakeholders (requirements, trade-offs, timelines).
• Strong focus on data quality and operational excellence: testing, monitoring, alerting, root-cause analysis, and continuous improvement.
• Strong SQL skills (complex joins, window functions, query tuning) and ability to translate business logic into reliable transformations.
• Proficiency with GitHub, CI/CD pipelines, and DevOps fundamentals; familiarity with Docker and Infrastructure as Code (Terraform).
• Demonstrated ability to mentor and coach engineers while remaining hands-on in delivery.
• Bachelor's degree (or equivalent experience) in Computer Science, Engineering, or a related field.

Nice to have

• AWS certification (Developer, Solutions Architect, Data Analytics) and/or Databricks certification.
• Experience with data transformation frameworks (e.g., dbt) and workflow orchestration tools (e.g., Airflow).
• Experience building and operating data products (product thinking, domain-aligned datasets, documentation, SLAs, adoption/usage metrics).
• Experience with data governance and access control tools such as Collibra and Immuta.
• Streaming data experience (e.g., Kafka, Kinesis) and building near real-time pipelines.
• Exposure to data mesh and data fabric patterns, and experience defining reusable platform capabilities.

Details

• Location: Hyderabad, Telangana, India (Hitec City Raidurg).
• Work arrangement: Hybrid.
• Employment is regular (full-time); no relocation; no visa sponsorship; no travel required.
• Posting end date: 10/30/2026.