Senior Software Engineer - Python, Data Engineering, AI

Overview

Senior Software Engineer responsible for building and operating production data pipelines in Python, developing lakehouse platforms, and supporting AI-ready data products.

What you'll do

• Design and operate batch and incremental ETL/ELT pipelines in Python.
• Model Iceberg and curated tables and tune Trino/Athena SQL.
• Build reliable incremental ingestion from external APIs.
• Extend data-quality checks, anomaly detection, alerting, and on-call operations.
• Lead Databricks-on-Azure lakehouse workstreams and migration activities.
• Ship tested code through pull requests, infrastructure-as-code, and CI/CD.
• Mentor junior engineers and improve testing, observability, and documentation practices.

What you'll need

• 5–7 years of production data engineering experience.
• Strong Python 3.x skills, including pandas/pyarrow, packaging, testing, and shared libraries.
• Scalable Apache Spark/PySpark experience, including partitioning, joins, skew, and shuffle tuning.
• Advanced SQL on a distributed engine.
• Hands-on lakehouse experience with Apache Iceberg or Delta Lake.
• Cloud data platform experience on AWS or Azure.
• Experience with DAG orchestration, incremental ingestion, Git, CI/CD, data quality, and observability.

Nice to have

• Databricks, Azure data stack, AWS depth, infrastructure as code, Docker, migration, streaming, BI serving, or AI/LLM data experience.
• Experience with relational sources, SaaS data, statistics for data quality, or mentoring engineers.

Details

• Location: Hyderabad, Telangana, India.