Senior Data Engineer - Vice President - Python Development

Overview

Senior Data Engineer and Vice President responsible for building the data foundation supporting greenfield Generative AI products across Markets. The role combines hands-on Python development with architecture, data quality, security, and operational ownership.

What you'll do

• Design and evolve data architecture for Generative AI products.
• Build Python pipelines for billions of records from diverse sources.
• Create curated datasets and low-latency data-serving layers.
• Optimise storage, indexing, ingestion, transformation, and query performance.
• Implement data quality, lineage, reconciliation, schema evolution, and recovery controls.
• Build observability, automated tests, and CI/CD pipelines.
• Collaborate with AI engineers, software engineers, architects, product partners, and Markets stakeholders.

What you'll need

• Extensive hands-on experience in data engineering, software engineering, or a related discipline.
• Deep practical Python expertise building production-grade software.
• Experience delivering large-scale data platforms or data-intensive applications.
• Strong SQL and relational database skills, particularly PostgreSQL.
• Experience with data modelling, indexing, partitioning, and query optimisation.
• Experience with pandas, PyArrow, SQLAlchemy, NumPy, and Apache Parquet.
• Knowledge of ETL/ELT, data governance, security, testing, code review, and CI/CD.

Nice to have

• Experience with Airflow, Dagster, or Prefect.
• Familiarity with Generative AI, retrieval architectures, LLMs, or agentic systems.
• Knowledge of financial instruments, trading concepts, or capital markets data.
• Bachelor’s or master’s degree in Computer Science, Engineering, or a relevant quantitative discipline.

Details

• Location: Pune, Maharashtra, India.
• Employment type: Full time.