Lead Python and Spark Data Engineer - Vice President

Overview

Lead the development of Citi's Futures and Derivatives Clearing data and reporting platform, owning the architecture, design and delivery of scalable data pipelines and warehouse solutions that support downstream reporting applications.

What you'll do

• Own a large-scale data warehouse, develop data products, and lead the architecture, design, development and maintenance of scalable and secure data architectures.
• Identify, evaluate and implement improvements to legacy systems, ensuring alignment with the target architecture design.
• Provide strong leadership to engineering teams, including mentoring, code review and enforcing engineering best practices.
• Own and define data models, taxonomies, metadata, data mappings, data standards, versioning and semantic layers for domains and data products, supporting data consumption, AI enablement and regulatory requirements.
• Build automated data integration solutions and data pipelines, including reference data, to meet business and reporting requirements.
• Implement data governance and management frameworks, quality standards and data controls to ensure data accuracy and security.
• Manage and maintain data process documentation.
• Define target-state data architecture and drive delivery against the strategy, managing execution through to delivery.
• Contribute to technical direction and strategic decisions as a technical expert in all development and implementation phases.
• Work closely with business users, stakeholders, BAs, product managers and other development teams to understand the problem space and develop solutions.
• Contribute to CI/CD implementation using DevOps best practices and ensure a robust testing framework for functional and nonfunctional testing across release cycles.
• Demonstrate a data-driven approach to problem-solving and decision-making, using metrics and performance indicators to assess team efficiency and identify improvement opportunities.
• Manage and maintain data processes and jobs.
• Partner with the Support team to ensure production stability through proactive monitoring and health checks.
• Partner with other Citi development teams to build APIs, integrations and shared components critical to the platform.

What you'll need

• Strong experience designing and building best-in-class enterprise data architecture in a highly matrixed, complex organization.
• Strong experience architecting complex, end-to-end data pipelines within highly available, distributed systems, covering extraction, ingestion and processing of large-scale datasets.
• Strong hands-on, advanced development experience with very large databases, complex data models, stored procedures, error handling, performance tuning and profiling, data security, database optimization and configurations.
• Strong experience in large complex physical and logical data model design and development for data warehouse, reporting and reference and master data.
• Hands-on experience with complex data integration and developing scalable, high-volume data pipelines using APIs, Kafka, Python, Java, Airflow and Spark.
• Proven experience developing data quality and reconciliation controls, data lineage tracking and data catalog documentation, establishing data control monitoring and proactively identifying and resolving data inconsistencies and gaps.
• Excellent verbal and written communication skills, enabling effective collaboration with cross-functional teams, stakeholders and management.
• Strong understanding of trading data and experience working within the financial services industry.
• Experience in implementing DevOps, such as CI/CD, modern best practices such as testing frameworks and the SOLID principles, and CI/CD tooling and test automation tools.
• Experience working in an Agile/Scrum work environment.
• Experience managing agile scrum teams and end-to-end delivery.
• Passion and commitment for adopting new technologies with an exploratory mindset.
• Innovative ideas and a can-do attitude to work under pressure.
• Bachelor's degree/University degree or equivalent experience in computer sciences, engineering, math/statistics.

Nice to have

• Strong hands-on experience in S3, Starburst, Oracle database, Databricks and Snowflake.
• Strong hands-on experience with modern software paradigms, including cloud-native (AWS, Docker) and microservices.
• Experience building observability dashboards for data processes and job monitoring.
• Experience in GenAI/Agentic, AI workflow, AI governance, AI agent and MCP automation development.

Details

• Location: Pune, Maharashtra, India.
• Employment type: Full time.