Distinguished Data Engineer

Overview

Hands-on AI data engineering role within the AMP Research & Sustainable Investing team, building the trusted data foundations that power machine learning, LLM, RAG, search and agentic AI workflows for investment research and sustainability use cases.

What you'll do

• Design, build, test, deploy and operate production-grade data pipelines for structured, semi-structured and unstructured research, sustainability and investment data.
• Build AI-ready datasets for machine learning and generative AI, including feature, training and evaluation datasets, embeddings, vector indexes and retrieval-augmented generation workflows.
• Engineer retrieval data flows that curate, enrich, index and serve trusted content for semantic search, vector search and LLM-based applications, with appropriate metadata and provenance.
• Help develop metadata and knowledge structures, including taxonomies, ontologies, entity resolution and knowledge graphs where appropriate, to improve data discovery, retrieval and contextual understanding.
• Build reusable data services and APIs that expose governed investment data to AI/ML models, applications, analytics and agentic workflows.
• Apply data quality, lineage, data contracts, security, privacy, entitlements, auditability and provenance controls across data pipelines and AI-ready data products.
• Use Snowflake, AWS, Kafka and enterprise data platforms to ingest and prepare data, integrating safely with Oracle and Microsoft SQL Server where source or legacy data is required.
• Apply strong software and data engineering practices, including Python/SQL development, Git, CI/CD, automated testing, observability, monitoring, performance tuning and secure production support.
• Partner with AI/ML engineers and investment stakeholders to translate AI use cases into reusable data products, trusted retrieval layers and measurable production outcomes.
• Investigate complex data, embedding, indexing, retrieval and pipeline issues, perform root-cause analysis and implement sustainable fixes with the Data Engineering Lead and wider team.

What you'll need

• Strong hands-on experience preparing structured, semi-structured and unstructured data for machine learning and generative AI, including feature, training and evaluation datasets.
• Practical experience building embedding pipelines, vector indexes, retrieval workflows and RAG data foundations for applications.
• Experience creating and curating evaluation datasets and applying data-quality, relevance, traceability and provenance checks to support reliable retrieval and model evaluation.
• Strong experience designing production-grade ingestion, transformation, enrichment and delivery pipelines using batch, near-real-time, streaming, API and event-based patterns.
• Strong proficiency in Python, Spark and SQL, with the ability to build maintainable, tested production code for data preparation, embeddings, indexing, retrieval and data services.
• Experience with Snowflake, AWS and Kafka, plus enterprise relational sources such as Oracle or Microsoft SQL Server; able to integrate modern data and retrieval workloads with existing platforms.
• Experience building reusable APIs and data services and using orchestration tools such as Airflow or Control-M to operate dependable production data workflows.
• Experience with Git, CI/CD, automated testing, infrastructure as code, data observability, monitoring, performance tuning and secure production engineering.
• Good understanding of data quality, lineage, data contracts, access control, privacy, entitlements, auditability, provenance and responsible AI controls across governed data products and AI-ready datasets.
• Good understanding of canonical and dimensional modelling, semantic layers and governed business concepts that can be consumed consistently by analytics and AI-enabled solutions.
• Demonstrated hands-on experience delivering production data engineering solutions for AI/ML use cases, particularly data pipelines and data foundations supporting machine learning, LLMs, search, retrieval and generative AI.
• Experience working closely with AI/ML engineers, data engineers, architects, analysts and business stakeholders to move data and retrieval workflows from experimentation into secure, reliable and supportable production services.
• Ability to own assigned data engineering components from design through build, testing, deployment, monitoring and production support.
• Strong communication skills, with the ability to explain data engineering, retrieval, metadata, data quality and governance topics clearly and translate investment and AI use cases into practical engineering solutions.

Details

• Location: Bengaluru
• Job type: Permanent
• Reports to: Data Engineering Lead, Research & Sustainable Investing
• Level: 5
• Application deadline: 14 November 2026
• Flexibility on where and when you work is offered, balancing team and individual needs.