Research Engineer, AI Evaluations

Overview

Design, build and run AI model evaluations end to end, measuring how well AI models serve people in the languages and contexts global benchmarks miss.

What you'll do

• Design benchmarks, rubrics and metrics, and choose between human, automated and hybrid evaluation based on reliability, cost and validity trade-offs.
• Deploy and serve models through APIs and self-hosted GPU inference, with versioned configs, logging and controls for reproducible results at scale.
• Analyse evaluation data with sound statistical methods (agreement, uncertainty, significance, disaggregation), and validate automated judges against human ground truth.
• Build and maintain modular evaluation pipelines and leaderboard infrastructure, covering sampling, inference, task packaging, scoring and reporting, with tested, documented code.
• Prototype new evaluation methods, and turn results into publications, technical reports and public releases.

What you'll need

• 4+ years in ML engineering or applied ML research, shipping production-quality systems.
• Strong software engineering, including clean, tested Python and end-to-end data and ML pipelines.
• Deep working knowledge of modern AI architectures and inference, including quality, latency and cost trade-offs.
• Rigorous grounding in probability and statistics on noisy real-world data.
• Ability to independently turn ambiguous questions into rigorous, executable technical work.
• Master's degree in Computer Science, Machine Learning, NLP or a related field, or equivalent industry or research experience.

Nice to have

• Experience building benchmarks or evaluation systems.
• Experience with human judgment data.
• Experience with speech or multilingual NLP, especially Indian languages.
• Publications or open-source work in AI or related domains.

Details

• Location: Bengaluru, India.
• Hybrid work options.