Research Engineering Intern, AI Evaluations

Overview

Research Engineering Intern on Karya's AI Evaluations team, building and running evaluations that measure how well AI models serve people in the languages and contexts global benchmarks miss.

What you'll do

• Prepare datasets, run model inference through APIs and self-hosted setups, and configure, batch and launch annotation tasks on the Karya platform.
• Monitor live evaluations, triage and reproduce annotation app issues, coordinate fixes with platform and operations teams, and document recurring problems.
• Run automated checks on annotation data and flag anomalies before they affect results.
• Compute agreement, scores and breakdowns; support focused studies such as validating automated judges against human ratings; and contribute to technical reports and public releases.
• Build and maintain tested, documented tooling for data processing, QC and analysis, with logged configs for reproducibility.

What you'll need

• Currently pursuing or recently completed a Bachelor's or Master's in Computer Science, Machine Learning, NLP or a related field.
• Strong Python and comfort working with data (pandas or similar).
• Working knowledge of LLMs and modern ML, and familiarity with calling models via APIs.
• Solid foundation in probability and statistics.
• Methodical debugging skills, and willingness to dig into unfamiliar systems when something breaks.

Nice to have

• Experience with ML projects involving evaluation, annotation or human feedback data.
• Exposure to speech or multilingual NLP, especially Indian languages.
• Experience with GPU inference, cloud tooling or Android/app debugging.

Details

• Location: Bengaluru, India.
• Employment type: internship.