AI Agent Evaluation Engineer

Overview

Own the evaluation of quality for Cresta's AI Receptionist, which answers calls for small businesses. As the company's first AI Agent Evaluation Engineer, define what quality means for every agent, build the systems that measure it across conversations, and be the person the company trusts on how good the agent really is.

What you'll do

• Define and maintain the evaluation rubric across adherence, outcome, and conversation experience (latency, turn-taking, tone, repetition).
• Write annotation guidelines and failure taxonomy; run calibration sessions and measure inter-rater agreement.
• Classify and prioritize failure modes by severity and frequency.
• Review production conversations by hand every week, leveraging QA partners; open-code failures, cluster them, and turn the clusters into evaluations worth automating.
• Trace each failure to its likely cause (prompt, knowledge gap, tool or flow bug, speech recognition) and route it to the right owner.
• Evaluate across languages and report quality by segment (language, accent, audio quality, intent), not just in aggregate.
• Build and operate the evaluation pipeline that scores every production conversation, combining automated checks, LLM-based judges, and human review.
• Calibrate every automated judge against human labels before trusting it.
• Design sampling strategies (stratified by intent, language, caller type, audio conditions) so reported quality changes are real.
• Build and maintain the golden set and regression suite that every prompt, model, knowledge, and flow change is validated against before release.
• Maintain the failure-mode coverage map and close gaps.
• Design evaluations for handoffs to a human, misuse and unwanted-caller handling, and resilience under upstream failures.
• Define the quality and latency thresholds the agent is held to.
• Report quality trends to internal stakeholders and customers, with uncertainty stated.
• Partner with forward-deployed engineers, backend engineers, product managers, and the voice/language engineering team.
• Document methodology so results are reproducible.

What you'll need

• 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles.
• At least 1–2 years working hands-on with LLM-based systems in production.
• Bachelor's or Master's degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.
• Built and operated an end-to-end evaluation system for a production LLM product (agent or otherwise): rubric, human labels, automated judge, regression suite.
• Deep understanding of evaluation methodology: what golden sets, judges, regression suites, and calibration are for, and when each fails.
• Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state; able to read a trace and tell what the agent was doing and why.
• Strong programming skills in Python and experience with LLM APIs; able to build and ship evaluation tooling independently.
• Able to write a rubric that independent reviewers would score the same way, and to measure whether they do.
• Able to design a test from an open-ended quality question.
• Willing to do non-scalable work weekly, both yourself and by directing QA partners: listening to calls, labeling conversations, building a failure taxonomy from scratch.
• Professional fluency in at least one language other than English, sufficient to evaluate multilingual agent behavior directly.
• Strong written communication.

Nice to have

• Fine-tuning and ML frameworks are not required.

Details

• Location: Canada (Remote); United States (Remote).
• Candidates must complete a self-paced CoderPad technical assessment after initial resume review.
• Recording and AI transcription of interviews is not authorized.
• Base salary is location-based; total compensation includes equity and benefits.