Research Engineer, Model Evaluations

Overview

Research Engineer for Model Evaluations, designing and running evaluations to measure what Claude can do and to support reliable model-health monitoring during training.

What you'll do

• Design and run new evaluations of Claude capabilities (reasoning, agentic behavior, knowledge, safety properties) and visualize results.
• Build and harden a distributed evaluation execution platform to run hundreds of evals against checkpoints during production RL training runs.
• Own dashboards used by researchers and leadership to monitor model health (signal-to-noise, latency, regression detection).
• Debug anomalous eval results during training runs and determine whether the cause is model change or infrastructure issue.
• Improve tooling, libraries, and workflows for implementing and iterating on evaluations.
• Partner with research teams across defining what to measure through interpreting results as training progresses.
• Run experiments to characterize how prompting, sampling, and scaffolding choices affect results on internal and industry benchmarks.
• Communicate evaluation results to internal stakeholders and, where appropriate, external audiences.

What you'll need

• Strong Python skills, including production or research infrastructure experience.
• Experience building or operating distributed systems and reliable-at-scale data pipelines or infrastructure.
• Clear written and verbal communication, especially when explaining technical results to non-specialists.
• Ability to operate in an on-call or production-support capacity while training runs are live.
• Care about societal impacts and interest in steering powerful AI to be safe and beneficial.

Details

• Annual compensation range: $500,000–$850,000 USD.
• Work mode: location-based hybrid policy; expected in an office at least 25% of the time.
• Visa sponsorship: provided, with reasonable efforts to get you a visa if you receive an offer.