Overview
Own how Arcfront measures what frontier models can and cannot do in a capability area, from what gets measured, to why the measurement is valid, to what should change as a result.
What you'll do
- Own a measurement roadmap for one capability area: what to measure now, what to retire, and what to deliberately leave for later.
- Design new evaluations from scratch, covering construct validity, item difficulty and headroom, the power to separate the models that matter, and a contamination threat model written before the first item is authored.
- Build agentic eval infrastructure: sandboxed environments, deterministic tool mocks, trajectory-level scoring and partial credit over long horizons.
- Validate judges: collect human labels, measure chance-corrected agreement, run bias audits and set clear rules on where an LLM judge is and isn't trusted.
- Apply adversarial pressure to our own numbers, hunting for reward hacking, shortcut solutions, grader exploits and prompt-format sensitivity.
- Make the argument yourself and raise the methodological bar for the engineers around you.
What you'll need
- 4–6 years in ML research engineering, applied research or quantitative measurement where being wrong was expensive.
- Experience shipping measurement that other people relied on, such as an eval suite, scoring harness or experimentation platform.
- Statistical depth in paired and clustered designs, power analysis, multiple-comparison control and calibration, including showing a team their study was underpowered with the arithmetic.
- Systems ability in large-scale batched inference, cost and rate-limit management, containerised sandboxes and reproducibility across thousands of runs.
- Design taste to quickly name how a proposed eval's score will stop meaning what it claims.
- Clear, persuasive writing whose memos change decisions.
Nice to have
- Built or substantially contributed to a public benchmark, eval harness or leaderboard.
- Published work on evaluation methodology, contamination or judge reliability.
- Background in psychometrics, econometrics, experimental design or causal inference applied to ML.
- A time you proved an exciting result was an artefact and convinced the team.
Details
Read the full description and apply on the company’s own careers page.