Software Engineer

Overview

Build and ship the infrastructure frontier AI labs use to evaluate coding agents and language models, including benchmark datasets, agent evaluation runs at scale, and automated task quality checks.

What you'll do

• Design and ship features end to end: data model, API, background workers, and the UI when the change needs it.
• Build and improve how agents and models are evaluated: running trials in sandboxed environments, scoring results, and turning thousands of runs into results people can trust.
• Improve how benchmark tasks are quality-checked: verifiers, rubric-based and LLM-as-a-judge grading, and deciding whether a task is clear, solvable, and fair to score.
• Integrate new models, agents, and benchmarks in a way that generalizes, without a special case for each one.
• Keep the system fast and reliable as usage grows: efficient queries, safe migrations, and concurrency that holds up under load.
• Review teammates' pull requests against engineering guidelines and help keep the codebase consistent.

What you'll need

• 5+ years building and operating production backend systems, with ownership of at least one service from design through on-call.
• Bachelor's degree or higher in Software Engineering, Computer Science, or a related field.
• Strong Python and experience with Django (or a similar framework), PostgreSQL, and a task queue such as Celery.
• Hands-on experience in LLM evaluation, beyond calling a model API.
• Understanding of how a benchmark task is built, how an agent trial is scored, and how results are aggregated and compared.
• Judgment about task quality, including spotting ambiguous instructions, weak or gameable verifiers, leaked answers, and checks that pass for the wrong reason.
• Good tradeoffs: the smallest design that solves the problem, extending what already exists before adding something new, and explaining what was chosen not to build.
• Ownership: taking a fuzzy request, finding the relevant code, asking questions only when the answer changes the design, and finishing the job including migrations, tests, and edge cases.
• Quick adoption of team conventions, with first pull requests matching written guidelines in naming, structure, tests, and error handling.
• Clear writing: pull request descriptions and design notes that explain why, not just what.

Nice to have

• Experience with Harbor, Terminal-Bench, SWE-bench, Inspect, or a similar harness.
• Experience running agents in sandboxed or containerized environments (Docker, Kubernetes, or cloud sandboxes such as Modal or Daytona).
• Experience with LLM-as-a-judge or rubric-based grading, and with reducing judge bias and noise.
• Comfort with a TypeScript and React frontend when a change is visible there.
• Familiarity with coding agents such as Claude Code, Codex, or OpenHands, and with MCP.
• Experience with distributed job scheduling, concurrency limits, or database locking at scale.
• Contributions to open-source evaluation or infrastructure projects.

Details

• Location: Global, remote; United States.
• Compensation: $xxx,000 - $xxx,000 base salary plus equity.