Software Engineering Director, Agentic Evaluations

Overview

Lead engineering for systems that deliver credible evaluations of software vendors’ AI agents. Own the core evaluation platform while managing and mentoring an engineering team.

What you'll do

• Scope the feasibility and effort of evaluating customer-facing software agents.
• Own the evaluation system across its admin interface, APIs, and workflow dispatch.
• Design reusable evaluation primitives across software verticals.
• Convert integration and maintenance processes into repeatable AI skills or agents.
• Track emerging agent-evaluation frameworks and techniques.
• Collaborate with data science peers on proprietary benchmarks.
• Mentor engineers and provide technical leadership on evaluation systems.

What you'll need

• 10+ years of professional programming experience in backend or full-stack environments.
• 2+ years of direct engineering management experience.
• Expert backend development skills in Python, Java/Kotlin, TypeScript/JavaScript, or Go.
• Strong proficiency with backend frameworks such as FastAPI or Node.js.
• Experience creating agent evaluations using trajectory traces and rubrics.
• Experience using OpenAI, Anthropic, or Google frontier models in LLM-as-a-judge applications.
• BS/BA degree in a related field.

Nice to have

• Experience with agent tool use through direct integrations or MCP servers.
• Knowledge of STATE-Bench, tau2-bench, or similar frameworks.
• Experience with Playwright, browser-use, or browser automation tools.
• Experience with durable execution, workflow, or agent-sandboxing frameworks.

Details

• Remote in the United States.