Senior Technical Product Manager - AI Agents, Evals & Reliability

Overview

Own the strategy, design, evaluation, and reliability of AI-agent products. Partner closely with engineering and ML teams to turn model capabilities into predictable, trusted user experiences.

What you'll do

• Define end-to-end AI system requirements from capabilities through user impact.
• Translate model capabilities, data constraints, and evaluation results into product decisions.
• Balance quality, latency, cost, reliability, and user experience trade-offs.
• Partner with ML, backend, and mobile engineers on system design and iteration.
• Define evaluation frameworks using offline metrics, online experiments, and human feedback.
• Drive execution through specifications, prioritization, and feedback loops.
• Own product quality, including correctness, predictability, and user trust.

What you'll need

• Strong computer science fundamentals in algorithms, data structures, and system design.
• Solid understanding of machine learning fundamentals and production AI systems.
• Experience with AI-powered products, including LLM-based systems.
• Experience with model evaluation, prompt or pipeline iteration, and feedback loops.
• Understanding of model limitations, hallucinations, bias, and drift.
• Significant experience owning complex technical products end-to-end.
• Ability to work closely with senior engineers and ML teams in ambiguous environments.

Nice to have

• Experience shipping AI-heavy consumer products.
• Engineering or highly technical product management background.
• Experience defining evaluation metrics for ML systems.
• Understanding of AI UX patterns and failure handling.
• Experience in zero-to-one product environments.

Details

• Location: United States.
• Interviews may be conducted virtually or onsite.