Senior Product Operations Manager, Evaluation Quality

Overview

Own the quality bar for Harvey’s human evaluation work so outputs are decision-grade, trusted, and scalable.

What you'll do

• Define what “good” looks like for evaluation methodology and data analysis.
• Author and maintain evaluation guidelines, instructions, and databases used by contract attorneys.
• Standardize rubric and evaluation design into repeatable templates and a documented methodology.
• Manage contract-attorney quality via onboarding, calibration, inter-rater reliability, and feedback loops.
• Run quantitative and qualitative analyses to diagnose errors and produce prioritized signals.
• Conduct QA on vendor and contract-attorney deliverables before results inform launch decisions.
• Ensure quality holds across jurisdictions and non-English geographies as coverage expands.

What you'll need

• 6+ years in product operations, research operations, evaluation/QA operations, or quality program management.
• Experience owning quality inputs (guidelines, instructions, benchmarks, QA procedures) for human-in-the-loop work.
• Experience onboarding, training, and calibrating a distributed pool of expert raters/annotators and running feedback loops.
• Familiarity with measurement concepts including calibration, inter-rater reliability, sampling, and rubric design.
• Experience running quantitative and qualitative analyses and communicating statistical conclusions to stakeholders.
• Ability to translate domain-expert judgment into repeatable, documented standards.
• Strong cross-functional coordination across Product, Engineering, Research, ALR, and data providers/vendors.

Details

• Role is part of the Evaluation Operations team.