Overview
Own the quality bar for Harvey’s human evaluation work so outputs are decision-grade, trusted, and scalable.
What you'll do
- Define what “good” looks like for evaluation methodology and data analysis.
- Author and maintain evaluation guidelines, instructions, and databases used by contract attorneys.
- Standardize rubric and evaluation design into repeatable templates and a documented methodology.
- Manage contract-attorney quality via onboarding, calibration, inter-rater reliability, and feedback loops.
- Run quantitative and qualitative analyses to diagnose errors and produce prioritized signals.
- Conduct QA on vendor and contract-attorney deliverables before results inform launch decisions.
- Ensure quality holds across jurisdictions and non-English geographies as coverage expands.
What you'll need
- 6+ years in product operations, research operations, evaluation/QA operations, or quality program management.
- Experience owning quality inputs (guidelines, instructions, benchmarks, QA procedures) for human-in-the-loop work.
- Experience onboarding, training, and calibrating a distributed pool of expert raters/annotators and running feedback loops.
- Familiarity with measurement concepts including calibration, inter-rater reliability, sampling, and rubric design.
- Experience running quantitative and qualitative analyses and communicating statistical conclusions to stakeholders.
- Ability to translate domain-expert judgment into repeatable, documented standards.
- Strong cross-functional coordination across Product, Engineering, Research, ALR, and data providers/vendors.
Details
- Role is part of the Evaluation Operations team.
Read the full description and apply on the company’s own careers page.
