AI Data Analyst

Overview

AI Data Analyst to support the human evaluation system behind Glean’s AI product quality, focusing on response quality, benchmarking, and release readiness.

What you'll do

• Run ongoing human labeling for priority AI workflows including response quality and task success.
• Triage bad-query reports, downvotes, escalations, and routed quality issues.
• Perform qualitative analysis to identify patterns like hallucinations and retrieval failures.
• Maintain high-quality labeled datasets, golden sets, and regression slices while monitoring coverage and drift.
• Participate in calibration exercises and validate LLM-as-a-judge outputs against human labels.
• Partner with QA, Engineering, Product, and eval owners on benchmark updates and release-readiness decisions.

What you'll need

• 3–5 years of experience in data labeling, data analysis, QA, or a related field.
• Strong analytical judgment and attention to detail to apply nuanced rubrics consistently.
• Experience evaluating AI-generated outputs or working with NLP/search/recommendation/ML systems.
• Familiarity with structured qualitative analysis and basic SQL/data retrieval.
• Clear written and verbal communication to explain findings to technical and non-technical audiences.
• Ability to work independently in ambiguous, fast-moving environments and collaborate across functions.

Details

• Hybrid role: 4 days a week in the offices.
• Location: Bangalore, India.