Quality Analyst

Overview

Own the answer key and quality measurement for image moderation at Roblox, calibrating vendor QA teams and driving root-cause fixes to improve moderation accuracy.

What you'll do

• Build and maintain the answer key by creating correctly labeled image cases for moderator training, QA, and policy work, adding new cases as policies and content trends change.
• Make final calls on disputed decisions and blind-audit sampled cases, resolving disagreements between moderators, vendor QA, and automated systems.
• Calibrate vendor QA teams by measuring vendor auditors' accuracy against the answer key and leading calibration sessions to close gaps.
• Find root causes and trends by tagging every error with a standard error taxonomy and analyzing quality data for patterns, bottlenecks, and improvements.
• Flag new risks early and run deep dives on the biggest problems, recommending fixes.
• Write short weekly stakeholder briefs that size each problem, describe its impact, and name the team best placed to fix it.
• Report on quality by pulling statistically valid samples and publishing weekly quality reports with accuracy metrics and confidence intervals.
• Use outside quality signals such as user appeals, escalations, partner feedback, and customer support to get a fuller picture of quality.
• Confirm that fixes worked by auditing affected queues again after each change.

What you'll need

• 3+ years in quality assurance, content moderation, Trust & Safety, or a related operations role.
• A strong understanding of content policy, plus sound judgment on close calls.
• Experience building answer keys, test sets, or calibration materials.
• Experience auditing and calibrating reviewers, including outsourced or vendor teams.
• Working knowledge of sampling and basic statistics, including accuracy rates, inter-rater agreement, and confidence intervals.
• Experience doing root-cause analysis and turning findings into practical recommendations.
• Skill with spreadsheets for pulling and analyzing data.
• Clear written and verbal communication, including writing short summaries for leadership.
• Experience working with cross-functional partners such as Policy, Operations, Engineering, and vendors.

Details

• Roles based in an office are onsite Tuesday, Wednesday, and Thursday, with optional presence on Monday and Friday (unless otherwise noted).