Conversation Quality Analyst

Overview

Listen to real customer calls from Bolna's voice AI agents, annotate what actually happened, and verify whether the company's automated LLM-as-judge evaluations and quantitative signal detection got it right. This is an internship role for someone early in their career.

What you'll do

• Listen to and annotate real customer calls, including transcription review, issue tagging and labeling, using tools such as Label Studio.
• Follow and help sharpen annotation guidelines for a multilingual environment (Hindi, English, Hinglish).
• For calls flagged by the automated eval pipeline, verify whether the model's call was correct, such as confirming whether a detected barge-in genuinely happened by listening to the audio.
• Mark agreements and disagreements clearly, with reasoning, so model accuracy can be measured and improved over time.
• Check system-flagged quantitative signals against the actual call, such as confirming whether a detected agent interruption occurred at that timestamp.
• Flag false positives and false negatives so detection logic can be tightened, and help identify edge cases that current rubrics or detection logic do not handle well.
• Regularly inspect calls beyond flagged ones to spot new or emerging issues, and bring these patterns back to the team so rubrics, prompts and detection logic keep improving.

What you'll need

• Strong attention to detail and the patience to do focused, high-precision work across many calls.
• Comfort learning new tools quickly, such as Label Studio, dashboards and internal QA apps.
• Genuine curiosity about AI and voice AI, with an interest in understanding why a call was flagged, not just completing a checklist.

Nice to have

• Multilingual comfort in Telugu, Tamil, Kannada, Marathi, Gujarati or Bengali, in addition to English and Hindi.
• Prior exposure to data annotation, labeling or QA work.
• Familiarity with spreadsheets, basic SQL or comfort reading dashboards such as Metabase.
• Background in linguistics, call center operations or content moderation.

Details

• Internship role.
• Based in Bengaluru, with in-person team collaboration at the Bengaluru office.