Quality Engineering Lead

Overview

Owns functional test planning and execution for Copilot Studio agents, validating conversational behavior, grounded answers, flows, and tool integrations, and ensuring quality and responsible AI gates are met before release.

What you'll do

• Design and execute test plans and test cases for Copilot Studio agents, covering topics, trigger phrases, entities, variables, and conversation/agent flows.
• Validate generative orchestration and generative answers for accuracy, grounding in knowledge sources, and consistency of responses.
• Test Power Automate flows and MCP Server tool/plugin integrations for functional correctness, error handling, and failure scenarios.
• Perform bias, safety, and hallucination checks as part of responsible AI validation.
• Test agent publishing and behavior across channels (Teams, web, custom apps) and across environments (dev-test-prod).
• Identify conversational breakdowns and connector/plugin issues, log, triage, and track defects through to closure, and verify fixes.
• Run functional and regression testing, including prompt and instruction changes, and maintain test evidence.
• Ensure solutions meet the agreed Definition of Done before UAT.
• Collaborate with developers, the Architect, and the Business Analyst to clarify requirements and edge cases.
• Support UAT coordination and release readiness checks.

What you'll need

• Minimum 5 years of experience required.
• 15 years full-time education required.
• Must have skills: Microsoft Functional Testing.
• Experience in functional/QA testing, preferably on conversational AI, chatbot, or low-code platforms.
• Hands-on exposure to testing Copilot Studio agents (topics, generative answers, knowledge sources, publishing).
• Working knowledge of Power Automate flows and API/tool integration testing concepts (including MCP Server tool connections).
• Understanding of responsible AI principles and bias/safety validation practices.
• Experience with defect management and test tracking tools (Azure DevOps/Jira).
• Strong attention to detail, with a structured approach to test design and conversational-flow troubleshooting.

Nice to have

• Exposure to Agile delivery environments.

Details

• Location: Pune.
• Full-time employment.