Quality Assurance Engineer – AI Agentic Applications, AVP

Overview

Quality Assurance Engineer responsible for validating AI agentic applications across development and production. The role combines specialized AI testing, automation, safety evaluation, and continuous quality measurement.

What you'll do

• Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows.
• Develop and implement specialized testing approaches for LLM-powered applications, validating response accuracy, relevance, and grounding, while identifying hallucinations and edge-case behaviours.
• Test and evaluate autonomous and semi-autonomous AI agents, including planning, tool selection and sequencing, agent-to-agent interactions, and graceful error and timeout recovery.
• Validate Retrieval-Augmented Generation (RAG) performance, including document ingestion, chunking, metadata quality, embedding accuracy, and citation/source attribution.
• Perform AI safety and security testing to identify prompt injection, data leakage, unauthorized tool execution, and privilege escalation vulnerabilities.
• Establish quantitative AI evaluation metrics and quality gates, including classification accuracy, groundedness, hallucination rate, tool-selection accuracy, cost, and latency.
• Build and maintain automated test suites covering APIs, UI, integration, RAG, and agent trajectories, progressively transitioning testing toward automated continuous evaluation.
• Analyze production AI interactions and failed agent journeys, identify recurring failure patterns, create regression tests from production incidents, and ensure decision auditability.

What you'll need

• Bachelor's degree or above in Computer Science, Engineering, Information Technology, or a related discipline.
• 5+ years of experience in software quality assurance and testing, with strong knowledge of test planning, functional, integration, regression, and end-to-end testing.
• 1–2 years of hands-on experience specifically testing GenAI, LLM, or Agentic AI applications.
• Strong experience with test automation and building automation frameworks and test utilities rather than relying solely on manual tools.
• Strong programming ability in at least one language: Java, Kotlin, or Python.
• Strong understanding of generative AI technologies, including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents/agentic workflows, and tool/function calling.
• Hands-on experience with LLM and agent evaluation tools such as Langsmith/Langfuse, Ragas, or equivalent frameworks for dataset-based testing, custom evaluation metrics, experiment tracking, tracing, and continuous regression evaluation.

Nice to have

• Experience testing API and data integrations, including REST APIs, JSON, SQL, databases, and messaging systems like Kafka.
• Good understanding of CI/CD integration and establishing automated quality gates.
• Experience working in the financial services industry, with knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controls.
• Experience with AI/LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologies.
• Hands-on experience with advanced AI testing capabilities such as prompt injection testing, AI evaluation/evals, and AI observability.
• Experience testing distributed systems.
• Proven ability to leverage AI tools to enhance productivity and optimise workflows to solve business problems, while applying critical judgment to ensure responsible and ethical use of data and AI outputs.

Details

• Location: Pune, India.
• Corporate title: AVP.