Fullstack AI Quality Engineer – Applied & Agentic AI Systems

Overview

Fullstack AI Quality Engineer focused on testing generative and agentic AI applications, including LLM/ML behavior, RAG pipelines, and agent/tool-use accuracy.

What you'll do

• Design and implement testing strategies for LLM-based applications and agentic systems, including response evaluation and RAG pipeline accuracy.
• Expand testing to evaluate autonomous multi-agent behaviors and tool-use accuracy.
• Build full-stack automated testing spanning integration tests, performance tests, and AI behavior tests across Python backend and TypeScript/React/Angular frontends.
• Use and co-create testing frameworks addressing AI-specific concerns like output consistency, contextual accuracy, and ethical compliance.
• Create and maintain AI-specific test documentation, test cases, and testing guidelines.
• Develop edge case, bias, and fairness test scenarios for ambiguous inputs and unexpected LLM response patterns.
• Collaborate on AI-specific security testing (e.g., prompt injection, data leakage) and conduct performance testing and resource monitoring.

What you'll need

• 7+ years of experience in software testing, with at least 1 year focused on genAI and/or agentic applications.
• Hold a B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent.
• Advanced command of Python and testing frameworks such as pytest and allure.
• High competence in TypeScript-driven test frameworks including Playwright, Jest, and Cypress.
• Experience integrating end-to-end automated testing suites in GitHub Actions or modern CI/CD pipelines.
• Expertise implementing evaluation harnesses such as RAGAS, DeepEval, and Promptfoo.
• Experience curating and versioning “Golden Datasets” for benchmarking against ground truth.

Details

• Location: Hyderabad, India.
• Working hours structured to capture overlap with Central European Time (IST evening).
• Regulated environment experience required, including understanding GxP vs non-GxP, and CSV and ITSM processes.