AI Test Engineer - VLabs

Overview

AI Test Engineer in VLabs Quality Engineering, validating AI-enabled solutions in production for performance, reliability, and integrity.

What you'll do

• Translate AI testing strategy into executable scenarios for LLM outputs, document classification, extraction, and agent workflows.
• Design adversarial and boundary test cases to reveal hallucinations, misclassifications, and failure modes.
• Build reusable Python-based evaluation frameworks and scoring mechanisms, including AI-as-Judge.
• Embed evaluation and drift checks into CI/CD and enable release gating by detecting regressions.
• Develop and maintain ground-truth datasets and data-quality standards with subject matter experts.
• Test end-to-end agent workflows and validate AI pipeline endpoints, data persistence, and integrity.

What you'll need

• 7+ years in software testing, including 2–3 years focused on AI/ML-enabled systems in production.
• Experience designing and executing AI evaluation frameworks and quality strategies.
• Strong track record building ground truth datasets, drift detection systems, and scalable evaluation pipelines.
• Advanced Python skills for evaluation frameworks, batch processing, and data analysis.
• Experience with LLM evaluation tools such as deepeval, RAGAS, promptfoo, or similar.
• SQL capability for data validation and pipeline integrity checks.

Details

• Location: Bengaluru, Karnataka, India.
• Work setup: based in Bangalore office with possibility of hybrid work mode.
• Education: Bachelor’s degree required.