AI Test Engineer - VLabs

Overview

Hands-on AI Test Engineering role within Vialto Labs (VLabs) Quality Engineering, validating the performance, reliability and integrity of AI-enabled solutions in production.

What you'll do

• Translate AI testing strategy into executable test scenarios across LLM outputs, document classification, extraction accuracy, agent workflows and edge cases
• Design adversarial and boundary test inputs to expose hallucination, misclassification and failure modes
• Validate AI outputs for structure, consistency, accuracy and production readiness against defined performance thresholds
• Build reusable Python-based evaluation frameworks, including output validation, hallucination detection and scoring mechanisms
• Develop parameterized test scripts reusable across features, models and releases
• Implement AI-as-Judge frameworks, including prompt design, scoring logic and calibration of evaluation reliability
• Embed evaluation frameworks into CI/CD pipelines to support continuous testing and deployment
• Design and operate drift detection frameworks using fixed baseline datasets and scheduled re-evaluation
• Establish thresholds to distinguish acceptable variation from performance degradation
• Enable release gating by identifying regressions prior to production deployment
• Build and maintain ground truth datasets in partnership with subject matter experts
• Define standards for classification, extraction accuracy and acceptable output characteristics
• Continuously update datasets to reflect evolving business requirements and use cases
• Test end-to-end agentic workflows, validating data integrity, error propagation and fallback behavior
• Perform API-level testing of AI pipeline endpoints using Python and Postman/Newman
• Validate data persistence and integrity across system layers using SQL
• Partner with engineering teams to ensure testability, observability and system reliability
• Define and scale standardized AI evaluation patterns and reusable quality frameworks across VLabs
• Contribute to enterprise AI quality standards and reference architectures
• Ensure adherence to Responsible AI, data privacy and governance requirements
• Support auditability, traceability and transparency of AI outputs and evaluation processes
• Translate evaluation results into actionable insights for engineering, product and business stakeholders
• Support decision-making on model readiness, release risk and performance trade-offs
• Proactively identify risks, patterns and systemic issues and escalate appropriately

What you'll need

• 7+ years in software testing, including 2–3 years focused on AI/ML-enabled systems in production environments
• Proven experience designing and executing AI evaluation frameworks and quality strategies
• Strong track record building ground truth datasets, drift detection systems and scalable evaluation pipelines
• Experience testing multi-step agentic workflows and AI-driven automation systems
• Experience operating in fast-paced, iterative delivery environments
• Advanced Python programming for evaluation frameworks, batch processing and data analysis
• Experience with LLM evaluation tools such as deepeval, RAGAS, promptfoo, or similar
• Strong capabilities in AI output validation, hallucination detection and grounding checks
• Drift detection frameworks and statistical evaluation methods
• OCR, VLM and document AI testing (classification, extraction, edge cases)
• API testing using Python (requests/httpx) and Postman/Newman
• SQL for data validation and pipeline integrity checks
• Ability to operate independently in fast-moving, ambiguous environments
• Strong analytical mindset with attention to detail and quality rigor
• Ability to balance speed and rigor in AI evaluation and delivery cycles
• Proactive communicator who identifies risks and drives resolution
• Ability to translate technical findings into business-relevant insights
• Bachelor's degree required

Nice to have

• Background in regulated or compliance-driven environments
• Familiarity with LangChain, LlamaIndex or similar frameworks
• Experience with cloud AI platforms such as Azure AI Foundry or AWS Bedrock
• Advanced degree in Computer Science, Data Science or related field

Details

• Based in the Bangalore office, with the possibility of hybrid work mode