Senior AI QA Engineer

Overview

Senior AI QA Engineer for the Precision Patient Care Pipeline, responsible for building the framework that ensures clinical insights are accurate, safe, and reliable. The role blends high-level software testing with data engineering to validate complex, non-deterministic medical outputs using a hybrid of automated grading methodologies.

What you'll do

• Architect multi-layered validation frameworks combining deterministic checks, semantic similarity metrics, and model-based evaluations.
• Develop automated model grading systems to evaluate clinical pipeline outputs for faithfulness, safety, and hallucination detection using techniques such as BERTScore, ROUGE, or custom heuristics.
• Leverage agentic AI tools to rapidly prototype complex test harnesses, red-team clinical logic, and build internal validation utilities at high velocity.
• Execute integration and regression tests for data-heavy backend processes, ensuring medical data remains consistent from ingestion to insight generation.
• Work closely with Data Scientists and Product Managers to define Ground Truth datasets and clinical evaluation rubrics.
• Deep-dive into complex system failures to identify whether issues stem from code logic, data drift, or model behavior.

What you'll need

• 6+ years of technical experience in Quality Assurance, with a strong focus on system architecture and backend data validation.
• Expert-level skills in Python for building custom test scripts and working within AI/ML ecosystems.
• Proven experience testing model outputs using diverse metrics (e.g., semantic similarity, NLP metrics, and custom scoring algorithms).
• Familiarity with tools like LangSmith, Arize Phoenix, or MLflow.
• Proficiency with AI-native development tools such as Cursor, Windsurf, or Claude Code.
• Proficiency with PyTest, and experience with traditional tools (Cypress, Playwright, or Selenium) for end-to-end flows.
• Solid understanding of API testing (Postman/RestAssured) and database validation (SQL/NoSQL).
• Experience serving as a Subject Matter Expert (SME) on test process development for large-scale systems.

Nice to have

• Familiarity with healthcare data standards such as FHIR, HL7, or SNOMED CT.
• Understanding of HIPAA regulations and the nuances of handling ePHI/PHI.
• Understanding of sensitivity, specificity, and F1-scores in a clinical context.

Details

• Location: Bengaluru (Bangalore), India.