Senior AI QA Engineer

Overview

Build the framework that ensures RhythmX's clinical insights are accurate, safe, and reliable, combining high-level software testing and data engineering to validate complex, non-deterministic medical outputs using a hybrid of automated grading methodologies.

What you'll do

• Architect multi-layered validation frameworks combining deterministic checks, semantic similarity metrics, and model-based evaluations.
• Develop systems to evaluate clinical pipeline outputs for faithfulness, safety, and hallucination detection using automated scoring techniques (e.g., BERTScore, ROUGE, or custom heuristics).
• Leverage agentic AI tools to rapidly prototype complex test harnesses, red-team clinical logic, and build internal validation utilities at high velocity.
• Execute integration and regression tests for data-heavy backend processes, ensuring medical data remains consistent from ingestion to insight generation.
• Work closely with Data Scientists and Product Managers to define Ground Truth datasets and clinical evaluation rubrics.
• Deep-dive into complex system failures to identify whether issues stem from code logic, data drift, or model behavior.

What you'll need

• 6+ years of technical experience in Quality Assurance, with a strong focus on system architecture and backend data validation.
• Expert-level Python skills for building custom test scripts and working within AI/ML ecosystems.
• Proven experience testing model outputs using diverse metrics (e.g., Semantic Similarity, NLP metrics, and custom scoring algorithms).
• Familiarity with tools like LangSmith, Arize Phoenix, or MLflow.
• Proficiency with AI-native development tools such as Cursor, Windsurf, Claude Code.
• Proficiency with PyTest, and experience with traditional tools (Cypress, Playwright, or Selenium) for end-to-end flows.
• Solid understanding of API testing (Postman/RestAssured) and database validation (SQL/NoSQL).
• Experience serving as a Subject Matter Expert (SME) on test process development for large-scale systems.

Nice to have

• Familiarity with healthcare data standards such as FHIR, HL7, or SNOMED CT.
• Understanding of HIPAA regulations and the nuances of handling ePHI/PHI.
• Understanding of sensitivity, specificity, and F1-scores in a clinical context.

Details

• Location: Bengaluru, Karnataka, India.