Senior AI Evals Engineer

Overview

Senior AI Evals Test Engineer responsible for defining, developing and executing evaluation frameworks and test strategies for AI agents built with the Google Agent Development Kit (ADK) or similar agentic frameworks, with a significant focus on Responsible AI and Safety Evals. The role is 70% automation and 30% manual testing, within the Applied AI: Test Engineering team.

What you'll do

• Design and build agent evaluation frameworks for LLM-based agents across correctness, latency, cost, safety and task completion rate
• Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions, including agents on Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models)
• Build golden datasets and test harnesses, curating domain-specific evaluation sets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance)
• Establish and enhance a centralized eval-as-a-service platform that IT team members can self-serve to test agents against standardized and custom benchmarks
• Automate regression and drift detection with CI/CD-integrated eval pipelines that catch regressions on every model or prompt change before production deployment
• Define eval taxonomies and rubrics for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate, policy adherence, error recovery)
• Instrument production observability for agent quality: success rates, user corrections, fallback triggers, escalation patterns
• Benchmark and select models through structured comparisons of cost, quality and latency trade-offs
• Analyze evaluation results and provide clear, actionable quality and safety reports with focus on risk metrics, trend analysis and mitigation recommendations
• Define end-to-end QA strategies covering functional, integration, regression and user acceptance testing for conversational and goal-oriented agents with tool use, memory and multi-turn reasoning
• Develop and maintain test plans, test cases, test scenarios and automated test suites covering agent features, tool invocation, memory persistence and reasoning accuracy across model versions
• Identify, document, prioritize and track defects, performance regressions and alignment failures in Jira, working with Applied AI Engineers on root cause analysis and fix validation
• Evaluate new technologies and patterns, run proofs-of-concept and influence the platform roadmap
• Partner with policy and legal teams so evaluations cover regulatory compliance, fairness objectives and AI governance requirements
• Provide technical leadership to engineers, including solutioning, design and code reviews, pair programming and mentoring of junior and contract engineers
• Collaborate cross-functionally with ML engineers, product teams, platform engineers and security to embed eval-driven quality gates in the agent development lifecycle
• Drive red-teaming and adversarial testing, designing suites for jailbreak resistance, data leakage prevention and boundary enforcement in enterprise-grade agents

What you'll need

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
• 8+ years of overall software test engineering experience
• 5+ years focused on test engineering, QA automation or developer tooling, with demonstrated ability to build evaluation systems across AI/ML domains, conversational agents or LLMs
• 2+ years working with ML/AI systems, building, testing or evaluating models in production, including hands-on experience with LLM-based applications, prompt engineering, RAG pipelines or agentic workflows
• 2+ years as a senior/lead, owning evaluation strategy, quality frameworks and technical decisions for complex AI agent systems in an enterprise product environment
• 1+ years designing or operating evaluation frameworks for generative AI / LLM agents, including benchmarking, red-teaming and production quality monitoring in the AI ADLC model
• Deep hands-on experience with LLM evaluation, familiarity with frameworks and libraries such as LangSmith, DeepEval, LLM as a Judge, Promptfoo, RAGAS, Hugging Face evaluation tooling, or custom eval harnesses
• Strong understanding of agentic AI patterns: tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, retrieval-augmented generation (RAG)
• Production experience building automated, CI-integrated evaluation infrastructure (not just notebooks)
• Proficiency in Python, with strong scripting and tooling skills; experience with pytest, async patterns and data processing at scale
• Familiarity with Google Cloud Platform or another cloud service relevant to AI/ML (e.g., Vertex AI), and experience integrating testing into MLOps workflows
• Statistical literacy to design experiments, interpret eval metrics with appropriate confidence and avoid overfitting to benchmarks and metric gaming
• Enterprise product sensibility: understanding of how AI quality translates to customer trust in a B2B context (security, compliance, reliability SLAs)
• Excellent communication, able to present eval results and quality trade-offs to engineering leadership, articulate risks to non-technical audiences and drive ship/no-ship decisions
• Hands-on experience with CI/CD pipelines and Git-based workflows, including automated testing, static analysis, deployment automation and environment configuration
• Python (must); DeepEval (must); LLM as a Judge (must); prompt injection testing experience (must)
• Familiarity with TypeScript and Java (optional), Selenium and Playwright, Anthropic Claude, OpenAI, Azure OpenAI and open-source models
• Familiarity with LangGraph, Claude Agent SDK and custom orchestrators, Kubernetes, CI/CD (Jenkins/GitHub Actions) and observability stacks (Datadog/Grafana)
• Ability to lead without direct authority, influencing cross-functional teams through clarity and data
• Experience working in agile environments with product owners, scrum teams and distributed stakeholders across time zones
• Excellent communication and documentation skills, comfortable presenting solutions and trade-offs to engineers, managers and senior business stakeholders
• Strong architectural thinking and problem-solving skills, including decomposing complex business processes into simple solutions

Nice to have

• Deep learning/NLP knowledge: fundamental understanding of how LLMs work, including RAG, prompting and alignment challenges
• Data analysis experience with tools such as Pandas and SQL for analyzing evaluation and safety results
• Familiarity with AI ethics principles and industry standards for responsible AI development
• Familiarity with tools that test model responses to highly ambiguous, misleading or malformed inputs for robustness and safety failures
• Experience testing complex APIs, especially for tool-integration and function-calling scenarios in agentic workflows
• Experience with data poisoning testing
• Certifications in AI/ML or cloud platforms (GCP, AWS Machine Learning Specialty, Google Professional ML Engineer, DeepLearning.AI specializations) and software testing methodologies
• Experience evaluating agents in cybersecurity, data protection or infrastructure domains
• Familiarity with Rubrik's product portfolio or similar enterprise data management platforms
• Hands-on experience with human-in-the-loop eval workflows (annotation platforms, inter-rater reliability)
• Experience with model fine-tuning evaluation (pre/post quality comparison)
• Background in building internal developer platforms or tooling teams
• Contributions to open-source eval frameworks or published work on AI evaluation methodology

Details

• Location: Bangalore