Software Development Engineer in Test - AI

Overview

Validate, evaluate and improve AI-powered systems, agents, copilots and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions.

What you'll do

• Design, validate and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines and autonomous AI agents.
• Develop AI evaluation (Evals) frameworks for hallucination detection, groundedness, toxicity and bias validation, response quality, context relevance, prompt regression testing, determinism and consistency, and safety and guardrail validation.
• Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings and evaluation metrics.
• Create synthetic and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and agent memory/context retention.
• Validate MCP (Model Context Protocol) integrations and tool interoperability.
• Work on Context Engineering, Prompt Engineering and Prompt Optimization strategies.
• Evaluate latency, token usage, model cost optimization, throughput and scalability.
• Develop and maintain scalable UI/API automation frameworks using Python, Robot Framework, Playwright, Selenium or Cypress.
• Automate testing for REST APIs, GraphQL APIs, event-driven systems, streaming systems, webhooks, microservices and MFE (Micro Frontend) architectures.
• Build reusable automation libraries, utilities and AI-assisted testing workflows.
• Integrate automation into CI/CD pipelines and quality gates.
• Validate RAG architectures including chunking strategies, embedding quality, retrieval accuracy, re-ranking and vector search relevance.
• Work with vector databases such as Pinecone, Weaviate, ChromaDB, Qdrant and FAISS.
• Build or validate AI agents leveraging LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel and OpenAI SDKs.
• Validate AI workflows involving tool calling, function calling, agent memory, agent planning/execution, multi-modal AI systems and structured outputs.
• Build and maintain quality workflows integrated with GitHub Actions, Jenkins, GitLab CI/CD and Azure DevOps.
• Work with Docker and Kubernetes environments and perform cloud-based testing and deployments across AWS, Azure, OCI and GCP.
• Validate distributed systems reliability and resiliency.
• Monitor AI system behaviors using observability platforms, logging and tracing tools.
• Contribute to shift-left and quality engineering best practices.

What you'll need

• Strong programming experience in Python.
• Strong understanding of software testing and quality engineering principles.
• Hands-on experience with AI/LLM testing, prompt engineering, context engineering, AI evals, RAG systems and AI agents.
• Experience with CI/CD pipelines.
• Experience with Docker and Kubernetes.
• Experience with cloud platforms (AWS, Azure, OCI or GCP).
• Experience with API testing and automation.
• Experience with microservices testing.
• Experience with UI automation.
• Experience with SQL and data validation.
• Strong debugging, analytical and troubleshooting skills.
• Experience working in Agile/Scrum environments.

Nice to have

• Bachelor's/Master's degree in Computer Science, Engineering, AI/ML or a related field, or equivalent hands-on experience and demonstrated projects.
• Experience with Robot Framework.
• Experience with Playwright.
• Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel and MCP (Model Context Protocol).
• Hands-on experience with Cursor, GitHub Copilot, Claude, Gemini and OpenAI APIs.
• Experience with vector databases and embeddings.
• Exposure to MLOps workflows and ML lifecycle testing.
• Knowledge of performance testing and security testing.
• Experience validating AI governance/compliance requirements.
• Experience with contract testing, chaos testing or resilience engineering.
• Familiarity with accessibility testing and cross-browser/device testing.
• Exposure to synthetic monitoring and production observability.
• AI red teaming and LLM security testing (prompt injection, jailbreaks, data leakage).
• Agent reliability and safety testing.
• AI cost optimization.
• Distributed systems testing.
• Test data engineering.
• Feature flag testing.
• AI observability and telemetry.
• Reinforcement learning feedback workflows.
• Human-in-the-loop evaluation systems.
• Knowledge graph integrations.
• Multi-modal AI testing (vision, audio, text).

Details

• Location: India, remote.