QA Engineer - AI Systems III

Overview

Build and maintain AI-powered applications using LLMs, RAG pipelines and agent-based workflows, combined with automated testing and evaluation frameworks that measure quality, accuracy, safety and performance.

What you'll do

• Build and maintain AI-powered applications using LLMs, RAG pipelines, and agent-based workflows.
• Develop integrations with foundation models and AI services using Python and modern AI frameworks.
• Design prompt workflows, structured outputs, tool calling, and retrieval-based systems.
• Build APIs, backend services, and internal tools using FastAPI or similar frameworks.
• Collaborate with stakeholders to translate business requirements into AI solutions.
• Design and execute automated test suites for AI applications and agentic workflows.
• Create and maintain evaluation datasets, benchmark scenarios, and regression test cases.
• Build pipelines to measure response quality, hallucinations, accuracy, latency, and cost.
• Implement LLM-as-a-judge evaluations and automated scoring frameworks using tools such as DeepEval, RAGAS, or PromptFoo.
• Develop AI-powered test generation for functional, edge-case, and negative testing scenarios.
• Perform adversarial testing including prompt injection, jailbreak attempts, and safety validation.
• Monitor production AI systems through logging, tracing, and quality dashboards.
• Work with engineering teams to establish quality gates before AI features are released.

What you'll need

• 2-4 years of software engineering, QA automation, AI engineering, or related experience.
• Strong Python and Golang programming skills and understanding of software engineering fundamentals.
• Familiarity with LLM applications, prompt engineering, RAG, or agent-based AI systems.
• Experience in test automation using frameworks such as PyTest or equivalent tools.
• Understanding of API testing, backend services, and REST architectures.
• Knowledge of AI evaluation concepts such as hallucination detection, response quality measurement, benchmark testing, and regression testing.
• Experience with SQL and basic cloud platforms (AWS, Azure, or GCP).
• Strong analytical and problem-solving skills.

Details

• Location: Hyderabad, India.
• Flexible work environment.