Senior Principal Engineer - IT Quality & Assurance Testing

Overview

Senior Principal Engineer leading IT Quality Engineering, Automation and AI/LLM testing for the MMB LAC L3 Squad at Marsh People and Investments.

What you'll do

• Input into planning and subsequent resource planning, including skills profiling and identifying training requirements for team members.
• Define and drive the Quality Engineering, Automation and AI Testing strategy across projects.
• Lead testing across UI, API, Database, Automation and AI/LLM applications.
• Establish AI testing and evaluation practices covering accuracy, relevance, faithfulness, groundedness, hallucination, safety and security.
• Drive testing of RAG applications, vector databases, retrieval pipelines, prompts and AI agent workflows.
• Evaluate and adopt appropriate AI testing, evaluation, observability and security tools based on project requirements.
• Lead automation initiatives using Playwright, WebdriverIO, JavaScript/TypeScript and Java, integrated with CI/CD and GitHub Actions.
• Establish scalable and maintainable automation frameworks, reusable utilities and test assets.
• Define and monitor test coverage, automation coverage, AI evaluation metrics, defect leakage and release quality.
• Drive shift-left testing, continuous testing, risk-based testing and quality engineering practices.
• Ensure effective root cause analysis, defect prevention and continuous quality improvement.
• Mentor and develop QA engineers across manual testing, automation and AI testing.
• Work closely with Product, Engineering, AI/ML, Data Science, Security and Business teams.
• Manage team performance, delivery risks, conflicts and stakeholder expectations.
• Ensure consistent testing standards, processes and quality practices across teams.

What you'll need

• 10+ years' experience in Software Testing / Quality Engineering with experience in managing end-to-end project delivery.
• Proven experience leading QA teams across Manual, Automation and AI/LLM testing.
• Strong understanding of STLC, SDLC, Agile, test strategy, test planning, risk management, defect management and quality metrics.
• Experience defining testing strategies for Web/UI, API, Database, AI/LLM, RAG and AI-powered applications.
• Strong automation experience using Python, JavaScript/TypeScript, Java, Playwright, Selenium and WebdriverIO (WDIO).
• Good understanding of API testing, SQL, relational databases and data validation.
• Understanding of RAG architecture, embeddings, vector databases, semantic search, retrieval and reranking.
• Experience with Generative AI, LLM evaluation, prompt engineering, AI agents and AI safety testing.
• Experience with AI/LLM testing and evaluation frameworks such as PromptFoo, Ragas, DeepEval, LangSmith and PyRIT, or similar tools.
• Strong understanding of AI quality and security risks including prompt injection, jailbreaks, hallucination, data leakage, guardrails and responsible AI.
• Experience with modern DevOps and test automation practices, including Git, GitHub, GitHub Actions and CI/CD pipeline integration.
• Strong communication, stakeholder management, analytical and people leadership skills.
• A visionary forward thinker who views Quality holistically and can drive a Quality Engineering mindset across traditional software and AI-powered applications.

Nice to have

• No items are explicitly marked as preferred in the source.

Details

• Location: Pune/Gurugram (Gurugram - DLF Building).
• Hybrid role: working at least three days a week in the office.
• Colleagues are expected to be in their local office or working onsite with clients at least three days per week, with one anchor day per week for office-based teams.