Overview
Build, evaluate, and harden Concierge, Guidewire's omnichannel AI agent for P&C carriers that handles first notice of loss over voice and chat.
What you'll do
- Own end-to-end slices of the agent: prompt and context architecture, retrieval over policy and coverage data, tool and API calls into core claims systems, and handoff to a human when the agent should stop.
- Build and run the eval loop, including golden sets, trajectory evaluation, LLM-as-judge, and error analysis on real production call traces, and make it the gate for releases.
- Drive containment and resolution rates up without trading away accuracy, and instrument the system so you can tell which of the two moved.
- Work directly on the voice path: turn-taking, interruption, latency budgets, and the telephony layer underneath (LiveKit, SIP).
- Sit in on carrier calls and listen to recordings to find where the conversation went sideways.
What you'll need
- Experience building and deploying an LLM application that real users depend on, with fluency in LLMs, context engineering, RAG, and agentic workflows.
- Real ML and deep learning depth, and a track record of making unpredictable systems behave predictably through disciplined evals, error analysis, and statistical measurement.
- Software engineering fundamentals: reasoning about cost, latency, reliability, and scalability as tradeoffs; designing data stores and distributed systems; writing tests; and taking security and privacy seriously, including advocating CI/CD process automation.
- Daily use of coding agents: managing their context, deciding when a spec is worth writing, giving them verifiers to close their own loops, and knowing when to intervene.
- Product sense and enough insurance context to argue about what the agent should do next, including when to ship an MVP to a carrier quickly and when a claims workflow requires getting it right first time.
- A commitment to continued learning as the field changes.
Nice to have
- Real-time or streaming voice systems (ASR, TTS, telephony, SBC/SIPREC).
- Insurance, claims, or another regulated domain where being wrong has a cost.
- Experience building internal eval or observability tooling that other engineers actually adopted.
- Machine learning principles, model experimentation, and training and deploying models in production.
- Experience in P&C insurance or exposure to Guidewire products and services.
- Prompt or pipeline optimization frameworks (DSPy, GEPA) applied to a production system.
Details
- Location: United States, Remote.
- Full-time position.
- Application: send a short note about something you shipped where the model's unpredictability was the central problem and what you did about it.
Read the full description and apply on the company’s own careers page.