Overview
Design, build and test the AI conversations and agents inside the client's learner-facing products, including advisory experiences, practice simulations and diagnostic coaching.
What you'll do
- Turn scenario specs, decision models and assessment rubrics into prompts, application logic and testable behavior.
- Spot missing requirements and technical constraints, and raise them with Product, Engineering and SMEs.
- Define how AI models connect to application services and approved data sources: APIs, tool interactions, data structures, validation, error handling, permitted actions.
- Design prompts, structured outputs and validation logic, and catch invalid responses before they reach learners.
- Build conversational runtimes with controlled personas, session state and explicit state transitions.
- Keep simulation behavior, assessment logic and feedback separate, so feedback cannot change assessment results.
- Implement grounding and retrieval from approved knowledge sources, with source references where required.
- Keep code, prompts, schemas and evaluation assets under version control, and document them.
- Run bounded experiments with Product, with clear questions and success measures.
- Take validated prototypes into production with Engineering, and check behavior stays the same.
- Help build the technical controls that keep AI features compliant with accreditation requirements (issue flagging, review steps, disclosure, logging).
What you'll need
- Expert prompt and context engineering: system prompts, few-shot examples, prompt chaining, routing, breaking complex tasks into steps.
- Structured outputs with JSON Schema, function calling and validation layers, with retry and fallback logic.
- RAG design: chunking, metadata filtering, hybrid search, reranking, citation grounding.
- Managing token budgets, prompt caching, context compaction and long-session memory.
- Agentic systems: tool use, planner/executor patterns, multi-agent orchestration, state machines, human-in-the-loop checkpoints.
- Defining tool schemas, permission scopes and action limits for agents.
- AI evaluation: golden datasets, rubric-based scoring, LLM-as-judge calibrated against human raters, regression testing.
- LLM observability and tracing.
- LLM security: prompt injection, data leakage, jailbreaks, unsafe tool actions, and guardrails.
- Comparing prompting, retrieval and model options on quality, latency and cost.
- Python or TypeScript, including API integrations and automated testing.
- Git (e.g. Bitbucket), reproducible tests and CI.
- Works with AI coding agents by default.
- Explains technical trade-offs to non-technical colleagues.
- Sound judgment on privacy, human oversight and the limits of AI-generated assessment or advice.
- Hands-on experience building LLM applications with multi-step conversational workflows.
- Experience collaborating with subject-matter experts.
Nice to have
- Background in learning products, assessment, simulations or regulated professional content.
Details
- Fully remote, long-term opportunity.
Read the full description and apply on the company’s own careers page.