Overview
Help build and operate an always-on adversarial testing capability for AI-enabled and agentic systems, integrating commercial, open-source, and Allstate-internal testing technologies into enterprise environments.
What you'll do
- Integrate commercial, open-source, and Allstate-built AI agentic evaluation and red-team technologies into Allstate environments using approved enterprise engineering and security patterns.
- Connect testing systems to AI products through APIs, agent and tool protocols, voice interfaces, and other application-specific integration patterns.
- Turn attack strategies into reliable, configuration-driven test capabilities that can be reused across applications and environments.
- Develop APIs, microservices, orchestration components, and automated triggers supporting continuous and large-scale adversarial testing.
- Build secure authentication, authorization, and credential-management patterns that allow automated testing to operate without manual intervention.
- Capture and persist test evidence including requests, responses, conversation history, tool interactions, configuration, grading results, and other relevant telemetry.
- Integrate test results with reporting, observability, compliance, and engineering workflows.
- Build CI/CD, monitoring, logging, tracing, alerting, and operational practices for the adversarial testing platform.
What you'll need
- Professional software engineering experience building and operating enterprise applications, services, or platforms.
- Strong proficiency in Python and/or Java.
- Experience designing and integrating REST APIs, microservices, asynchronous workflows, or event-driven systems.
- Experience integrating third-party, vendor, or open-source technologies into enterprise environments.
- Experience deploying applications using cloud, container, or enterprise PaaS technologies.
- Experience with CI/CD, automated testing, source control, configuration management, and modern software delivery practices.
- Understanding of authentication and authorization, service identities, secrets management, network controls, and secure application integration.
- Experience building observable services using logs, metrics, tracing, alerts, and operational dashboards.
- Strong problem-solving and troubleshooting skills across application, integration, deployment, and infrastructure boundaries.
Nice to have
- Experience with AI/LLM evaluation, red-team, security testing, or quality-assurance frameworks.
- Experience with tools such as Coval, Promptfoo, PyRIT, Garak, Inspect AI, or similar technologies.
- Experience with agentic AI systems, including tool-calling agents, MCP, multi-agent systems, conversational agents, or voice agents.
- Familiarity with LLM evaluation, LLM-as-a-judge techniques, AI guardrails, or AI observability.
- Familiarity with AI security risks such as prompt injection, sensitive-data disclosure, excessive agency, memory poisoning, or insecure tool use.
Details
- Location: US, remote.
- Allstate generally does not sponsor individuals for employment-based visas for this position.
- The hiring manager has flexibility to hire at the Senior Consultant I to Lead Consultant level, determined by the selected applicant's skills and competencies.
- Candidates offered this position will be required to submit to a background investigation.
Read the full description and apply on the company’s own careers page.