Agentic AI, Senior Software Development Engineer

Overview

Senior Software Development Engineer for Zillow’s Agentic AI team, focused on building evaluation and observability capabilities for production AI agents.

What you'll do

• Design, build, and scale evaluation frameworks for Zillow’s agentic AI experiences.
• Build tracing, observability, and quality measurement systems for production AI agents.
• Create platform infrastructure so domain teams can build on the Evals platform.
• Integrate new AI evaluation capabilities into production systems with applied scientists and ML engineers.
• Help evolve how Zillow measures AI quality, reliability, trustworthiness, and product impact.
• Support scaling, reliability, performance optimization, incident response, and cost management for the evaluation layer.
• Stay current with agentic AI paradigms, evaluation techniques, and LLM tooling and translate them into platform improvements.

What you'll need

• 4+ years of backend engineering experience designing, shipping, and operating scalable production ML services.
• Experience building platform infrastructure or developer-facing services consumed by multiple teams.
• Hands-on experience with ML Evals & Observability frameworks (Databricks MLflow, with evolving LLM frameworks preferred such as LangSmith, Braintrust, or Promptfoo).
• Experience evaluating third-party AI platforms and making build-vs-buy decisions.
• Fluency working across applied science, ML, product, design, and engineering.
• Comfort with LLMs, agentic systems, and evaluation frameworks, including familiarity with orchestration tools (LangChain, LangGraph, MCP, or similar).
• Strong ownership and growth mindset in ambiguous environments.

Details

• Role is categorized as Remote in the U.S.