Overview
Senior Software Engineer on the Reliability team, reporting to Tushar Roy, focused on the reliability of operating Temporal Cloud end to end, including building an AI-first Reliability Agent and making reliability knowledge accessible across engineering.
What you'll do
- Own reliability outcomes for operating Temporal Cloud end to end, partnering across engineering, infrastructure, and product to drive measurable improvements.
- Build and evolve the Reliability Agent, including workflows that gather context from telemetry, incidents, runbooks, deployments, and service history to support faster, more consistent triage.
- Design safe, useful agent experiences for handling routine reliability questions and issues, with appropriate human oversight, escalation paths, auditability, and safeguards against incorrect actions.
- Strengthen incident response for Temporal Cloud, from detection and triage through escalation, mitigation, and communication, and lead post-incident reviews that turn into durable corrective actions.
- Define and evolve reliability practices, including service-level objectives, alerting thresholds, operational readiness criteria, and escalation paths.
- Plan and run gamedays and safely expand chaos testing to validate failure response, evaluate agent-assisted operations, and reduce operational risk.
- Build and maintain reliability scorecards that make service health, operational readiness, agent effectiveness, and improvement priorities visible.
- Lead load, performance, and capacity testing, and improve observability standards across services.
What you'll need
- Strong computer science fundamentals, especially in distributed systems, concurrency, performance, and fault-tolerant design.
- Experience designing, building, or operating complex systems under high load and partial failure.
- Experience building AI-powered tools, agents, or production automation, with an understanding of tool use, evaluation, state, failure modes, and human-in-the-loop design.
- A track record of driving reliability improvements across multiple services, systems, or engineering teams.
- Hands-on experience with incident response and post-incident reviews, including leading incidents and following corrective actions through to completion.
- Hands-on experience with one or more of gamedays, chaos testing, load testing, capacity planning, reliability scorecards, or observability programs.
- Sound judgment in ambiguous situations, with the ability to balance reliability, agent autonomy, safety, customer impact, and delivery speed.
- Clear communication skills and the ability to align stakeholders, share operational knowledge, mentor others, and participate in an on-call rotation.
Nice to have
- Experience operating multi-tenant systems and designing protections against noisy-neighbor behavior.
- Experience building internal platforms or tooling that enables teams to meet reliability standards.
- Familiarity with LLM observability, agent evaluation, retrieval-augmented generation, or workflow orchestration systems.
- Open source contributions in infrastructure, distributed systems, AI, or reliability engineering.
Details
- Remote, United States role.
- No regular travel expected.
- Participation in an on-call rotation.
Read the full description and apply on the company’s own careers page.