Temporal logo

Senior Software Engineer, Reliability

Temporal
NewPosted yesterday

LOCATION

United States · Remote

EXPERIENCE

Not specified

TYPE

FullTime

SKILLS REQUIRED

Distributed SystemsIncident ResponseSite Reliability EngineeringObservability

Job description

Overview

Senior Software Engineer on the Reliability team, reporting to Tushar Roy, focused on the reliability of operating Temporal Cloud end to end, including building an AI-first Reliability Agent and making reliability knowledge accessible across engineering.

What you'll do

  • Own reliability outcomes for operating Temporal Cloud end to end, partnering across engineering, infrastructure, and product to drive measurable improvements.
  • Build and evolve the Reliability Agent, including workflows that gather context from telemetry, incidents, runbooks, deployments, and service history to support faster, more consistent triage.
  • Design safe, useful agent experiences for handling routine reliability questions and issues, with appropriate human oversight, escalation paths, auditability, and safeguards against incorrect actions.
  • Strengthen incident response for Temporal Cloud, from detection and triage through escalation, mitigation, and communication, and lead post-incident reviews that turn into durable corrective actions.
  • Define and evolve reliability practices, including service-level objectives, alerting thresholds, operational readiness criteria, and escalation paths.
  • Plan and run gamedays and safely expand chaos testing to validate failure response, evaluate agent-assisted operations, and reduce operational risk.
  • Build and maintain reliability scorecards that make service health, operational readiness, agent effectiveness, and improvement priorities visible.
  • Lead load, performance, and capacity testing, and improve observability standards across services.

What you'll need

  • Strong computer science fundamentals, especially in distributed systems, concurrency, performance, and fault-tolerant design.
  • Experience designing, building, or operating complex systems under high load and partial failure.
  • Experience building AI-powered tools, agents, or production automation, with an understanding of tool use, evaluation, state, failure modes, and human-in-the-loop design.
  • A track record of driving reliability improvements across multiple services, systems, or engineering teams.
  • Hands-on experience with incident response and post-incident reviews, including leading incidents and following corrective actions through to completion.
  • Hands-on experience with one or more of gamedays, chaos testing, load testing, capacity planning, reliability scorecards, or observability programs.
  • Sound judgment in ambiguous situations, with the ability to balance reliability, agent autonomy, safety, customer impact, and delivery speed.
  • Clear communication skills and the ability to align stakeholders, share operational knowledge, mentor others, and participate in an on-call rotation.

Nice to have

  • Experience operating multi-tenant systems and designing protections against noisy-neighbor behavior.
  • Experience building internal platforms or tooling that enables teams to meet reliability standards.
  • Familiarity with LLM observability, agent evaluation, retrieval-augmented generation, or workflow orchestration systems.
  • Open source contributions in infrastructure, distributed systems, AI, or reliability engineering.

Details

  • Remote, United States role.
  • No regular travel expected.
  • Participation in an on-call rotation.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Senior Software Engineer, Reliability