(Senior/Junior AI Inference Reliability Engineer, Compute Infra

Overview

Own the production reliability of the Kubernetes-native control plane that provisions and runs Together AI's GPU inference fleet, including the host lifecycle pipeline, automated self-healing, and SLOs for inference serving.

What you'll do

• Operate the provisioning state machine in production: own the health of the host lifecycle pipeline (discovery, inference bring-up, GPU driver/CUDA validation, health checks, decommission/RMA).
• Triage stuck or failed state transitions at fleet scale and feed real failure patterns back into how the states and transitions are designed.
• Triage and extend automated self-healing, diagnosing new failure modes live (including InfiniBand issues in disaggregated prefill/decode serving, GPU Xid variants, and hero-model or LoRA-specific faults) and turning each into a new detection/remediation plugin.
• Own end-to-end reliability of the pipeline: hold SLOs and error budgets for provisioning and fleet health, lead incident response, and drive systemic fixes.
• Partner with the inference/ML platform team to feed real cluster topology, interconnect, and scheduling failure modes back into platform abstractions.
• Work with ML researchers, engineers, and product managers to bring new technologies into production and improve performance, latency, throughput, and efficiency of the inference stack.
• Build tools that give visibility into bottlenecks and sources of instability, and design and implement solutions for the highest-priority issues.
• Lead reliability cross-functionally across infra, ML platform, and product teams, and represent reliability in planning.

What you'll need

• Experience operating a provisioning or host-lifecycle pipeline in production, including troubleshooting stuck or failed state transitions at fleet scale.
• Deep experience owning production reliability for large-scale distributed systems.
• Hands-on experience troubleshooting InfiniBand/RDMA networking issues, ideally in a multi-node training or disaggregated-inference context, including reading IB port counters and subnet manager logs.
• Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
• Familiarity with the operational realities of multi-tenant model serving (LoRA/adapter-based serving) and with running differentiated SLOs across model pools of different priority.
• A demonstrated track record of eliminating operational toil through automation and tooling you built yourself.
• A software-engineering mindset: you write and ship production code and treat GPU infrastructure as software to be engineered.

Nice to have

• GPU/ML-serving infrastructure experience.
• Comfort with event-driven architectures, backpressure/circuit-breaking design, and built-in observability (metrics, logs, traces).

Details

• Location: San Francisco.
• Employment type: full-time.
• US base salary range: $160,000 - $280,000 plus equity and benefits.