Senior Backend Engineer - Databases Tempo | US | Remote

Overview

Senior Backend Engineer on the Tempo distributed tracing backend team at Grafana Labs, owning technical direction on major roadmap problems for Grafana Cloud Traces and Grafana Enterprise Traces.

What you'll do

• Deliver significant technical projects, taking well-scoped, multi-month initiatives from problem framing through rollout, such as trace aggregation APIs, Limitless Tempo, autoscaling cells and customer limits, or query engine improvements.
• Partner with Staff engineers and the tech lead on broader, cross-team direction.
• Own major components of Tempo, contributing to and owning parts of ingestion, storage, query, and metrics generation.
• Write and present design docs, participate in design reviews, make sound trade-offs on performance, cost, and complexity, and document the why for the team.
• Help shape the next generation of Tempo's interfaces (structured, deterministic, discoverable) so that Act 3 products, LLM-driven assistants, and external integrators can build on Tempo reliably.
• Own outcomes for areas you build against concrete SLOs (P99 write latency, incident recurrence, TCO per ingested GB), and help move the team toward Zero Ops through automation, parameterized rollouts, and actionable alerts.
• Work with PMs and with App Observability, Asserts, Drilldown, and Grafana Assistant teams to understand how Tempo gets consumed and ship what unblocks them.
• Raise the bar through thorough code review, constructive design feedback, and pairing, especially with junior and mid-level engineers.
• Participate in on-call for the services you help build, and contribute to incident response and post-incident learning.
• Engage the open-source community, review external contributions, and contribute to Tempo in the open.

What you'll need

• A track record of delivering complex, multi-month projects that spanned design, delivery, and operations, and of helping teammates grow.
• Significant hands-on experience building and operating distributed data systems in production: ingestion pipelines, storage engines, query execution, or similar.
• Strong software craftsmanship: clean, robust, performant software that others can maintain, and knowing when to optimize vs. when to ship.
• Strong Go, or a path to it. Solid experience in other systems languages (Rust, C, C++) translates well.
• Operational mindset: have owned production services, carried a pager, reduced toil, and treated SLOs as a product feature.
• Customer focus and pragmatism: break complex problems into short feedback loops (analyze, design, deliver an MVP, learn, iterate).
• Collaboration and clear writing: contribute through design docs, reviews, and shipped code, and communicate clearly in a fully remote, asynchronous environment.

Nice to have

• Experience with tracing, OpenTelemetry, or large-scale observability systems.
• Experience designing query languages, SQL/TraceQL-like engines, or APIs intended to be consumed programmatically (by services or agents).
• Experience with columnar storage formats (e.g., Parquet) or purpose-built on-disk formats for analytical workloads.
• Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
• Experience building for AI/LLM consumers: structured APIs, metadata/discovery endpoints, deterministic outputs, evaluation harnesses.
• Open-source contribution or maintainership, and comfort engaging a community in the open.
• Experience as an on-call user of Grafana, Prometheus, Loki, or Tempo in a previous role (or on a homelab).
• Experience in a fully remote, globally distributed team.

Details

• Remote position; Grafana Labs is seeking candidates in the US and Canada.
• US OTE/Base compensation range: USD $154,445 - $185,334; actual compensation may vary by level, experience, and skillset.
• Includes Restricted Stock Units (RSUs).
• In-person onboarding provided.