Staff Software Engineer - OpenTelemetry Python instrumentation | UK | Remote

Overview

Staff Software Engineer working on the OpenTelemetry Python SDK and instrumentation libraries, with a focus on OpenTelemetry for Generative AI and agent workflows. Works upstream-first within the open-source OpenTelemetry community and brings the results into Grafana Cloud.

What you'll do

• Develop and maintain the OpenTelemetry Python SDK and instrumentation libraries, working with maintainers and other vendors on specifications, APIs, and implementation decisions.
• Build Python instrumentation for AI agents that connects agent runs, model calls, and tool execution, including across asynchronous code.
• Contribute to evolving OpenTelemetry conventions for generative AI, with care for sensitive data and metric cardinality.
• Improve Python SDK performance using profiling and benchmarks with the upstream community, making sound tradeoffs around runtime overhead, compatibility, and maintainability.
• Be visible as an OpenTelemetry expert by teaching others and promoting best practices around OpenTelemetry, semantic conventions, and related technologies and frameworks, within Grafana Labs and the broader OpenTelemetry community.
• Support the technical direction and vision of the team and contribute to business conversations on OpenTelemetry strategy at Grafana Labs.
• Own projects from an ambiguous problem through design, implementation, testing, documentation, and release.
• Partner with GTM, Product and Support teams, join customer conversations, investigate instrumentation issues, and turn findings into fixes and practical guidance.

What you'll need

• 8+ years of experience in software development.
• Strong proficiency in Python.
• Experience debugging production software, language internals, libraries, and asynchronous code.
• An interest in AI applications and agent systems, and curiosity about how to instrument and troubleshoot them.
• Experience working in open-source communities.
• Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry.
• Experience with cloud-based software or systems engineering.
• Understanding of production environments and the ability to assess operational risks.
• High agency and good communication skills.

Nice to have

• Prior experience building AI systems.
• Experience working on open source observability projects, ideally OpenTelemetry or Prometheus.
• Experience in Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
• Experience in a hands-on technical leadership role, setting technical direction and leading project teams.
• Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI.
• Experience contributing to or maintaining open source projects, with evidence of successful pull requests and community collaboration.
• Rust experience for exploring Python SDK performance improvements, including Rust-backed implementations.

Details

• Remote role; applications considered from Germany, UK, Ireland and Spain.
• Compensation range in the UK: GBP 107,000 - GBP 134,000, with Restricted Stock Units (RSUs) included.
• Remote-first, with regular video-call meetings.
• Annual leave of 30 days per annum, including 3 Grafana Shutdown Days.