Staff Software Engineer - OpenTelemetry Python instrumentation | Germany | Remote

Overview

Staff Software Engineer focused on the OpenTelemetry Python SDK and instrumentation libraries, with an emphasis on instrumenting AI applications and agent workflows. Remote role based in Germany.

What you'll do

• Develop and maintain the OpenTelemetry Python SDK and instrumentation libraries, working with maintainers and other vendors on specifications, APIs, and implementation decisions.
• Build Python instrumentation that connects agent runs, model calls, and tool execution, including across asynchronous code.
• Contribute to evolving OpenTelemetry conventions for generative AI, with care for sensitive data and metric cardinality.
• Improve Python SDK performance using profiling and benchmarks with the upstream community, making sound tradeoffs around runtime overhead, compatibility, and maintainability.
• Be visible as an OpenTelemetry expert by teaching others and promoting best practices for OpenTelemetry, semantic conventions, and related technologies, within Grafana Labs and the broader OpenTelemetry community.
• Support the technical direction and vision of the team, and contribute to business conversations on the OpenTelemetry strategy at Grafana Labs.
• Own projects from an ambiguous problem through design, implementation, testing, documentation, and release.
• Partner with GTM, Product and Support teams, join customer conversations, investigate instrumentation issues, and turn what you learn into fixes and practical guidance.

What you'll need

• 8+ years of experience in software development.
• Strong proficiency in Python and experience debugging production software, language internals, libraries, and asynchronous code.
• An interest in AI applications and agent systems, and curiosity about how to instrument and troubleshoot them.
• Experience working in open-source communities.
• Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry, alongside experience with cloud-based software or systems engineering.
• Understanding of production environments and the ability to assess operational risks.

Nice to have

• Prior experience building AI systems.
• Experience working on open source observability projects, ideally in OpenTelemetry or Prometheus.
• Experience in Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
• Experience in a hands-on technical leadership role, setting technical direction and leading project teams.
• Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI.
• Experience contributing to or maintaining open source projects, with evidence of successful pull requests and community collaboration.
• Rust experience for exploring Python SDK performance improvements, including Rust-backed implementations.

Details

• Location: Germany (Remote). Applications are also welcome from the UK, Ireland and Spain.
• 100% remote role with high agency and good communication skills required; guidance and regular meetings take place via video calls.