Staff Software Engineer - OpenTelemetry Python instrumentation | Ireland | Remote

Overview

Staff Software Engineer focused on the OpenTelemetry Python SDK and instrumentation libraries, with emphasis on instrumenting AI applications and agent workflows. This is a remote role based in the Republic of Ireland.

What you'll do

• Develop and maintain the OpenTelemetry Python SDK and instrumentation libraries, working with maintainers and other vendors on specifications, APIs, and implementation decisions.
• Build instrumentation that connects agent runs, model calls, and tool execution, including across asynchronous code.
• Contribute to evolving OpenTelemetry conventions for generative AI, with care for sensitive data and metric cardinality.
• Improve Python SDK performance by using profiling and benchmarks to evaluate implementation approaches with the upstream community, making sound tradeoffs around runtime overhead, compatibility, and maintainability.
• Be visible as an OpenTelemetry expert by teaching others and promoting best practices around OpenTelemetry, semantic conventions, and related technologies and frameworks, within Grafana Labs and the broader OpenTelemetry community.
• Support the technical direction and vision of the team, and contribute to business conversations on the OpenTelemetry strategy at Grafana Labs.
• Own projects from an ambiguous problem through design, implementation, testing, documentation, and release.
• Partner with GTM, Product and Support teams, join customer conversations, investigate instrumentation issues, and turn what you learn into fixes and practical guidance.

What you'll need

• 8+ years of experience in software development.
• Strong proficiency in Python and experience debugging production software, language internals, libraries, and asynchronous code.
• An interest in AI applications and agent systems, and curiosity about how to instrument and troubleshoot them.
• Experience working in open-source communities.
• Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry, alongside experience with cloud-based software or systems engineering.
• Understanding of production environments and the ability to assess operational risks.

Nice to have

• Prior experience building AI systems.
• Experience working on open source observability projects, ideally in OpenTelemetry or Prometheus.
• Experience in Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
• Experience in a hands-on technical leadership role, setting technical direction and leading project teams.
• Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI.
• Experience contributing to or maintaining open source projects, with evidence of successful pull requests and community collaboration.
• Rust experience for exploring Python SDK performance improvements, including Rust-backed implementations.

Details

• Location: Republic of Ireland (Remote).
• Remote role; high agency and good communication skills are required, as the team meets regularly via video calls.