Principal Software Engineer

Overview

Principal Software Engineer for HubSpot’s Observability platform, serving as the technical anchor for distributed tracing, metrics, and structured logging—plus AI/agent observability.

What you'll do

• Define patterns and evolution of the telemetry platform architecture at large scale.
• Set instrumentation standards across a large, polyglot engineering organization.
• Lead technical strategy for tracing and understanding AI agents and ML systems in production.
• Architect high-cardinality, high-throughput telemetry pipelines and storage with cost/query tradeoffs.
• Ship production code, lead design reviews, and drive initiatives from prototype to production.
• Design observability instrumentation APIs/libraries and drive OpenTelemetry adoption.
• Define SLO/SLI and alerting patterns to move from reactive to proactive incident response.

What you'll need

• Proven experience building observability/telemetry tooling for internal engineering teams.
• Deep expertise in telemetry pipeline trade-offs (high-cardinality, sampling, query latency, retention, fidelity).
• Strong fluency with OpenTelemetry, distributed tracing, metric backends, and log ingestion infrastructure.
• Track record linking telemetry signals to automated operational workflows and remediation.

Details

• Location: Remote - USA.
• Annual cash compensation range: $313,800 - $502,080 USD.
• In-person onboarding required at a regional HubSpot office for engineering team hires.