Senior DevOps Engineer

Overview

Senior DevOps Engineer, Observability to design, build, operate, and continuously improve a high-scale observability platform built around ClickHouse, HyperDX, OpenTelemetry, Kubernetes, and modern DevOps automation practices, taking end-to-end ownership of observability infrastructure for production environments.

What you'll do

• Design, build, and operate scalable observability platforms using ClickHouse, HyperDX, OpenTelemetry, Kubernetes, Prometheus, Grafana, Alertmanager, Fluent Bit, and Filebeat.
• Architect and optimize ClickHouse for high-volume observability workloads, including logs, traces, metrics, and telemetry analytics.
• Design and manage ClickHouse schemas, partitioning strategies, ordering keys, TTLs, materialized views, retention policies, storage efficiency, and query optimization.
• Build, operate, and improve HyperDX for log search, distributed tracing, service analysis, dashboards, telemetry correlation, troubleshooting, and root-cause analysis.
• Build and maintain scalable OpenTelemetry Collector pipelines for collecting, processing, enriching, filtering, sampling, and routing telemetry data.
• Implement reliable correlation across logs, metrics, and traces to support faster application troubleshooting, service dependency analysis, and incident resolution.
• Design resilient telemetry pipelines with batching, queuing, retries, backpressure handling, sampling, rate limiting, and cardinality controls.
• Deploy and operate observability infrastructure on Kubernetes, with focus on scalability, high availability, capacity planning, resiliency, and operational safety.
• Automate infrastructure deployment, configuration management, platform upgrades, application onboarding, and recurring operational tasks using DevOps best practices.
• Build and maintain CI/CD workflows using Jenkins, infrastructure automation using Terraform and Ansible, and service discovery or secrets management integrations using HashiCorp Consul and Vault.
• Partner with application engineering, platform engineering, SRE, and operations teams to troubleshoot production issues, improve observability coverage, and reduce mean time to detect and resolve incidents.
• Analyze production performance issues across infrastructure, applications, telemetry pipelines, and ClickHouse queries, and drive corrective actions to closure.
• Define and improve standards for telemetry instrumentation, log quality, metric hygiene, trace propagation, dashboard design, alert quality, and production readiness.
• Participate in incident response, post-incident reviews, capacity planning, operational reviews, and remediation tracking for observability services.
• Mentor junior engineers, review designs and automation changes, and raise the overall technical and operational maturity of the team.

What you'll need

• 6+ years of experience in DevOps, SRE, Platform Engineering, Infrastructure Engineering, or Observability Engineering roles.
• Strong production experience with ClickHouse, including architecture, administration, schema design, performance tuning, query optimization, troubleshooting, retention management, and operations at scale.
• Hands-on experience deploying and operating HyperDX, including integration with ClickHouse and usage for log search, distributed tracing, dashboards, service analysis, and troubleshooting.
• Strong experience with OpenTelemetry and OpenTelemetry Collector, including receiver, processor, exporter, sampling, enrichment, batching, and routing configurations.
• Strong working knowledge of Fluent Bit, Filebeat, Prometheus, Alertmanager, Grafana, ClickHouse, HyperDX, and OpenTelemetry.
• Strong DevOps and automation experience with Jenkins, CI/CD pipelines, Ansible, Terraform, HashiCorp Consul, HashiCorp Vault, and Kubernetes.
• Strong understanding of Linux, networking, microservices, REST, gRPC, distributed systems, high-availability architectures, and production operations.
• Experience operating and troubleshooting high-volume production systems with focus on reliability, scalability, capacity, and performance.
• Ability to debug complex issues across application telemetry, Kubernetes infrastructure, data ingestion pipelines, storage systems, and query performance.
• Strong scripting and automation skills, with the ability to reduce manual operations and improve repeatability, reliability, and operational efficiency.
• Ability to take ownership of production systems, drive problems to closure, communicate clearly during incidents, and collaborate effectively with engineering and operations teams.

Nice to have

• Development experience with Java and Python, including the ability to understand application code, troubleshoot instrumentation issues, and analyze performance behavior.
• Experience developing automation, internal tools, APIs, platform services, or self-service onboarding workflows.
• Experience with Kafka or other high-throughput messaging and streaming platforms.
• Strong understanding of APM, distributed tracing, OpenTelemetry instrumentation, context propagation, service maps, and service dependency analysis.
• Experience with cloud platforms such as AWS, Azure, GCP, or OCI.
• Experience designing observability solutions for large-scale Kubernetes, microservices, or distributed application environments.
• Experience improving alert quality, reducing noise, defining SLOs or SLIs, and supporting production readiness reviews.