Overview
Build and maintain the tools and frameworks that manage the full lifecycle of Kafka clusters as part of Sumo Logic's internal Platform-as-a-Service, within the Kafka Infrastructure team.
What you'll do
- Build and maintain automation and tooling for the Kafka cluster lifecycle: provisioning, upgrades, scaling, failover and migration
- Develop and own the self-service interfaces teams use to consume Kafka (Terraform modules, CLI tools, APIs, dashboards), with predictable, versioned behavior
- Maintain and evolve the team's core components, including cluster balancing, consumer-lag monitoring, schema management, cluster-to-cluster data migration, Kafka-to-S3 backup and hardened machine images
- Improve observability and operational practices for Kafka across the organization, and drive standardization of tooling
- Provide second-level support for Kafka clusters, and participate in the on-call rotation (with an eventual move into the PaaS global on-call rotation)
- Educate and collaborate with application teams through onboarding sessions, documentation and migration guides, and consult on partitioning, consumer-group and schema best practices
- Evaluate and test new Kafka technologies and designs, and make recommendations
- Write detailed design documents and documentation on system design and implementation
What you'll need
- Bachelor's or Master's degree in Computer Science or another scientific or technical discipline
- Ability to influence: understand people's values and motivations and influence them towards making good architectural choices
- Collaborative working style: you can work with other engineers to come up with good decisions
- Bias towards action: you need to make things happen; be an enabler, not an inhibitor of progress
- Flexibility: you are willing to learn and change, and admit past approaches might not be the right ones now
- 2+ years of experience in software development in Java/Scala/Golang or similar
- 1+ years of hands-on experience with Apache Kafka, including brokers, topics, partitions, replication and consumer groups
- Good understanding of Kafka internals and operations: partition reassignment and rebalancing, leader election and in-sync replicas, retention and compaction, and broker and consumer performance tuning
- Experience with cloud platforms like AWS/Azure/GCP (AWS preferred)
- Skilled in Infrastructure-as-Code (IaC) tools, specifically Terraform
- Proficient in scripting languages such as Python/Bash
- Comfortable with Linux administration and troubleshooting distributed systems in production
- Familiarity with monitoring tools like Prometheus, Grafana, or similar, including consumer-lag monitoring
- Working knowledge of CI/CD and version control workflows
- Knowledgeable in the security aspects of Kafka (TLS, authentication and authorization)
Nice to have
- Experience with Cruise Control, Schema Registry, Kafka Connect or MirrorMaker
- Experience building hardened machine images (for example with Packer)
- Experience with Kubernetes and GitOps
- Experience building CLI tools or internal developer platforms
Read the full description and apply on the company’s own careers page.