Staff Backend Engineer - K8 (Envoy)

Overview

Lead the design and delivery of scalable, reliable AI inference infrastructure as a Staff Backend Engineer. Provide technical direction across teams building platform services for machine learning and generative AI workloads.

What you'll do

• Architect and build an AI Inference Gateway platform.
• Design request routing, endpoint abstraction, traffic shaping, load balancing, failover, caching, and policy enforcement.
• Develop infrastructure components in Go, Java, or Python.
• Design multi-tenant capabilities for authentication, authorization, quotas, cost attribution, rate limiting, and governance.
• Lead architecture reviews and mentor senior engineers across teams.
• Define observability standards and resolve complex production issues.
• Optimize performance, latency, throughput, and infrastructure efficiency.

What you'll need

• 8+ years of professional software development experience.
• 5+ years designing and operating large-scale distributed systems in production.
• Strong programming expertise in Go, Java, or Python.
• Experience building highly available, mission-critical platform or infrastructure services.
• Experience with Kubernetes and containerized production workloads.
• Experience operating services on AWS, Azure, or GCP.
• Understanding of microservices, distributed systems, observability, reliability engineering, and production operations.

Nice to have

• Experience with AI/ML inference platforms, LLM gateways, model serving, or GPU workloads.
• Expertise in Kubernetes networking technologies such as Gateway API, Envoy, or service mesh.
• Experience with inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, KServe, or SGLang.
• Experience with Kafka, Cassandra, Redis, MongoDB, or similar distributed data systems.

Details

• Location: Bengaluru.
• Work model: Hybrid, onsite, or remote; the hybrid model requires at least 3 office days per week.