Senior Principal Engineer - Kubernetes

Overview

Design, build and operate Lilly's Kubernetes-based hosting platform across multiple clouds, owning platform capabilities end to end and leading a small team of platform engineers.

What you'll do

• Build, deploy and operate Kubernetes clusters end to end, from provisioning and configuration through upgrades, networking, scaling and decommissioning, on a managed offering such as EKS or ROSA, or an equivalent platform.
• Design, operate, and troubleshoot Kubernetes networking, including CNI plugins (Cilium, AWS VPC CNI), Services and kube-proxy replacement, CoreDNS, Ingress and Gateway API, network policies, pod-to-pod and cross-cluster connectivity, and egress control.
• Develop and maintain custom Kubernetes controllers and operators in Go (controller-runtime, Kubebuilder, or Operator SDK), including CRD design, reconciliation logic, finalizers, and admission webhooks.
• Operate and extend controller-based platform tooling such as AWS Controllers for Kubernetes (ACK), Crossplane, kro, Argo CD, and cert-manager.
• Deploy and operate the Istio service mesh (sidecar and ambient modes), including mTLS, traffic management (Gateway, VirtualService, DestinationRule, ServiceEntry), authorization policies, mesh observability, and version upgrades.
• Author and maintain automation, tooling, and services in Go or Python that streamline platform operations, onboarding, and review processes.
• Build and manage infrastructure as code using Terraform, Helm, and Kustomize, with strong module design, state management, and policy-as-code practices.
• Own and evolve CI/CD and GitOps pipelines using GitHub, GitHub Actions, Argo CD, Artifactory, and Checkmarx, with a security-first approach to software supply chain.
• Implement node and workload autoscaling with Karpenter, Cluster Autoscaler, HPA/VPA, and KEDA, including node image and upgrade strategies.
• Review and approve pull requests for platform hosting, deployment configurations, and infrastructure changes, ensuring they are secure, reliable, and aligned with platform standards.
• Lead a small team of platform engineers, providing technical direction, architecture and design guidance, and mentoring to grow engineers' skills and independence.
• Own team delivery: break down roadmap items into sprint-sized work, plan and track sprints, manage dependencies and risks, and communicate status to stakeholders and leadership.
• Partner with application teams during onboarding, migration, and go-live, providing hands-on technical guidance and clear documentation.
• Participate in on-call and incident response for platform issues, including root cause analysis (RCA).

What you'll need

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
• 9+ years of experience in Platform or DevOps engineering, including running Kubernetes in production.
• Experience technically leading or mentoring a small team of engineers, with a track record of guiding design decisions and delivering platform work on schedule.
• Strong, hands-on experience with at least one managed Kubernetes offering (EKS, AKS, ROSA, or ARO).
• Strong programming experience in Go, including hands-on experience building Kubernetes controllers and operators.
• Hands-on experience with Kubernetes manifests, rollouts and Helm charts (deployments, rollbacks, upgrades), along with a solid understanding of core Kubernetes concepts such as storage (PV/PVC, StorageClasses, CSI), networking (Services, Ingress, DNS, network policies) and scheduling.
• Strong Kubernetes networking knowledge: CNI plugins, Service and kube-proxy modes, CoreDNS, Ingress and Gateway API, network policies, and troubleshooting pod, node, and VPC connectivity.
• Hands-on experience operating the Istio service mesh in production, including mTLS, traffic management, authorization policies, ingress and egress gateways, and upgrades.
• Hands-on experience with CI/CD and GitOps tooling: GitHub, GitHub Actions, Argo CD or Flux, and artifact management (Artifactory or equivalent).
• Infrastructure-as-Code experience with Terraform.
• Working knowledge of AWS and/or Azure core services, networking, and IAM.
• Practical experience with observability tooling (Prometheus, Grafana, Loki, Tempo, Mimir, or equivalents) and log analysis in Splunk or similar.
• Experience with Linux, containers, and container runtimes, and strong troubleshooting skills across the network and OS layers.

Nice to have

• Python for tooling and automation.
• Experience with ACK, Crossplane, kro, or similar infrastructure-from-Kubernetes approaches.
• Experience with security scanning and policy tools such as Checkmarx, Trivy, Kyverno, or OPA.
• Experience supporting platforms in regulated industries (Life Sciences, Healthcare).
• Kubernetes certifications (CKA, CKAD, CKS) or cloud certifications (AWS or Azure professional level).
• Experience with multi-cluster management, disaster recovery, and cost optimization (FinOps).
• Exposure to Agile methodologies and SDLC.

Details

• Location: Bengaluru, India.
• Full-time employee position.
• Availability for flexible hours may be needed to support platform operations, and some on-call or non-standard-hours support may be required depending on platform incident volume.