Principal Production Engineer

Overview

Principal Production Engineer role in Zscaler's Production Engineering team within Cloud Infrastructure & Operations, providing technical vision and hands-on execution to drive an automation-first culture and improve reliability of a global platform.

What you'll do

• Design and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environments
• Drive an automation-first culture by writing code (Python/Go) to eliminate manual toil and build self-healing systems
• Implement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgets
• Act as a lead Incident Commander (TDO on-call), develop response playbooks, and conduct deep-dive post-incident analyses
• Partner with Engineering and partner teams to conduct operability reviews

What you'll need

• Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes within your functional domain
• 10+ years of experience managing reliability, scalability, and availability for large-scale production services
• Deep expertise in programming (e.g., Python, Go, or C/C++)
• Strong background in networking protocols, Linux/RHEL systems, and distributed architecture
• Experience in high-stakes incident management and participation in a 24/7 on-call rotation
• Proficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews

Nice to have

• Proven ability to leverage AI technologies and workflows to drive measurable operational efficiency
• Extensive experience with public cloud (AWS, Azure, GCP) and Infrastructure-as-Code (Ansible, Terraform, Helm, Temporal)
• Experience with chaos engineering and disaster recovery planning at scale
• Expertise in global routing (BGP) and traffic tunneling (GRE, IPSec) with a deep understanding of L7 proxy architectures (HAProxy), DNS at scale, and OS networking stack internals

Details

• Hybrid: 3 days a week in San Jose, CA, or Remote
• Reports to Production Engineering in the Cloud Infrastructure & Operations department
• Full-time position
• Base pay range $164,500 – $235,000 USD (excludes commission/bonus/equity and benefits)