Site Reliability Engineer

Overview

Act as the incident commander for PayPal's most critical incidents, directing application and infrastructure teams and authorizing critical actions such as rollbacks and regional failovers. Work within the Site Health Engineering (Command Center) team across PayPal, Venmo, Xoom, Zettle and Braintree platforms.

What you'll do

• Act as incident commander with final decision authority, directing engineering teams, authorizing rollbacks, and commanding regional failovers.
• Direct application and infrastructure teams during incidents by making work assignments and prioritizing troubleshooting paths.
• Rapidly assess incidents by reading Infrastructure as Code (Terraform, CloudFormation), Kubernetes manifests, and CI/CD configurations.
• Give final authorization for critical actions including production rollbacks, regional failovers, and emergency changes.
• Interface with executive leadership during critical incidents and post-mortems to provide technical guidance and impact assessments.
• Identify when incidents stem from teams deviating from established cloud-native patterns.
• Command cross-functional teams during high-severity incidents affecting PayPal core and brand platforms (Venmo, Xoom, Zettle, Braintree).
• Lead blameless postmortem sessions and contribute to Root Cause Analysis (RCA) processes.
• Drive continuous improvement initiatives based on incident learnings.
• Serve as the primary technical escalation point during critical incidents.
• Accelerate incident response times through standardized playbooks and automated workflows.
• Leverage knowledge in cloud platforms (AWS, GCP, Azure) to drive incident resolution.
• Support Braintree and Venmo cloud infrastructure operations.
• Guide teams toward solutions by providing direction during incidents.
• Stay current with emerging cloud technologies and best practices.
• Implement automation, dashboards, and tooling to enhance the team's incident response capabilities.
• Build runbooks and playbooks for cloud-native incident scenarios.
• Develop internal tools and scripts to improve operational efficiency.
• Drive projects that advance the Command Center's operational capabilities.
• Proactively identify and address vulnerabilities in cloud (AWS, GCP, Azure) and on-premises infrastructure.
• Review Infrastructure as Code changes for reliability risks as part of change approval process.
• Identify architectural anti-patterns in Kubernetes deployments and cloud migrations.
• Conduct regular disaster recovery drills and readiness tests before major events (Thanksgiving, Cyber 5, peak shopping seasons).
• Participate in situation room activities for new product rollouts.
• Drive site resilience projects to enhance system reliability and uptime.
• Lead new datacenter and CDN certification initiatives.
• Actively monitor and analyze system metrics to ensure the availability, performance, and reliability of digital platforms and applications.
• Diagnose and resolve complex system issues, perform root cause analysis, and implement long-term fixes to prevent recurrence.
• Create and maintain automation scripts, tools, and processes to streamline operations, reduce manual effort, and enhance reliability.
• Configure and improve monitoring and alerting tools to provide actionable insights into system health and performance.
• Analyze system usage trends and forecast future resource requirements to ensure scalability and prevent capacity-related issues.
• Work with development teams to design and implement reliable, fault-tolerant systems, incorporating best practices for high availability.
• Ensure smooth and reliable software releases by managing continuous integration and continuous delivery (CI/CD) pipelines.
• Perform failure simulations, such as chaos engineering, to identify weaknesses and improve system robustness.
• Develop and maintain detailed documentation for system configurations, operational procedures, and incident handling workflows.
• Partner with development, operations, and product teams to enhance system architecture for improved performance, reliability, and scalability.

What you'll need

• 1+ years relevant experience and a Bachelor's degree OR any equivalent combination of education and experience.
• Hands-on experience with at least one major cloud provider (AWS or GCP required).
• Hands-on experience with Kubernetes and CNCF ecosystem tools, including troubleshooting K8s deployments, manifests, and cluster issues.
• Proven experience managing critical incidents in Infrastructure-as-Code driven environments, including troubleshooting IaC state issues, GitOps failures, and cloud-native deployment problems.
• Experience with monitoring and observability tools (Splunk, Datadog, Prometheus, Grafana).
• Strong knowledge of networking, load balancing, CDN technologies, and DNS management.
• 2+ years of experience in site reliability engineering, infrastructure operations, or similar technical operations roles.
• Deep understanding of distributed systems, microservices architecture, and containerization (Docker, Kubernetes).
• Exceptional communication skills with ability to articulate complex technical issues to both technical and non-technical stakeholders.
• Executive presence and ability to effectively communicate with senior leadership during high-pressure incidents and post-mortems.
• Strong analytical and problem-solving abilities with a systematic approach to troubleshooting.
• Ability to remain calm under pressure and make critical decisions during incidents.
• Excellent collaboration skills with experience working across global, cross-functional teams.
• Strong documentation skills and attention to detail.

Details

• Location: Chennai, Tamil Nadu, India.
• Work mode: hybrid, with 3 days in the office and 2 days at your choice of the PayPal office or home workspace, for most employees.
• Shift pattern: 3 to 4 days of alternating 12-hour shifts.
• Subsidiary: PayPal. Travel: 0%.