Principal Production EngineerWe are looking for a Principal Production Engineer to join our team. This role is available as a hybrid opportunity 3 days a week in San Jose, CA or Remote reporting to Production Engineering in the Cloud Infrastructure & Operations department. Join Zscaler to be a force multiplier for the reliability of a global platform processing 200+ billion transactions daily across tens of millions of enterprise users.In this role, you will provide the technical vision and hands-on execution to drive an "automation-first" culture across the company.
By maturing our observability and architectural standards, you will directly reduce our Mean Time to Mitigate (MTTM) and shape the scalability of our globally distributed, multi-cloud infrastructure.Role ExpectationsDesign and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environmentsDrive an "automation-first" culture by writing code (Python/Go) to eliminate manual toil and build self-healing systemsImplement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgetsAct as a lead Incident Commander (TDO on-call), develop response playbooks, and conduct deep-dive post-incident analysesPartner with Engineering and partner teams to conduct operability reviewsSuccess ProfileYou act like an owner with a bias for action and integrity.You are a pragmatic builder obsessed with creating, iterating, and shipping.You champion simplicity by distilling complex problems into clear, actionable plans.You are data-driven, valuing evidence over assumptions.You think at scale, building solutions and processes built to last a high-growth global organization.Minimum Qualifications10+ years of experience managing reliability, scalability, and availability for large-scale production servicesDeep expertise in programming (e.g., Python, Go, or C/C++)Strong background in networking protocols, Linux/RHEL systems, and distributed architectureExperience in high-stakes incident management and participation in a 24/7 on-call rotationProficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviewsPreferred QualificationsExtensive experience with public cloud (AWS, Azure, GCP) and Infrastructure-as-Code (Ansible, Terraform, Helm, Temporal)Experience with chaos engineering and disaster recovery planning at scaleExpertise in global routing (BGP) and traffic tunneling (GRE, IPSec) with a deep understanding of L7 proxy architectures (HAProxy), DNS at scale, and OS networking stack internalsBase Pay Range $164,500 - $235,000 USDZscaler is committed to building a team that reflects the communities we serve and the customers we work with. We foster an inclusive environment that values all backgrounds and perspectives, emphasizing collaboration and belonging. Join us in our mission to make doing business seamless and secure.Our Benefits program is one of the most important ways we support our employees.
Zscaler proudly offers comprehensive and inclusive benefits to meet the diverse needs of our employees and their families throughout their life stages, including:Various health plansTime off plans for vacation and sick timeParental leave optionsRetirement optionsEducation reimbursementIn-office perks, and more!