Responsibilities:
- Own the architecture health of a billion-scale distributed system, its failure modes, capacity limits, consistency guarantees, and the interactions between 50+ deployments that no single team can see.
- Approve critical-path designs. Changes that touch the system's core go through you, not as bureaucracy, but as the person accountable for the whole staying sound. When there's a disagreement, you make your case on merit.
- Hunt gaps proactively, single points of failure, unbounded queues, missing idempotency, thundering herds, and quiet data-loss windows, and drive the fixes before they become incidents.
- Build the parts nobody else can. No sprint tickets. You prototype the risky architectural bets yourself, ship the remediation after serious incidents, and pair into the gnarliest cross-team bugs, roughly a quarter to a third of your time in code, all of it on the hardest problems.
- Make resilience a property of the system, not a heroic act, degradation strategies, backpressure, isolation boundaries, and capacity models that survive 10%+ month-over-month growth.
- Raise the teams around you. Design reviews that teach, post-mortems that change architecture (not just add alerts), and patterns that 80+ engineers build on.
- Set the standard for how AI-assisted engineering works safely on systems this critical, where a bad merge doesn't cost a demo; it costs real businesses their revenue.
Requirements:
- 10+ years of engineering experience with deep, hands-on ownership of large-scale distributed systems; a comparable scale is strongly preferred: hundreds of services or thousands of instances, billions of daily events.
- You've carried sole accountability for a production system through real failures, not adjacent to it, not advising on it. Owned it.
- Deep command of queueing and async architectures, delivery semantics, ordering, backpressure, idempotency, exactly-once myths, and at-least-once realities.
- Strong with multiple storage engines (SQL and NoSQL), you reason about consistency models, indexing at scale, and when each engine is the wrong choice.
- Expert-level depth in Redis or comparable in-memory systems, including their failure modes under memory pressure and network partition.
- Production experience on Kubernetes at scale, resource limits, autoscaling behavior, and what actually happens when a node pool dies.
- Exceptional design communication, docs, diagrams, and RCAs that drive decisions across multiple teams.
- Fluent in Node.js and/or Go, enough to prototype your own proposals and ship fixes on the critical path.
The terrain:
- Runtime: Node.js (TypeScript), Go, and thousands of pods on GKE.
- Messaging and async: GCP Pub/Sub, Cloud Tasks, Redis.
- Storage: MongoDB, Firestore, ClickHouse, ElasticSearch.
- Scale: 21.5B automation actions/month, 2.6B messages/month, 28.5K req/s peaks, ~226B async events across the org.
What Success Looks Like:
- The critical paths of workflows have named owners, capacity models, and tested failure modes, because you made it so.
- Incident count trends down while traffic grows double-digit percent month over month.
- Your design reviews are the ones engineers want their proposals to survive.
- Your scope has expanded on results, more systems, more surface, and more trust.
Staff Engineer in tx at Unknown Company
This position is listed as full time and onsite.