Up to $300k base salary + Bonus + Equity.
Salary can be open for the right engineer.
San Francisco, Bay Area (Remote or Hybrid)
About the Role
Our client is scaling a rapidly growing experimentation program, 120+ experiments already running, with an aggressive growth roadmap behind them — and needs someone to serve as the statistical brain behind it. You'll own the methodology layer that makes every experiment trustworthy, sensitive, and correctly interpreted.
This is not a platform-engineering role, that team builds the infrastructure. Your job is to design, validate, and continuously improve the statistical frameworks that sit on top of it, and to be the person the team turns to when they ask "can we trust this result?"
Duties
- Guardrail metrics & SRM detection: design and maintain automated anomaly detection for live experiments, including Sample Ratio Mismatch checks and traffic split validation; define alerting thresholds and circuit-breaking criteria so compromised experiments get flagged or stopped before polluting decisions; define and validate guardrail metrics for sensitivity, directionality, and interpretability
- Variance reduction & sensitivity: implement and iterate on CUPED and related pre-experiment covariate adjustment methods; develop techniques to strip noise from user historical behaviour, enabling faster detection of true treatment effects in limited-traffic or high-priority scenarios (target: shortening average experiment duration from 14 days to 9, unlocking 40%+ more experiments per quarter)
- Continuous A/A monitoring: design and run always-on A/A experiments as a health check on the data pipeline, from client-side event reporting through message queues to the real-time data warehouse; define what "healthy" looks like for metric baseline volatility and pipeline stability
- Partner with product, engineering, and growth teams on experiment design — sample size calculations, metric selection, duration estimation, result interpretation — and advise on causal inference methods (DID, synthetic control, RDD) when randomisation isn't feasible
- Translate methods into specifications, validation scripts, and decision frameworks the team can self-serve against while maintaining rigour
Target Candidates
You'll own the methodology from day one at a company serving 80M+ users, with direct access to leadership when a result challenges an existing plan — this team runs experiments to discover the right decision, not confirm one already made. It's a small, focused US R&D team where data scientists and engineers sit together: you define the science, engineers build it into production, with no political overhead and no six-month wait to see your methods deployed.
- MS or PhD in Statistics, Biostatistics, Economics/Econometrics, Computer Science, or a related quantitative field
- 3+ years designing and analysing online controlled experiments at a tech company with meaningful user scale — not exclusively survey experiments or clinical trials
- Solid foundations in hypothesis testing, power analysis, multiple testing correction, and sequential testing; hands-on with at least two of: variance reduction (CUPED or similar), SRM detection, or continuous data quality monitoring
- Proficient in Python (scipy, statsmodels or equivalent) and complex SQL (window functions, CTEs)
- Strong "AI sense" — extensive hands-on use of AI coding tools (Claude Code, OpenClaw, or similar) as a daily productivity multiplier, not a novelty
- Able to explain complex statistical concepts to non-technical stakeholders and write specifications engineers can implement directly
- Fluent in English and Mandarin, able to bridge the US R&D center with Asia-Pacific engineering and product teams in Singapore, Dubai, and beyond (comfortable with occasional early-morning/evening cross-timezone meetings)