Unknown Company

Physics Researcher for AI Model Evaluation

Apply Now Save
san francisco, ca • Posted 4 days ago
Remote Full Time science
Role Overview

Create and verify human-quality reference solutions for the CritPt benchmark (arXiv: v3), a frontier research-level physics benchmark. The role produces fully human-verified reference data used to evaluate large language model performance on frontier physics reasoning. Work includes solving CritPt research-level problems end-to-end, auditing other experts'' solutions, and adjudicating between parallel solution attempts to determine the golden reference.

Physics subdomains covered

  • High Energy Physics and Mathematical Physics
  • Biophysics and Statistical Physics
  • Condensed Matter and AMO
  • Gravitation, Cosmology, and Astrophysics
  • Quantum Information
  • Optical Properties of Materials
  • Magnetic Materials
  • Measurements in Quantum Mechanics
Key Responsibilities
  • Solve research-level physics challenges end-to-end, with verifiable derivations, runnable code, and peer-reviewed references.
  • Decompose challenges into standalone checkpoint sub-problems that require genuine physical reasoning and can be independently verified.
  • Author Python answer templates that include automated grading functions for symbolic and numerical answers.
  • Audit submitted solutions for correctness, scope, and soundness of method, providing actionable feedback across iterations.
  • Adjudicate between parallel solver attempts and decide which solution becomes the golden reference for a problem.
  • Document detailed chain-of-thought reasoning, specify error tolerances, present equivalent symbolic forms, and supply verification test cases.
Qualifications
  • Solver track: PhD or postdoc in the relevant subfield, senior PhD student minimum.
  • Auditor track: Postdoc or junior professor in the relevant subfield, PhD minimum.
  • Adjudicator track: Full professor or industry research principal investigator in the relevant subfield, senior postdoc or junior professor minimum.
  • Hands-on familiarity with at least two canonical methods of the target subfield, demonstrated through publications, broader coverage preferred.
  • Provide 3 to 5 representative publications, with arXiv ID or DOI, ideally within the last approximately 5 years and in the target subfield.
  • Working proficiency with LaTeX, Python, Jupyter, and SymPy.
  • Strong written English, B2, C1, or C2 level minimum; native or near-native preferred.
Work Terms
  • Location: Remote.
  • Employment type: hourly.
  • Expected commitment: approximately 10 hours per week, sustained across an 8 to 10 week window per task pool.
  • Work is asynchronous.
Compensation
  • Pay range: $80 to $135 per hour, based on role and demonstrated expertise.
Eligibility
  • Candidates must hold the academic or research standing specified under Qualifications for the track they apply to.
  • Applicants must be able to provide the requested publications (arXiv ID or DOI) and demonstrate working proficiency with the listed tools.
  • Strong written English is required to prepare the human-verified reference solutions and feedback.
Back to Job Search