Unknown Company

SWE-Bench Task Auditor

Downey, California • Posted Today
Remote Full Time technology

Help safeguard the quality and reliability of repository-level software engineering benchmarks used to train and evaluate advanced AI models. You will review benchmark tasks, reference patches, test harnesses, and grading logic, then deliver clear written feedback using defined rubrics.

Key Responsibilities
  • Evaluate task quality, correctness, and reproducibility across software engineering benchmark repositories.
  • Audit reference patches, test runners, Docker isolation, and grading integrity.
  • Identify potential answer leakage, reward hacking, and other issues that could compromise benchmark validity.
  • Provide precise, rubric-based written assessments.
Qualifications
  • At least 3 years of professional software engineering experience.
  • Open-source contributor or maintainer experience, such as merged pull requests or committer and maintainer roles.
  • Strong ability to review reference patches, test runners, and Docker isolation environments.
  • Fluency in Python plus at least one of Java, Go, TypeScript, or C++.
Preferred Qualifications
  • Familiarity with SWE-Bench Verified or comparable repository-level benchmarks.
  • Maintainer experience with major Python open-source projects, such as Django, Flask, scikit-learn, SymPy, or pytest.
  • Previous code review or task grading experience.
Work Terms
  • Remote, United States.
  • Hourly engagement.
Compensation
  • $70 to $90 per hour.

SWE-Bench Task Auditor in Downey at Unknown Company

This position is listed as full time and able to be worked remotely.

Back to Job Search