Help ensure software engineering benchmark tasks are accurate, reproducible, and resistant to flawed grading. You will evaluate repository-level tasks used to train and assess advanced AI models, reviewing reference patches, test harnesses, grading integrity, and written evaluation criteria.
Key Responsibilities- Assess the quality, correctness, and reproducibility of software engineering benchmark tasks.
- Review repository-level tasks, reference patches, test runners, and Docker isolation.
- Identify answer leakage, reward hacking risks, and grading issues.
- Provide clear written feedback using defined evaluation rubrics.
- At least 3 years of professional software engineering experience.
- Open-source contribution or maintainer experience, such as merged pull requests or committer and maintainer roles.
- Strong ability to audit reference patches, test runners, and Docker isolation environments.
- Fluency in Python and at least one of Java, Go, TypeScript, or C++.
- Familiarity with SWE-Bench Verified or similar repository-level benchmarks.
- Maintainer experience with major Python open-source projects, such as Django, Flask, scikit-learn, SymPy, or pytest.
- Prior experience with code review or task grading.
- Remote role open to candidates located in the United States.
- Hourly engagement.
- $70 to $90 per hour.
SWE-Bench Task Auditor in los angeles at Unknown Company
This position is listed as full time and able to be worked remotely.