Generalist Evaluator Expert
Organization: Mercor
Referral Partner: Crossing Hurdles
Type: Hourly Contract
Compensation: $35–$40/hour
Location: Remote
Duration: Approximately 1 month
Commitment: 10-40 hours/week, flexible and asynchronous
Role Responsibilities
- Design and optimize detailed prompts with multiple constraints and clear instructions for language model evaluation
- Define and document evaluation standards and develop comprehensive rubrics for assessing correct responses in consumer contexts
- Conduct testing and grading by running prompts through AI models and assessing outputs against expectations
- Support benchmarking and quality assurance to ensure prompt rigor and consistency before integration into official evaluation sets
- Collaborate with team members in quality assurance review processes to maintain evaluation reliability
Requirements
- BS or BA degree from a reputable institution (completed or in progress)
- Strong writing skills, critical thinking, and instructional clarity
- Ability to work independently and meet deadlines
- Significant familiarity with ChatGPT or similar language tools for personal or professional use
- Experience in teaching or research is preferred
- Access to a desktop or laptop computer (Chromebooks are not supported)
Application Process
- Upload resume
- AI interview based on your resume (15 min)
Seniority Level: Mid-Senior level
Employment Type: Contract
Job Function: Quality Assurance, Writing/Editing, and Research
Industries: E-Learning Providers, Education, and Information Services
#J-18808-Ljbffr