Role Overview
Define what excellent digital product design looks like by creating task-specific grading criteria and by evaluating AI-generated and human design work. Instead of delivering designs yourself, you will specify rigorous standards for real-world product deliverables and score completed work samples with detailed, evidence-based written justifications.
Key Responsibilities- Design precise, task-specific grading rubrics for real-world product and UX deliverables, including design systems and component libraries, responsive UI layouts and dashboards, end-to-end product flows such as onboarding, checkout, and booking, logged-in customer portals, UX writing and product content, and design-to-engineering handoff.
- Score AI-generated and human work samples against those criteria, providing a clear, written justification for every score.
- Apply consistent, evidence-based judgment so scores are reproducible and defensible.
- Incorporate structured feedback from senior reviewers and iterate quickly on grading criteria and scoring approaches.
- Minimum 5 years of professional experience in product design, UI/UX, or design systems for digital products, not brand or marketing sites.
- Experience at leading digital-product and UX studios, in-house product-design teams at product-led companies, or in e-commerce and marketplace companies with sophisticated customer portals and dashboards.
- Relevant roles may include Product / UI / Design Systems Designer, UX Lead, UX Writer / Content Designer, or digital Product / Business Analyst.
- Deep fluency in interaction and visual design, design systems, responsive layouts, end-to-end product flows, prototyping and production handoff in Figma, and the ability to articulate critique as a design or UX lead would in a review.
- Exceptionally strong written communication, able to explain precisely why a piece of work does or does not meet the bar for a real product and its users.
- Detail-oriented and consistent, comfortable having your judgment reviewed and calibrated against peers.
- Prior experience with AI training, evaluation, or human-data projects is a strong plus.
- Location: Remote.
- Employment type: Hourly engagement.
- Pay range: 80 - 150 hourly.
The listing does not specify work-authorization or sponsorship details. Applicants should ensure they are eligible to work where they reside and able to perform remote work.
Application Process- To apply, submit your resume and portfolio or other documentation of relevant professional experience.
- Qualified applicants may be asked to complete a brief assessment or provide additional information as part of the evaluation.