Overview
At Mindrift, we connect domain experts with cutting-edge AI projects. We are seeking an Evaluation Scenario Writer - QA for a project focused on ensuring the quality and correctness of evaluation scenarios created for LLM agents. This is a flexible, project-based opportunity that blends manual scenario validation, automated test thinking, and collaboration with writers and engineers.
Responsibilities
- Reviewing and validating test scenarios from Evaluation Writers
- Spotting logical inconsistencies, ambiguities, or missing checks
- Suggesting improvements to structure, edge cases, or scoring logic
- Collaborating with infrastructure and tool developers to automate parts of the review
- Creating clean and testable examples for others to follow
Qualifications
- Strong QA background (manual or automation), preferably in complex testing environments
- Understanding of test design, regression testing, and edge case detection
- Ability to evaluate logic and structure of test scenarios (even if written by others)
- Experience reviewing and debugging structured test case formats (JSON, YAML)
- Familiarity with Python and JS scripting for test automation or validation
- Clear communication and documentation skills
- Willingness to occasionally write or refactor test scenarios
- Experience testing AI-based systems or NLP applications (valued)
- Familiarity with scoring systems and behavioral evaluation
- Git/GitHub workflow familiarity (PR review, versioning of test cases)
- Experience using test management systems or tracking tools
Benefits
- Get paid for your expertise, with rates that can go up to $55/hour depending on your skills, experience, and project needs
- Take part in a flexible, remote, freelance project that fits around your commitments
- Participate in an advanced AI project and gain valuable experience for your portfolio
- Influence how future AI models understand and communicate in your field of expertise
How To Get Started
Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule.
Employment details
- Seniority level: Internship
- Employment type: Part-time