LLM Evaluation Researcher/Benchmarking Engineer

Budget: $2000 fixed

I'm developing a proprietary AI-assisted story evaluation system for film, television and literature.

I need an experienced Python/LLM evaluation engineer to implement and stress-test an existing 20-dimension narrative assessment framework.

The framework, evaluation protocol and initial global benchmark manifest are already prepared.

Responsibilities:

• Build a reproducible Python-based LLM evaluation harness.
• Prepare and test 16 global benchmark works.
• Evaluate eight additional controlled works and their supplied test variations.
• Run blind, repeatability and diagnostic sensitivity tests.
• Track ratings, source evidence, model consistency, token usage and API costs.
• Deliver working source code, JSON/CSV results and a concise technical report.

Requirements:

• Strong Python and LLM API experience.
• Experience with LLM evaluation, benchmarking or AI quality assurance.
• Structured outputs, automated testing, source-document parsing and reliable logging.
• Ability to work independently within a fixed budget.

This is not a website development, chatbot, creative writing or model-training project.

The objective is to test an existing diagnostic system rigorously, identify weaknesses and produce reproducible results.

Please begin your proposal with THALBERG TEST and briefly describe a comparable LLM evaluation project you've completed.