Research Scientist

About the Role

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make informed decisions before deploying models in production.

What You'll Do

• Evaluate newly released AI models across a range of use cases and report on their capabilities.
• Build benchmarks from the ground up, including dataset development and coordination with labelers.
• Advance automated evaluation methods for generated text to improve scalability and accuracy.
• Partner with engineers to implement and scale evaluation approaches in production systems.
• Work with AI labs and enterprise customers to understand evaluation needs and shape research priorities.
• Share research findings, including through publications where appropriate.

What We're Looking For

• At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation methods, NLP, or language models.
• A record of published research, with applied research experience and practical systems work valued.
• Strong Python skills and experience building clean, maintainable software; familiarity with Django, Flask, or similar HTTP frameworks.
• Hands-on experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant AI/ML techniques.
• Experience creating benchmarks or evaluation methodologies and translating research into scalable systems.
• A master's or PhD in computer science, machine learning, or a related field is preferred; candidates with a bachelor's degree and 1 to 3 years of applied experience may also be considered.
• Experience collaborating in engineering teams, including Git workflows, code review, and incorporating feedback.

Compensation & Benefits

Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California, United States.