Member of Technical Staff, Coding Research

Role Overview

Advance the evaluation and development of frontier coding agents by designing the benchmarks, methodologies, and data systems used to measure and improve next-generation coding models. This role combines AI research, software engineering, and model evaluation in a remote, full-time technical research environment.

Key Responsibilities

• Design and own evaluation frameworks for coding agents, including benchmark specifications, scoring methods, rubrics, and quality standards.
• Lead end-to-end research initiatives that measure and improve coding-model performance across a range of software engineering tasks.
• Create high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Analyze model behavior and failure modes, identify systematic weaknesses, and turn findings into improvements for training and evaluation.
• Build tooling and infrastructure for large-scale experiments, data generation, review workflows, and evaluation pipelines.
• Establish rigorous, reproducible best practices for coding-agent assessment and measurement quality.
• Partner with researchers, engineers, and applied AI teams to design experiments and assess emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research initiatives that communicate performance findings and insights.

Qualifications

• Strong software engineering background and proficiency in Python, C++, or comparable programming languages.
• At least 3 years of experience in software engineering, machine learning, AI research, evaluation, or a related technical field.
• Experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems.
• Ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical ability to investigate model behavior and derive insights from complex technical systems.
• Excellent written and verbal communication skills, including the ability to explain technical findings to varied audiences.
• Comfort working in a fast-moving research environment with ambiguity and evolving priorities.

Preferred Experience

• Experience with frontier AI systems, coding agents, or model-evaluation research.
• Deep interest in how data, evaluations, and feedback mechanisms influence model capabilities.
• A record of independently taking ambiguous technical or research projects from concept through execution.
• Experience designing benchmarks or datasets for machine learning systems at scale.
• Familiarity with agentic workflows, tool use, reinforcement learning, or post-training methods.
• Publications, open-source contributions, or demonstrated technical leadership.

Work Terms

• Full-time remote role.

Compensation

• Annual compensation range: $400, 000 to $800, 000.