Software Engineer, AI Evaluation

Build the measurement systems that decide whether an agent change is actually an improvement.

About Sycamore

Sycamore is building the trusted agent operating system for the enterprise. Our platform helps companies build, deploy, and orchestrate AI agents that take on real operational work, with the security and control large organizations need.

We are a small, engineering-led team working directly with Fortune 500 enterprises. We have raised $65M from Coatue and Lightspeed, along with other investors and industry leaders.

The role

You will build the shared runtime and learning systems that Sycamore’s product engineers and customer applications use to create dependable agents.

One half of the work is the agent runtime: multi-turn sessions, model routing, tool execution, connectors, memory, human approvals, durable workflows, and the APIs and libraries that expose those capabilities.

The other half is the intelligence loop around that runtime: collecting useful signals from production behavior, connecting feedback to business outcomes, reproducing failures offline, evaluating changes, and turning what we learn into safer and more capable versions of agents. That can include improvements to prompts, tools, skills, context, memory, retrieval, routing, policies, and, where the evidence supports it, learning or optimization techniques.

This is not about letting agents silently rewrite themselves in production. Improvements should be grounded in evidence, versioned, measured against reproducible tasks, reviewed when risk requires it, and deployable with clear rollback paths. The goal is a closed loop from observation to learning to governed application.

You will also carry these systems all the way into Sycamore Build, our primary product surface. You will build React interfaces for composing and running agents, inspecting live traces and tool calls, reviewing evaluations, resolving approvals, and understanding why a system behaved the way it did. Core AI engineers are expected to own coherent end-to-end slices across backend services, APIs, state, and user experience, not stop at a service boundary.

You will work closely with Product engineers when a customer workflow exposes a missing primitive or a difficult failure mode. You will find the durable abstraction, build it without overfitting to one deployment, and operate it in production.

Product owns the customer-specific application and last mile. Infrastructure owns the execution and deployment substrate. Core AI owns the horizontal runtime, intelligence, and improvement capabilities that both depend on, along with the Sycamore Build experiences that make those capabilities usable.

Working on AI Evaluation

Agent quality is genuinely hard to measure. A change that looks better on a handful of examples often is not, a judge model can be confidently wrong in the same direction as the system it grades, and most of the metrics that are easy to collect are proxies for something you actually care about.

This specialization owns that problem. You will build the evaluation systems that gate what ships: offline task suites, replay corpora that catch model drift, scorers that actually exercise the artifact rather than pattern-matching its output, and the statistical discipline to say whether a difference is real.

You will also own the human side of measurement. Blinded side-by-side review, agreement between human reviewers and automated judges, and the calibration that makes an automated score trustworthy enough to gate a release.

This is the specialization where being rigorous matters more than being fast. A wrong evaluation is worse than no evaluation, because it moves the whole team confidently in the wrong direction.

What the work looks like

In one week, you might:

• Trace a failed long-horizon agent run across model responses, tool calls, retries, context, memory, and durable workflow state, then remove the underlying failure mode.
• Build a React product flow in Sycamore Build that streams agent progress, tool calls, artifacts, and approval requests while keeping complex runtime state understandable.
• Prototype an end-to-end improvement workflow: surface a failure cluster, let an operator inspect evidence and compare variants, then safely promote or reject a change.
• Turn a vague production complaint into a reproducible evaluation, a useful success or reward signal, and a regression test.
• Replay production-shaped tasks offline, cluster recurring failure patterns, and identify whether the right intervention belongs in the prompt, tool contract, context, retrieval, memory, model routing, or runtime.
• Build a system that proposes and compares agent variants, records why a change was suggested, and sends the result through an appropriate review and deployment path.
• Add a provider-independent tool or agent capability with typed inputs, explicit permissions, safe write behavior, and graceful degradation.
• Improve a long-running session so it survives partial failures, provider interruptions, retries, and context growth without losing correctness.
• Turn a pattern discovered in a customer deployment into a documented library or cloud service used by multiple products.
• Give an experiment framework real statistics: variance modeling, a minimum-trials calculation, and confidence intervals on deltas, then re-analyze past model comparisons and report which conclusions actually held.
• Audit a two-stage judge for correlated bias by swapping and matching the reasoning and grading models, and compare the result against blinded human review.
• Build a record-and-replay evaluation slice so that a model provider’s next release cannot silently change behavior without failing CI.
• Calibrate a release gate’s thresholds for agreement, latency, and cost against real evidence, then wire it in so a prompt change that breaks grounding or citations cannot merge.
• Run a blinded human-review campaign end to end, from generating the comparison set through aggregating results into a score you would stake a release on.

What you will do

• Develop the learning data plane around agents: structured trajectories, feedback and outcome signals, offline datasets, lineage, privacy controls, and reliable links between an agent version and its behavior.
• Design experiment and versioning systems for comparing changes through replay, shadow traffic, canaries, or controlled rollouts, with clear promotion and rollback criteria.
• Design typed tool interfaces and protocol-based execution across internal capabilities and customer-authorized services.
• Develop memory extraction and retrieval while enforcing tenant, user, and project visibility boundaries.
• Publish reliable APIs, event-driven interfaces, reusable libraries, and pluggable improvement strategies that work across different agent categories and enterprise deployments.
• Join customer conversations when a repeated requirement or production outcome reveals a missing horizontal capability.
• Own the evaluation systems that decide whether an agent change ships, and the credibility of the numbers they produce.
• Bring statistical discipline to model, prompt, and configuration comparisons, and push back when a result does not support the conclusion being drawn from it.

The environment you will work in

Our current Core AI environment includes Python cloud services; React and TypeScript product surfaces in Sycamore Build; asynchronous and streaming systems; typed APIs and data models; relational and vector data; durable workflows; protocol-based tool execution; multiple model providers; and cloud-native deployment.

This is context, not a checklist. We do not require previous experience with every language, framework, model provider, cloud platform, database, or infrastructure tool in our stack. Comparable experience building distributed runtimes, experimentation platforms, retrieval or recommendation systems, workflow engines, developer platforms, or production AI systems is highly relevant.

What we are looking for

• 5-12 years of software engineering experience. We will make exceptions for exceptional people in either direction.
• Strong backend and distributed-systems fundamentals, including typed API design, asynchronous workflows, persistence, reliability, and production debugging.
• An empirical approach to AI quality: you can form a hypothesis, design a useful evaluation, interpret noisy evidence, and distinguish a real improvement from movement in a proxy metric.
• A security-minded approach to multi-tenant systems, identity, authorization, credentials, tool execution, privacy, and auditability.
• AI-native. You use coding agents and modern models as a force multiplier while still owning architecture, correctness, evidence, and operational outcomes.
• Comfort building product-facing software. You can work in React and TypeScript when a capability needs a great interface, and you can reason about streaming state, accessibility, and end-to-end user experience.
• Comfort with startup ambiguity, fast feedback loops, and broad ownership.
• Genuine comfort with statistics: variance, power, significance, and the ways an experiment can mislead you.
• Experience designing evaluations for systems whose output is not a single correct answer, and skepticism about metrics that are easy to move.

Experience with reinforcement learning, preference learning, reward modeling, post-training, continual learning, ranking, causal inference, or large-scale experimentation is valuable but not required. We care more about whether you can connect learning ideas to production evidence and reliable systems than whether you have used a particular technique.

Engineers from agent-runtime, distributed-workflow, retrieval, recommendation, experimentation-platform, integration-platform, developer-platform, or production applied-AI backgrounds often do well here. We care more about the systems you personally built, measured, and operated than a particular company, school, language, or model vendor.

• A 30-minute introductory conversation.
• Two 60-minute technical interviews, one focused on systems design and one on coding.
• A take-home assignment where you build and present a real solution using the tools you would use on the job.
• Build the cloud services that help enterprise agents learn from production experience.
• Ship Core AI capabilities end to end in Sycamore Build, from cloud service and API design through the React experience customers use.
• Turn production outcomes into governed improvements used across customers.
• Shape how feedback-driven learning, agent evaluation, and safe self-improvement work in a high-trust enterprise setting.
• Join early enough to shape the Core AI architecture and engineering team.
• Receive competitive cash compensation and meaningful equity in the company you are helping build.

Hard problems, real impact

Trust architectures, memory systems, multi-agent coordination. The foundational layer that makes AI agents work in production.

Small team, high ownership

Every engineer shapes the product and the culture. No layers of process between you and the work that matters.

Backed by the best

$65M from Coatue, Lightspeed, Abstract Ventures, Dell Technologies Capital, 8VC, and notable industry angels.

Grow with us

Competitive compensation, meaningful equity, and a genuine focus on your growth as the company scales.