LLM / Reasoning Engineer
. Locations are Austin, Charlotte, San Diego, NYC-Onsite
Responsibilities
Design the reasoning tier: structured-output disposition schemas, context-engineered prompts and skills, and a curated exemplar set drawn from the team's own prior decisions.
Build and own the gold set - and measure precision, recall and confidence calibration against it.
Work along with the existing domain team for execution.
Implement abstention as a first-class output (insufficient evidence must escalate, never guess) and self-consistency sampling for high-stakes items.
Mine reasoning traces to extract new deterministic rules, continuously pushing items down from the model tier to the rule tier - driving cost and latency down while consistency rises.
Drive building the regression suite for the AI layer so that a prompt, rule or model-version change cannot silently degrade quality.
Partner with domain SMEs to convert their verdicts into exemplars, noise filters and rules - closing and compounding the loop.
Own model routing (inexpensive models for the mechanical majority, frontier models for the hard tail) and the per-cycle token budget.
Qualifications
Python - production-grade.
Deep production LLM application engineering: structured / schema-constrained generation, context engineering, tool use, and evaluation - not prototypes.
Evaluation engineering: golden sets, LLM-as-judge (and a clear-eyed view of its limits), calibration, trajectory and trace scoring, regression benchmarking.
A track record of making an LLM system measurably better sprint over sprint, with numbers you can quote.
RAG and grounding quality - retrieval precision, recall and faithfulness scoring.
Sound statistics and ML fundamentals; understands why same-model self-validation has correlated failure modes and designs around it.
AWS Bedrock / Claude (or equivalent frontier models) in production.
Working knowledge
Code graphs and static analysis; SQL; relational data models.
Fine-tuning / PEFT; agentic orchestration frameworks; MLOps.
. Locations are Austin, Charlotte, San Diego, NYC-Onsite
Responsibilities
Design the reasoning tier: structured-output disposition schemas, context-engineered prompts and skills, and a curated exemplar set drawn from the team's own prior decisions.
Build and own the gold set - and measure precision, recall and confidence calibration against it.
Work along with the existing domain team for execution.
Implement abstention as a first-class output (insufficient evidence must escalate, never guess) and self-consistency sampling for high-stakes items.
Mine reasoning traces to extract new deterministic rules, continuously pushing items down from the model tier to the rule tier - driving cost and latency down while consistency rises.
Drive building the regression suite for the AI layer so that a prompt, rule or model-version change cannot silently degrade quality.
Partner with domain SMEs to convert their verdicts into exemplars, noise filters and rules - closing and compounding the loop.
Own model routing (inexpensive models for the mechanical majority, frontier models for the hard tail) and the per-cycle token budget.
Qualifications
Python - production-grade.
Deep production LLM application engineering: structured / schema-constrained generation, context engineering, tool use, and evaluation - not prototypes.
Evaluation engineering: golden sets, LLM-as-judge (and a clear-eyed view of its limits), calibration, trajectory and trace scoring, regression benchmarking.
A track record of making an LLM system measurably better sprint over sprint, with numbers you can quote.
RAG and grounding quality - retrieval precision, recall and faithfulness scoring.
Sound statistics and ML fundamentals; understands why same-model self-validation has correlated failure modes and designs around it.
AWS Bedrock / Claude (or equivalent frontier models) in production.
Working knowledge
Code graphs and static analysis; SQL; relational data models.
Fine-tuning / PEFT; agentic orchestration frameworks; MLOps.