LLM / Reasoning Engineer

LLM / Reasoning Engineer

Locations are Austin, Charlotte, San Diego, NYC-Onsite

Responsibilities

• Design the reasoning tier: structured-output disposition schemas, context-engineered prompts and skills, and a curated exemplar set drawn from the team's own prior decisions.
• Build and own the gold set — and measure precision, recall and confidence calibration against it.
• Work along with the existing domain team for execution.
• Implement abstention as a first-class output (insufficient evidence must escalate, never guess) and self-consistency sampling for high-stakes items.
• Mine reasoning traces to extract new deterministic rules, continuously pushing items down from the model tier to the rule tier — driving cost and latency down while consistency rises.
• Drive building the regression suite for the AI layer so that a prompt, rule or model-version change cannot silently degrade quality.
• Partner with domain SMEs to convert their verdicts into exemplars, noise filters and rules — closing and compounding the loop.
• Own model routing (inexpensive models for the mechanical majority, frontier models for the hard tail) and the per-cycle token budget.

Qualifications

• Python — production-grade.
• Deep production LLM application engineering: structured / schema-constrained generation, context engineering, tool use, and evaluation — not prototypes.
• Evaluation engineering: golden sets, LLM-as-judge (and a clear-eyed view of its limits), calibration, trajectory and trace scoring, regression benchmarking.
• A track record of making an LLM system measurably better sprint over sprint, with numbers you can quote.
• RAG and grounding quality — retrieval precision, recall and faithfulness scoring.
• Sound statistics and ML fundamentals; understands why same-model self-validation has correlated failure modes and designs around it.
• AWS Bedrock / Claude (or equivalent frontier models) in production.

Working knowledge

• Code graphs and static analysis; SQL; relational data models.
• Fine-tuning / PEFT; agentic orchestration frameworks; MLOps.