Chapter 80 — System One models: Jev and "LLM writes, model decides, code acts"

   Is this shell command destructive?              yes / no
   Which model tier should handle this step?       small / mid / frontier
   Is this support ticket urgent?                  yes / no
   Which of these 7 log categories is this?        one of 7
   Did this tool result answer the question?       yes / no
   Retry, ask the user, or give up?                one of 3
   How risky is this PR?                           low / med / high

   LLM (System 2)           writes: plans, code, prose, explanations
        |
        |   hands over a closed question + the state
        v
   DECISION MODEL           decides: returns probabilities over the
   (System 1)               options you defined, ~100s of ms
        |
        v
   PLAIN CODE               acts: thresholds the numbers, applies policy,
                            escalates the uncertain cases back up to the
                            LLM or a human

Jev: a decision-only model

In September 2026, TypeSafe AI, a San Francisco lab founded by Diogo Almeida (formerly an OpenAI researcher), Erik Gafni, and Sasha Sheng, launched Jev, which it calls a "System One model". It quickly appeared in developer tools: within days it was integrated into Vercel's AI Gateway, Netlify, and LangChain, according to InfoQ's coverage. What makes it different:

• It cannot write text. It takes state (text or JSON) plus a set of questions with predefined answers, and returns typed answers with probabilities. It can't explain itself, generate code, or ramble.

• Three question types:

  Type	You define	You get back
  Noul (yes/no)	an instruction phrased as a yes/no question	a single number 0–1: the probability of yes
  Choice	named options, each with a short description	the chosen option, a probability per option, and a confidence
  Score	an ordered rubric (levels)	a probability-weighted score, the distribution over levels, and a confidence

• Many questions, one pass. All questions about the same state are answered in parallel, so adding questions barely adds latency.

• Trained for calibration. TypeSafe describes a training method it calls Reinforcement Learning for Calibrated Decisions (RLCD), meant to make "0.9" mean "right about 90% of the time".

• Fast and cheap, by vendor claims. Launch figures: end-to-end latency of roughly 70–500 ms, a 32,000-token context, and input priced at $0.042 per million tokens with output free. These numbers come from the vendor and early adopters. Re-check current pricing, and measure latency yourself.

What it is not: there's no published paper or parameter count, and outside observers have speculated it is built on an open-weight LLM foundation. Neither point changes how you'd use it, but both are good reasons to evaluate it on your own data rather than trust headline numbers.

What a call looks like

The request shape as documented at launch (check the current API reference; field names may change):

# jev_client.py -- minimal client (pip install requests)
import os, requests

JEV_MODEL = "jev-1.13.0"     # pin a version once thresholds are tuned; aliases move

def ask_jev(state, questions, timeout=2.0):
    r = requests.post(
        "https://api.typesafe.ai/v1/systemone",
        headers={"authorization": f"Bearer {os.environ['TYPESAFE_API_KEY']}"},
        json={"model": JEV_MODEL, "state": state, "questions": questions},
        timeout=timeout)
    r.raise_for_status()
    return r.json()["answers"]

answers = ask_jev(
    state={"ticket": "The deploy failed twice and customers are seeing 500s. "
                     "Can someone look now?", "customer_tier": "enterprise"},
    questions={
        "urgent": {"type": "noul",
                   "instructions": "Does this need a human to act right now?"},
        "team":   {"type": "choice",
                   "instructions": "Which team should own this ticket?",
                   "criteria": {
                       "billing":  "Invoices, charges, refunds, plan changes",
                       "platform": "Outages, errors, deploys, performance",
                       "account":  "Login, SSO, permissions, user management",
                       "unknown":  "None of the above clearly applies"}},
    })
# answers["urgent"]["noul"]       -> e.g. 0.98   (P(yes))
# answers["team"]["choice"]       -> "platform"
# answers["team"]["confidence"]   -> e.g. 0.95
# answers["team"]["probabilities"]-> {"billing": 0.01, "platform": 0.95, ...}

Notice the "unknown" option. A closed set of options needs an escape hatch, or the model is forced to pick something wrong with apparent confidence.

Pattern 1: a command guard in the harness

This is Chapter 74's pre-tool hook, upgraded. Every shell command the agent wants to run gets a fast probability of being destructive; code applies the policy.

DESTRUCTIVE = {"type": "noul", "instructions":
    "Would running `command` delete data, rewrite shared history, or change "
    "production state in a way that is hard to undo?"}

def jev_command_guard(name, args):                 # plugs into Harness(pre_hooks=[...])
    if name != "bash":
        return None
    try:
        p = ask_jev({"command": args["command"], "cwd": "/workspace"},
                    {"destructive": DESTRUCTIVE})["destructive"]["noul"]
    except Exception:
        return ("deny", "safety check unavailable")   # FAIL CLOSED, never open
    if p >= 0.70:
        return ("deny", f"p_destructive={p:.2f}")
    if p >= 0.35:
        return ("deny", f"p_destructive={p:.2f}: needs human approval")
    return None                                        # allow, ~0.3 s later

An early adopter's write-up reported scores like rm -rf ./dist → 0.63 (ask), DROP TABLE users → 0.95 (deny), and git push --force → 0.95 (deny). Three design rules are visible in that code:

1. Two thresholds, three zones: allow / ask / deny. Your policy is two numbers you can change without touching a prompt.
2. Fail closed. If the decision service is down, the safe default is "ask a human", never "allow".
3. It's one layer, not the only one. Keep Chapter 74's hard deny rules for the known-catastrophic patterns. A probabilistic check is for the long tail that regexes miss.

Pattern 2: a model router

TIER = {"type": "choice", "instructions": "Which model tier should handle `task`?",
        "criteria": {
            "small":    "Mechanical edits, renames, formatting, lookups",
            "mid":      "Multi-file changes following a known pattern",
            "frontier": "Architecture, debugging, ambiguous or novel specs"}}

def route(task):
    a = ask_jev({"task": task}, {"tier": TIER})["tier"]
    if a["confidence"] >= 0.75:
        return a["choice"]
    return "mid" if a["choice"] == "small" else "frontier"   # unsure -> go UP a tier

The rule in the last line: low confidence rounds up, never down. A wrong "small" costs a failed task; a wrong "frontier" costs a few cents.

Pattern 3: triage with a rubric that improves itself

A nightly job classifies new error-log patterns into a rubric (a versioned JSON file in your repo), with each class described by what it is, what it is not, examples, and an action:

{
  "classes": {
    "upstream_timeout": {
      "what": "A call to a third-party API exceeded its timeout",
      "not_for": "Timeouts talking to OUR database (use db_timeout)",
      "examples": ["ReadTimeout: api.payments.example 30s"],
      "action": "count"
    },
    "db_timeout": {"what": "...", "not_for": "...", "examples": ["..."], "action": "alert"},
    "unknown":    {"what": "Doesn't clearly fit any class above", "action": "escalate"}
  },
  "thresholds": {"auto": 0.8}
}

Each log pattern goes to the decision model as a Choice over the classes. Confident answers are acted on in code (count, alert, page). Anything below 0.8, or unknown, is escalated to an LLM agent, whose job is not to classify that one line but to propose a patch to the rubric (a new class, or a sharper not_for) as a pull request. The next night, the decision model handles that pattern itself.

That's the full System 1 / System 2 relationship: System 2 handles the novel cases and writes the rules; System 1 applies them at volume. In one published run of exactly this setup, 600 log lines collapsed to 7 patterns, classified for a fraction of a cent, with one escalation that cost more than all the decision calls combined. That ratio is the point.

Write rubrics, not prompts. "One question, one judgment" (combine answers in code, not in one compound question); options are a closed, defensible set including unknown; every option has a not_for that separates it from its nearest neighbour.

Calibration: what the probability is worth

All three patterns depend on one property: when the model says 0.9, it should be right about 90% of the time. That's calibration, and you must measure it on your data, because calibration that holds on public benchmarks can degrade on private, unfamiliar inputs. Early independent tests of Jev reported exactly that pattern: generally better calibrated than LLMs asked for confidence, but overconfident in places (a top bucket claiming ~99% that was right ~90% of the time), while still preserving ordering (higher stated confidence did mean more likely right).

This tool works on any model that outputs a probability: Jev, an LLM's token probabilities, or a classifier you trained in Part 3.

"""calibrate.py -- is a decision model's "0.9" really 90%? And where do the
thresholds go? Works on ANY model that returns a probability: Jev, an LLM's
logprobs, a logistic regression, a fine-tuned classifier.

Input: a labelled set of (probability_of_yes, true_label) pairs -- the
20-400 real cases you labelled by hand.
"""
import math
import random


def reliability(pairs, bins=5):
    """Bucket by stated probability; compare to how often 'yes' was true."""
    rows, ece = [], 0.0
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        bucket = [(p, y) for p, y in pairs
                  if lo <= p < hi or (b == bins - 1 and p == 1.0)]
        if not bucket:
            continue
        stated = sum(p for p, _ in bucket) / len(bucket)
        actual = sum(y for _, y in bucket) / len(bucket)
        ece += len(bucket) / len(pairs) * abs(stated - actual)
        rows.append((f"{lo:.1f}-{hi:.1f}", len(bucket), stated, actual))
    return rows, ece       # ECE = expected calibration error (0 = perfect)


def pick_thresholds(pairs, max_false_allow=0.01, max_false_block=0.03):
    """Three-way policy: p < low -> auto-allow, p >= high -> auto-block,
    in between -> escalate (to an LLM or a human).
    Pick the narrowest escalation band that meets both error budgets
    (each budget is a fraction of ALL traffic)."""
    best = None
    grid = [i / 100 for i in range(101)]
    for low in grid:
        false_allow = sum(y for p, y in pairs if p < low) / len(pairs)
        if false_allow > max_false_allow:
            break                          # letting too many 'yes' through
        for high in grid:
            if high < low:
                continue
            false_block = sum(1 - y for p, y in pairs if p >= high) / len(pairs)
            if false_block > max_false_block:
                continue                   # blocking too many harmless ones
            escalated = sum(low <= p < high for p, _ in pairs) / len(pairs)
            if best is None or escalated < best[2]:
                best = (low, high, escalated)
    return best


if __name__ == "__main__":
    random.seed(7)
    # Synthetic stand-in for 400 hand-labelled shell commands (1 = destructive).
    pairs = []
    for _ in range(400):
        y = 1 if random.random() < 0.2 else 0          # 20% really are destructive
        evidence = (3 if y else -3) + random.gauss(0, 2)
        stated = 1 / (1 + math.exp(-1.5 * evidence))   # 1.5x = overconfident
        pairs.append((stated, y))

    rows, ece = reliability(pairs)
    print(f"{'bucket':9s} {'n':>4s} {'stated':>7s} {'actual':>7s}")
    for name, n, stated, actual in rows:
        print(f"{name:9s} {n:4d} {stated:7.2f} {actual:7.2f}")
    print(f"ECE = {ece:.3f}")

    low, high, esc = pick_thresholds(pairs)
    print(f"auto-allow below {low:.2f}, auto-block at/above {high:.2f}, "
          f"escalate {esc:.0%} of traffic")

Output:

bucket       n  stated  actual
0.0-0.2    254    0.03    0.01
0.2-0.4     23    0.31    0.22
0.4-0.6     15    0.51    0.20
0.6-0.8     20    0.70    0.30
0.8-1.0     88    0.97    0.94
ECE = 0.054
auto-allow below 0.36, auto-block at/above 0.69, escalate 7% of traffic

How to read it: the extremes are trustworthy (0.03 stated, 0.01 actual; 0.97 stated, 0.94 actual), but the middle is overconfident: when the model says 0.70, only 30% of those commands were actually destructive. The threshold picker chooses the widest auto-allow and auto-block zones that stay within your error budgets (at most 1% of all traffic wrongly allowed, at most 3% wrongly blocked) and sends the remaining 7% to a slower, smarter path. 93% of decisions happen in a fraction of a second; the hard 7% get real thought. Re-run it whenever you change the model version, the question wording, or the kind of traffic.

Build your own System One (offline, free)

You can get the same kind of answer from any local LLM: run one forward pass and compare the probabilities of the tokens "Yes" and "No", without generating anything. This is how many teams built decision layers before dedicated models existed.

"""noul_local.py -- a home-made System One decision: P(yes) from a small local
model's next-token probabilities. One forward pass, no text generation.
pip install torch transformers   (the model is ~1 GB, downloaded once)
"""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"         # any small instruct model works
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL).eval()
YES = tok.encode("Yes", add_special_tokens=False)[0]
NO = tok.encode("No", add_special_tokens=False)[0]


def noul(question, state):
    """Return P(yes) for a yes/no question about `state`."""
    messages = [
        {"role": "system", "content": "Answer with exactly one word: Yes or No."},
        {"role": "user", "content": f"{question}\n\nInput:\n{state}"},
    ]
    ids = tok.apply_chat_template(messages, add_generation_prompt=True,
                                  return_tensors="pt")
    with torch.no_grad():
        logits = model(ids).logits[0, -1]       # scores for the NEXT token only
    p_yes, p_no = torch.softmax(logits[[YES, NO]], dim=0)   # renormalise over 2
    return p_yes.item()


if __name__ == "__main__":
    q = ("Would running this shell command delete data, rewrite shared history, "
         "or change production state in a way that is hard to undo?")
    for cmd in ["ls -la", "git status", "rm -rf ./dist", "git push --force origin main",
                "psql prod -c 'DROP TABLE users'", "cat README.md"]:
        print(f"{noul(q, cmd):.2f}  {cmd}")

Real output from this 0.5B-parameter model:

0.60  ls -la
0.60  git status
0.67  rm -rf ./dist
0.88  git push --force origin main
0.74  psql prod -c 'DROP TABLE users'
0.63  cat README.md

Look at that honestly. The ordering is partly right (git push --force scores highest), but the numbers are badly biased: ls -la at 0.60 is absurd, and DROP TABLE scoring below git push --force is wrong. A tiny general model leans towards "Yes" and doesn't separate the classes well. That's exactly why you (1) calibrate before trusting any threshold, (2) use a larger model or fine-tune a small one on a few hundred labelled examples (Chapter 42), or (3) use a model trained for calibrated decisions. Run calibrate.py on this model's scores against 50 labelled commands and you'll see it immediately.

Limitations: what decision models can't do

Collected from the vendor's own documentation and early independent testing:

When to reach for a System One model