Decision model · v2.1
Manchego
A decision is a distribution.
A 4B model for typed decisions. State, question, and options in; a probability for every option out, in one forward pass.
- Parameters
- 4B
- Weights
- bf16 · 9.3 GB
- Options
- 2 to 255
- Passes
- One per question
- License
- Apache-2.0
- Tagged
- v2.1 · 21 Sep 2026
Overview
Manchego answers a question about a state by scoring the options your software supplies. It never writes text. Probabilities are read from the logits of the option codes at the last prompt position, so one forward pass returns the full distribution.
Three question types share that readout. Choice picks one of up to 255 options. Noul gives the probability that a yes/no condition holds. Score returns a distribution over ordered levels; its expected value is the answer.
v2.1 is a calibration release. A short second stage on human-written task definitions removed v2’s overconfidence on unfamiliar tasks: calibration error on a sealed set of 25 unseen tasks fell from 0.153 to 0.040. Accuracy did not change, and the card says so.
Fig. 01 · Recorded response
Three questions about one state
Each question is its own prompt and its own forward pass. The distribution is the answer.
What the model reads
Which team should handle this message?
State:
Customer: the blender I bought last week smells of burning and stopped working. Order 5521.
Options:
A = returns — refunds, exchanges and defective items
B = shipping — delivery status and lost parcels
C = billing
Reply with only the letter of the best option.What it returns
choiceconfidence 0.998
What the model reads
Does the customer report a defective product?
State:
Customer: the blender I bought last week smells of burning and stopped working. Order 5521.
Options:
A = yes — the condition holds
B = no — the condition does not hold
Reply with only the letter of the best option.What it returns
noulP(yes) 0.992
What the model reads
How urgent is this message?
State:
Customer: the blender I bought last week smells of burning and stopped working. Order 5521.
Options:
A = level 0 — routine
B = level 1 — soon
C = level 2 — immediately
Reply with only the letter of the best option.What it returns
scoreexpected level 1.34 · confidence 0.18
Interface
Two prompt contracts, one readout. The model was trained on both. Use the renderings exactly: the short prompt is three to seven points more accurate on JevBench’s short items, and the state-first prompt is the only one for large menus.
| Contract | Options | Codes | Use it when |
|---|---|---|---|
| Short prompt | 2 to 26 | A–Z | You have at most 26 options. |
| State-first prompt | 2 to 255 | A–Z up to 26; beyond that, two-letter codes (AA, AB, …), each one token | The menu is large, or the state is untrusted text. The state is fenced by markers that carry a nonce, so it cannot close its own fence. |
- Readout
- Softmax over the offered option codes at the last prompt position
- Temperature
- 1.0, no fitted calibration
- Confidence
- (K · max p − 1) / (K − 1), K = number of options
- Chat template
- Thinking disabled, generation prompt added
- Context
- Trained on prompts up to about 9,000 tokens
- Language
- English
<question>
State:
<state text, or JSON pretty-printed>
Options:
A = <value> — <description>
B = <value> — <description>
Reply with only the letter of the best option.import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("oraculumai/Manchego", revision="v2.1")
model = AutoModelForCausalLM.from_pretrained("oraculumai/Manchego", revision="v2.1",
dtype=torch.bfloat16, device_map="auto").eval()
def decide(question, state, options): # options: {value: description or None}, at most 26
L = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"[: len(options)]
menu = "\n".join(f"{c} = {v} — {d}" if d else f"{c} = {v}" for c, (v, d) in zip(L, options.items()))
user = f"{question}\n\nState:\n{state}\n\nOptions:\n{menu}\n\nReply with only the letter of the best option."
text = tok.apply_chat_template([{"role": "user", "content": user}], tokenize=False,
add_generation_prompt=True, enable_thinking=False)
ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.no_grad():
logits = model(**ids).logits[0, -1].float()
probs = torch.softmax(logits[[tok.encode(c, add_special_tokens=False)[0] for c in L]], 0)
return dict(zip(options, probs.tolist()))
print(decide("Is this message spam?", "WIN a free cruise! Reply YES now.", {"yes": None, "no": None}))Evaluation
Every trained open ~4B decision model that could be run as shipped (28 in all, counting v2.1) was scored on identical rows. Each other model ran through its own pinned code and prompt on a disposable cloud A100; Manchego and the untrained base were scored on an RTX 5090. Refusals count as wrong. Intervals are paired 95% bootstraps. No open ~4B model is ahead on all three benchmarks.
Fig. 02 · Accuracy
Where v2.1 stands in the open ~4B field
Every trained open ~4B decision model that could be run as shipped, on identical rows. Refusals count as wrong.
JevBench v1.2
231 public decisions
0.8053= of 28
Best other: alibiserikbay/JevK5, 0.861. One model is resolved ahead.
Held-out task types
27 Natural Instructions tasks, source-clean for Manchego · 1,561 rows
0.7015 of 28
Best other: Mapika/decider-4b, 0.717. None is resolved ahead.
Public8
1,600 rows from eight public real-text datasets
0.73215 of 28
Best other: Mapika/decider-4b, 0.800. 7 models are resolved ahead; each trained on, or cannot be cleared of, Public8’s source data.
View as table
| Model | JevBench v1.2 | Held-out task types | Public8 |
|---|---|---|---|
| alibiserikbay/JevK5 | 0.861 | 0.641 | 0.733 |
| HopitAI/hopper | 0.827 | 0.698 | 0.740 |
| Manchego v2.1 | 0.805 | 0.701 | 0.732 |
| abhishek085/spark-s1-4b-v6 | 0.805 | 0.683 | 0.728 |
| Praveenrajus/jevify-qwen3.5-4b-t2 | 0.797 | 0.687 | 0.729 |
| kshetrajna12/reflex-qwen3.5-4b-lora | 0.788 | 0.696 | 0.711 |
| wayfind/metask-jev-4b-policy-mix | 0.788 | 0.703 | 0.739 |
| AstroHan/decision-head-qwen3.5-4b-rlcd-32k | 0.779 | 0.677 | 0.723 |
| apus-ailab/APUS-OpenJev-v1-4B | 0.779 | 0.582 | 0.750 |
| Qwen3.5-4B (untrained base) | 0.775 | 0.678 | 0.686 |
| Mapika/decider-4b | 0.766 | 0.717 | 0.800 |
| jbrashear/jebadiah-4b-v0 | 0.749 | 0.695 | 0.733 |
| llm-semantic-router/Decision-1.0-Nox-4B | 0.749 | 0.657 | 0.709 |
| ThakiCloud/kd-4b-ko-v0 | 0.745 | 0.704 | 0.763 |
| n4ze3m/Qwen3.5-4B-Hmm | 0.745 | 0.694 | 0.771 |
| jaredpalmer/kev-4b | 0.723 | 0.688 | 0.762 |
| Berk/assay-4b | 0.719 | 0.675 | 0.764 |
| marcmagn1/kev-4b-typed-v1 | 0.719 | 0.699 | 0.756 |
| AmeenAhmed2/zico | 0.710 | 0.706 | 0.786 |
| pngwn/system-one-qwen3.5-4b-scorer-v2b | 0.697 | 0.553 | 0.717 |
| yah01/vjev-vision | 0.688 | 0.668 | 0.738 |
| szybkie-ai/jeb-4b | 0.684 | 0.684 | 0.750 |
| GestaltLabs/Jeff-1 | 0.680 | 0.645 | 0.729 |
| OzLabs/typical-medium | 0.662 | 0.593 | 0.670 |
| nullsilver/alpha-sys-1-3B | 0.662 | 0.600 | 0.637 |
| AlexWortega/openjev | 0.632 | 0.586 | 0.666 |
| Okura66/Kahn1-Qwen2.5-3B | 0.597 | 0.632 | 0.669 |
| Edoigtrd/Mirave-4.2B-xlm-roberta-xl | 0.446 | 0.354 | 0.391 |
| cua-ai/cua-s1-4b-0.1 | 0.368 | 0.380 | 0.402 |
Fig. 03 · Held-out tests
What training bought
Accuracy of the untrained base and Manchego v2.1 on held-out test sets. The largest gains are on the project’s own decision families; the smallest are on public real text.
Large menus (27–255 options)300 rows
1.000+0.590
Lookup chains1,003 rows
0.955+0.472
Governed decisions1,003 rows
0.845+0.396
Temporal reasoning1,000 rows
0.832+0.389
Governed, held-out domain1,000 rows
0.703+0.389
Adequacy1,000 rows
0.846+0.344
Prompt-injection probe2,794 rows
0.906+0.251
Real text, human labels9,054 rows
0.819+0.134
Score levels800 rows
0.976+0.123
Public8 real text1,600 rows
0.732+0.046
View as table
| Test set | Rows | Untrained base | Manchego v2.1 |
|---|---|---|---|
| Large menus (27–255 options) | 300 | 0.410 | 1.000 |
| Lookup chains | 1,003 | 0.483 | 0.955 |
| Governed decisions | 1,003 | 0.449 | 0.845 |
| Temporal reasoning | 1,000 | 0.443 | 0.832 |
| Governed, held-out domain | 1,000 | 0.314 | 0.703 |
| Adequacy | 1,000 | 0.502 | 0.846 |
| Prompt-injection probe | 2,794 | 0.655 | 0.906 |
| Real text, human labels | 9,054 | 0.685 | 0.819 |
| Score levels | 800 | 0.853 | 0.976 |
| Public8 real text | 1,600 | 0.686 | 0.732 |
Fig. 04 · Calibration
Overconfidence, repaired where it mattered
Expected calibration error, lower is better. v2.1 improves on every set except held-out real text, where v2 was already well calibrated.
Sealed set, 25 unseen tasks
0.040−0.113
Public8 real text
0.053−0.093
JevBench, 231 public
0.072−0.010
Held-out real text
0.042+0.026
View as table
| Set | v2 | v2.1 | Null (95th pct.) |
|---|---|---|---|
| Sealed set, 25 unseen tasks | 0.153 | 0.040 | — |
| Public8 real text | 0.146 | 0.053 | 0.033 |
| JevBench, 231 public | 0.082 | 0.072 | 0.070 |
| Held-out real text | 0.016 | 0.042 | 0.013 |
| Accuracy | Label NLL | Brier | ECE | |
|---|---|---|---|---|
| Untrained base | 0.723 | 0.576 | 0.380 | 0.040 |
| Manchego v2 | 0.684 | 0.774 | 0.456 | 0.153 |
| v2, continued without the new rows | 0.689 | 0.747 | 0.448 | 0.137 |
| Manchego v2.1 | 0.695 | 0.599 | 0.399 | 0.040 |
Read the sealed set honestly: v2.1 returns the probabilities to the untrained base’s level. It is not a gain over the base in accuracy (−0.027 on unrounded scores; the interval includes zero). On 27 source-clean held-out Natural Instructions tasks, v2.1 is +0.024 above the base on unrounded scores, a gain from the first training stage.
| Easy | Standard | Hard | All 231 | Label NLL | ECE | |
|---|---|---|---|---|---|---|
| Untrained Qwen3.5-4B | 1.000 | 0.931 | 0.577 | 0.775 | 0.493 | 0.062 |
| Manchego v2 | 1.000 | 0.972 | 0.595 | 0.797 | 0.480 | 0.082 |
| Manchego v2.1 | 1.000 | 0.972 | 0.613 | 0.805 | 0.423 | 0.072 |
| v2.1, two prompts averaged | 1.000 | 0.958 | 0.631 | 0.810 | 0.445 | 0.054 |
Formats
Every conversion is its own numeric series, measured on the same 80 held-out task rows and 200 human-labeled real-text rows. Eight bits keeps the bf16 decisions within one item. Four bits costs six of the 80 task rows.
Fig. 05 · Formats
Size against fidelity
Accuracy on the same held-out rows, one prompt per forward pass. The tick on each track is the untrained base.
View as table
| Format | Size | 80 task rows (CE) | 200 real-text rows (CE) |
|---|---|---|---|
| bf16 · Transformers | 9.3 GB | 0.950 (0.177) | 0.905 (0.280) |
| MLX 8-bit | 4.5 GB | 0.938 (0.180) | 0.910 (0.279) |
| MLX 4-bit | 2.4 GB | 0.875 (0.288) | 0.900 (0.290) |
| Untrained base, bf16 | — | 0.650 | 0.690 |
Serving it
manchego-serve serves exactly the policy behind the card’s numbers: automatic contract selection, one pass per question, temperature 1.0, and the client’s option order. Weights are fetched once, pinned by commit, and checked against the published SHA-256 of every weight file and of the chat template, tokenizer and config files. After that the server makes no outbound connections, and every response reports whether the bytes it loaded are the published ones.
Fig. 06 · Measured by the author
One question, about 80 milliseconds
81 ms
Median per question
NVIDIA A10, CUDA image; p95 82 ms. Serial, reference linear-attention kernels
≈70 ms
Per short question
Apple M3 Max, MLX 8-bit
2.0 s
A 7,511-token request
NVIDIA A10
13.5 GB
Peak GPU memory
NVIDIA A10, bf16
docker build -t manchego-serve:2.1-cuda . # downloads and verifies the weights (9.3 GB)
docker run --rm --gpus all -p 127.0.0.1:8000:8000 manchego-serve:2.1-cuda
curl -s localhost:8000/healthzRead before relying on it
- For typed decisions over options your software supplies: routing, policy checks, triage, graded judgments, where you read the probabilities.
- Not for unreviewed high-stakes decisions (medical, legal, financial, safety) without a person in the loop, open-ended text generation, or images (not measured).
- Limits: up to 26 options with the short prompt and 255 with the state-first prompt; trained on prompts up to about 9,000 tokens (longer inputs are untested); English.
Disclosures
- JevBench-informed. The training families of v2 and v2.1 were designed from an earlier version’s per-family JevBench hard-tier scores (the published specification only, no item text). Read its JevBench result with that in mind.
- Phrase audit. An early audit used Jev, a hosted decision service, to flag phrases in two data pools; 86 phrases were dropped. Jev never produced a training label.
- Selection. v2’s registered checkpoint selection read one development group whose targets came from Jev; a rule that excludes that group selects the same checkpoint.
- Labels. Every training label was computed by code or is a public dataset’s original human annotation. No model wrote a label.
Nine known weaknesses
01No measured gain in accuracy over v2
v2.1 is a better probability model than v2, not a more accurate one. On unseen task definitions the untrained base is two to three points more accurate (interval through zero).
02Level on JevBench’s hard tier
The hard tier is where a decision model should matter: 0.61 against the base’s 0.58. Long policy documents, multi-hop lookups and judging between drafts are the weak families.
03A cost on policy documents
One to two points on the project’s own policy-document test; a same-seed control shows part of it is the second stage itself.
04Arithmetic in one pass
Date and number arithmetic is unreliable outside its own templates. Do the arithmetic in code and ask the model the judgment.
05Prompt sensitivity
The short prompt is better on short items, the state-first prompt on others. Averaging the two is the best single readout measured.
06Serving numerics
Scoring several prompts in one padded batch moves probabilities slightly at bf16 or 8 bits. Score one prompt at a time when reproducibility matters.
07Adversarial input, partly measured
The state-first prompt fences the state; the prompt-injection probe (0.906) is the project’s own, and no adaptive attack was run.
08Seeds
Two seeds for the second stage, one for the first. The released weights are seed 0; seed 1 agrees on every reading on the card.
09A 4B model
Its world knowledge is the base model’s. It will be literal, and wrong on questions that need facts it does not have.
Provenance
Fig. 07 · Lineage
From base model to every format
- Qwen/Qwen3.5-4BUpstream · Apache-2.0
- LoRA fine-tune, two stagesOraculum · 491,520 + 24,576 rows
| Stage | Rows | Updates | What the labels are |
|---|---|---|---|
| One (Manchego v2) | 491,520 | 30,720 × 16 rows | Computed decision families (governed decisions, lookup chains, temporal, fault environments with exact posteriors, large menus, and more) plus 78,884 human-labeled real-text rows |
| Two (v2.1) | 24,576 | 1,536 × 16 rows | 7,372 rows from 101 human-written task definitions in Super-NaturalInstructions (42 upstream datasets, permissive licenses), with their original labels; the rest a stratified replay of stage one |
- Base
- Qwen/Qwen3.5-4B @ 851bf6e8, hybrid Gated DeltaNet and attention
- Adapter
- LoRA, rank 32, scale 20, last 16 layers; 32.5M trainable parameters, merged
- Stage two
- One A100-40GB, 1.2 hours; the final update, nothing selected inside the run
- Memorization
- Continues training text verbatim for 0.22–0.34 tokens on average, less than the untrained base
Versions
Fig. 08 · Release history
Three versions in three days, then the fine print
v1
First release
The first public release of the decision model, still available at the tag v1.
v2
Trained from the base
491,520 rows with computed labels and original human annotations.
v2.1
Calibration release
A second stage on human-written task definitions repairs v2’s overconfidence on unfamiliar tasks.
Cards
The whole field, and the fine print
A comparison across 28 open ~4B models, counting v2.1; the JevBench-informed and phrase-audit disclosures.
Attribution
Base model: Qwen3.5-4B (Apache-2.0, Alibaba Cloud). Real-text training corpora, each under its own license with its human labels: WANLI (CC BY 4.0), MultiNLI (OANC, with named CC BY and CC BY-SA works), the Bitext customer-support dataset (CDLA-Sharing-1.0, rows not redistributed), and banking77 (CC BY 4.0). Stage-two task definitions come from Super-NaturalInstructions; every upstream dataset is listed with its license in NATURAL_TASKS_ATTRIBUTION.md. Doom states were recorded from ViZDoom (MIT) with Freedoom assets (BSD-3).
The interface follows TypeSafe AI’s System One contract. Manchego is an independent project, not affiliated with or endorsed by TypeSafe AI.
About this page
Summarized from the public model card and its published files. Tagged release 21 Sep 2026; reviewed 26 Sep 2026. The Hugging Face card is the source of truth for licenses and current details.
Upstream: Qwen / Qwen3.5-4B