Decision model · v2.1
Manchego · MLX 8-bit
The whole decision, on a laptop.
Manchego v2.1 for Apple silicon. 8-bit MLX, 4.5 GB, within one decision of the bf16 weights on the format test.
- Runtime
- MLX · mlx-lm ≥ 0.31.3
- Precision
- 8-bit, group size 64
- Size
- 4.5 GB
- Speed
- About 70 ms per short question, M3 Max
- License
- Apache-2.0
- Updated
- 23 Sep 2026
Overview
Manchego v2.1 converted with mlx_lm at 8 bits for Apple silicon. It keeps the main model’s interface: state, question, and options in; a probability for every option out, in one forward pass.
On the format test it is within one decision of the bf16 weights on 80 held-out task rows, one better on 200 real-text rows, and within 0.003 in cross-entropy. The manchego-serve reference answers come from this build.
Fig. 01 · Formats
Size against fidelity
Accuracy on the same held-out rows, one prompt per forward pass. The tick on each track is the untrained base.
View as table
| Format | Size | 80 task rows (CE) | 200 real-text rows (CE) |
|---|---|---|---|
| bf16 · Transformers | 9.3 GB | 0.950 (0.177) | 0.905 (0.280) |
| MLX 8-bit | 4.5 GB | 0.938 (0.180) | 0.910 (0.279) |
| MLX 4-bit | 2.4 GB | 0.875 (0.288) | 0.900 (0.290) |
| Untrained base, bf16 | — | 0.650 | 0.690 |
Usage
from mlx_lm import load
import mlx.core as mx
model, tok = load("oraculumai/Manchego-MLX-8bit")
prompt = "Is this message spam?\n\nState:\nWIN a free cruise! Reply YES now.\n\nOptions:\nA = yes\nB = no\n\nReply with only the letter of the best option."
letters = "AB" # the letters you offered
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
add_generation_prompt=True, enable_thinking=False)
logits = model(mx.array([tok.encode(text, add_special_tokens=False)]))[0, -1]
cand = [tok.encode(L, add_special_tokens=False)[0] for L in letters]
probs = mx.softmax(logits[mx.array(cand)].astype(mx.float32))
print(dict(zip(letters, probs.tolist())))- Text-only conversion. The vision tower and the multi-token-prediction head are dropped, so load it with
mlx_lm, notmlx-vlm. - One prompt at a time. Scoring several prompts as one padded batch moves probabilities slightly at this precision: up to 0.09 on one item in 164, with one changed decision.
- Up to 26 options with the short prompt. For larger menus or untrusted state, use the state-first prompt from
contract_v2.pyin the main repository.
Read the main card first
What the model is, how it was trained, every result and every weakness are on the main Manchego card. v2.1 repairs v2’s overconfidence on unfamiliar task definitions, but it is not more accurate than v2, and on the sealed set of unseen tasks it is not more accurate than the untrained base.
Disclosures
- JevBench-informed. The training families of v2 and v2.1 were designed from an earlier version’s per-family JevBench hard-tier scores (the published specification only, no item text). Read its JevBench result with that in mind.
- Phrase audit. An early audit used Jev, a hosted decision service, to flag phrases in two data pools; 86 phrases were dropped. Jev never produced a training label.
About this page
Summarized from the public model card and its published files. Repository updated 23 Sep 2026; reviewed 26 Sep 2026. The Hugging Face card is the source of truth for licenses and current details.
Upstream: Oraculum / Manchego