Manchego

Manchego evaluates options supplied by your software. A single forward pass reads the whole question, then a softmax over the offered option-code logits returns their probabilities. Choice, yes/no conditions (noul) and ordered scores share that readout.

V3 starts again from the untrained Qwen3.5-4B base. It uses a fresh rank-16 LoRA, the SemIf direct-options prompt, harder computed families, answer judging, more public corpora and self-distilled replay. It is not continued training from V2.1.

SemIf supplies a system message and one JSON user message with evidence, criterion and lettered options. The trained path covers 2–16 options with a nonempty state. The server reports which contract each question received.

The public JevBench and Public8 comparisons resolve improvements over V2.1. Held-out task-type and sealed unseen-task intervals still cross zero, and the sealed computed-family comparison resolves a regression.

These are the project’s own matched-runtime evaluations. The public JevBench accuracy is not an official JevBench composite score. V2.1’s historical official score of 68.8 belongs to that earlier version and evaluation regime.

The model card documents prompt changes, grouped resampling, runtime, earlier task exposure and merged-weight numerics. Read the published evaluation before comparing these rows with another leaderboard.

V3 improves the public comparisons, but it is overconfident on unfamiliar task definitions. On the 27 held-out tasks, mean confidence is 0.829 against accuracy of 0.709; expected calibration error is 0.129, versus 0.083 for V2.1.

The packaged bf16 serving map uses temperatures of 1.5 for choice, 0.2 for yes/no, and 1.0 for score. The yes/no setting deliberately sharpens uncertain answers for the published scoring rule; it is not a general calibration guarantee. A positive temperature does not change the winning option.

Every evaluation number reproduced here uses temperature 1. Set --temperature-map off to request that policy, and inspect the temperatures and contract reported with each response. Consult the pinned format card before assuming a map applies to different weight bytes.

The task rows come from families V3 trains on. The text check uses development rows from earlier corpora: three also appear in V3’s training data, and 50 come from banking77, which V3 did not train on. Read these as conversion checks, alongside the public and sealed evaluations on the main card.

The merged bf16 model is 9.3 GB. The MLX builds are text-only Apple-silicon conversions: 4.5 GB at 8 bits and 2.4 GB at 4 bits. Four-bit results are lower on both published conversion checks; choose it when the memory budget requires it.

Use manchego-serve 0.2.0 with contract semif. The examples below explicitly disable serving temperature maps to match the evaluation policy. Setup downloads and verifies weights; requests then run offline.

Read answers.route.probabilities for the option distribution. Responses report per-question contracts, temperatures and weight verification. The schema demonstration is not a claim that a model runs on this website.

CUDA graph speedups are platform-specific. The release keeps graphs off by default: the published Linux A10 test was slower with them, while the Windows RTX 5090 test was faster. Neither result is a universal latency promise.

Training combines computed decision families, answer judging, original public annotations, domain generators, calibration rows, and replay toward the untrained base’s probabilities. Self-distilled replay contributes 15% of the estimated prompt tokens.

Some upstream datasets include model-generated text or labels. The published card describes those sources, the inherited phrase audit, development-informed design, the MathDial reproduction probe, and the account holder’s release decisions. These disclosures are part of the release, not an assertion that every source is interchangeable.

The base is Qwen3.5-4B; the prompt format comes from SemIf (MIT). Full training-data attributions and declarations are in the published NOTICE. Manchego is independent of TypeSafe AI; using the System One interface does not imply endorsement.

V3 is the current public release. V2.1 remains available with its original prompt and evidence. Earlier research iterations are historical; do not assume their tags are still distributed.