Test Qwen, GLM, DeepSeek, Kimi, MiniMax, or another model against the work you actually do. Keep the assignment, sources, reasoning setting, timeout, and acceptance check fixed, then read the machine result, raw receipts, retry history, provider cost, and human repair time together.
July 25, 2026 Last verifiedOne bounded job Start withAccepted results Compare cost by
01Compare one bounded job under conditions you can explain.Start with one baseline, one candidate, one acceptance standard, and a clear name for the kind of comparison you are running.
Open sectionSection open
What you will have at the end
You will have one bounded job; one baseline and one candidate model; the same assignment, sources, reasoning setting, timeout, and acceptance check for both; a machine-readable result from each model; proof of which model and harness actually ran; first-try and retry results kept separate; and a place to record provider cost and human repair time.
The companion article explains what work to test, what failure means, what belongs on a frontier model, and what counts as a cheaper accepted result. This guide gives you the terminal setup: a clean Ringer install, pinned endpoints, an executed check, raw attempt history, and a score sheet you can keep using.
Read the Ringer guide first if the harness is new to you
This guide assumes you understand Ringer's basic loop: a worker produces an artifact, Ringer executes your check, and exit code zero is the only PASS. The main guide covers installation, manifests, worker lanes, retries, Ringside, and the model scoreboard.
Name the test you are actually running
Model-only comparison — the harness, gateway, tools, prompt, reasoning setting, timeout, and check stay fixed; only the model ID changes. This answers which model performs better under mostly fixed system conditions.
Operational-lane comparison — the job, sources, timeout, and acceptance standard stay fixed; the model, harness, and possibly the access plan change. This answers which setup should do this job in your real workflow.
The starter manifest is a model-only comparison. Both candidates run through OpenCode and OpenRouter. One can be an American frontier baseline and the other a Chinese-model candidate. Only the task-level model ID changes.
This does not isolate the trained model as cleanly as a laboratory would. OpenRouter may serve the two models through different underlying providers, with different throughput or quantization. Record the route and host when they are exposed, and keep provider failures separate from model failures.
If your real baseline is Codex CLI on a ChatGPT plan and your candidate is Qwen through OpenCode and OpenRouter, run that too. Call it an operational-lane comparison: you are testing the model plus the harness and access path, which may be exactly what the customer cares about.
A task name containing qwen, deepseek, or kimi is not proof. A family name is not an endpoint. The task-level model field, resolved run metadata, and any inner product's own session metadata are the receipts.
Keep the first run serial and keep credentials out of the work
Ringer needs macOS or Linux, or Windows through WSL, plus Python 3.11 or newer. It also needs at least one authenticated worker CLI.
Keep credentials out of prompts, manifests, source packets, and chat. Complete each tool's own sign-in. No agent needs to read the key.
Start the first comparison with max_parallel set to 1. Ringer's field notes include OpenCode database-lock failures at higher parallelism and, in one case, with only two simultaneous workers. Parallelism is useful after the lane is clean. It is an extra variable during the first smoke test.
The kit contains a README, the already-named check_result.py validator, a model-only Ringer manifest, an accepted-result score sheet, sample claims, two sample source files, and known-good and known-bad fixtures. You do not need to rename the validator.
The sample worker must classify every claim as supported, unsupported, or unclear. A supported claim needs an exact passage from an allowed source. The Python check verifies that every requested claim appears once; no unknown claim appears; every row has the required fields; every status is allowed; every supported claim names a real source; every quoted passage appears in that source; unsupported rows do not smuggle in invented citations; and unclear rows explain what the sources do not establish.
The check does not decide whether every unsupported or unclear judgment is intellectually correct. No generic script can do that for an arbitrary research question. The blind review still matters. For a production replay pack, add known-answer tests where the answer is genuinely deterministic, then keep a human grade for cases that require judgment.
The five-claim task is small enough to understand and strict enough to catch plausible-looking nonsense. Replace it with your own recurring work before you draw a business conclusion.
Browse the files before you replace the sample. The validator uses only the Python standard library, runs locally, and makes no network calls.
Freeze the harness for the comparison
This guide was checked on July 25, 2026 against Ringer commit 4ac3791e00276106be962d72ca2b55f4802d8fa2. Ringer can update itself, so record the commit you use and freeze it for the duration of the comparison.
02Install Ringer and prove the shared OpenCode lane.Prove the base harness before you configure the two endpoints that will use it.
Open sectionSection open
Step 1 — Install Ringer and prove the base setup
Check Python first. If it is older than 3.11, install a current Python before continuing.
Codex CLI is Ringer's built-in worker lane and runs the demo. The base setup is good when all three demo workers report PASS, the verdict table prints, and Ringside opens. Use the official quickstart in the pinned Ringer README as the source of truth if the installation changes.
The agent integration is optional. Before a controlled comparison, update once and record the exact commit in your score sheet or run notes. Every later evaluation command uses --no-self-update so Ringer does not change between the baseline and candidate.
You do
Complete the Codex sign-in yourself and save the recorded Ringer commit with the evaluation.
The AI does
Checks Python, installs the worker CLI if needed, clones and configures Ringer, runs the demo, and reports the commit that will stay fixed.
Install OpenCode with one official method, then run its own sign-in and choose OpenRouter. Paste the API key into OpenCode's sign-in when it asks. Do not put the key in swarm.json.
On macOS, enable the OpenCode engine in ~/.config/ringer/config.toml and replace the bin value with the absolute path inside your Ringer clone. Leave model_default out for a bakeoff so a missing task-level model stops the run instead of quietly sending both cells to a default.
OpenCode's --dangerously-skip-permissions flag suppresses its prompts. It is not the macOS sandbox. Ringer's wrapper uses Seatbelt to allow reads and network access while confining writes to the task directory, a temporary scratch directory, and OpenCode's state folders.
On Linux or WSL, the supplied Seatbelt wrapper does not work. Set bin to the absolute path returned by command -v opencode. That gives you the model lane without the same operating-system write confinement, so keep the assignment and writable surface narrow. No ordinary bakeoff cell needs full_access.
You do
Choose OpenRouter during sign-in and paste the API key into OpenCode's own prompt.
The AI does
Installs OpenCode, confirms the binary and version, enables the Ringer engine, and verifies the wrapper or binary path.
03Pin exact model IDs, then make the checker reject bad work.Prove model identity and checker behavior before you spend tokens on the comparison.
Open sectionSection open
Step 3 — Refresh the catalog and pin the models
Force the first OpenRouter catalog download, then find the exact two IDs you want. If the catalog reports z-ai/glm-5.2, the OpenCode manifest value is openrouter/z-ai/glm-5.2. Do the same for Qwen, DeepSeek, Kimi, MiniMax, or the frontier baseline.
Do not copy a model ID from this guide and assume it is still the model you mean. Catalogs, aliases, prices, and access routes move.
Record the exact model ID, lab, harness, provider or plan, reasoning setting, test date, and Ringer commit. Ringer keeps those fields separate. Grok Build, for example, is a harness, not a model.
Use the model identity rules in Ringer's pinned docs when a harness, provider route, and trained model are easy to conflate.
Step 4 — Prove the checker before paying a model
Go to the included kit and run the known-good fixture. It must print the expected PASS. Then run the known-bad fixture. That command must fail and name the invented quotation, missing source, invalid status, and omitted claim.
A check that accepts the bad fixture will flatter both models. A check that rejects the good fixture will create phantom failures and waste the retry.
Ringer gives a check 60 seconds. Keep it deterministic and fast. If it needs a remote model to decide whether the work is good, you have created another model evaluation, not a clean acceptance check.
Show the full prompt
<prompt>
<task>
cd '/ABSOLUTE/PATH/TO/ringer-bakeoff-kit'
python3 check_result.py \
fixtures/good-result.json \
--claims source-packet/claims.json \
--sources source-packet/sources
PASS: every requested claim is present, the schema is valid, and all cited evidence appears in an allowed source
python3 check_result.py \
fixtures/bad-result.json \
--claims source-packet/claims.json \
--sources source-packet/sources
</task>
</prompt>
04Build one clean manifest and preflight it four ways.Give every run a fresh directory, understand every field, and read each preflight stage for the specific failure it can reveal.
Open sectionSection open
Step 5 — Make a run-specific manifest
Copy the example instead of editing the source file. In swarm.json, replace every /ABSOLUTE/PATH/TO/ringer-bakeoff-kit, both current model-ID placeholders, and the workdir.
Use a new work directory for every real run. Change the date or suffix on the next run. Normal scratch-directory runs do not clear old task output, so reusing a directory can let a stale result.json contaminate the next check.
The starter keeps max_parallel at 1. That makes the first comparison slower and cleaner. Once the OpenCode lane has run without database locks, try two workers. If the harness starts failing under concurrency, record that as a harness result instead of turning it into a model-quality claim.
Step 6 — Understand what every manifest field is doing
key — names the cell and becomes its task-directory name. It is a label, not model proof.
spec — gives the worker the complete assignment. It should be identical across models for the same case.
check — executes the Python validator. Exit zero is the only PASS Ringer accepts.
expect_files — requires a nonempty result.json after the worker and check finish.
engine — selects OpenCode for both cells in the model-only example.
model — pins the exact model for that cell. This is where the comparison changes.
engine_args — fixes OpenCode's reasoning variant. Keep it the same unless effort is part of the test.
task_type — puts the result in the research evidence bucket used by the model scoreboard.
timeout_s — gives both workers the same 15-minute limit.
verified — says exactly what the machine check proved, without pretending it judged every research decision.
The task descriptions are duplicated on purpose. If you improve the candidate prompt after watching it struggle, make the same change for the baseline and rerun both.
The sample uses task_type research because its evidence should inform future research routing. For code fixes, document reviews, or format conversion, use the real job shape consistently. The official bakeoff template uses bakeoff when you want a separate experiment bucket. Pick one deliberately.
Workers start in the path shown below. That is why the source packet and validator use absolute paths. Telling a worker to read the source packet does not work if it has no idea where the packet is.
<workdir>/<task.key>/
Step 7 — Run four preflight stages
Lint looks for checks that cannot fail, silent checks, underspecified work, path collisions, and worktree mistakes. Fix anything that makes the result harder to trust. A larger controlled matrix with max_parallel 1 may produce a serial-fan-out warning; note that as an intentional choice for the first run.
Baseline runs checks without spawning workers or writing evaluation rows. In this starter, both cells should fail because result.json does not exist. If either passes, stop and look for an old file or a check that proves nothing. A baseline can also contain assertions about existing behavior, and those should pass. The rule is not that every row fails: new-output assertions should fail, while unchanged-behavior assertions should pass. The command itself exits zero even when its table contains expected failures, so read the table.
Dry run prints resolved paths and worker commands without spending model tokens. It does not fully prove every engine binary is available, which is why you already confirmed OpenCode and checked the wrapper.
The real run spends tokens. Ringer executes the check, retries a FAIL or TIMEOUT once with the failure output, and preserves both attempts. A setup ERROR does not get a retry.
Real-run exit code 0 means every task passed. Exit code 1 means at least one task failed. Exit code 2 means a manifest, configuration, or setup error. Exit code 130 means the run was interrupted.
Show the full prompt
<prompt>
<task>
./ringer.py --no-self-update lint '/ABSOLUTE/PATH/TO/swarm.json'
./ringer.py --no-self-update run \
'/ABSOLUTE/PATH/TO/swarm.json' \
--baseline
./ringer.py --no-self-update run \
'/ABSOLUTE/PATH/TO/swarm.json' \
--dry-run
./ringer.py --no-self-update run \
'/ABSOLUTE/PATH/TO/swarm.json'
</task>
</prompt>
05Confirm the route in raw metadata, then review the output blind.The task label is not evidence. Read the run state and attempt history before judging anonymized artifacts against the same criteria.
Open sectionSection open
Step 8 — Confirm which models actually ran
Start in Ringside, not in a terminal. It opens automatically during a run; if the tab is gone, reopen it and switch to the Models view. That view is built from your own attempt history and already shows, per model: lab, harness, API or plan, tier, task count, first-try rate, eventual pass rate, median tokens, median speed, and when you last used it. Most of what you want to know after a run is already on that screen — read it before you reach for anything else.
The one thing the scoreboard does not do is verify identity per attempt, and that is the check a bakeoff lives or dies on. Run the two queries below for that and only that: confirm the requested and resolved model match, that both cells used the intended harness and reasoning settings, and that any harness-reported model agrees with the resolved model. If the task keys say frontier and qwen but both metadata rows name the same model, you did not run a bakeoff — and the scoreboard cannot tell you that, because it believes the labels.
When a cell fails, go to the raw files. Task output — <workdir>/<task key>/. Normal worker log — <workdir>/<task key>/worker.log. Worktree-mode log — <workdir>/logs/<task key>.worker.log. Run state — ~/.ringer/runs/<run_id>.json. Attempt history — ~/.ringer/runs.jsonl. Final HTML report — ~/.ringer/artifacts/<run_id>-report.html. Artifact index — ~/.ringer/artifacts/index.html. OpenRouter catalog — ~/.ringer/openrouter-catalog.json.
Read the raw failure before you label a model. A bad check, wrong route, provider stall, missing binary, and wrong answer are different results.
Show the full prompt
<prompt>
<task>
# Ringside — read the Models view first
./ringer.py hud
# then open http://127.0.0.1:8700 and switch to Models
# Identity cross-check — the one thing the scoreboard cannot do.
# Copy the run_id printed by the real run.
jq '.tasks[] | {
key,
engine,
model,
status,
verdict,
attempts,
tokens,
check_returncode,
check_timed_out,
setup_error,
taskdir,
log_path
}' ~/.ringer/runs/RUN_ID.json
jq -c --arg run "RUN_ID" '
select(.run_id == $run) |
{
task_key,
worker_engine,
model,
reported_model,
expected_model,
reasoning_effort,
retry,
verdict,
duration_ms,
worker_tokens,
logged_at
}
' ~/.ringer/runs.jsonl
</task>
</prompt>
Step 9 — Do the blind review
Rename or copy the two result.json files so the reviewer does not see the model label. Review them against the source packet and the same written grading criteria.
For the sample research task, ask whether it classified each claim correctly; used the right source; quoted a passage that actually supports the claim; marked missing evidence instead of filling the gap; explained the ambiguous case cleanly; stopped at the allowed sources; and how long review and repair took.
The machine check earns a technical PASS. The blind review decides whether the result is accepted for the job. Record both. Do not rewrite a machine failure as a human pass without noting why, and do not treat a clean schema as proof that the judgment was good.
You do
Review anonymized outputs against one written rubric and record repair time.
The AI does
Keeps the model labels out of the review packet and preserves the machine verdict alongside the human decision.
06Count accepted results, then replace the sample with real work.Cost only means something after acceptance. Build a replay pack, keep retries visible, and use your own history as an early routing signal.
Open sectionSection open
Step 10 — Count the accepted result
Do not hand-collect what Ringer already tracks. Eight of the score sheet's columns are on the Models view in Ringside, computed from your own attempt history — exact model ID, lab, harness, provider or plan, first-try acceptance, acceptance after retry, total duration, and reported tokens. Copy those across, or just read them there.
The six that are genuinely yours to fill are the ones Ringer cannot see: case type, Ringer commit, human repair minutes, provider charge, tool and infrastructure cost, and failure type. Treat score-sheet.csv as the layer on top of the scoreboard — the money and the minutes — not as a manual re-entry of the machine's own record.
That split is the whole point. Ringer knows what happened; it does not know what it cost you. It cannot replace the provider invoice, and it cannot know the value of your review time. Those two numbers are what turn a pass rate into a business decision.
Keep first-try pass rate separate from eventual pass rate. A model that passes after Ringer's retry may still be useful, but you are buying more latency and another failure path.
Step 11 — Replace the sample with your replay pack
The included task is only a plumbing test. It proves that routing, output, and checking work, but says nothing about whether a model should own research.
Start with three to five cases: ordinary, hard, ugly, changed-constraint, and correct-stop when that case fits. Stop early if the candidate obviously cannot do the job. If it survives, expand to five to twenty known cases from your own work. Give both models the same case distribution. Preserve old failures and expensive edge cases. Review blind where practical. Use a new work directory for every run.
Duplicate the two model cells for every case. For each case row, the model IDs differ. The case wording, source paths, output shape, checker, timeout, reasoning setting, and number of attempts do not.
If the worker drives another product, agent, or application instead of writing the result directly, use Ringer's official bakeoff template. Its session validator proves that the inner product used the expected model. A generic session validator would be fake precision; it must read that product's actual transcript or run metadata.
After enough real cases, ask Ringer what your own history says. The same scoreboard lives in two places: the Models view in Ringside, which is where you will actually keep looking, and the CLI below, which is what you want when you need to filter by task type or date. Both read the same attempt history. The scoreboard separates tasks from attempts and first-try pass rate from eventual pass rate. That gap shows how much the retry lane is rescuing.
The current exploration view calls a model proven for a task type after at least three tasks and a first-try pass rate of at least two-thirds. Two of the first three tasks is enough to cross that internal line. Treat it as an early signal to keep testing, not a production standard.
models --explore only shows candidates. It does not add a task to your next run. Take one promising candidate from the list and give it a low-stakes audition yourself.
This is the part that compounds. One bakeoff answers one question; the scoreboard turns every run you ever do into routing evidence, and a model that is proven in someone else's history is still untested in yours.
07Make a narrow lane decision, then test the next exact model.Promote only the job shape the evidence supports, with a rollback condition and a fresh comparison for every new model artifact.
Open sectionSection open
Step 13 — Decide whether the candidate gets a lane
Keep it out. The candidate misses the job's important failures, needs too much repair, or creates an unacceptable data or operating path.
Keep it as a challenger. Give it a small amount of low-risk live work so you can collect more receipts without leaning on it.
Give it a specialist lane. Put one bounded task type there when the accepted-result cost is better, while the frontier model keeps planning, difficult review, and exceptions.
Make it the default for that job. Promote it only after it survives the replay pack, continues to pass live checks, and has a clear rollback path.
Do not promote a model because it crossed Ringer's first three tasks. Before it becomes a recurring default, look for enough cases to include the failures that matter; strong first-try performance, not only rescued passes; stable provider behavior; acceptable human repair time; confirmed model identity; a data path you can explain; saved prompts, checks, cases, and prior outputs; and a named condition that sends the work back to the baseline.
If the same correction appears three times, stop correcting outputs. Inspect the assignment, check, harness, provider, or role.
Step 14 — Add another Chinese model
Start a new dated run. Refresh the provider catalog, record any Ringer update, and pin the new exact ID.
For a model-only comparison, change only the model field shown below. Keep the harness, prompt, sources, checker, reasoning setting, timeout, and review criteria fixed. If you change any of those, say so and rerun the baseline under the new conditions.
Do not transfer a result from one family member to another. A small Qwen checkpoint, a hosted Qwen Max model, and a quantized local build are different artifacts. The same is true of DeepSeek, GLM, Kimi, and MiniMax releases.
08Separate setup, harness, provider, checker, and model failures.Preserve the raw evidence, fix the right layer, and rerun whenever a condition that could change the result moves.
Open sectionSection open
Troubleshooting reference
unknown worker engine: opencode — the engine block is still commented out, or Ringer loaded another config file — enable the full block in ~/.config/ringer/config.toml, or pass the intended config explicitly.
opencode not found on PATH — the macOS wrapper cannot locate the OpenCode binary — run command -v opencode, fix the install or shell path, then retry.
Engine binary preflight fails — bin is relative, wrong, or not executable — use the absolute wrapper path on macOS and verify it with test -x.
Ringer refuses a missing model — the OpenCode template contains {model} and neither the task nor engine supplied one — pin the task-level model. This is desirable in a bakeoff.
Ringer refuses a noncanonical route — the registry knows a sanctioned route for that model — use the named canonical route. Use --allow-noncanonical-route only for a deliberate comparison and expect the result to remain unranked.
Both task names differ but the model is the same — the task key was treated as routing, or an engine default overrode the intended setup — inspect the task-level model, dry-run command, run state, and JSONL identity fields.
Baseline passes when new output is required — old output exists or the checker cannot fail — use a new work directory and strengthen the check.
Baseline prints FAIL but the shell exit status is zero — baseline is diagnostic and reports cell results in its table — read the table. Expected new-output failures are healthy.
A correct fixture fails — the checker is wrong — fix and rerun fixtures before paying workers.
A bad fixture passes — the checker is too weak — add the missing rejection and prove the fixture fails.
Check times out — Ringer's check window is 60 seconds — make the check deterministic and local, or split it into smaller checks.
database is locked — OpenCode workers are contending for local state — lower max_parallel to 1 and rerun in a fresh work directory.
Worker cannot find the source packet — shared inputs were given as vague relative paths — put absolute source paths in the spec and check.
Worker says it finished but Ringer reports FAIL — the output did not pass the executed check — read the raw check output. The worker summary is not the verdict.
Correct work is marked failed — the check or fixture may be wrong — inspect the artifact and check before blaming the model; preserve the row with the right failure type.
Rerun seems to use yesterday's result — the old task directory still contains output — give every real run a unique work directory.
Provider stalls or times out — the access path may be the failure — preserve the result and label it provider or harness behavior unless the raw evidence supports a model verdict.
Output disappears in worktree mode — passing task worktrees are removed — write the deliverable outside the worktree or have the check export it before PASS.
Product-session check cannot prove the model — the inner product has no usable session receipt — add a validator for its transcript or run metadata. A task name is not enough.
Rerun when a condition that matters changes
Keep the cases. Rerun them when the exact model or checkpoint changes; the provider or region changes; the price changes; the harness or tool permissions change; the prompt or reasoning setting changes; the acceptance check changes; the source shape changes; the quantization or serving stack changes; or the data-handling requirements change.
Update Ringer before the new comparison, record the new commit, and freeze it again. Do not compare a result from a moving harness with one from a frozen harness and call the difference a model improvement.
Quick command sheet
Update and record the harness once, freeze it during the run, prove the manifest before spending tokens, then read your own history.
Show the full prompt
<prompt>
<task>
# Update once and record the version
./ringer.py self-update
git rev-parse HEAD
# Refresh exact provider IDs
./ringer.py --no-self-update catalog --refresh
./ringer.py --no-self-update catalog --json
# Prove the manifest before spending tokens
./ringer.py --no-self-update lint '/ABSOLUTE/PATH/TO/swarm.json'
./ringer.py --no-self-update run '/ABSOLUTE/PATH/TO/swarm.json' --baseline
./ringer.py --no-self-update run '/ABSOLUTE/PATH/TO/swarm.json' --dry-run
# Run
./ringer.py --no-self-update run '/ABSOLUTE/PATH/TO/swarm.json'
# Read your history
./ringer.py --no-self-update models --task-type research
./ringer.py --no-self-update models --explore --task-type research
./ringer.py --no-self-update models --open
</task>
</prompt>
Write the decision this narrowly: On this job, under this harness, with this exact model, check, provider, and review cost, the candidate did—or did not—earn the lane.