Balanov/verdict

What it does

Verdict targets a common blind spot in agent and CI automation: a zero exit code does not mean the task actually succeeded. A test command can exit 0 after collecting zero tests, and a non-zero exit can reflect a missing dependency rather than a failed assertion. Verdict reads the command, exit code, stdout, and stderr, then returns one of four labels: success, failure, blocked, or unknown project overview.

It is designed as an advisory semantic layer for coding agents, CI runners, and other tool-using systems. The model is a hashed n-gram feature extractor feeding a multiclass logistic regression, with about 196K int8 weights. Inference runs in pure Python on CPU with no tokenizer files, GPU, network calls, or telemetry model details.

Quick start

Requirements: a Python installation and a local clone of the repository. The package itself is dependency-free at inference time.

git clone https://github.com/Balanov/verdict.git
cd verdict
python -m pip install .

Then call the Python API with command, exit code, and captured output install and Python example:

from verdict_tiny import Verdict

model = Verdict()
result = model.predict(
    command="pytest tests/",
    exit_code=0,
    stdout="collected 0 items\nno tests ran in 0.01s",
)
print(result)

For the example input above, the README shows an unknown label with high confidence, because the output indicates no tests ran rather than a successful test run example output.

The same model is available as a CLI. Pipe stdout into it and pass the command and exit code as flags CLI usage:

printf 'collected 0 items\nno tests ran\n' | verdict \
  --command 'pytest tests/' --exit-code 0

Use --stdout output.log, --stderr error.log, or --weights /path/to/verdict-q8.bin when you need file inputs or custom weights. Release weights are also published on Hugging Face, and a browser demo runs the same quantized weights locally browser demo.

Output and limits

The output contract distinguishes completed operations, concrete failures, environment blockers, and inconclusive evidence. Probabilities come from a linear model and are not calibrated safety confidence; for consequential actions, apply thresholds plus deterministic checks or human review output contract. Verdict classifies observed CLI evidence only. It does not judge task completion, code correctness, safety, or truthfulness.

On the reported benchmark, Verdict 0.1 (int8) reached 93.2% accuracy and macro-F1 on 192 synthetic examples from 48 CLI families excluded from training, versus 47.9% accuracy for an exit-code baseline. Median inference was 0.54 ms across 576 predictions on Windows 11 / AMD64 / Python 3.12, including feature extraction and the API call benchmark. The corpus is synthetic and English-biased, so the numbers measure generalization across held-out templates rather than reliability on production agent traces; real-trace validation is listed as future work.

Reproduction and license

To retrain and re-evaluate from scratch, install the training extras, run the training script, and execute the test suite reproduction steps:

python -m pip install -e '.[train,dev]'
python training/train.py
pytest -q

Code and model weights are Apache-2.0, and the synthetic dataset is CC0-1.0 license. The repository separates inference code, training code, benchmark results, synthetic data, and release weights, so the model behavior can be inspected and rebuilt from the published corpus.

HotRepo's take

Verdict entered HotRepo's new-project list with 136 stars gained between 2026-11-02 19:14 UTC and 2026-11-03 02:45 UTC, its first recorded snapshot. That is early uptake rather than a stable trend, and no GitHub daily rank was recorded for it. The useful signal in the listing is the project's stated focus: a tiny, dependency-free classifier that challenges the assumption that exit code 0 equals success. The benchmark caveat matters more than the headline accuracy: it is measured on synthetic, English-biased data, so the project should be treated as a promising advisory layer awaiting real-trace validation.