heterodoxin/AbliteratedOutTheBox

This is an inference-time refusal-removal fork of llama.cpp, so it belongs in the hands of people already comfortable converting and serving GGUF models. The README is unusually explicit about the mechanism and the measured trade-offs; the interesting question is not whether it works on its stated test models, but how its gate behaves on your prompts and your hardware.

The problem

A user has a local model they want to steer toward answering prompts that the model's safety alignment normally refuses, without retraining it or storing a second copy of the weights. Weight-baked abliteration changes the stored weights, so every prompt sees a modified model and the result depends on architecture-specific surgery. Here the workflow stays close to ordinary llama.cpp: convert a Hugging Face model to GGUF, then launch llama-abliterate with the prompt. The runtime decides per prompt whether to remove a refusal direction from the activations, so the model file on disk is not rewritten and ordinary prompts can be served from the same copy.

What's different

The edit is computed from the prompt itself. For each prompt the runtime forces the assistant into a few comply versus refuse openings, diffs their activations to derive a refusal direction, and adds a second direction for safety-disclaimer tone. A look-ahead gate generates up to 24 of the model's own tokens and fires when the peak refusal projection, band-averaged across middle layers, crosses --ff-threshold. The edit is a scale-free projection at the build_cvec layer-output site shared by 131 of 165 architectures, so one recipe is meant to cover many model families rather than requiring per-architecture code.

The author's measurements on Qwen3-8B give the clearest picture: refusal falls from 96.3% to 7.9% on broad held-out harmful prompts and from 82.0% to 0.0% on JailbreakBench, while the gate fires on 0.4% of alpaca prompts (1 of 256), and when it does not fire the output is reported as bit-identical to base. The same recipe reportedly works on Llama-3.1-8B and Gemma-4-e4b. The trade-off is on the disclaimer side: raising --disc-strength removes more hedging but can degrade longer outputs, and answers that stay coherent still show a lower distinct-token ratio (~0.65 versus 0.84 base). The runtime also handles one sequence at a time because the gate scalar is per context. These are the author's numbers on three models, not independent reproductions.

Who should try it

Good fit: a developer who already runs llama.cpp and wants to study or demonstrate how refusal behavior is represented and can be changed in activations, on models and hardware they already have. The CPU-friendly runtime matters here because the direction work is done on prompt prefill rather than training. Poor fit: someone expecting a general-purpose safety-bypass service for many concurrent users; the one-sequence-at-a-time gate and the fixed set of forced openings both constrain that use. A user who only needs an uncensored model for a single task may find a pre-abliterated checkpoint simpler, though this project avoids maintaining a second weights file.

Requirements and limitations

Setup assumes a llama.cpp-style C++ build, one of the usual backends (CUDA, HIP or CPU), and Python with torch, transformers and gguf only for converting a Hugging Face checkpoint to GGUF. The README points to tool details for the abliterate utility, and the general project README is the place to confirm current flags. If you already have a GGUF model, the Python conversion step can be skipped.

Ongoing constraints matter more than install: the gate threshold trades harmful recall (0.89 at the default 0.70) against benign false fires (about 0.4%), and both are measured on the author's Qwen3-8B test sets. Unusual prompts may gate or edit less well because directions come from a fixed opening set. Throughput is claimed unchanged for decode, but flagged prompts pay an extra prefill pass, and a high disclaimer strength can degrade long generations. The project documents no evaluation on models outside its three reported families, so architecture coverage is a claim to verify on your own checkpoint rather than a guarantee.

HotRepo's take

HotRepo first recorded the project at 3 days old on 2026-10-24, with a single NEW-REPOSITORY board snapshot on 2026-10-25 showing 104 total stars at #18 in the cpp list. That is not a daily growth measurement, and the stored coverage has one distinct date, so persistence cannot yet be judged. Before adopting, check that your target architecture is among the covered set and run the gate on a sample of your own benign prompts; the documented false-positive rate is specific to the reported models.