huawei-csl/MACaQ

What it does

MACaQ is a model-agnostic quantization method and toolkit for producing low-precision weights and activations (notably W4A4, plus 2/3/4-bit GGUF weights) that can be run on common local inference stacks. It targets text LLMs (dense and MoE), diffusion transformers for image, video and image-to-video generation, and speech generators. The method uses calibration data to fit quantization parameters, and MACaQ-R further refines weight codes inside an existing quantized GGUF while keeping its native formats and block scales.

The project accompanies a paper on fast calibrated quantization; reported results include quantizing a 122B-parameter model in 64 minutes with 32.5 GB of device memory, lower perplexity than MR-GPTQ and DuQuant++, and matching Llama-3-8B perplexity with 11× less quantization time. These are the authors' claims and should be read alongside the paper's evaluation setup.

Backends

Each backend has its own environment and guide:

• llama.cpp — 2/3/4-bit GGUF for text generation; CPU execution is supported. MACaQ-R GGUFs run with a standard llama.cpp installation.
• vLLM — W4A4 or weight-only checkpoints for text generation and serving over an HTTP API.
• Diffusers — quantization of diffusion transformers saved as safetensors, for image, video, image-to-video and audio generation.
• PyTorch — W4A4 speech generators (Qwen3-TTS, Chatterbox Turbo) that save WAV audio.
• Ascend — W4A4 dense and grouped matmul kernels contributed as CANN pull requests; full-model execution requires integration with an Ascend inference runtime.

Quick start

Use Python 3.12 and a separate environment for each backend. The repository does not put install commands in the top-level README; each backend guide has its own install, quantize and generate steps. For llama.cpp specifically:

1. Create the llama.cpp environment and install dependencies by following the llama.cpp setup guide.
2. Prepare the original model and its FP16 teacher, plus a calibration text file, as described in prepare a model.
3. Refine a quantized MoE GGUF into MACaQ-R:

.venv-llamacpp/bin/python llamacpp/macaq.py refine \
  --model models/model-q4.gguf --hf models/original \
  --teacher models/original-f16.gguf --calibration data/calibration.txt \
  --output models/model-macaq-r.gguf --work-dir work/refinement

4. Run the resulting file with standard llama.cpp:

llama cli -m models/model-macaq-r.gguf

Use llama serve -m models/model-macaq-r.gguf to serve it instead. A pre-refined Qwen3.6-35B-A3B MACaQ-R GGUF is also published on Hugging Face.

Constraints

• Python 3.12 is required; keep a separate environment per backend to avoid dependency conflicts.
• Full MACaQ with activation transforms requires the bundled llama.cpp runtime; only MACaQ-R refined weights are portable to a stock llama.cpp build.
• vLLM W4A4 is the default inference mode, and the reported speedups (2.33× prefill, 2.58× decode for Qwen3.8-27B) are specific to that setup.
• Ascend support is currently kernel-level; running whole models there needs an Ascend inference runtime.
• Calibration data and an FP16 teacher are needed for refinement; MACaQ-R preserves the source GGUF's formats and block scales.
• Licensed under Apache 2.0; bundled third-party components keep their own licenses.

HotRepo's take

MACaQ is a recent project with a short observation window: it first appears in HotRepo's records on 2026-11-12 on the C++-language new-project list, with 112 stars and 5 forks, and a measured net gain of 7 stars and 2 forks over a roughly one-hour window that morning. There is no all-language daily rank recorded and no Hacker News discussion, so the data supports only a single-day snapshot, not a trend or adoption claim. What is notable from the material itself is breadth: one quantization method spanning llama.cpp, vLLM, Diffusers and PyTorch, including a refinement path that produces GGUFs runnable by stock llama.cpp.