AIWatch.

Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit.

📖 Documentation | 🤗 Hugging Face | 🤖 ModelScope | 💬 WeChat

Hy3 llama.cpp Quantization

Low-bit quantized Hy3 (hy_v3) GGUFs for llama.cpp — ready to run out of the box, with MTP self-speculative decoding. This repo ships the quantized models, plus the mixed-precision recipes to build your own from a calibration set.

Two parts below: Deploy (build & run) and Quantization.

Quick Start

Build

Clone the latest llama.cpp (must include the hy_v3 support merged in PR #25395 — i.e. any commit after 505b1ed; the current master is fine) and build it following the official build guide:

git clone https://github.com/ggml-org/llama.cpp llama.cpp-hyv3
# build per docs/build.md (CUDA / Metal / CPU as appropriate for your machine)

Binaries land in llama.cpp-hyv3/build/bin/.

Already downloaded an older GGUF? Earlier GGUFs here were built against a patched llama.cpp and won't load correctly on today's upstream. To build and run those, follow the previous README (the setup_hyv3_llama.sh + patches flow) instead.

Run

# plain serve (no speculative decoding)
./llama.cpp-hyv3/build/bin/llama-server -m /path/to/Hy3.gguf -ctk q8_0 -ctv q8_0 -fa on -c 65536

# serve + MTP self-speculative decoding (needs an MTP gguf, i.e. converted WITHOUT --no-mtp)
./llama.cpp-hyv3/build/bin/llama-server -m /path/to/Hy3-mtp.gguf --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 1 -ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0 -fa on -c 65536