Field guide Hy4 Flash

Checked

Unofficial guide

Hy4 Flash GGUF

Last updated

The GGUF files Tencent's model card points to live in AngelSlim/Hy4-flash-GGUF. AngelSlim describes five builds. Three are split into a trunk and a second shard. Two Spark builds are a single file.

The five builds

Sizes are the ones printed on the AngelSlim card, confirmed against the file bytes on 21 November 2026. Bits per weight are AngelSlim's figures. “Excl. Engram” is their column, not a second download.
Build Files Size bpw overall / excl. Engram
Q4_K_M 00001 is 97.78 GiB, 00002 is 59.01 GiB 156.79 GiB 6.05 / 5.27
SPARK-4Bit One file 151.3 GiB 5.84 / 4.78
SPARK-4Bit-Fast One file, 2.7 GiB smaller than SPARK-4Bit 148.6 GiB 5.74 / 4.63
Q3_K 00001 is 85.69 GiB, 00002 is 59.01 GiB 144.70 GiB 5.58 / 4.62
IQ2-AH File name uses IQ2_AH. 00001 is 66.72 GiB, 00002 is 59.01 GiB 125.73 GiB 4.85 / 3.60

AngelSlim's notes on the recipes: Q4_K_M keeps most non-expert weights in Q8_0, quantizes expert gate and up as Q4_K, and gives down Q6_K on 19 layers and Q4_K on the other 22. They say to use it unless you are memory-constrained. Q3_K drops expert gate and up to Q3_K on 22 layers chosen by sensitivity and keeps Q4_K on 19. IQ2-AH runs expert gate and up at 2.0625 bits (IQ2_XXS) and down at Q4_K, with most non-expert weights left in BF16. The two SPARK-4Bit files use NVIDIA's 4-bit floating-point format. SPARK-4Bit-Fast is the smaller of the two. The card says both are suited to Spark devices. Image input uses mmproj-Hy4-flash-bf16.gguf (0.97 GiB). There is no speed row for IQ2-AH.

Which machine they measured

The speed table on the GGUF card is labeled as a test in real WorkBuddy scenarios. Two machines are named. Nothing else is.

Tokens per second from AngelSlim. Each cell is 4K / 16K / 32K context. This site did not re-run the test.
Machine Format Prefill Decode
Mac M5 Max 128 GB Q4_K_M 629.8 / 448.1 / 418.6 48.13 / 45.18 / 44.15
Mac M5 Max 128 GB Q3_K 743.2 / 563.0 / 520.6 50.34 / 49.47 / 47.37
RTX Spark 128 GB SPARK-4Bit-Fast 799 / 717 / 707 31.95 / 33.14 / 33.39
RTX Spark 128 GB SPARK-4Bit 765 / 686 / 666 25.80 / 28.16 / 30.3

They also say Q3_K reaches up to 70 tokens per second on code generation, and SPARK-4Bit-Fast reaches up to 40. A separate Mac note, on their patch rather than pull request 32517, reports about 36 tokens per second at 64K in plain autoregressive mode and 44–50 tokens per second at 4K–32K with MTP draft depth 2. Treat that as a different build. The memory budget in that note is on the requirements page.

Build llama.cpp

The main GGUF README says architecture support is in llama.cpp pull request 32517 and was not merged, so the build checks that pull request out. On 21 November 2026 the pull request was still open. Add -DGGML_CUDA=ON on NVIDIA. Metal is on by default on macOS in their note.

AngelSlim's general build
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/32517/head:hy4-flash
git checkout hy4-flash

cmake -B build
cmake --build build --config Release -j

The Mac README takes a different path: check out llama.cpp commit 2cbca389f and apply patch/hy4-flash-mac-minimal.patch from the GGUF repo. The Spark README pins 2cbca389f515df455412dafa07e2b1359cc72a6d, applies patch/hy4-flash-spark.patch, and compiles CUDA architecture 121a for Spark / GB10. Those READMEs are the place for the sysctl, the CUDA paths, and the server flags. The Mac README's example paths say A12B-Q4_K_M-.... That filename is not in the repository. The published trunk is Hy4-flash-Q4_K_M-00001-of-00002.gguf.

Run the Q4_K_M pair

Commands from the GGUF card
./build/bin/llama-cli -m Hy4-flash-Q4_K_M-00001-of-00002.gguf -c 8192 -ngl 99 --jinja

./build/bin/llama-server -m Hy4-flash-Q4_K_M-00001-of-00002.gguf -c 8192 -ngl 99 --jinja --port 8080

./build/bin/llama-mtmd-cli -m Hy4-flash-Q4_K_M-00001-of-00002.gguf --mmproj mmproj-Hy4-flash-bf16.gguf \
    --image photo.png -p "Describe this image." -c 8192 -ngl 99 --jinja

Point the tool at the first shard. The second shard has to sit beside it. Context in these examples is 8,192, and -ngl 99 asks to offload as many layers as possible. --jinja turns on the chat template.

Common questions

Which Hy4 Flash GGUF quant should I use?

AngelSlim calls Hy4-flash-Q4_K_M the safe default unless you are memory-constrained. Q3_K is the mixed 3-bit and 4-bit expert build. IQ2_AH is the smallest pair, 125.73 GiB, and has no speed table on the card. SPARK-4Bit and SPARK-4Bit-Fast are the builds they say are for NVIDIA Spark devices.

Does stock llama.cpp run Hy4 Flash?

Not in the build AngelSlim documents. Their main README says support is in llama.cpp pull request 32517 and was not merged, so you check out that pull request. That pull request was still open on 21 November 2026. Their Mac and Spark notes instead pin commit 2cbca389f and apply a patch from the GGUF repo.

Can a 128 GB Mac run the Q4_K_M GGUF?

AngelSlim publishes prefill and decode speeds for Q4_K_M and Q3_K on a Mac M5 Max with 128 GB. The Q4_K_M pair is 156.79 GiB. Their Mac notes say the trunk is about 97.4 GiB and the roughly 59 GiB Engram table is lazily mapped, and they raise the Metal wired limit to 112 GiB.

Sources

  1. AngelSlim/Hy4-flash-GGUF Official GGUF builds linked from the model card.
  2. AngelSlim, Apple silicon notes
  3. AngelSlim, NVIDIA Spark notes
  4. llama.cpp pull request 32517 Open and unmerged when this guide was checked on 21 November 2026.
  5. tencent/Hy4-flash on Hugging Face Model card, architecture table, deployment commands, and license pointer.