Field guide Hy4 Flash

Checked

Unofficial guide

Hy4 Flash errors and troubleshooting

Last updated

These are the failures the official docs already warn about. Each fix is the one those docs describe. If a command here disagrees with a newer model card, follow the card.

vLLM from pip does not know the model

The recipe says a released vLLM wheel does not register HYV4FlashVLForConditionalGeneration or the DFly drafter, so a pip install is not the supported path. Run vllm/vllm-openai:hy4-flash. The model card and the recipe both use that image. The full command is on the vLLM page.

The server dies as soon as context is large

Two different documents tell you not to start at the full window. The vLLM recipe advertises 1,048,576 positions and then starts the sample at --max-model-len 8192, --max-num-seqs 4, and one image. Engram is offloaded to host RAM, so an out-of-memory error can be host memory rather than GPU memory. The 529 GB and 345 GB numbers on the recipe are download size times 1.2, not a measured minimum you should try to allocate.

On Apple silicon, AngelSlim's Mac notes say kIOGPUCommandBufferCallbackErrorOutOfMemory means you shrink the KV pool before the micro-batch, then restart. Their failing example is four slots and a 512k pool at micro-batch 1,024, totaling 113.3 GiB. The working example shares a 256k pool. They also set iogpu.wired_limit_mb=114688 (112 GiB), and that sysctl resets on reboot.

AMD GPU, or a GPU count other than 8

The recipe says AMD Instinct is not supported because no ROCm image ships Hy4 Flash. The published Docker commands set tensor parallel size to 8. This guide does not have an official command for two or four GPUs. Data parallelism, pipeline parallelism, context parallelism, EPLB, DBO, and LoRA are listed as unsupported. MTP is not the speculative method in this vLLM implementation. The method string in the examples is dfly.

The client gets 404 for the model name

The model card's Docker command passes --served-model-name hy4-flash, and its Python client sends model="hy4-flash". The recipe's command omits that flag and tells you to send the checkpoint id, tencent/Hy4-flash or tencent/Hy4-flash-FP8. Mixing the two clients is a 404. The SGLang command on the card also omits a served name, so read /v1/models before reusing the vLLM client.

llama.cpp does not recognize the GGUF

Stock llama.cpp, as of AngelSlim's README, does not contain this architecture. Check out pull request 32517, which was open on 21 November 2026, or apply the patch they ship for Mac or Spark on commit 2cbca389f. The Mac README's sample path A12B-Q4_K_M-... is not a file in the repo. Use Hy4-flash-Q4_K_M-00001-of-00002.gguf and keep the -00002-of-00002 shard in the same folder. Image runs need mmproj-Hy4-flash-bf16.gguf and llama-mtmd-cli. The Mac patch note says to use a revision that still has tools/mtmd, or the projector fails to load.

The base checkpoint will not chat

tencent/Hy4-flash-base is a pretrained model. The model card says the chat and reasoning examples apply to tencent/Hy4-flash and tencent/Hy4-flash-FP8 only. If answers ignore the template, you downloaded the base weights.

Transformers cannot import the architecture

The model card says support will land on the transformers main branch later, and until then to install from pull request 50903:

pip install "transformers[torch,vision] @ git+https://github.com/huggingface/transformers.git@refs/pull/50903/head"

Re-read the card before you pin that URL. If the pull request has merged, the pin is stale. device_map="auto" still has to place a checkpoint of about 440 GB.

Common questions

Why does pip install vLLM fail to load Hy4 Flash?

The vLLM recipe says no released wheel registers HYV4FlashVLForConditionalGeneration or the DFly drafter. Use the image vllm/vllm-openai:hy4-flash instead of a pip install.

Why does Hy4 Flash run out of memory at 1M context?

The checkpoint length is 1,048,576 tokens. The vLLM recipe's sample starts at 8,192 tokens, four sequences, and one image. AngelSlim's Mac notes say a 512k pool with four slots totals 113.3 GiB and fails on their setup, while a shared 256k pool fits. Shrink context before you blame the quant.

Why does llama.cpp say the architecture is unknown?

AngelSlim says Hy4 Flash support is in llama.cpp pull request 32517 and was not merged when they wrote the card. That pull request was still open on 21 November 2026. Their Mac and Spark notes use commit 2cbca389f plus a patch in the GGUF repo instead.

Sources

  1. vLLM recipe for tencent/Hy4-flash
  2. tencent/Hy4-flash on Hugging Face Model card, architecture table, deployment commands, and license pointer.
  3. AngelSlim/Hy4-flash-GGUF Official GGUF builds linked from the model card.
  4. AngelSlim, Apple silicon notes
  5. llama.cpp pull request 32517 Open and unmerged when this guide was checked on 21 November 2026.
  6. tencent/Hy4-flash-base on Hugging Face Pre-trained base checkpoint, not an instruction-tuned chat model.