Unofficial guide
Hy4 Flash errors and troubleshooting
Last updated
These are the failures the official docs already warn about. Each fix is the one those docs describe. If a command here disagrees with a newer model card, follow the card.
vLLM from pip does not know the model
The recipe says a released vLLM wheel does not register
HYV4FlashVLForConditionalGeneration or the DFly drafter, so a pip install is not
the supported path. Run vllm/vllm-openai:hy4-flash. The model card and the recipe
both use that image. The full command is on the vLLM page.
The server dies as soon as context is large
Two different documents tell you not to start at the full window. The vLLM recipe advertises
1,048,576 positions and then starts the sample at --max-model-len 8192,
--max-num-seqs 4, and one image. Engram is offloaded to host RAM, so an
out-of-memory error can be host memory rather than GPU memory. The 529 GB and 345 GB numbers on
the recipe are download size times 1.2, not a measured minimum you should try to allocate.
On Apple silicon, AngelSlim's Mac notes say kIOGPUCommandBufferCallbackErrorOutOfMemory
means you shrink the KV pool before the micro-batch, then restart. Their failing example is four
slots and a 512k pool at micro-batch 1,024, totaling 113.3 GiB. The working example shares a
256k pool. They also set iogpu.wired_limit_mb=114688 (112 GiB), and that sysctl
resets on reboot.
AMD GPU, or a GPU count other than 8
The recipe says AMD Instinct is not supported because no ROCm image ships Hy4 Flash. The
published Docker commands set tensor parallel size to 8. This guide does not have an official
command for two or four GPUs. Data parallelism, pipeline parallelism, context parallelism,
EPLB, DBO, and LoRA are listed as unsupported. MTP is not the speculative method in this vLLM
implementation. The method string in the examples is dfly.
The client gets 404 for the model name
The model card's Docker command passes --served-model-name hy4-flash, and its
Python client sends model="hy4-flash". The recipe's command omits that flag and
tells you to send the checkpoint id, tencent/Hy4-flash or
tencent/Hy4-flash-FP8. Mixing the two clients is a 404. The SGLang command on the
card also omits a served name, so read /v1/models before reusing the vLLM client.
llama.cpp does not recognize the GGUF
Stock llama.cpp, as of AngelSlim's README, does not contain this architecture. Check out
pull request 32517, which was open
on 21 November 2026, or apply the patch they ship for Mac or Spark on commit
2cbca389f. The Mac README's sample path A12B-Q4_K_M-... is not a file
in the repo. Use Hy4-flash-Q4_K_M-00001-of-00002.gguf and keep the
-00002-of-00002 shard in the same folder. Image runs need
mmproj-Hy4-flash-bf16.gguf and llama-mtmd-cli. The Mac patch note says
to use a revision that still has tools/mtmd, or the projector fails to load.
The base checkpoint will not chat
tencent/Hy4-flash-base is a pretrained model. The model card says the chat and
reasoning examples apply to tencent/Hy4-flash and tencent/Hy4-flash-FP8
only. If answers ignore the template, you downloaded the base weights.
Transformers cannot import the architecture
The model card says support will land on the transformers main branch later, and until then to install from pull request 50903:
pip install "transformers[torch,vision] @ git+https://github.com/huggingface/transformers.git@refs/pull/50903/head"
Re-read the card before you pin that URL. If the pull request has merged, the pin is stale.
device_map="auto" still has to place a checkpoint of about 440 GB.
Common questions
Why does pip install vLLM fail to load Hy4 Flash?
The vLLM recipe says no released wheel registers HYV4FlashVLForConditionalGeneration or the DFly drafter. Use the image vllm/vllm-openai:hy4-flash instead of a pip install.
Why does Hy4 Flash run out of memory at 1M context?
The checkpoint length is 1,048,576 tokens. The vLLM recipe's sample starts at 8,192 tokens, four sequences, and one image. AngelSlim's Mac notes say a 512k pool with four slots totals 113.3 GiB and fails on their setup, while a shared 256k pool fits. Shrink context before you blame the quant.
Why does llama.cpp say the architecture is unknown?
AngelSlim says Hy4 Flash support is in llama.cpp pull request 32517 and was not merged when they wrote the card. That pull request was still open on 21 November 2026. Their Mac and Spark notes use commit 2cbca389f plus a patch in the GGUF repo instead.
Sources
- vLLM recipe for tencent/Hy4-flash
- tencent/Hy4-flash on Hugging Face Model card, architecture table, deployment commands, and license pointer.
- AngelSlim/Hy4-flash-GGUF Official GGUF builds linked from the model card.
- AngelSlim, Apple silicon notes
- llama.cpp pull request 32517 Open and unmerged when this guide was checked on 21 November 2026.
- tencent/Hy4-flash-base on Hugging Face Pre-trained base checkpoint, not an instruction-tuned chat model.