Unofficial guide
Hy4 Flash on vLLM
Last updated
Tencent's model card and the vLLM recipe both tell you to serve Hy4 Flash from the image
vllm/vllm-openai:hy4-flash. They publish two different commands. Use the one that
matches the checkpoint you downloaded.
FP8 command from the model card
This is the command on the Hugging Face model card.
It serves tencent/Hy4-flash-FP8, names the model hy4-flash, and turns on
DFly speculative decoding plus the hy_v4 tool and reasoning parsers.
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:hy4-flash tencent/Hy4-flash-FP8 \
--tensor-parallel-size 8 \
--speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \
--attention-backend FLASHMLA_SPARSE \
--tool-call-parser hy_v4 \
--reasoning-parser hy_v4 \
--enable-auto-tool-choice \
--port 8000 \
--served-model-name hy4-flash BF16 command from the vLLM recipe
The recipe sample fetched for this guide
serves tencent/Hy4-flash in bfloat16. It adds a privileged container, the Rust
frontend, an 8,192-token context, four sequences, and one image per prompt. It does not set
--served-model-name. The recipe says to use the checkpoint id as the client model
name in that case: tencent/Hy4-flash, or tencent/Hy4-flash-FP8 if you
selected the FP8 variant.
docker run --gpus all \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e VLLM_USE_RUST_FRONTEND=1 \
vllm/vllm-openai:hy4-flash tencent/Hy4-flash \
--dtype bfloat16 \
--max-model-len 8192 \
--max-num-seqs 4 \
--max-num-batched-tokens 8192 \
--limit-mm-per-prompt '{"image":1,"video":0}' \
--tensor-parallel-size 8 \
--enable-auto-tool-choice \
--reasoning-parser hy_v4 \
--tool-call-parser hy_v4 \
--speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' What the recipe says about hardware
- GPUs: H100, H200, B200, GB200, B300, and GB300. AMD Instinct is not supported. There is no ROCm image for this model.
- Tensor parallelism is supported, with optional expert parallelism. Data parallelism, pipeline parallelism, context parallelism, EPLB, DBO, and LoRA are not supported.
- DFly speculative decoding is the supported drafter. The recipe says MTP is not supported in this implementation, and that turning speculative decoding off serves the target model alone.
- Engram tables are offloaded to host RAM by default. The recipe says to provision host memory as well as GPU memory. It does not print a host-RAM number.
- The VRAM hints on the recipe, 529 GB for BF16 and 345 GB for FP8, are the download sizes 440.5 GB and 287.1 GB multiplied by 1.2. The page says those are conservative aggregate estimates, not measured GPU minima, because Engram on the CPU lowers the GPU footprint.
The recipe also lists a single-node deployment and says tensor-parallel size is set to the GPU count at deploy time. The concrete commands above use 8. This guide does not have a published command for a smaller GPU count.
Client
The model card's client matches the FP8 command's --served-model-name hy4-flash.
Sampling in that example is temperature 0.9 and top_p 1.0, with reasoning turned off for the demo.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="hy4-flash",
messages=[{"role": "user", "content": "Hello! Can you briefly introduce yourself?"}],
temperature=0.9,
top_p=1.0,
extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)
SGLang has its own official Docker command on the model card. It is on the SGLang page.
Transformers support, as of the model card, is installed from
pull request 50903 until it
lands on the main branch. Loading tencent/Hy4-flash with device_map="auto"
is still a multi-hundred-gigabyte load.
Common questions
Can I pip install vLLM for Hy4 Flash?
The vLLM recipe says no released vLLM wheel registers HYV4FlashVLForConditionalGeneration or the DFly drafter, so pip installs are not supported. Use the Docker image vllm/vllm-openai:hy4-flash.
How many GPUs does the Hy4 Flash vLLM example use?
Both published examples set tensor parallel size to 8. The recipe's hardware list is Hopper and Blackwell data-center GPUs: H100, H200, B200, GB200, B300, and GB300. It says AMD Instinct is not supported because no ROCm image ships Hy4 Flash.
Does the vLLM command start at the full 1M context?
The checkpoint advertises 1,048,576 positions. The recipe's sample command starts at --max-model-len 8192, four concurrent sequences, and one image per prompt. Raise those flags to match memory. The recipe says its 529 GB and 345 GB figures are download size times 1.2, not measured GPU minima.
Sources
- tencent/Hy4-flash on Hugging Face Model card, architecture table, deployment commands, and license pointer.
- vLLM recipe for tencent/Hy4-flash