Field guide Hy4 Flash

Checked

Unofficial guide

Hy4 Flash on vLLM

Last updated

Tencent's model card and the vLLM recipe both tell you to serve Hy4 Flash from the image vllm/vllm-openai:hy4-flash. They publish two different commands. Use the one that matches the checkpoint you downloaded.

FP8 command from the model card

This is the command on the Hugging Face model card. It serves tencent/Hy4-flash-FP8, names the model hy4-flash, and turns on DFly speculative decoding plus the hy_v4 tool and reasoning parsers.

Model card · FP8 · 8 GPUs
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy4-flash tencent/Hy4-flash-FP8 \
    --tensor-parallel-size 8 \
    --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \
    --attention-backend FLASHMLA_SPARSE \
    --tool-call-parser hy_v4 \
    --reasoning-parser hy_v4 \
    --enable-auto-tool-choice \
    --port 8000 \
    --served-model-name hy4-flash

BF16 command from the vLLM recipe

The recipe sample fetched for this guide serves tencent/Hy4-flash in bfloat16. It adds a privileged container, the Rust frontend, an 8,192-token context, four sequences, and one image per prompt. It does not set --served-model-name. The recipe says to use the checkpoint id as the client model name in that case: tencent/Hy4-flash, or tencent/Hy4-flash-FP8 if you selected the FP8 variant.

vLLM recipe · BF16 · 8 GPUs
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:hy4-flash tencent/Hy4-flash \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v4 \
  --tool-call-parser hy_v4 \
  --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}'

What the recipe says about hardware

The recipe also lists a single-node deployment and says tensor-parallel size is set to the GPU count at deploy time. The concrete commands above use 8. This guide does not have a published command for a smaller GPU count.

Client

The model card's client matches the FP8 command's --served-model-name hy4-flash. Sampling in that example is temperature 0.9 and top_p 1.0, with reasoning turned off for the demo.

OpenAI-compatible client
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="hy4-flash",
    messages=[{"role": "user", "content": "Hello! Can you briefly introduce yourself?"}],
    temperature=0.9,
    top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

SGLang has its own official Docker command on the model card. It is on the SGLang page. Transformers support, as of the model card, is installed from pull request 50903 until it lands on the main branch. Loading tencent/Hy4-flash with device_map="auto" is still a multi-hundred-gigabyte load.

Common questions

Can I pip install vLLM for Hy4 Flash?

The vLLM recipe says no released vLLM wheel registers HYV4FlashVLForConditionalGeneration or the DFly drafter, so pip installs are not supported. Use the Docker image vllm/vllm-openai:hy4-flash.

How many GPUs does the Hy4 Flash vLLM example use?

Both published examples set tensor parallel size to 8. The recipe's hardware list is Hopper and Blackwell data-center GPUs: H100, H200, B200, GB200, B300, and GB300. It says AMD Instinct is not supported because no ROCm image ships Hy4 Flash.

Does the vLLM command start at the full 1M context?

The checkpoint advertises 1,048,576 positions. The recipe's sample command starts at --max-model-len 8192, four concurrent sequences, and one image per prompt. Raise those flags to match memory. The recipe says its 529 GB and 345 GB figures are download size times 1.2, not measured GPU minima.

Sources

  1. tencent/Hy4-flash on Hugging Face Model card, architecture table, deployment commands, and license pointer.
  2. vLLM recipe for tencent/Hy4-flash