RedHatAI/Qwen3-4B-speculator.dflash2

This is a DFlash2 speculator model for Qwen/Qwen3-4B.

Training Details

This model was trained using the Speculators library on inference-optimization/Qwen3-8B-Regenerated-Collection. DFlash2 extends DFlash with local dynamic convolutions and a candidate selector for improved token prediction.

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3-4B \
  --data ./data/qwen3_8b_regen/autoif_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/evol_codealpaca_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/lmsys_arena_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/magpie_output.jsonl \
  --data ./data/qwen3_8b_regen/metamathqa_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/nemotron_chat_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/nemotron_math_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/nemotron_stem_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/orca_math_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/tulu3_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/ultrachat_output.jsonl \
  --data ./data/qwen3_8b_regen/ultrafeedback_train_sft_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/ultrainteract_train_Qwen3-8B.jsonl \
  --output ./output/dflash2_qwen3_4b_8spec \
  --seq-length 8192

Launch vLLM

# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0 python scripts/launch_vllm.py \
  Qwen/Qwen3-4B \
  --target-layer-ids 1 9 17 25 33 \
  -- --port 8000 \
  --max-model-len 8194 \
  --enforce-eager

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=1,2,3,4 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-4B \
  --speculator-type dflash2 \
  --data-path ./output/dflash2_qwen3_4b_8spec \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/dflash2_qwen3_4b_8spec/checkpoints \
  --block-size 9 \
  --max-anchors 512 \
  --target-layer-ids 1 9 17 25 33 \
  --num-layers 5 \
  --conv-kernel-size 2 \
  --conv-group-size 16 \
  --selector-rank 256 \
  --selector-top-k 16 \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --dflash-decay-gamma 4.0 \
  --per-position-loss-weight fixed-exp-decay \
  --optimizer adamw \
  --lr 6e-4 \
  --weight-decay 0.0 \
  --scheduler-type cosine \
  --scheduler-warmup-ratio 0.04 \
  --epochs 1 \
  --total-seq-len 8192 \
  --seed 42 \
  --fsdp-shard \
  --on-missing generate \
  --on-generate delete \
  --checkpoint-freq 0.1

Model Specifications

Base Model	Qwen/Qwen3-4B
Chat Template	Qwen/Qwen3-4B (use /chat/completions endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia H100

Deployment

pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head"

# Deploy with speculative decoding
vllm serve Qwen/Qwen3-4B \
    --speculative-config '{
        "model": "RedHatAI/Qwen3-4B-speculator.dflash2",
        "num_speculative_tokens": 8,
        "method": "dflash2"
    }'

Acceptance Rates

Per-position token acceptance rates across datasets:

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	89.3%	77.9%	67.0%	56.9%	48.2%	40.6%	33.9%	5.14
math_reasoning	92.5%	84.4%	76.2%	68.4%	61.0%	54.3%	47.8%	5.85
qa	80.1%	63.7%	50.7%	41.4%	33.9%	28.1%	23.5%	4.21
question	81.7%	64.8%	51.7%	41.8%	34.0%	27.9%	23.3%	4.25
rag	82.3%	65.8%	52.6%	42.0%	33.6%	27.1%	22.1%	4.25
summarization	74.5%	52.2%	35.5%	23.3%	15.7%	10.3%	6.6%	3.18
tool_call	81.0%	63.6%	49.9%	38.7%	30.5%	24.1%	19.5%	4.07
translation	84.6%	69.6%	56.5%	44.9%	35.4%	28.3%	22.6%	4.42
writing	81.6%	64.8%	51.6%	41.6%	33.9%	27.8%	23.2%	4.24