Qwen3.6-27B-DMTP

Blog | GitHub | SGLang PR

A DMTP speculative draft for Qwen3.6-27B, combining native MTP first-token prediction with DFlash2 parallel block drafting.

This repository contains draft weights only, not the 27B target model. The target model, its native MTP module, and the shared token embedding and language-model head must be supplied separately.

DMTP: MTP proposes the first candidate, then DFlash drafts the remaining block in parallel

Method

  1. Native MTP proposes the first candidate token, T1.

  2. DFlash2 receives [A, T1, MASK, MASK, MASK, MASK, MASK, MASK], where A is the last emitted token.

  3. The MTP representation that predicted T1 is normalized and passed through a rank-256 residual network. Its output is added only to the T1 input embedding:

    E(T1) <- E(T1) + W_up SiLU(W_down RMS(h_MTP))

  4. DFlash2 proposes T2 through T7. The target verifies all seven candidates.

The A embedding, six MASK embeddings, and original target-context feature path remain unchanged. The residual output projection was zero-initialized at the start of training.

Model and Training

  • Five draft layers; hidden size 5120; intermediate size 6144.
  • Target features from layers 0, 15, 30, 45, and 60.
  • DFlash2 convolution modules and rank-256 candidate selector, top-k 16.
  • BF16 weights.
  • Dataset: Huang2020/qwen3.6-27B-reasoning-regen.
  • Maximum training sequence length 3072; global batch size 128; three epochs; final step 41,209.
  • Target and native MTP frozen during draft training. T1 uses the ground-truth token during training and an MTP proposal during inference.

Evaluation

We compare dual-anchor DMTP with a DFlash2 baseline trained with the same training configuration. Both models use token-level rejection sampling (RS) with seven draft proposals per verification round.

Evaluation uses TP4, concurrency up to 32, a 262,144-token context limit, and up to 28,000 output tokens, with natural EOS termination. Both models use temperature 1, top-p 1, unrestricted target top-k, and decode CUDA Graphs. Each task uses the same 64 fixed inputs for both models.

We report per-verify accepted length, aggregated over verification rounds rather than averaged over requests:

Lverify=1+βˆ‘rNaccepted drafts,rβˆ‘rNverify,r. L_{\mathrm{verify}} = 1 + \frac{\sum_r N_{\mathrm{accepted\ drafts},r}}{\sum_r N_{\mathrm{verify},r}}.

The extra 1 counts the guaranteed token advanced per verification round. Higher is better.

Task  DFlash2 Β· per-verify acc len    DMTP (dual anchor) Β· per-verify acc len  
Code3.96214.6022
Math4.29094.8871
MT-Bench4.00444.1826
Agent3.81974.6871
SWE2.92733.9427
Long-horizon3.19294.3310

Neither draft was trained on agent-specific data. DMTP nevertheless improves accepted length on Agent, SWE, and long-horizon tasks, suggesting that incorporating MTP's pretrained representations strengthens DFlash2's generalization to agentic workloads. These results measure drafting acceptance, not task-solving accuracy.

Quick Start

Install the DMTP branch from SGLang PR #50270 with matching FA3 dependencies:

git clone --depth 1 --branch feat/dmtp-upstream-20261113 \
  https://github.com/Sylvan820/sglang-dmtp.git
cd sglang-dmtp
pip install -e python

export SGLANG_DMTP_T1_RESIDUAL=1
export SGLANG_USE_SGL_FA3_KERNEL=1
export SGLANG_MTP_PREFIX_MTP_GRAPH=1
export SGLANG_MTP_PREFIX_MTP_GRAPH_IMPL=native_bridge
export SGLANG_MTP_PREFIX_MTP_GRAPH_CHECK=1
export SGLANG_MTP_PREFIX_ANCHOR_WINDOW_GRAPH=1
export SGLANG_MTP_PREFIX_CONDITIONER_GRAPH=1
export SGLANG_MTP_PREFIX_HEAD_CONDITIONER_GRAPH=1
export SGLANG_MTP_PREFIX_SKIP_UNUSED_LOGITS=1
export SGLANG_MTP_PREFIX_TARGET_ONLY=0
export SGLANG_MTP_PREFIX_HEAD_GRAPH=0
export SGLANG_DMTP_FA3_SKIP_DENSE_MASK=1
export SGLANG_MTP_PREFIX_PERF_PATH=''

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.6-27B \
  --speculative-draft-model-path Piggy12/Qwen3.6-27B-DMTP \
  --speculative-algorithm DFLASH \
  --speculative-num-draft-tokens 8 \
  --attention-backend fa3 \
  --speculative-draft-attention-backend fa3 \
  --tp-size 8 \
  --dtype bfloat16 \
  --page-size 1 \
  --max-running-requests 32 \
  --context-length 3072 \
  --max-total-tokens 65536 \
  --mem-fraction-static 0.75 \
  --disable-overlap-schedule \
  --disable-radix-cache \
  --disable-cuda-graph-padding \
  --cuda-graph-backend-decode full \
  --cuda-graph-backend-prefill disabled \
  --skip-server-warmup \
  --host 127.0.0.1 --port 39532 --random-seed 42

TP8, concurrency up to 32, context length 3072. Decode CUDA Graphs, MTP graphs, and the combined T1 graph are enabled; prefill stays eager. A configurable launcher is available as serve_dmtp.sh.

Integration and Performance

See the Blog for the method and the GitHub repository for implementation references, including architecture-specific experiments. With an AI coding tool, these references can help you quickly adapt DMTP to an existing inference framework. Check hidden-state alignment, conditioning, and cache management when porting it.

Inference performance is still being optimized. Scheduling, cache management, and CUDA Graph coverage need further refinement; speedup depends on the workload and runtime implementation.

Citation

If you find this work useful, please consider citing this technical blog:

@misc{shen2026dmtp,
  author = {Shen, Yuhao and Huang, Jianuo},
  title  = {{DMTP}: A Better First Token for Block Drafting via MTP},
  year   = {2026},
  note   = {Technical blog}
}
Downloads last month
84
Safetensors
Model size
1B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Piggy12/Qwen3.6-27B-DMTP

Base model

Qwen/Qwen3.6-27B
Finetuned
(440)
this model