Safety Alignment Collapses Without apply_chat_template(): An Empirical Study
text
Safety Alignment Collapses Without apply_chat_template(): An Empirical Study
new_line
new_line
new_line
new_line
This weekend, I ran an experiment on the safety alignment of several small-scale open models (Qwen2.5, Qwen3, Gemma-3, SmolLM). My objective was to measure the robustness of refusal mechanisms when deviating from canonical chat templates.
text
This weekend, I ran an experiment on the safety alignment of several small-scale open models (Qwen2.5, Qwen3, Gemma-3, SmolLM). My objective was to measure the robustness of refusal mechanisms when deviating from canonical chat templates.
new_line
new_line
The finding: Safety guarantees effectively collapse when apply_chat_template() is omitted.
text
The finding: Safety guarantees effectively collapse when apply_chat_template() is omitted.
new_line
new_line
new_line
METHODOLOGY
text
METHODOLOGY
new_line
new_line
I evaluated models in two states:
text
I evaluated models in two states:
new_line
new_line
• In-Distribution: Input wrapped in standard <|im_start|> instruction tokens
text
• In-Distribution: Input wrapped in standard <|im_start|> instruction tokens
new_line
• Out-of-Distribution: Input provided as a raw string
text
• Out-of-Distribution: Input provided as a raw string
new_line
new_line
For scalable evaluation, I used Qwen3Guard-Gen-4B as an automated judge, classifying responses as Safe, Unsafe, or Controversial.
text
For scalable evaluation, I used Qwen3Guard-Gen-4B as an automated judge, classifying responses as Safe, Unsafe, or Controversial.
new_line
new_line
new_line
KEY FINDINGS: REFUSAL COLLAPSE
text
KEY FINDINGS: REFUSAL COLLAPSE
new_line
new_line
When "Assistant" formatting tokens are removed, models undergo a distributional shift—reverting from a helpful assistant to a raw completion engine.
text
When "Assistant" formatting tokens are removed, models undergo a distributional shift—reverting from a helpful assistant to a raw completion engine.
new_line
new_line
Gemma-3: 100% refusal (aligned) → 60% (raw)
text
Gemma-3: 100% refusal (aligned) → 60% (raw)
new_line
Qwen3: 80% refusal (aligned) → 40% (raw)
text
Qwen3: 80% refusal (aligned) → 40% (raw)
new_line
SmolLM2-1.7B: 0% → 0% (no safety tuning to begin with)
text
SmolLM2-1.7B: 0% → 0% (no safety tuning to begin with)
new_line
new_line
new_line
QUALITATIVE FAILURES
text
QUALITATIVE FAILURES
new_line
new_line
The failure modes were not minor. Without the template, models that previously refused harmful queries began outputting high-fidelity harmful content:
text
The failure modes were not minor. Without the template, models that previously refused harmful queries began outputting high-fidelity harmful content:
new_line
new_line
• Explosives: Qwen3 generated technical detonation mechanisms
text
• Explosives: Qwen3 generated technical detonation mechanisms
new_line
• Explicit content: Requests flatly refused by aligned models were fulfilled with graphic narratives by unaligned versions
text
• Explicit content: Requests flatly refused by aligned models were fulfilled with graphic narratives by unaligned versions
new_line
new_line
This suggests instruction tuning acts as a "soft mask" over the pre-training distribution rather than removing harmful latent knowledge.
text
This suggests instruction tuning acts as a "soft mask" over the pre-training distribution rather than removing harmful latent knowledge.
new_line
new_line
new_line
👉 Read the full analysis: 
text
👉 Read the full analysis: 
https://teendifferent.substack.com/p/apply_chat_template-is-the-safety
https://teendifferent.substack.com/p/apply_chat_template-is-the-safety
link
new_line
💻 Reproduction Code: 
text
💻 Reproduction Code: 
https://github.com/REDDITARUN/experments/tree/main/llm_alignment
https://github.com/REDDITARUN/experments/tree/main/llm_alignment
link
Safety Alignment Collapses Without apply_chat_template(): An Empirical Study



This weekend, I ran an experiment on the safety alignment of several small-scale open models (Qwen2.5, Qwen3, Gemma-3, SmolLM). My objective was to measure the robustness of refusal mechanisms when deviating from canonical chat templates.

The finding: Safety guarantees effectively collapse when apply_chat_template() is omitted.


METHODOLOGY

I evaluated models in two states:

• In-Distribution: Input wrapped in standard <|im_start|> instruction tokens
• Out-of-Distribution: Input provided as a raw string

For scalable evaluation, I used Qwen3Guard-Gen-4B as an automated judge, classifying responses as Safe, Unsafe, or Controversial.


KEY FINDINGS: REFUSAL COLLAPSE

When "Assistant" formatting tokens are removed, models undergo a distributional shift—reverting from a helpful assistant to a raw completion engine.

Gemma-3: 100% refusal (aligned) → 60% (raw)
Qwen3: 80% refusal (aligned) → 40% (raw)
SmolLM2-1.7B: 0% → 0% (no safety tuning to begin with)


QUALITATIVE FAILURES

The failure modes were not minor. Without the template, models that previously refused harmful queries began outputting high-fidelity harmful content:

• Explosives: Qwen3 generated technical detonation mechanisms
• Explicit content: Requests flatly refused by aligned models were fulfilled with graphic narratives by unaligned versions

This suggests instruction tuning acts as a "soft mask" over the pre-training distribution rather than removing harmful latent knowledge.


👉 Read the full analysis: https://teendifferent.substack.com/p/apply_chat_template-is-the-safety
💻 Reproduction Code: https://github.com/REDDITARUN/experments/tree/main/llm_alignment
🔥
unmodeled-tyler John6666
6758a9850e3fff481964ca6d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/EolfJfjW25hC4Bt_hCPq8.png
Jean Louis
JLouisBiz
user
686c460ba3fc457ad14ab6f8
https://cdn-avatars.huggingface.co/v1/production/uploads/686c460ba3fc457ad14ab6f8/QFa608PO_fSX8WCmi_T1s.jpeg
Tyler Williams
unmodeled-tyler
user