Muon vs MuonClip vs Muon+Adamw
text
Muon vs MuonClip vs Muon+Adamw
new_line
new_line
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fine‑tuning? We ran head‑to‑head tests on Qwen3‑4B (10k+ high‑quality instruction rows) to find out.
text
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fine‑tuning? We ran head‑to‑head tests on Qwen3‑4B (10k+ high‑quality instruction rows) to find out.
new_line
new_line
Short story: Pure Muon converged fastest at the start, but its gradient‑norm spikes made training unstable. MuonClip (Kimi K2’s clipping) stabilizes long pretraining runs, yet in our small‑scale fine‑tune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
text
Short story: Pure Muon converged fastest at the start, but its gradient‑norm spikes made training unstable. MuonClip (Kimi K2’s clipping) stabilizes long pretraining runs, yet in our small‑scale fine‑tune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
new_line
new_line
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
text
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
new_line
new_line
Next Step: scale to larger 
text
Next Step: scale to larger 
models/datasets
text
models/datasets
 to see if Muon’s spikes become catastrophic or if clipping wins out.
text
 to see if Muon’s spikes become catastrophic or if clipping wins out.
new_line
new_line
Full Blog Link: 
text
Full Blog Link: 
https://huggingface.co/blog/KingNish/optimizer-part1
https://huggingface.co/blog/KingNish/optimizer-part1
link
Muon vs MuonClip vs Muon+Adamw

Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fine‑tuning? We ran head‑to‑head tests on Qwen3‑4B (10k+ high‑quality instruction rows) to find out.

Short story: Pure Muon converged fastest at the start, but its gradient‑norm spikes made training unstable. MuonClip (Kimi K2’s clipping) stabilizes long pretraining runs, yet in our small‑scale fine‑tune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.

Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.

Next Step: scale to larger models/datasets to see if Muon’s spikes become catastrophic or if clipping wins out.

Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
🔥
KingNish Winky24 siddhantsingh victor Abhaykoul John6666 WarlordHermes BootsofLagrangian ChuGyouk