I played around with the new RXTX paper (XX^T) and was able to train nanogpt with 4x4 RXTX matmuls in both attention layer and optimizer🤕 
text
I played around with the new RXTX paper (XX^T) and was able to train nanogpt with 4x4 RXTX matmuls in both attention layer and optimizer🤕 
new_line
It just works (well I had to add some guardrails) but still saves 5% of memory usage:
text
It just works (well I had to add some guardrails) but still saves 5% of memory usage:
new_line
The Patch:
text
The Patch:
new_line
- Computes attention scores with a 4x4 blockwise RXTX matmuls (no pytorch dot prod)
text
- Computes attention scores with a 4x4 blockwise RXTX matmuls (no pytorch dot prod)
new_line
- Handles arbitrary sequence lengths by padding to the nearest multiple of 4.
text
- Handles arbitrary sequence lengths by padding to the nearest multiple of 4.
new_line
- An RXTX variant of shampoo with params reshaped into 4x4 blocks during each optimizer step.
text
- An RXTX variant of shampoo with params reshaped into 4x4 blocks during each optimizer step.
new_line
- Uses 5% less ops
text
- Uses 5% less ops
new_line
Code: 
text
Code: 
https://github.com/Jaykef/ai-algorithms/blob/main/nanogpt-rxtx.ipynb
https://github.com/Jaykef/ai-algorithms/blob/main/nanogpt-rxtx.ipynb
link
new_line
Paper: 
text
Paper: 
https://arxiv.org/pdf/2505.09814
https://arxiv.org/pdf/2505.09814
link
I played around with the new RXTX paper (XX^T) and was able to train nanogpt with 4x4 RXTX matmuls in both attention layer and optimizer🤕 
It just works (well I had to add some guardrails) but still saves 5% of memory usage:
The Patch:
- Computes attention scores with a 4x4 blockwise RXTX matmuls (no pytorch dot prod)
- Handles arbitrary sequence lengths by padding to the nearest multiple of 4.
- An RXTX variant of shampoo with params reshaped into 4x4 blocks during each optimizer step.
- Uses 5% less ops
Code: https://github.com/Jaykef/ai-algorithms/blob/main/nanogpt-rxtx.ipynb
Paper: https://arxiv.org/pdf/2505.09814
👍
victor GrayGentleman zhang123123 AtAndDev Gatozu35 tiendung paresh2806
😎
tiendung