This recent paper points to an explanation for the unreasonable effectiveness of Frankenmerges: 
text
This recent paper points to an explanation for the unreasonable effectiveness of Frankenmerges: 
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth
  Approach (2502.05171)
https://huggingface.co/papers/2502.05171
2502.05171
paper
resource
https://huggingface.co/papers/2502.05171
new_line
new_line
Specifically, the duplication of layers in Frankenmerges serves a purpose similar to what occurs in their recurrent-depth architecture. Successful frankenmerges that operate without additional fine-tuning are able to recover or "heal" from any damage due to abrupt transitions between layer blocks. Operational replicated layer blocks can provide functional benefits grounded in latent reasoning. Frankenmerges can also result in hybrid reasoning, by splicing together the latent reasoning of different models.
text
Specifically, the duplication of layers in Frankenmerges serves a purpose similar to what occurs in their recurrent-depth architecture. Successful frankenmerges that operate without additional fine-tuning are able to recover or "heal" from any damage due to abrupt transitions between layer blocks. Operational replicated layer blocks can provide functional benefits grounded in latent reasoning. Frankenmerges can also result in hybrid reasoning, by splicing together the latent reasoning of different models.
new_line
new_line
Back in April 2024, I was able to duplicate a few layers in the Llama 3 8B model, turning it into a 9B model, without harming benchmarks significantly, despite any transition damage.
text
Back in April 2024, I was able to duplicate a few layers in the Llama 3 8B model, turning it into a 9B model, without harming benchmarks significantly, despite any transition damage.
new_line
https://huggingface.co/grimjim/llama-3-experiment-v1-9B
grimjim/llama-3-experiment-v1-9B
model
resource
https://huggingface.co/grimjim/llama-3-experiment-v1-9B
new_line
My informal experimentation suggested that latent reasoning circuits could occupy continguous stacks of 2-4 layers, though the result was highly sensitive to the choice of transition location between layers.
text
My informal experimentation suggested that latent reasoning circuits could occupy continguous stacks of 2-4 layers, though the result was highly sensitive to the choice of transition location between layers.
This recent paper points to an explanation for the unreasonable effectiveness of Frankenmerges: https://huggingface.co/papers/2502.05171

Specifically, the duplication of layers in Frankenmerges serves a purpose similar to what occurs in their recurrent-depth architecture. Successful frankenmerges that operate without additional fine-tuning are able to recover or "heal" from any damage due to abrupt transitions between layer blocks. Operational replicated layer blocks can provide functional benefits grounded in latent reasoning. Frankenmerges can also result in hybrid reasoning, by splicing together the latent reasoning of different models.

Back in April 2024, I was able to duplicate a few layers in the Llama 3 8B model, turning it into a 9B model, without harming benchmarks significantly, despite any transition damage.
https://huggingface.co/grimjim/llama-3-experiment-v1-9B
My informal experimentation suggested that latent reasoning circuits could occupy continguous stacks of 2-4 layers, though the result was highly sensitive to the choice of transition location between layers.
👍
John6666 muhtasham FiditeNemini dantezxcd
🚀
John6666 spuun dantezxcd
🔥
John6666
❤️
nicolollo
66f4dcba00d2de86872a126f
https://cdn-avatars.huggingface.co/v1/production/uploads/66f4dcba00d2de86872a126f/4SG4Jch3ntQEjYGF09yow.jpeg
Jonathan Harrison
Raiff1982
user