Layer-Feedback Transformer (LFT)

Most transformer language models process hidden states in a strictly feed-forward sequence. This only evaluates each layer once per foward pass. After information leaves a layer, that layer does not process the representation produced by deeper layers. We introduce the Layer-Feedback Transformer (LFT) architecture, a recurrent depth-routing architecture in which adjacent Transformer layers are reused before computation proceeds deeper into the network; we test whether this through the same layers produces higher benchmark scores than a standard forward pass, without adding any new parameters.

For example on a five-layer model, the execution path becomes:

L1 → L2 → L1 → L2 → L3 → L2 → L3 → L4 → L3 → L4 → L5

Instead of:

L1 → L2 → L3 → L4 → L5

This may provide higher quality at the cost of more compute, with no additional parameters.

Therefore a five-layer LFT model only contains five unique Transformer layers, but performs eleven layer executions.

Architecture

Let Li denote Transformer block i.

A conventional Transformer performs:

h1 = L1(h0)
h2 = L2(h1)
h3 = L3(h2)
...

In that conventional transformer each layer is only used once.

Our LFT architecture introduces a local feedback cycle between adjacent layers.

For two neighboring blocks Li and Li+1:

h1 = Li(h)
h2 = Li+1(h1)
h3 = Li(h2)
h4 = Li+1(h3)

The second pass allows the earlier layer to process a representation that has already undergone deeper transformation.

Then the refined representation is passed onward.

Here is an example of the execution sequence on a five layer model.

L1
L2 → L1 → L2
L3 → L2 → L3
L4 → L3 → L4
L5

Expanded directly:

x = L1(x)
x = L2(x)
x = L1(x)
x = L2(x)
x = L3(x)
x = L2(x)
x = L3(x)
x = L4(x)
x = L3(x)
x = L4(x)
x = L5(x)

The compact implementation is:

x = self.layers[0](x)
for i in range(1, len(self.layers)):
    x = self.layers[i](x)
    if i < len(self.layers) - 1:
        x = self.layers[i - 1](x)
        x = self.layers[i](x)

The Transformer layers themselves do not require any modification.

Our implementation includes:

• pre-normalization
• RMSNorm
• causal self-attention
• RoPE
• SwiGLU
• residual connections
• tied input and output embeddings

A block is otherwise conventional:

def forward(self, x, cos, sin):
    x = x + self.attn(self.norm1(x), cos, sin)
    x = x + self.mlp(self.norm2(x))
    return x

What defines the LFT architecture is its depth execution graph: this controls how many times the hidden state is transformed, and it does so without adding parameters.

---

Parameter Depth vs. Execution Depth

Conventional Transformers usually couple two different quantities:

Parameter depth

The number of unique Transformer blocks stored in the model.

Execution depth

The number of Transformer-block transformations applied sequentially to a hidden state.

For a conventional Transformer, these are normally equal.

A five-layer Transformer has:

5 unique blocks
5 block executions

A five-layer LFT has:

5 unique blocks
11 block executions

Our LFT architecture therefore gets more block executions per parameter than a standard Transformer of the same size.

---

Computational Cost

For N unique Transformer layers, this LFT routing performs:

E(N) = 3N − 4

block executions.

The execution multiplier relative to a conventional N-layer Transformer is:

(3N − 4) / N = 3 − 4/N

Here are some examples:

Bold benchmark values indicate the better result within each same-parameter Standard/LFT pair.
Tied results are bold for both models.
Base Bench category values use weighted accuracy.

Controlled Experimental Setup

To isolate the effect of Layer-Feedback routing, we trained matched Standard and LFT models at three parameter scales.

Each Standard/LFT pair uses the same:

• parameter count
• number of unique Transformer layers
• hidden size
• attention configuration
• FFN size
• initialization seed
• tokenizer
• training data
• token order
• context length
• batch size
• optimizer
• learning-rate schedule
• training-token budget

The only architectural difference between each pair is the execution path through the Transformer layers.

All models use a single shared 3,072-token byte-level BPE tokenizer and a context length of 768 tokens.

All controlled 500M-token experiments were trained on the same FineWeb-Edu token stream.

The tested model configurations were:

This makes the comparison parameter-matched, parameter-matched but not compute-matched.compute-matched, so LFT's gains may partly reflect the extra computation rather than architectural advantage. LFT deliberately performs more Transformer block executions for each token.

---

Results

At 2.5M parameters, the Standard Transformer remains stronger overall. Its Base Bench 1.1 accuracy is 32.57%, compared with 30.86% for LFT. The Standard model also performs slightly better on PIQA, ARC-Easy, and HellaSwag.

At 10M parameters, however, LFT takes the lead. The LFT model increases Base Bench 1.1 accuracy from 32.57% to 36.86%, an improvement of 4.29 percentage points.

The largest differences for the 10M model appear in several Base Bench categories:

The 10M LFT model answers 129 of 350 Base Bench questions correctly, compared with 114 for the Standard model.

At 25M parameters, LFT also finishes ahead overall, although by a smaller margin. Base Bench accuracy increases from 35.43% to 36.57%, while LM-Eval also shows small improvements on all three evaluated tasks:

The average of these three LM-Eval benchmarks increases from 38.64% to 38.84%.

These results suggest that the usefulness of Layer-Feedback may depend on needs both sufficient model capacity and enough training dynamics tokens to pay off, rather than simply producing an immediate improvement at every scale.

---

Training-Dependent Behavior

Our earlier 10M parameter model on 200M tokens produced:

At a training budget of only 200M tokens, LFT was clearly behind.

When we trained the same parameter size from initialization for 500M tokens, the result reversed:

The LFT difference therefore changed from (showing LFT needs more training to pay off):

200M tokens: -2.57 percentage points
500M tokens: +4.29 percentage points

While our LFT architecture doesen't add parameters, its additional layer executions are not computationally free.

For N unique layers, LFT performs:

E(N) = 3N − 4

For the models tested here:

Therefore the current experiments compare models at equal parameter count and equal training-token count, but not equal FLOPs, so LFT's apparent gains may partly reflect more compute rather than a better architecture.

---

---