GUIDE.md

Making your own vectors

This is the whole pipeline that produced the files in vectors/: fit a vector to a fine-tune, score it to pick its strength, export it for llama.cpp, and (optionally) build a merge's vector from its parents' vectors. Every command here was run to make this release.

What you need

• An Apple silicon Mac, for the code as shipped. The fit and the score run on MLX. They build one decoder layer at a time, so peak memory stays around 11 GB whatever the checkpoint size; measured on an M1 Max with 32 GB of unified memory. Nothing in the method needs MLX: it needs forward passes of both models over the same text, so it could be ported to PyTorch (NVIDIA or otherwise) or another engine. That port has not been written.
• Disk for two checkpoints: stock google/gemma-4-31B-it and the fine-tune, about 62.5 GB each in bf16. A LoRA fine-tune can be fitted from its adapter instead, which is a few hundred MB to a few GB.
• uv for Python. The dependencies are pinned in pyproject.toml; mlx-lm is pinned exactly, because its Gemma 4 layer code is what runs each layer.
• llama.cpp (or ik_llama.cpp) to use the result. See the README for launch flags.

Only Gemma 4 31B (dense, 60 layers) is supported. The 26B-A4B MoE shares its model_type, and the code refuses it.

git clone <this repo> ecce-vector && cd ecce-vector
uv sync                                     # creates .venv with the pinned dependencies
hf download google/gemma-4-31B-it --local-dir gemma-4-31B-it \
    --include "*.safetensors" "*.json" "tokenizer*" "chat_template.jinja"
hf download SomeAuthor/Some-Finetune-31B --local-dir Some-Finetune-31B \
    --include "*.safetensors" "*.json"

1. Fit

uv run python -m ecce_vector.fit --base gemma-4-31B-it --tune Some-Finetune-31B \
    --out fits --name some-finetune
# a LoRA release: fit from the adapter, applied on stock
uv run python -m ecce_vector.fit --base gemma-4-31B-it --adapter Some-LoRA \
    --out fits --name some-lora

About 5.5 minutes on an M1 Max, most of it reading weights. Run with --smoke first to time two layers.

For each of the 60 layers, the fit runs stock's layer and the fine-tune's layer on the fine-tune's own hidden states over 8 short prompts (4 roleplay, 4 novel prose; ecce_vector/data/fit_corpus.json) and averages the difference of their outputs. That average is the part of the fine-tune's edit that is the same at every token: all that a single vector per layer can carry. The tracking fit (vectors_track, the one released here) goes one step further. It runs stock plus the vectors fitted so far and fits each layer's vector to bring that steered stream back to the fine-tune's, so the drift that early layers' vectors cause is corrected as it builds up. It is fitted the way llama.cpp applies a vector: nothing at layer 0, and added at every position including the first.

It writes fits/some-finetune.npz (the vectors, and the fine-tune's final hidden states on both prompt splits, which the score compares against) and fits/some-finetune.json (per-layer diagnostics). The printed constant share is the fraction of the edit's energy that is the same at every token. It ranks how well a vector will carry a fine-tune about as well as anything tried (Spearman +0.76 over the releases here), but it misjudges some tunes one by one, so score anyway.

The fine-tunes a vector cannot carry at all are refused: an untied lm_head (the output head is not in the residual stream). A fine-tune that changed its embedding (so its tied output head) is fitted, and its own log-probs are saved beside the fit as NAME.logp.npz (about 1 GB), because stock's head reads its hidden states differently. How much that costs varies a lot: see the README's note on melody1437-v2.

2. Score, and pick the strength

uv run python -m ecce_vector.score --base gemma-4-31B-it --ref fits/some-finetune.npz \
    --vector fits/some-finetune.npz::vectors_track --alphas 0.5 0.75 1.0 1.25 1.5 \
    --out fits/some-finetune.score.json

About 5 minutes for five strengths. Each strength is one extra stream through stock's layers, computed as a difference from the stock stream, so extra arms are cheap.

The score runs stock with the vector added, on 16 prompts (the 8 fit prompts and 8 held-out ones), and compares its next-token distributions with the fine-tune's at 128 reference positions per prompt:

• recovery = 1 − KL(tune ‖ stock + vector) / KL(tune ‖ stock). 1 is a perfect copy, 0 is no better than stock, and below 0 means the vector moves stock away from the fine-tune.
• top-1 recovery: the same for the share of positions where the fine-tune's most likely next token differs from stock's and the vector restores it.

Pick the strength on the fit split and quote the held-out split; the output's best does exactly that, for each vector scored. The released files use the tracking fit at that strength. Most tunes here pick 1.25.

A vector's score measures it against the fine-tune's full-precision weights on these prompts: both read the same text and their next-word predictions are compared at every position (teacher-forced), and neither writes anything. It says nothing about how replies read, and it charges a miss less where the fine-tune is unsure of its next token, which flatters tunes that make stock less certain. Read top-1 recovery beside it. In the README's blind test, judges reading the replies found only a faint hint of each fine-tune in its vector's output, including vectors scoring above 90%.

3. Export for llama.cpp

uv run python -m ecce_vector.export fits/some-finetune.npz::vectors_track \
    vectors/some-finetune.gguf --alpha 1.25 --name some-finetune

The GGUF holds alpha × c_L for layers 1–59 as F32 (llama.cpp rejects other types and never applies a layer-0 vector), so it loads at strength 1.0 and weighs 1.27 MB.

4. A merge's vector from its parents' vectors (advanced)

A mergekit merge in the linear family (linear, task_arithmetic, della_linear, dare_linear, …) on stock moves the model by a weighted sum of its parents' edits, so the merge's vector is the same weighted sum of the parents' vectors, layer by layer. bake reads the recipe exactly as mergekit resolves it (gradients across depth, per-module filters, normalisation, lambda) and writes one file:

uv run python -m ecce_vector.bake recipe.yaml vectors/my-merge.gguf \
    --parent ./ReadyArt/gemma-4-31B-it-scotoma-2=vectors/scotoma-2.gguf \
    --parent ./Blazed-Forge/Gemma-4-Giftige-Blume-31B-v1=vectors/giftige-blume-v1.gguf \
    --parent ./zerofata/G4-MeroMero-v2-31B=vectors/meromero-v2.gguf

Each --parent maps a model: entry of the recipe, spelled exactly as the recipe spells it, to that parent's vector. examples/schattenblume-recipe.yaml is the real recipe of Nimbz/Schattenblume-31B. The file baked from it recovers 0.894 of the merge, against 0.899 for a vector fitted to the merge's own weights: no model weights needed.

Why bake rather than blend at launch? llama.cpp's --control-vector-scaled takes one weight per file, and a recipe usually weights each parent differently at different depths. For schattenblume, one constant weight per parent (the recipe's weights averaged over layers) recovers only 0.844. Blending at launch is still the easy way to invent a mix, and you can score any blend:

uv run python -m ecce_vector.score --base gemma-4-31B-it --ref fits/my-merge.npz \
    --gguf vectors/my-merge.gguf --blend vectors/a.gguf:0.5,vectors/b.gguf:0.5

Refused: methods that are not a weighted sum (slerp, model_stock, arcee_fusion, …), a base other than stock, and a recipe that weights a layer's attention and MLP differently (one vector per layer cannot express that). ties/dare_ties/della are baked without their sign-election step, with a warning; that case is untested.

5. Generate comparisons

uv run python -m ecce_vector.examples --server llama-server --model gemma-4-31B-it-Q4_K_M.gguf \
    --prompts examples/prompts.json --out runs/stock
uv run python -m ecce_vector.examples --server llama-server --model gemma-4-31B-it-Q4_K_M.gguf \
    --prompts examples/prompts.json --out runs/some-finetune \
    --control-vector-scaled vectors/some-finetune.gguf:1.0

One llama-server per run (control vectors are fixed at launch), Google's recommended sampling for Gemma 4 (temperature 1.0, top-k 64, top-p 0.95, nothing else), a fixed seed, and llama-server's own prompt and generation timings saved beside each reply in run.json.

Reproducing this release

The prompts are fixed (ecce_vector/data/fit_corpus.json holds the exact token ids), and the fit and score are deterministic on a given MLX build. The code in this repository was checked against the research code that produced the released numbers:

check	result
the corpus module rebuilds the original token arrays, masks and scored rows	identical
bake on the schattenblume recipe against the original baker	identical (0.0 difference)
score of schattenblume's vector / its recipe bake, held out	0.89876 / 0.89388, against 0.89876 / 0.89386
fit of storymaxxed2 from its released weights against the stored fit	bit-identical, every layer

Details in metrics/package-checks.md.