AGENTS.md

AGENTS.md

Guidance for a coding agent working in this repository: what is here, how to run it, how to add a vector, and the mistakes that are easy to make.

What this repository is

A collection of llama.cpp control vectors for google/gemma-4-31B-it, one per released fine-tune, each an approximation of what that fine-tune does to stock (vectors/*.gguf, 1.27 MB each), with metrics on how close each one gets (metrics/), example generations (examples/), and the code that made them (ecce_vector/). The README is the Hugging Face model card; GUIDE.md is the user guide to the code.

The summary every edit must keep accurate: a vector reproduces most of the next-word shift of a small, broad style tune and much less of a large or token-dependent edit (README → How close are they, by the numbers?), top-token agreement is far lower than the KL figure suggests, and blind judges find only a faint hint of a fine-tune in its vector's replies (README → Will you hear the real fine-tune?). Do not round a result up in prose, and do not describe the vectors as reproducing their fine-tunes.

Map

path	what
ecce_vector/io.py	safetensors reader (by byte offset, bf16 upcast), Checkpoint, PEFT Adapter, GGUF read/write
ecce_vector/corpus.py, data/fit_corpus.json	the 16 fit / held-out prompts, stored as exact token ids
ecce_vector/model.py	one Gemma 4 decoder layer at a time on MLX (mlx_lm.models.gemma4_text), the Side projection wrapper, the output head and KL arithmetic
ecce_vector/fit.py	fit plain and tracking vectors from a fine-tune's weights or adapter
ecce_vector/score.py	recovery and top-1 recovery of vectors, released GGUFs, or blends
ecce_vector/export.py	fitted vector to llama.cpp GGUF with the strength baked in
ecce_vector/bake.py	a mergekit recipe applied to parent vectors, per layer
ecce_vector/examples.py	llama-server generations with timings
vectors/	the released GGUFs; vectors/merges/ holds baked merges
metrics/metrics.json, .csv	one row per vector: recovery, top-1 recovery, KL from stock, strength, share, size, source repo, licence, flags
examples/	prompts.json (from IlyaGusev/gpt_roleplay_realm, CC-BY-4.0), runs/<arm>/ outputs and run.json timings, EXAMPLES.md

Running things

• Python through uv: uv sync, then uv run python -m ecce_vector.<tool> --help. The code is MLX, so as shipped it runs on Apple silicon only. Never run two MLX or llama.cpp GPU jobs at once on one machine: they compete for unified memory, and the timings in run.json become meaningless.
• A fit takes about 5.5 minutes and a five-strength score about 5 minutes on an M1 Max (32 GB); both peak near 11 GB. --smoke on fit times two layers.
• The stock and fine-tune checkpoints are ~62.5 GB each. Fitting from a PEFT adapter needs only stock plus the adapter.

Adding a vector for a new fine-tune

1. Fit: python -m ecce_vector.fit --base <stock dir> --tune <tune dir> --out fits --name <name> (--adapter <dir> for a LoRA release).
2. Score over a strength grid: python -m ecce_vector.score --base <stock> --ref fits/<name>.npz --vector fits/<name>.npz::vectors_track --alphas 0.25 0.5 0.75 1.0 1.25 1.5 --out fits/<name>.score.json. The strength is the one with the lowest fit-split KL (best[<vector>] in the output); report its held-out numbers, never the fit split's.
3. Export at that strength: python -m ecce_vector.export fits/<name>.npz::vectors_track vectors/<name>.gguf --alpha <a> --name <name>.
4. Add its row to metrics/metrics.json and .csv and the README table, with the source repo, its declared licence, and the flags below. Generate its examples with the same seed and sampler as the rest.

Invariants

• Gemma 4 31B dense only: 60 decoder layers, no MoE. The 26B-A4B shares model_type: gemma4; the code asserts the layer count and refuses it. The vision tower's blocks reuse the language stack's module names, so select layers by the language-model prefix, never by module name alone (Checkpoint does).
• llama.cpp semantics: no vector at layer 0, added after the layer at every position including BOS. The tracking fit is fitted that way; do not "fix" either.
• Do not change the fit corpus (data/fit_corpus.json) or the scoring positions without refitting and rescoring every vector: the metrics are only comparable on one prompt set.
• Pin mlx-lm. The fit runs gemma4_text.DecoderLayer directly; a version with different layer internals changes the numbers.
• GGUF tensors stay F32 (direction.1 … direction.59); llama.cpp rejects other types.
• Do not publish to Hugging Face or push anywhere without the owner's explicit approval.

Flags in the metrics, and what they mean

flag	meaning
own_head	the fine-tune changed its embedding, which is also its (tied) output head, so it is scored against its own log-probs; a vector acts on the hidden stream only and cannot follow a changed head. It can cost little (equinox, about 2%) or most of the shift (melody1437-v2, whose stream is carried at 0.87 and its output at 0.35)
adapter_scored	fitted and scored against stock plus the release's published adapter, which reconstructs it
low_signal	the fine-tune barely moves stock on these prompts (KL below 0.1), so its recovery is measured on almost no signal; for a Heretic/ARA abliteration that is because its main edit, refusal removal, never fires in roleplay
grid_edge	the chosen strength is the lowest on the grid scored; the optimum may be lower

Checks to rerun after changing the code

The package was validated against the research code that produced the release (metrics/package-checks.md): the corpus rebuilds identical token arrays; bake matches the original baker exactly; score reproduces schattenblume's released recovery (0.899) and its recipe bake's (0.894); fit of storymaxxed2 matches its stored fit. If you change model.py, fit.py or score.py, rerun at least the score check: a held-out recovery that moves by more than about 2e-4 means the numerics changed.

Traps

• A negative recovery is a real result (the vector moves stock away from the fine-tune), not a bug.
• KL recovery flatters broadening tunes: a tune that makes stock less certain is charged less for each miss. Always read top-1 recovery beside it.
• A fine-tune with an untied lm_head is refused, deliberately: its output head is not in the stream.
• Recovery is teacher-forced on 16 prompts: both models read the same text and their next-token predictions are compared, with no generation. It is not a measure of generation quality or of whether the output sounds like the fine-tune, and a small difference between two vectors (under ~0.02) is inside the prompt-to-prompt noise. Blind judges reading the replies (README → Will you hear the real fine-tune?) found only a faint, fine-tune-specific hint in vectors' output, so never describe a vector as reproducing its fine-tune on the strength of its recovery.