# AGENTS.md Guidance for a coding agent working in this repository: what is here, how to run it, how to add a vector, and the mistakes that are easy to make. ## What this repository is A collection of llama.cpp **control vectors** for `google/gemma-4-31B-it`, one per released fine-tune, each an approximation of what that fine-tune does to stock (`vectors/*.gguf`, 1.27 MB each), with metrics on how close each one gets (`metrics/`), example generations (`examples/`), and the code that made them (`ecce_vector/`). The README is the Hugging Face model card; `GUIDE.md` is the user guide to the code. The summary every edit must keep accurate: a vector reproduces most of the next-word shift of a small, broad style tune and much less of a large or token-dependent edit (README → *How close are they, by the numbers?*), top-token agreement is far lower than the KL figure suggests, and blind judges find only a faint hint of a fine-tune in its vector's replies (README → *Will you hear the real fine-tune?*). Do not round a result up in prose, and do not describe the vectors as reproducing their fine-tunes. ## Map | path | what | |---|---| | `ecce_vector/io.py` | safetensors reader (by byte offset, bf16 upcast), `Checkpoint`, PEFT `Adapter`, GGUF read/write | | `ecce_vector/corpus.py`, `data/fit_corpus.json` | the 16 fit / held-out prompts, stored as exact token ids | | `ecce_vector/model.py` | one Gemma 4 decoder layer at a time on MLX (`mlx_lm.models.gemma4_text`), the `Side` projection wrapper, the output head and KL arithmetic | | `ecce_vector/fit.py` | fit plain and tracking vectors from a fine-tune's weights or adapter | | `ecce_vector/score.py` | recovery and top-1 recovery of vectors, released GGUFs, or blends | | `ecce_vector/export.py` | fitted vector to llama.cpp GGUF with the strength baked in | | `ecce_vector/bake.py` | a mergekit recipe applied to parent vectors, per layer | | `ecce_vector/examples.py` | llama-server generations with timings | | `vectors/` | the released GGUFs; `vectors/merges/` holds baked merges | | `metrics/metrics.json`, `.csv` | one row per vector: recovery, top-1 recovery, KL from stock, strength, share, size, source repo, licence, flags | | `examples/` | `prompts.json` (from `IlyaGusev/gpt_roleplay_realm`, CC-BY-4.0), `runs//` outputs and `run.json` timings, `EXAMPLES.md` | ## Running things - Python through **uv**: `uv sync`, then `uv run python -m ecce_vector. --help`. The code is MLX, so as shipped it runs on Apple silicon only. Never run two MLX or llama.cpp GPU jobs at once on one machine: they compete for unified memory, and the timings in `run.json` become meaningless. - A fit takes about 5.5 minutes and a five-strength score about 5 minutes on an M1 Max (32 GB); both peak near 11 GB. `--smoke` on `fit` times two layers. - The stock and fine-tune checkpoints are ~62.5 GB each. Fitting from a PEFT adapter needs only stock plus the adapter. ## Adding a vector for a new fine-tune 1. Fit: `python -m ecce_vector.fit --base --tune --out fits --name ` (`--adapter ` for a LoRA release). 2. Score over a strength grid: `python -m ecce_vector.score --base --ref fits/.npz --vector fits/.npz::vectors_track --alphas 0.25 0.5 0.75 1.0 1.25 1.5 --out fits/.score.json`. The strength is the one with the lowest **fit-split** KL (`best[]` in the output); report its **held-out** numbers, never the fit split's. 3. Export at that strength: `python -m ecce_vector.export fits/.npz::vectors_track vectors/.gguf --alpha --name `. 4. Add its row to `metrics/metrics.json` and `.csv` and the README table, with the source repo, its declared licence, and the flags below. Generate its examples with the same seed and sampler as the rest. ## Invariants - **Gemma 4 31B dense only**: 60 decoder layers, no MoE. The 26B-A4B shares `model_type: gemma4`; the code asserts the layer count and refuses it. The vision tower's blocks reuse the language stack's module names, so select layers by the language-model prefix, never by module name alone (`Checkpoint` does). - **llama.cpp semantics**: no vector at layer 0, added after the layer at every position including BOS. The tracking fit is fitted that way; do not "fix" either. - **Do not change the fit corpus** (`data/fit_corpus.json`) or the scoring positions without refitting and rescoring every vector: the metrics are only comparable on one prompt set. - **Pin `mlx-lm`.** The fit runs `gemma4_text.DecoderLayer` directly; a version with different layer internals changes the numbers. - **GGUF tensors stay F32** (`direction.1` … `direction.59`); llama.cpp rejects other types. - Do not publish to Hugging Face or push anywhere without the owner's explicit approval. ## Flags in the metrics, and what they mean | flag | meaning | |---|---| | `own_head` | the fine-tune changed its embedding, which is also its (tied) output head, so it is scored against its own log-probs; a vector acts on the hidden stream only and cannot follow a changed head. It can cost little (`equinox`, about 2%) or most of the shift (`melody1437-v2`, whose stream is carried at 0.87 and its output at 0.35) | | `adapter_scored` | fitted and scored against stock plus the release's published adapter, which reconstructs it | | `low_signal` | the fine-tune barely moves stock on these prompts (KL below 0.1), so its recovery is measured on almost no signal; for a Heretic/ARA abliteration that is because its main edit, refusal removal, never fires in roleplay | | `grid_edge` | the chosen strength is the lowest on the grid scored; the optimum may be lower | ## Checks to rerun after changing the code The package was validated against the research code that produced the release (`metrics/package-checks.md`): the corpus rebuilds identical token arrays; `bake` matches the original baker exactly; `score` reproduces `schattenblume`'s released recovery (0.899) and its recipe bake's (0.894); `fit` of `storymaxxed2` matches its stored fit. If you change `model.py`, `fit.py` or `score.py`, rerun at least the `score` check: a held-out recovery that moves by more than about 2e-4 means the numerics changed. ## Traps - A **negative recovery** is a real result (the vector moves stock away from the fine-tune), not a bug. - **KL recovery flatters broadening tunes**: a tune that makes stock less certain is charged less for each miss. Always read top-1 recovery beside it. - A fine-tune with an **untied `lm_head`** is refused, deliberately: its output head is not in the stream. - Recovery is teacher-forced on 16 prompts: both models read the same text and their next-token predictions are compared, with no generation. It is not a measure of generation quality or of whether the output sounds like the fine-tune, and a small difference between two vectors (under ~0.02) is inside the prompt-to-prompt noise. Blind judges reading the replies (README → *Will you hear the real fine-tune?*) found only a faint, fine-tune-specific hint in vectors' output, so never describe a vector as reproducing its fine-tune on the strength of its recovery.