bge-small-euaiact

BAAI/bge-small-en-v1.5 fine-tuned for retrieval over one text: the EU AI Act, Regulation (EU) 2024/1689 as amended by Regulation (EU) 2026/1744 (consolidated text of 27 July 2026). It maps a question to the paragraph of the Act that answers it. 384 dimensions, 512 tokens, cosine similarity.

It belongs to eu-ai-act-rag, where the training script, the training questions and the evaluation live. The Act can be read paragraph by paragraph at https://aghasalim.github.io/eu-ai-act-rag/.

Use

Queries take the same instruction prefix as the base model. Passages take none.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("aghasalim/bge-small-euaiact")
prefix = "Represent this sentence for searching relevant passages: "
q = model.encode([prefix + "What fine applies to a prohibited practice?"],
                 normalize_embeddings=True)
docs = model.encode(["[CHAPTER XII - PENALTIES] Article 99(3) - Penalties\n3. ..."],
                    normalize_embeddings=True)
print(q @ docs.T)

In the project: EMBED_MODEL=aghasalim/bge-small-euaiact make index.

How it was trained

A local model (Qwen2.5-14B, 4-bit, through mlx-lm) wrote three questions for each of the 779 binding chunks of the 2026 text: a short search, a concrete scenario and one in everyday words. That gave 2,267 questions (train/synthetic_queries.jsonl). A tenth of the chunks were held out with their questions. The other 2,042 pairs were trained with MultipleNegativesRankingLoss (in-batch negatives, no duplicate passages in a batch) for 4 epochs at batch size 64, learning rate 2e-5, seed 13. It was one run with fixed settings. Nothing was tuned against the evaluation questions, and the script refuses to start if a synthetic question is also an evaluation question.

Results

Held-out synthetic questions (225, about chunks the model never trained on), dense search over all 994 chunks:

model recall at 6
bge-small-en-v1.5 60.0%
bge-small-euaiact 86.7%

Hand-written questions from the project's evaluation set (110 with an answer, 2026 text, paragraph chunks, k=6). Dense search is fused with BM25 here, so these are system numbers and not this model alone. The last column is the paired comparison with the first row on full recall: questions that got better / worse, and the exact two-sided sign test.

retrieval hit rate full recall multi-hop full recall better / worse (p)
hybrid, bge-small-en-v1.5 97.3% 84.5% 26.3% n/a
hybrid, bge-small-euaiact 99.1% 88.2% 36.8% 5 / 1 (0.219)
split search, bge-small-en-v1.5 97.3% 90.0% 57.9% 6 / 0 (0.031)
split search, bge-small-euaiact 99.1% 92.7% 63.2% 9 / 0 (0.004)

On its own, swapping the embedder is not a significant gain on 110 questions. With split search it is. make experiments in the repository reproduces the table.

Limits

  • It is adapted to one text. It has seen most of the passages the evaluation questions are about, with other questions attached, so these numbers say nothing about another law, or about this one after the next amendment.
  • The training questions come from one model and one prompt.
  • English only. The Act is equally valid in 24 languages.
  • The questions were written from the consolidated text of 27 July 2026. The consolidated text has no legal effect; only the Official Journal texts are authentic.
  • This is a student project and it isn't legal advice.

Licence and credit

MIT, like the base model. The corpus is Regulation (EU) 2024/1689 from the EU Publications Office, reused under Decision 2011/833/EU. Not affiliated with or endorsed by the EU.

Downloads last month
41
Safetensors
Model size
33.4M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for aghasalim/bge-small-euaiact

Finetuned
(561)
this model