bm-xlm-roberta-base

XLM-RoBERTa base adapted to Bambara (Bamanankan, bm): continued masked-language-model pretraining with a vocabulary-extended tokenizer that adds Bambara words and the characters Ɛ Ɔ Ɲ Ŋ.

This is a masked-LM checkpoint, not a task model. It predicts <mask> and produces contextual representations; use it as the initialisation for fine-tuning on classification, token tagging, QA, or as a retrieval encoder.

Usage

The mask token is <mask>, not [MASK].

from transformers import pipeline

fill = pipeline("fill-mask", model="djelia/bm-xlm-roberta-base")

for p in fill("Mɔgɔw bɛɛ bɛ bange hɔrɔnya <mask> danbe la."):
    print(f"{p['score']:.4f}  {p['token_str']!r}")

Always load the tokenizer from this repo — token ids are not interchangeable with stock XLM-R.

Architecture

Class	XLMRobertaForMaskedLM (encoder-only)
Parameters	280,924,397 (F32)
Layers / hidden / heads / FFN	12 / 768 / 12 / 3072
Max sequence length	512
vocab_size	253,421
Special tokens	<s>=0, <pad>=1, </s>=2, <unk>=3, <mask>=253420

Notes

The checkpoint carries no pooler weights, so pooler_output from AutoModel is randomly initialised. Take last_hidden_state and pool it yourself:

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("djelia/bm-xlm-roberta-base")
encoder = AutoModel.from_pretrained("djelia/bm-xlm-roberta-base").eval()

hidden = encoder(**tokenizer(["Aw ni ce."], return_tensors="pt")).last_hidden_state

Mean-pooled token vectors are not sentence embeddings — no contrastive objective was applied here. Fine-tune with a sentence-similarity objective if you need retrieval.