PrometheanStudio/talos-mini-254k-oasst1

Talos Mini 254K — OASST1

Parameters: 254,272

Talos Mini is a 254,272-parameter experimental decoder-only language model trained on a 2,000-example subset of OpenAssistant (OASST1).

Model specifications

• Parameters: 254,272
• Layers: 2
• Hidden size: 64
• Attention heads: 4
• KV heads: 2
• Attention: GQA
• Vocabulary size: 1,024
• Maximum sequence length: 512
• Tokenizer: Talos-native BPE

Training

• Dataset: OpenAssistant / OASST1
• Examples: 2,000
• Training split: 1,800
• Validation split: 200
• Sequence length: 64
• Hardware: NVIDIA Tesla T4
• Final training step: 7,680
• Final training loss: 1.7859
• Final validation loss: 1.9177

Training progress

Step	Train Loss	Validation Loss
480	3.0289	2.9331
2400	2.2165	2.3276
3840	2.0245	2.1350
7680	1.7859	1.9177

Files

• step-7680.pt — trained Talos Mini checkpoint
• tokenizer.json — Talos tokenizer
• metrics.json — training metrics
• config.json — model configuration and parameter metadata

Limitations

This is a very small experimental model. Its free-running generations are often incoherent or repetitive, and the model should not be compared directly with modern production-scale language models.

The checkpoint is primarily a research and scaling baseline for Talos.