Create README.md

README.md

@@ -0,0 +1,146 @@
+---
+pipeline_tag: audio-classification
+library_name: omar_rq
+license: cc-by-nc-sa-4.0
+tags:
+  - audio
+  - music
+  - self-supervised-learning
+  - audio-representation
+  - music-tagging
+  - pitch-estimation
+  - chord-recognition
+  - beat-tracking
+  - segmentation
+  - difficulty-estimation
+---
+
+# OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction
+
+**OMAR-RQ**  is an open-source foundation model for music audio understanding, presented in the paper [OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction](https://huggingface.co/papers/2507.03482).
+
+OMAR-RQ is trained with self-supervision via masked token classification methodologies using a large-scale dataset with over 330,000 hours of music audio. It offers powerful, multipurpose representations essential for advancing research in music information retrieval. The model achieves state-of-the-art performance among open self-supervised models across various tasks:
+*   **Music Tagging**
+*   **Pitch Estimation**
+*   **Chord Recognition**
+*   **Beat Tracking**
+*   **Segmentation**
+*   **Difficulty Estimation**
+
+For the full training, validation, and inference code, please refer to the [official GitHub repository](https://github.com/MTG/omar-rq).
+
+## Installation
+
+For embedding extraction or fine-tuning:
+
+```bash
+pip install .
+```
+
+For development including pre-training your own models:
+
+```bash
+pip install -e .[train]
+```
+
+## Inference
+
+You can load an OMAR-RQ model by specifying its Hugging Face model ID:
+
+```python
+import torch
+from omar_rq import get_model
+
+# Embedding extraction example
+x = torch.randn(1, 16000 * 4).cpu() # Example: 4 seconds of mono audio at 16kHz
+
+# Load a specific model, e.g., "mtg-upf/omar-rq-multifeature-25hz-fsq"
+model_id = "mtg-upf/omar-rq-multifeature-25hz-fsq" 
+model = get_model(model_id=model_id, device="cpu") # Use "cuda" if a GPU is available
+
+# Extract embeddings from layer 6
+embeddings = model.extract_embeddings(x, layers=[6])
+
+# Use the `model.eps` field to compute timestamps for the extracted embeddings
+timestamps = torch.arange(embeddings.shape[2]) / model.eps
+
+print(f"Extracted embeddings shape: {embeddings.shape}")
+print(f"First 5 timestamps: {timestamps[:5]}")
+```
+
+**`get_model` reference:**
+
+```
+Returns an OMAR-RQ Module from the provided  model_id or config_file.
+
+Args:
+    model_id (str): Hugging Face's Model ID or local path to the model
+    config_file (Path): Path to the model config of a trained model.
+    device (str): Device to use for the model. Defaults to "cpu".
+    quantization_targets (bool): If True, it will create the quantization
+        targets for SSL pre-training of the model. Defaults to False.
+
+Output:
+    module: The model from the provided config file.
+
+
+Module usage:
+
+Args:
+    audio (torch.Tensor): 2D mono audio tensor (B, T'). Where B is
+        the batch size and T' is the number of samples.
+    layers (set): Set of layer indices to extract embeddings from.
+        By default, it extracts embeddings from the last layer (logits).
+
+Output:
+    torch.Tensor: Extracted embeddings. The output tensor has shape
+        (L, B, T, C,) where L = len(layers), B is the batch size, T is
+        the number of output timestamps, and C = embedding dimension.
+```
+
+**`extract_embeddings` reference:**
+
+```
+Extract embeddings from an input audio batch.
+
+Args:
+    audio (torch.Tensor): 2D mono audio tensor (B, T'). Where B is 
+        the batch size and T' is the number of samples.
+    layers (set): Set of layer indices to extract embeddings from.
+        By default, it extracts embeddings from the last layer (logits).
+
+Output:
+    torch.Tensor: Extracted embeddings. The output tensor has shape 
+        (L, B, T, C,) where L = len(layers), B is the batch size, T is
+        the number of output timestamps, and C = embedding dimension.
+```
+
+## Available Models
+
+OMAR-RQ models are offered in different configurations, each with its own strengths and weaknesses. Models based on mel spectrogram (**base** and **multicodebook**) tend to perform better on semantic tasks such as auto-tagging, structure recognition, and difficulty estimation. On the other hand, **multifeature-25hz-fsq** offers the best performance in tonal and temporal tasks such as pitch and chord estimation, and beat tracking.
+
+| Model                     | Input  | Rate   | Tagging | Difficulty | Pitch    | Chord    | Beat    | Structure | Hugging Face Model ID                                       |
+|:--------------------------|:-------|:-------|:--------|:-----------|:---------|:---------|:--------|:----------|:------------------------------------------------------------|
+|                           |        | Hz     | _mAP_   | _MSE_      | _acc._   | _acc._   | _F1_    | _acc._    |                                                             |
+| **base**                  | mel    | 15.63  | .482    | **1.65**   | .892     | .657     | .783    | **.647**  | [`mtg-upf/omar-rq-base`](https://huggingface.co/mtg-upf/omar-rq-base)                   |
+| **multicodebook**         | mel    | 15.63  | **.488**| 1.66       | .897     | .675     | .775    | .639      | [`mtg-upf/omar-rq-multicodebook`](https://huggingface.co/mtg-upf/omar-rq-multicodebook) |
+| **multifeature**          | audio  | 18.75  | .467    | 1.76       | .938     | .734     | .833    | .623      | [`mtg-upf/omar-rq-multifeature`](https://huggingface.co/mtg-upf/omar-rq-multifeature)   |
+| **multifeature-25hz**     | audio  | 25     | .463    | 1.79       | .932     | .728     | .848    | .628      | [`mtg-upf/omar-rq-multifeature-25hz`](https://huggingface.co/mtg-upf/omar-rq-multifeature-25hz) |
+| **multifeature-25hz-fsq** | audio  | 25     | .463    | 1.71       | **.940** | **.749** | **.855**| .628      | [`mtg-upf/omar-rq-multifeature-25hz-fsq`](https://huggingface.co/mtg-upf/omar-rq-multifeature-25hz-fsq) |
+
+## Licensing Information
+
+The code in the [GitHub repository](https://github.com/MTG/omar-rq) is available under the [AGPL-3.0 license](https://www.gnu.org/licenses/agpl-3.0.en.html). The model weights are available under the [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license for non-commercial applications.
+
+## Citation
+
+If you find this work useful, please cite the paper:
+
+```bibtex
+@article {alonso2025omarrq,
+  title={OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction},
+  author={Alonso-Jim\'enez, Pablo and Ramoneda, Pedro and Araz, R. Oguz and Poltronieri, Andrea and Bogdanov, Dmitry},
+  journal={arXiv preprint arXiv:2507.03482},
+  year={2025}
+}
+```