As the amount of datasets for fine tuning chat models has grown, there's been a plethora of dataset formats emerge. The most popular of these include the formats used by Alpaca, ShareGPT and Open Assistant datasets. The datasets and their formats have also evolved from single-turn conversation to multi-turn. Many of these formats share similarities (and they all have the same goal), but handling the variations in formats across datasets is often a hassle, and source of potential bugs.
text
As the amount of datasets for fine tuning chat models has grown, there's been a plethora of dataset formats emerge. The most popular of these include the formats used by Alpaca, ShareGPT and Open Assistant datasets. The datasets and their formats have also evolved from single-turn conversation to multi-turn. Many of these formats share similarities (and they all have the same goal), but handling the variations in formats across datasets is often a hassle, and source of potential bugs.
new_line
new_line
Luckily the community seems to be converging on a simple and elegant chat dataset format: a list with each record being an array with each conversation turn being an object with a role (system, assistant or user) and content. Hugging Face uses this input format in the [Templates for Chat Models](
text
Luckily the community seems to be converging on a simple and elegant chat dataset format: a list with each record being an array with each conversation turn being an object with a role (system, assistant or user) and content. Hugging Face uses this input format in the [Templates for Chat Models](
https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates
https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates
link
) docs:
text
) docs:
new_line
new_line
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
python
```python
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
```
code_fence
new_line
new_line
Popular datasets like 
text
Popular datasets like 
https://huggingface.co/datasets/HuggingFaceH4/no_robots
HuggingFaceH4/no_robots
dataset
resource
https://huggingface.co/datasets/HuggingFaceH4/no_robots
 follow this format.
text
 follow this format.
new_line
new_line
To encourage usage of this format, I propose we give it a name: Hugging Face MessagesList format.
text
To encourage usage of this format, I propose we give it a name: Hugging Face MessagesList format.
new_line
new_line
The format is defined as:
text
The format is defined as:
new_line
new_line
- Having at least one 
text
- Having at least one 
messages
`messages`
inline_code
 column of type list.
text
 column of type list.
new_line
- Each messages record is an array containing one or more message turn objects.
text
- Each messages record is an array containing one or more message turn objects.
new_line
- A message turn must have 
text
- A message turn must have 
role
`role`
inline_code
 and 
text
 and 
content
`content`
inline_code
 keys.
text
 keys.
new_line
- 
text
- 
role
`role`
inline_code
 should be one of 
text
 should be one of 
system
`system`
inline_code
, 
text
, 
assistant
`assistant`
inline_code
 or 
text
 or 
user
`user`
inline_code
.
text
.
new_line
- 
text
- 
content
`content`
inline_code
 is the text content of the message.
text
 is the text content of the message.
new_line
new_line
This may be a small thing, but having a common dataset format will reduce wasted time data wrangling and help everyone.
text
This may be a small thing, but having a common dataset format will reduce wasted time data wrangling and help everyone.
As the amount of datasets for fine tuning chat models has grown, there's been a plethora of dataset formats emerge. The most popular of these include the formats used by Alpaca, ShareGPT and Open Assistant datasets. The datasets and their formats have also evolved from single-turn conversation to multi-turn. Many of these formats share similarities (and they all have the same goal), but handling the variations in formats across datasets is often a hassle, and source of potential bugs.

Luckily the community seems to be converging on a simple and elegant chat dataset format: a list with each record being an array with each conversation turn being an object with a role (system, assistant or user) and content. Hugging Face uses this input format in the [Templates for Chat Models](https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates) docs:

```python
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
```

Popular datasets like https://huggingface.co/datasets/HuggingFaceH4/no_robots follow this format.

To encourage usage of this format, I propose we give it a name: Hugging Face MessagesList format.

The format is defined as:

- Having at least one `messages` column of type list.
- Each messages record is an array containing one or more message turn objects.
- A message turn must have `role` and `content` keys.
- `role` should be one of `system`, `assistant` or `user`.
- `content` is the text content of the message.

This may be a small thing, but having a common dataset format will reduce wasted time data wrangling and help everyone.
❤️
davanstrien osanseviero Weyaxi lhoestq julien-c clem merve alielfilali01 andysalerno dvilasuero samusenps afrideva IFSDrake sbrandeis
👍
victor julien-c Tonic dvilasuero TuringsSolutions madroid davanstrien clefourrier Dashanka
5f17f0a0925b9863e28ad517
https://cdn-avatars.huggingface.co/v1/production/uploads/5f17f0a0925b9863e28ad517/fXIY5i9RLsIa1v3CCuVtt.jpeg
Victor Mustar
victor
user
639afe83611f99c4be1e370a
https://cdn-avatars.huggingface.co/v1/production/uploads/639afe83611f99c4be1e370a/-TtkPQJyeqwoXA-MmEO1b.jpeg
Damien Tanner
dctanner
user
60107b385ac3e86b3ea4fc34
https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg
Daniel van Strien
davanstrien
user
65820f41283fdb0c8dd5b140
/avatars/40bb534399cd10cdb853213edb5b7509.svg
Gokul
gokulavasan
user