Do you have a hidden massive storage leak thanks to HF hub models and datasets revisions adding up and not getting automatically deleted?
text
Do you have a hidden massive storage leak thanks to HF hub models and datasets revisions adding up and not getting automatically deleted?
new_line
new_line
Here is how to delete all old revisions and only keeping 
text
Here is how to delete all old revisions and only keeping 
main
`main`
inline_code
 in a few quick steps and no tedious manual editing.
text
 in a few quick steps and no tedious manual editing.
new_line
new_line
In terminal A:
text
In terminal A:
new_line
$ pip install huggingface_hub["cli"] -U
$ huggingface-cli delete-cache --disable-tui
File to edit: /tmp/tmpundr7lky.txt
0 revisions selected counting for 0.0. Continue ? (y/N)
```
$ pip install huggingface_hub["cli"] -U
$ huggingface-cli delete-cache --disable-tui
File to edit: /tmp/tmpundr7lky.txt
0 revisions selected counting for 0.0. Continue ? (y/N)
```
code_fence
new_line
Do not answer the prompt and proceed with my instructions.
text
Do not answer the prompt and proceed with my instructions.
new_line
new_line
(note your tmp file will have a different path, so adjust it below)
text
(note your tmp file will have a different path, so adjust it below)
new_line
new_line
In terminal B:
text
In terminal B:
new_line
$ cp /tmp/tmpedbz00ox.txt cache.txt
$ perl -pi -e 's|^#(.*detached.*)|$1|' cache.txt
$ cat cache.txt >>  /tmp/tmpundr7lky.txt
```
$ cp /tmp/tmpedbz00ox.txt cache.txt
$ perl -pi -e 's|^#(.*detached.*)|$1|' cache.txt
$ cat cache.txt >>  /tmp/tmpundr7lky.txt
```
code_fence
new_line
The perl one-liner uncommented out all lines that had 
text
The perl one-liner uncommented out all lines that had 
(detached)
`(detached)`
inline_code
 in it - so can be wiped out. And then we pasted it back into the tmp file 
text
 in it - so can be wiped out. And then we pasted it back into the tmp file 
huggingface-cli
`huggingface-cli`
inline_code
 expects to be edited.
text
 expects to be edited.
new_line
new_line
Now go back to terminal A and hit: N, Y, Y, so it looks like:
text
Now go back to terminal A and hit: N, Y, Y, so it looks like:
new_line
new_line
0 revisions selected counting for 0.0. Continue ? (y/N) n
89 revisions selected counting for 211.7G. Continue ? (y/N) y
89 revisions selected counting for 211.7G. Confirm deletion ? (Y/n) y
```
0 revisions selected counting for 0.0. Continue ? (y/N) n
89 revisions selected counting for 211.7G. Continue ? (y/N) y
89 revisions selected counting for 211.7G. Confirm deletion ? (Y/n) y
```
code_fence
new_line
Done.
text
Done.
new_line
new_line
If you messed up with the prompt answering you still have 
text
If you messed up with the prompt answering you still have 
cache.txt
`cache.txt`
inline_code
 file which you can feed again to the new tmp file it'll create when you run 
text
 file which you can feed again to the new tmp file it'll create when you run 
huggingface-cli delete-cache --disable-tui
`huggingface-cli delete-cache --disable-tui`
inline_code
 again.
text
 again.
new_line
new_line
For more details and additional techniques please see 
text
For more details and additional techniques please see 
https://github.com/stas00/ml-engineering/tree/master/storage#huggingface-hub-caches
https://github.com/stas00/ml-engineering/tree/master/storage#huggingface-hub-caches
link
Do you have a hidden massive storage leak thanks to HF hub models and datasets revisions adding up and not getting automatically deleted?

Here is how to delete all old revisions and only keeping `main` in a few quick steps and no tedious manual editing.

In terminal A:
```
$ pip install huggingface_hub["cli"] -U
$ huggingface-cli delete-cache --disable-tui
File to edit: /tmp/tmpundr7lky.txt
0 revisions selected counting for 0.0. Continue ? (y/N)
```
Do not answer the prompt and proceed with my instructions.

(note your tmp file will have a different path, so adjust it below)

In terminal B:
```
$ cp /tmp/tmpedbz00ox.txt cache.txt
$ perl -pi -e 's|^#(.*detached.*)|$1|' cache.txt
$ cat cache.txt >>  /tmp/tmpundr7lky.txt
```
The perl one-liner uncommented out all lines that had `(detached)` in it - so can be wiped out. And then we pasted it back into the tmp file `huggingface-cli` expects to be edited.

Now go back to terminal A and hit: N, Y, Y, so it looks like:

```
0 revisions selected counting for 0.0. Continue ? (y/N) n
89 revisions selected counting for 211.7G. Continue ? (y/N) y
89 revisions selected counting for 211.7G. Confirm deletion ? (Y/n) y
```
Done.

If you messed up with the prompt answering you still have `cache.txt` file which you can feed again to the new tmp file it'll create when you run `huggingface-cli delete-cache --disable-tui` again.

For more details and additional techniques please see https://github.com/stas00/ml-engineering/tree/master/storage#huggingface-hub-caches
❤️
vicgalle cahlen osanseviero not-lain samusenps clem victor Wauplin afrideva sethuiyer sbrandeis
👍
neph1