Huggingface.js documentation
π€ Hugging Face Hub API
π€ Hugging Face Hub API
Hugging Face Hub APIλ₯Ό μ¬μ©νλ 곡μ μ νΈλ¦¬ν°μ λλ€.
μ€μΉ
pnpm add @huggingface/hub npm add @huggingface/hub yarn add @huggingface/hub
Deno
// esm.sh
import { uploadFiles, listModels } from "https://esm.sh/@huggingface/hub"
// or npm:
import { uploadFiles, listModels } from "npm:@huggingface/hub"μ¬μ©
μΌλΆ νΈμΆμ κ²½μ° κ³μ μ μμ±νκ³ μ‘μΈμ€ ν ν°μ μμ±ν΄μΌ ν©λλ€.
μ΄ μΈν°λν°λΈ νν 리μΌμμ hub ν¨ν€μ§λ₯Ό μ¬μ©νμ¬ λ¬΄λ£ λͺ¨λΈμ μ°Ύλ λ°©λ²μ λ°°μ보μΈμ.
import * as hub from "@huggingface/hub";
import type { RepoDesignation } from "@huggingface/hub";
const repo: RepoDesignation = { type: "model", name: "myname/some-model" };
const {name: username} = await hub.whoAmI({accessToken: "hf_..."});
for await (const model of hub.listModels({search: {owner: username}, accessToken: "hf_..."})) {
console.log("My model:", model);
}
const specificModel = await hub.modelInfo({name: "openai-community/gpt2"});
await hub.checkRepoAccess({repo, accessToken: "hf_..."});
await hub.createRepo({ repo, accessToken: "hf_...", license: "mit" });
await hub.uploadFiles({
repo,
accessToken: "hf_...",
files: [
// path + blob content
{
path: "file.txt",
content: new Blob(["Hello World"]),
},
// Local file URL
pathToFileURL("./pytorch-model.bin"),
// Local folder URL
pathToFileURL("./models"),
// Web URL
new URL("https://huggingface.co/xlm-roberta-base/resolve/main/tokenizer.json"),
// Path + Web URL
{
path: "myfile.bin",
content: new URL("https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin")
}
// Can also work with native File in browsers
],
});
// or
for await (const progressEvent of await hub.uploadFilesWithProgress({
repo,
accessToken: "hf_...",
files: [
...
],
})) {
console.log(progressEvent);
}
// Edit a file in place, without downloading/uploading its unchanged data
const file = await hub.downloadFile({ repo, path: "myfile.bin", accessToken: "hf_..." });
await hub.uploadFile({
repo,
accessToken: "hf_...",
file: {
path: "myfile.bin",
originalContent: file,
edits: [{
// Replace bytes [0, 4)
start: 0,
end: 4,
content: new Blob(["new prefix"])
}, {
// Append at the end
start: file.size,
end: file.size,
content: new Blob(["suffix"])
}]
}
});
await hub.deleteFile({repo, accessToken: "hf_...", path: "myfile.bin"});
await (await hub.downloadFile({ repo, path: "README.md" })).text();
for await (const fileInfo of hub.listFiles({repo})) {
console.log(fileInfo);
}
await hub.deleteRepo({ repo, accessToken: "hf_..." });λ²ν·
Storage Bucketsλ Hubμμ λ³κ²½ κ°λ₯νκ³ λ²μ κ΄λ¦¬λμ§ μλ μ μ₯μμ
λλ€. λμΌν ν¨μλ₯Ό μ¬μ©νμ¬ buckets/ μ λμ¬(λλ { type: "bucket", name })λ₯Ό μ¬μ©νμΈμ:
const bucket = "buckets/myname/my-bucket";
await hub.createRepo({ repo: bucket, accessToken: "hf_..." });
await hub.uploadFiles({ repo: bucket, accessToken: "hf_...", files: [pathToFileURL("./checkpoints")] });
for await (const fileInfo of hub.listFiles({ repo: bucket, recursive: true, accessToken: "hf_..." })) {
console.log(fileInfo.path, fileInfo.uploadedAt);
}
// Copy files from a model or dataset repo into a bucket server-side, without downloading them
await hub.copyFolder({
source: { repo: { type: "dataset", name: "myname/my-dataset" }, path: "data" },
destination: { repo: bucket, path: "data" },
accessToken: "hf_...",
});CLI μ¬μ©
CLI λͺ¨λλ‘ @huggingface/hubλ₯Ό μ¬μ©νμ¬ λ¦¬ν¬μ§ν λ¦¬λ‘ νμΌμ μ
λ‘λνκ±°λ λ€μ΄λ‘λν μ μμ΅λλ€.
npx @huggingface/hub upload coyotte508/test-model .
npx @huggingface/hub upload datasets/coyotte508/test-dataset .
# Same thing
npx @huggingface/hub upload --repo-type dataset coyotte508/test-dataset .
# Upload new data with 0 history in a separate branch
npx @huggingface/hub branch create coyotte508/test-model release --empty
npx @huggingface/hub upload coyotte508/test-model . --revision release
# Download files to the current directory (or --local-dir)
npx @huggingface/hub download coyotte508/test-model config.json
npx @huggingface/hub download coyotte508/test-model config.json README.md --local-dir ./test-model
# Download an entire repo, optionally filtered with --include / --exclude
npx @huggingface/hub download coyotte508/test-model --local-dir ./test-model
npx @huggingface/hub download --repo-type dataset coyotte508/test-dataset --include "*.json"
npx @huggingface/hub --help
npx @huggingface/hub upload --help
npx @huggingface/hub download --helpnpm install -g @huggingface/hubλ₯Ό μ¬μ©νμ¬ μ μμΌλ‘ μ€μΉν μλ μμ΅λλ€. κ·Έλ¬λ©΄ λ€μμ μνν μ μμ΅λλ€:
hfjs upload coyotte508/test-model . hfjs branch create --repo-type dataset coyotte508/test-dataset release --empty hfjs upload --repo-type dataset coyotte508/test-dataset . --revision release hfjs download coyotte508/test-model config.json hfjs download coyotte508/test-model --local-dir ./test-model hfjs --help hfjs upload --help hfjs help jobs hfjs download --help
OAuth λ‘κ·ΈμΈ
OAuth(βSign in with HFβ)λ₯Ό μ¬μ©νμ¬ λ‘κ·ΈμΈν μ μμ΅λλ€.
Space λλ OAuth Appμ μ€μ λ μ€μ½νμ λ°λΌ APIμ μΌλΆμ μ‘μΈμ€ν μ μλ μ‘μΈμ€ ν ν°μ κ°μ Έμ¬ μ μμ΅λλ€.
import { oauthLoginUrl, oauthHandleRedirectIfPresent } from "@huggingface/hub";
const oauthResult = await oauthHandleRedirectIfPresent();
if (!oauthResult) {
// If the user is not logged in, redirect to the login page
window.location.href = await oauthLoginUrl();
}
// You can use oauthResult.accessToken, oauthResult.accessTokenExpiresAt and oauthResult.userInfo
console.log(oauthResult);λ°λͺ¨ νμΈ: https://huggingface.co/spaces/huggingfacejs/client-side-oauth
Hugging face μΊμ
@huggingface/hub ν¨ν€μ§λ μΊμ λλ ν 리λ₯Ό μ€μΊνλ κΈ°λ³Έ κΈ°λ₯μ μ 곡ν©λλ€. huggingface_hub μΊμ μμ€ν
κ΄λ¦¬μ λν΄ μμΈν μμ보μΈμ.
scanCacheDir
scanCacheDir ν¨μλ₯Ό μ¬μ©νμ¬ μΊμλ μ μ₯μ λͺ©λ‘μ κ°μ Έμ¬ μ μμ΅λλ€.
import { scanCacheDir } from "@huggingface/hub";
const result = await scanCacheDir();
console.log(result);μ°Έκ³ : λΈλΌμ°μ μμλ μλνμ§ μμ΅λλ€
downloadFileToCacheDir
downloadFileToCacheDir ν¨μλ₯Ό μ¬μ©νμ¬ μ μ₯μμ νμΌμ μΊμμ μ μ₯ν μ μμ΅λλ€.
import { downloadFileToCacheDir } from "@huggingface/hub";
const file = await downloadFileToCacheDir({
repo: 'foo/bar',
path: 'README.md'
});
console.log(file);μ°Έκ³ : λΈλΌμ°μ μμλ μλνμ§ μμ΅λλ€
snapshotDownload
snapshotDownload ν¨μλ₯Ό μ¬μ©νμ¬ νΉμ 리λΉμ μ μ 체 μ μ₯μλ₯Ό μΊμ λλ ν°λ¦¬μ λ€μ΄λ‘λν μ μμ΅λλ€.
import { snapshotDownload } from "@huggingface/hub";
const directory = await snapshotDownload({
repo: 'foo/bar',
});
console.log(directory);μ΄ μ½λλ λ΄λΆμ μΌλ‘ downloadFileToCacheDir ν¨μλ₯Ό μ¬μ©ν©λλ€.
μ°Έκ³ : λΈλΌμ°μ μμλ μλνμ§ μμ΅λλ€
μ±λ₯ κ³ λ € μ¬ν
λμ©λ νμΌμ μ
λ‘λν λ, sha256 κ³μ°μ μ€νλ‘λνκΈ° μν΄ commit νΈμΆμ μ컀μμ μ€ννλ κ²μ΄ μ’μ΅λλ€.
μ격 리μμ€μ λ‘컬 νμΌμ RAM μ¬μ©λμ μ€μ΄κΈ° μν΄ μ²ν¬ λ¨μλ‘ μ§μ° λ‘λν μ μλλ‘ κ°λ₯ν κ²½μ° URLλ‘ μ λ¬ν΄μΌ ν©λλ€. λΈλΌμ°μ 컨ν
μ€νΈ λ΄μμ Fileλ₯Ό μ λ¬νλ κ²μ Blobμ²λΌ κΈ°λ³Έμ μΌλ‘ λμνλ―λ‘ λ¬Έμ μμ΅λλ€.
λ΄λΆμ μΌλ‘ @huggingface/hubλ νμΌμ λ‘λνκΈ° μν΄ μ§μ°(blob) ꡬνμ μ¬μ©ν©λλ€.
κΈ°μ‘΄ νμΌμ νΈμ§ν λλ downloadFileμ΄ λ°νν blobμ originalContentλ‘ μ¬μ©νμ¬ edit μ»€λ° μμ
μ μ¬μ©νλ κ²μ΄ μ’μ΅λλ€. μ΄λ κ² νλ©΄ νΈμ§λ μμλ§ λ€μ΄λ‘λ, μ¬μ²ν¬ λ° ν΄μ μ²λ¦¬λ©λλ€. λμΌν νμΌμ λ°λ³΅μ μΌλ‘ μΆκ°ν λλ commit νΈμΆμ 곡μ λ rangeEditCache: new Map()μ μ λ¬νμ¬ μ€ν λ¦¬μ§ λ©νλ°μ΄ν° μ볡λ 건λλ°λλ‘ νμΈμ.
μμ‘΄μ±
@huggingface/tasks: νμ μ μλ§ ν¬ν¨