Huggingface.js documentation

πŸ€— Hugging Face Hub API

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

πŸ€— Hugging Face Hub API

Hugging Face Hub APIλ₯Ό μ‚¬μš©ν•˜λŠ” 곡식 μœ ν‹Έλ¦¬ν‹°μž…λ‹ˆλ‹€.

μ„€μΉ˜

pnpm add @huggingface/hub

npm add @huggingface/hub

yarn add @huggingface/hub

Deno

// esm.sh
import { uploadFiles, listModels } from "https://esm.sh/@huggingface/hub"
// or npm:
import { uploadFiles, listModels } from "npm:@huggingface/hub"

μ‚¬μš©

일뢀 호좜의 경우 계정을 μƒμ„±ν•˜κ³  μ•‘μ„ΈμŠ€ 토큰을 생성해야 ν•©λ‹ˆλ‹€.

이 μΈν„°λž™ν‹°λΈŒ νŠœν† λ¦¬μ–Όμ—μ„œ hub νŒ¨ν‚€μ§€λ₯Ό μ‚¬μš©ν•˜μ—¬ 무료 λͺ¨λΈμ„ μ°ΎλŠ” 방법을 λ°°μ›Œλ³΄μ„Έμš”.

import * as hub from "@huggingface/hub";
import type { RepoDesignation } from "@huggingface/hub";

const repo: RepoDesignation = { type: "model", name: "myname/some-model" };

const {name: username} = await hub.whoAmI({accessToken: "hf_..."});

for await (const model of hub.listModels({search: {owner: username}, accessToken: "hf_..."})) {
  console.log("My model:", model);
}

const specificModel = await hub.modelInfo({name: "openai-community/gpt2"});
await hub.checkRepoAccess({repo, accessToken: "hf_..."});

await hub.createRepo({ repo, accessToken: "hf_...", license: "mit" });

await hub.uploadFiles({
  repo,
  accessToken: "hf_...",
  files: [
    // path + blob content
    {
      path: "file.txt",
      content: new Blob(["Hello World"]),
    },
    // Local file URL
    pathToFileURL("./pytorch-model.bin"),
    // Local folder URL
    pathToFileURL("./models"),
    // Web URL
    new URL("https://huggingface.co/xlm-roberta-base/resolve/main/tokenizer.json"),
    // Path + Web URL
    {
      path: "myfile.bin",
      content: new URL("https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin")
    }
    // Can also work with native File in browsers
  ],
});

// or

for await (const progressEvent of await hub.uploadFilesWithProgress({
  repo,
  accessToken: "hf_...",
  files: [
    ...
  ],
})) {
  console.log(progressEvent);
}

// Edit a file in place, without downloading/uploading its unchanged data
const file = await hub.downloadFile({ repo, path: "myfile.bin", accessToken: "hf_..." });
await hub.uploadFile({
  repo,
  accessToken: "hf_...",
  file: {
    path: "myfile.bin",
    originalContent: file,
    edits: [{
      // Replace bytes [0, 4)
      start: 0,
      end: 4,
      content: new Blob(["new prefix"])
    }, {
      // Append at the end
      start: file.size,
      end: file.size,
      content: new Blob(["suffix"])
    }]
  }
});

await hub.deleteFile({repo, accessToken: "hf_...", path: "myfile.bin"});

await (await hub.downloadFile({ repo, path: "README.md" })).text();

for await (const fileInfo of hub.listFiles({repo})) {
  console.log(fileInfo);
}

await hub.deleteRepo({ repo, accessToken: "hf_..." });

버킷

Storage BucketsλŠ” Hubμ—μ„œ λ³€κ²½ κ°€λŠ₯ν•˜κ³  버전 κ΄€λ¦¬λ˜μ§€ μ•ŠλŠ” μ €μž₯μ†Œμž…λ‹ˆλ‹€. λ™μΌν•œ ν•¨μˆ˜λ₯Ό μ‚¬μš©ν•˜μ—¬ buckets/ 접두사(λ˜λŠ” { type: "bucket", name })λ₯Ό μ‚¬μš©ν•˜μ„Έμš”:

const bucket = "buckets/myname/my-bucket";

await hub.createRepo({ repo: bucket, accessToken: "hf_..." });

await hub.uploadFiles({ repo: bucket, accessToken: "hf_...", files: [pathToFileURL("./checkpoints")] });

for await (const fileInfo of hub.listFiles({ repo: bucket, recursive: true, accessToken: "hf_..." })) {
  console.log(fileInfo.path, fileInfo.uploadedAt);
}

// Copy files from a model or dataset repo into a bucket server-side, without downloading them
await hub.copyFolder({
  source: { repo: { type: "dataset", name: "myname/my-dataset" }, path: "data" },
  destination: { repo: bucket, path: "data" },
  accessToken: "hf_...",
});

CLI μ‚¬μš©

CLI λͺ¨λ“œλ‘œ @huggingface/hubλ₯Ό μ‚¬μš©ν•˜μ—¬ λ¦¬ν¬μ§€ν† λ¦¬λ‘œ νŒŒμΌμ„ μ—…λ‘œλ“œν•˜κ±°λ‚˜ λ‹€μš΄λ‘œλ“œν•  수 μžˆμŠ΅λ‹ˆλ‹€.

npx @huggingface/hub upload coyotte508/test-model .
npx @huggingface/hub upload datasets/coyotte508/test-dataset .
# Same thing
npx @huggingface/hub upload --repo-type dataset coyotte508/test-dataset .
# Upload new data with 0 history in a separate branch
npx @huggingface/hub branch create coyotte508/test-model release --empty
npx @huggingface/hub upload coyotte508/test-model . --revision release

# Download files to the current directory (or --local-dir)
npx @huggingface/hub download coyotte508/test-model config.json
npx @huggingface/hub download coyotte508/test-model config.json README.md --local-dir ./test-model
# Download an entire repo, optionally filtered with --include / --exclude
npx @huggingface/hub download coyotte508/test-model --local-dir ./test-model
npx @huggingface/hub download --repo-type dataset coyotte508/test-dataset --include "*.json"

npx @huggingface/hub --help
npx @huggingface/hub upload --help
npx @huggingface/hub download --help

npm install -g @huggingface/hubλ₯Ό μ‚¬μš©ν•˜μ—¬ μ „μ—­μœΌλ‘œ μ„€μΉ˜ν•  μˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€. 그러면 λ‹€μŒμ„ μˆ˜ν–‰ν•  수 μžˆμŠ΅λ‹ˆλ‹€:

hfjs upload coyotte508/test-model .

hfjs branch create --repo-type dataset coyotte508/test-dataset release --empty
hfjs upload --repo-type dataset coyotte508/test-dataset . --revision release

hfjs download coyotte508/test-model config.json
hfjs download coyotte508/test-model --local-dir ./test-model

hfjs --help
hfjs  upload --help

hfjs help jobs
hfjs download --help

OAuth 둜그인

OAuth(β€œSign in with HF”)λ₯Ό μ‚¬μš©ν•˜μ—¬ λ‘œκ·ΈμΈν•  수 μžˆμŠ΅λ‹ˆλ‹€.

Space λ˜λŠ” OAuth App에 μ„€μ •λœ μŠ€μ½”ν”„μ— 따라 API의 일뢀에 μ•‘μ„ΈμŠ€ν•  수 μžˆλŠ” μ•‘μ„ΈμŠ€ 토큰을 κ°€μ Έμ˜¬ 수 μžˆμŠ΅λ‹ˆλ‹€.

import { oauthLoginUrl, oauthHandleRedirectIfPresent } from "@huggingface/hub";

const oauthResult = await oauthHandleRedirectIfPresent();

if (!oauthResult) {
  // If the user is not logged in, redirect to the login page
  window.location.href = await oauthLoginUrl();
}

// You can use oauthResult.accessToken, oauthResult.accessTokenExpiresAt and oauthResult.userInfo
console.log(oauthResult);

데λͺ¨ 확인: https://huggingface.co/spaces/huggingfacejs/client-side-oauth

Hugging face μΊμ‹œ

@huggingface/hub νŒ¨ν‚€μ§€λŠ” μΊμ‹œ 디렉토리λ₯Ό μŠ€μΊ”ν•˜λŠ” κΈ°λ³Έ κΈ°λŠ₯을 μ œκ³΅ν•©λ‹ˆλ‹€. huggingface_hub μΊμ‹œ μ‹œμŠ€ν…œ 관리에 λŒ€ν•΄ μžμ„Ένžˆ μ•Œμ•„λ³΄μ„Έμš”.

scanCacheDir

scanCacheDir ν•¨μˆ˜λ₯Ό μ‚¬μš©ν•˜μ—¬ μΊμ‹œλœ μ €μž₯μ†Œ λͺ©λ‘μ„ κ°€μ Έμ˜¬ 수 μžˆμŠ΅λ‹ˆλ‹€.

import { scanCacheDir } from "@huggingface/hub";

const result = await scanCacheDir();

console.log(result);

μ°Έκ³ : λΈŒλΌμš°μ €μ—μ„œλŠ” μž‘λ™ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€

downloadFileToCacheDir

downloadFileToCacheDir ν•¨μˆ˜λ₯Ό μ‚¬μš©ν•˜μ—¬ μ €μž₯μ†Œμ˜ νŒŒμΌμ„ μΊμ‹œμ— μ €μž₯ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

import { downloadFileToCacheDir } from "@huggingface/hub";

const file = await downloadFileToCacheDir({
  repo: 'foo/bar',
  path: 'README.md'
});

console.log(file);

μ°Έκ³ : λΈŒλΌμš°μ €μ—μ„œλŠ” μž‘λ™ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€

snapshotDownload

snapshotDownload ν•¨μˆ˜λ₯Ό μ‚¬μš©ν•˜μ—¬ νŠΉμ • λ¦¬λΉ„μ „μ˜ 전체 μ €μž₯μ†Œλ₯Ό μΊμ‹œ 디렉터리에 λ‹€μš΄λ‘œλ“œν•  수 μžˆμŠ΅λ‹ˆλ‹€.

import { snapshotDownload } from "@huggingface/hub";

const directory = await snapshotDownload({
  repo: 'foo/bar',
});

console.log(directory);

이 μ½”λ“œλŠ” λ‚΄λΆ€μ μœΌλ‘œ downloadFileToCacheDir ν•¨μˆ˜λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€.

μ°Έκ³ : λΈŒλΌμš°μ €μ—μ„œλŠ” μž‘λ™ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€

μ„±λŠ₯ κ³ λ € 사항

λŒ€μš©λŸ‰ νŒŒμΌμ„ μ—…λ‘œλ“œν•  λ•Œ, sha256 계산을 μ˜€ν”„λ‘œλ“œν•˜κΈ° μœ„ν•΄ commit ν˜ΈμΆœμ„ μ›Œμ»€μ—μ„œ μ‹€ν–‰ν•˜λŠ” 것이 μ’‹μŠ΅λ‹ˆλ‹€.

원격 λ¦¬μ†ŒμŠ€μ™€ 둜컬 νŒŒμΌμ€ RAM μ‚¬μš©λŸ‰μ„ 쀄이기 μœ„ν•΄ 청크 λ‹¨μœ„λ‘œ μ§€μ—° λ‘œλ“œν•  수 μžˆλ„λ‘ κ°€λŠ₯ν•œ 경우 URL둜 전달해야 ν•©λ‹ˆλ‹€. λΈŒλΌμš°μ € μ»¨ν…μŠ€νŠΈ λ‚΄μ—μ„œ Fileλ₯Ό μ „λ‹¬ν•˜λŠ” 것은 Blob처럼 기본적으둜 λ™μž‘ν•˜λ―€λ‘œ λ¬Έμ œμ—†μŠ΅λ‹ˆλ‹€.

λ‚΄λΆ€μ μœΌλ‘œ @huggingface/hubλŠ” νŒŒμΌμ„ λ‘œλ“œν•˜κΈ° μœ„ν•΄ μ§€μ—°(blob) κ΅¬ν˜„μ„ μ‚¬μš©ν•©λ‹ˆλ‹€.

κΈ°μ‘΄ νŒŒμΌμ„ νŽΈμ§‘ν•  λ•ŒλŠ” downloadFile이 λ°˜ν™˜ν•œ blob을 originalContent둜 μ‚¬μš©ν•˜μ—¬ edit 컀밋 μž‘μ—…μ„ μ‚¬μš©ν•˜λŠ” 것이 μ’‹μŠ΅λ‹ˆλ‹€. μ΄λ ‡κ²Œ ν•˜λ©΄ νŽΈμ§‘λœ μ˜μ—­λ§Œ λ‹€μš΄λ‘œλ“œ, 재청크 및 ν•΄μ‹œ μ²˜λ¦¬λ©λ‹ˆλ‹€. λ™μΌν•œ νŒŒμΌμ— 반볡적으둜 μΆ”κ°€ν•  λ•ŒλŠ” commit ν˜ΈμΆœμ— 곡유된 rangeEditCache: new Map()을 μ „λ‹¬ν•˜μ—¬ μŠ€ν† λ¦¬μ§€ 메타데이터 왕볡도 κ±΄λ„ˆλ›°λ„λ‘ ν•˜μ„Έμš”.

μ˜μ‘΄μ„±

  • @huggingface/tasks : νƒ€μž… μ •μ˜λ§Œ 포함
Update on GitHub