Skip to content

[Utils] Add get_cached_repo_tree utility - #4513

Merged
Wauplin merged 6 commits into
mainfrom
get-cached-repo-tree
Jul 9, 2026
Merged

[Utils] Add get_cached_repo_tree utility#4513
Wauplin merged 6 commits into
mainfrom
get-cached-repo-tree

Conversation

@Wauplin

@Wauplin Wauplin commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Follow-up PR after #4394.

This PR adds a utility to retrieve the cached repo tree from snapshot_download. First mentioned in huggingface/diffusers#14118 (comment). Should help diffusers integration to define the structure of repo (with its variants, etc.) even in offline mode. cc @sayakpaul

Summary

Adds a power-user helper get_cached_repo_tree(repo_id, repo_type=..., revision=..., cache_dir=...) that returns the repo's cached /tree listing (populated as a side effect of snapshot_download) as a list[RepoFile], without any network call.

  • Kept separate from HfApi.list_repo_tree: it always returns the full list, RepoFile only (no RepoFolder), and has no recursive/expand/path/... options — everything cached is returned.
  • Raises a new errors.CachedRepoTreeNotFoundError when nothing is cached for the requested revision.

Example

>>> from huggingface_hub import snapshot_download, get_cached_repo_tree
>>> snapshot_download("gpt2")
>>> files = get_cached_repo_tree("openai-community/gpt2")
>>> [f.path for f in files][:3]
['.gitattributes', '64-8bits.tflite', '64-fp16.tflite']

>>> get_cached_repo_tree("gpt2", revision="does-not-exist")
huggingface_hub.errors.CachedRepoTreeNotFoundError: No cached tree listing found for 'openai-community/gpt2' ...

Note

Low Risk
Additive public API over existing on-disk tree cache; read-only local filesystem access with clear errors when cache is missing.

Overview
Adds get_cached_repo_tree, a read-only helper that returns the full cached repo file tree as list[RepoFile] from the trees/<commit_hash>.json data written by snapshot_download, with no Hub request. Revisions can be commit hashes or branch/tag names resolved via cached refs/; local_dir reads trees from local_dir/.cache/huggingface/ while still resolving refs from the main cache.

Introduces CachedRepoTreeNotFoundError when no ref or tree cache exists, documents both in the package reference, exports get_cached_repo_tree from the public API, and adds unit tests in test_tree_cache.py. CLI repo-create tests expect space_template=None on create_repo (unrelated assertion fix).

Reviewed by Cursor Bugbot for commit 6557128. Bugbot is set up for automated code reviews on this repo. Configure here.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/huggingface_hub/_snapshot_download.py
@bot-ci-comment

bot-ci-comment Bot commented Jul 9, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

return [
RepoFile(path=path, size=entry.size, oid=entry.blob_id, xetHash=entry.xet_hash)
for path, entry in tree_entries.items()
]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LFS metadata omitted from RepoFile

Medium Severity

get_cached_repo_tree builds each RepoFile from TreeCacheEntry with path, size, oid, and xetHash only. Cached lfs_sha256 and lfs_size are never passed into RepoFile, so LFS-tracked files always have lfs=None despite the tree cache storing that metadata when snapshot_download runs.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b907c2f. Configure here.

@Wauplin Wauplin Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on purpose, not needed in practice
let's add them in the future if they are really needed. path + size + xet_hash is already enough

@Wauplin
Wauplin marked this pull request as ready for review July 9, 2026 10:03
@Wauplin
Wauplin requested a review from hanouticelina July 9, 2026 10:03
@Wauplin Wauplin changed the title [Utils] Add get_cached_repo_tree utility [Utils] Add get_cached_repo_tree utility Jul 9, 2026

@hanouticelina hanouticelina left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f10a15f. Configure here.

Comment thread src/huggingface_hub/_snapshot_download.py Outdated
@Wauplin
Wauplin merged commit 923acb0 into main Jul 9, 2026
19 of 26 checks passed
@Wauplin
Wauplin deleted the get-cached-repo-tree branch July 9, 2026 11:07
@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v1.23.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants