Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
5d47e5d
first httpx integration
Wauplin Sep 3, 2025
10ee9c7
more migration
Wauplin Sep 3, 2025
7f6fbfd
some fixes
Wauplin Sep 3, 2025
d937c15
Merge branch 'main' into httpx-migration
Wauplin Sep 3, 2025
49888c2
download workflow should work
Wauplin Sep 3, 2025
92649df
Fix repocard and error utils tests
Wauplin Sep 4, 2025
de73d83
fix hf-file-system
Wauplin Sep 4, 2025
798d0ec
gix http utils tests
Wauplin Sep 4, 2025
6db2a57
more fixes
Wauplin Sep 4, 2025
811769e
fix some inference tests
Wauplin Sep 4, 2025
bce2db0
fix test_file_download tests
Wauplin Sep 4, 2025
978937f
async inference client
Wauplin Sep 4, 2025
5ff9b65
async code should be good
Wauplin Sep 4, 2025
8d9719c
Define RemoteEntryFileNotFound explicitly (+some fixes)
Wauplin Sep 5, 2025
861009e
fix async code quality
Wauplin Sep 5, 2025
6753ad5
torch ok
Wauplin Sep 5, 2025
d05e41d
fix hf_file_system
Wauplin Sep 5, 2025
e098da9
fix errors tests
Wauplin Sep 5, 2025
405d291
mock
Wauplin Sep 5, 2025
afb3f20
fix test_cli mock
Wauplin Sep 5, 2025
bf459ca
fix conflict
Wauplin Sep 5, 2025
3fdf5dd
fix commit scheduler
Wauplin Sep 5, 2025
f033dde
add fileno test
Wauplin Sep 5, 2025
50eaaf6
no more requests anywhere
Wauplin Sep 5, 2025
01a547f
fix test_file_download
Wauplin Sep 5, 2025
5b89a4f
tmp requests
Wauplin Sep 5, 2025
7167514
Update src/huggingface_hub/utils/_http.py
Wauplin Sep 8, 2025
9a4038f
Update src/huggingface_hub/utils/_http.py
Wauplin Sep 8, 2025
c99087f
Update src/huggingface_hub/hf_file_system.py
Wauplin Sep 8, 2025
0b8b279
Merge branch 'main' into httpx-migration
Wauplin Sep 8, 2025
7e28029
not async
Wauplin Sep 8, 2025
abbbdde
fix tests
Wauplin Sep 8, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/conda/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ requirements:
- pip
- fsspec
- filelock
- requests
- httpx
- tqdm
- typing-extensions
- packaging
Expand All @@ -26,7 +26,7 @@ requirements:
- python
- pip
- filelock
- requests
- httpx
- tqdm
- typing-extensions
- packaging
Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/guides/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ By default, the `hf download` command will be verbose. It will print details suc

On machines with slow connections, you might encounter timeout issues like this one:
```bash
`requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host='cdn-lfs-us-1.huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: a33d910c-84c6-4514-8362-c705e2039d38)')`
`httpx.TimeoutException: (TimeoutException("HTTPSConnectionPool(host='cdn-lfs-us-1.huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: a33d910c-84c6-4514-8362-c705e2039d38)')`
```

To mitigate this issue, you can set the `HF_HUB_DOWNLOAD_TIMEOUT` environment variable to a higher value (default is 10):
Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/package_reference/environment_variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ Set to disable using `hf-xet`, even if it is available in your Python environmen

Set to `True` for faster uploads and downloads from the Hub using `hf_transfer`.

By default, `huggingface_hub` uses the Python-based `requests.get` and `requests.post` functions.
By default, `huggingface_hub` uses the Python-based `httpx.get` and `httpx.post` functions.
Although these are reliable and versatile,
they may not be the most efficient choice for machines with high bandwidth.
[`hf_transfer`](https://github.com/huggingface/hf_transfer) is a Rust-based package developed to
Expand Down
22 changes: 13 additions & 9 deletions docs/source/en/package_reference/utilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,35 +177,39 @@ Here is a list of HTTP errors thrown in `huggingface_hub`.
the server response and format the error message to provide as much information to the
user as possible.

[[autodoc]] huggingface_hub.utils.HfHubHTTPError
[[autodoc]] huggingface_hub.errors.HfHubHTTPError

#### RepositoryNotFoundError

[[autodoc]] huggingface_hub.utils.RepositoryNotFoundError
[[autodoc]] huggingface_hub.errors.RepositoryNotFoundError

#### GatedRepoError

[[autodoc]] huggingface_hub.utils.GatedRepoError
[[autodoc]] huggingface_hub.errors.GatedRepoError

#### RevisionNotFoundError

[[autodoc]] huggingface_hub.utils.RevisionNotFoundError
[[autodoc]] huggingface_hub.errors.RevisionNotFoundError

#### BadRequestError

[[autodoc]] huggingface_hub.errors.BadRequestError

#### EntryNotFoundError

[[autodoc]] huggingface_hub.utils.EntryNotFoundError
[[autodoc]] huggingface_hub.errors.EntryNotFoundError

#### BadRequestError
#### RemoteEntryNotFoundError

[[autodoc]] huggingface_hub.utils.BadRequestError
[[autodoc]] huggingface_hub.errors.RemoteEntryNotFoundError

#### LocalEntryNotFoundError

[[autodoc]] huggingface_hub.utils.LocalEntryNotFoundError
[[autodoc]] huggingface_hub.errors.LocalEntryNotFoundError

#### OfflineModeIsEnabled

[[autodoc]] huggingface_hub.utils.OfflineModeIsEnabled
[[autodoc]] huggingface_hub.errors.OfflineModeIsEnabled

## Telemetry

Expand Down
36 changes: 20 additions & 16 deletions docs/source/ko/package_reference/utilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,39 +125,43 @@ except HfHubHTTPError as e:

여기에는 `huggingface_hub`에서 발생하는 HTTP 오류 목록이 있습니다.

#### HfHubHTTPError[[huggingface_hub.utils.HfHubHTTPError]]
#### HfHubHTTPError[[huggingface_hub.errors.HfHubHTTPError]]

`HfHubHTTPError`는 HF Hub HTTP 오류에 대한 부모 클래스입니다. 이 클래스는 서버 응답을 구문 분석하고 오류 메시지를 형식화하여 사용자에게 가능한 많은 정보를 제공합니다.

[[autodoc]] huggingface_hub.utils.HfHubHTTPError
[[autodoc]] huggingface_hub.errors.HfHubHTTPError

#### RepositoryNotFoundError[[huggingface_hub.utils.RepositoryNotFoundError]]
#### RepositoryNotFoundError[[huggingface_hub.errors.RepositoryNotFoundError]]

[[autodoc]] huggingface_hub.utils.RepositoryNotFoundError
[[autodoc]] huggingface_hub.errors.RepositoryNotFoundError

#### GatedRepoError[[huggingface_hub.utils.GatedRepoError]]
#### GatedRepoError[[huggingface_hub.errors.GatedRepoError]]

[[autodoc]] huggingface_hub.utils.GatedRepoError
[[autodoc]] huggingface_hub.errors.GatedRepoError

#### RevisionNotFoundError[[huggingface_hub.utils.RevisionNotFoundError]]
#### RevisionNotFoundError[[huggingface_hub.errors.RevisionNotFoundError]]

[[autodoc]] huggingface_hub.utils.RevisionNotFoundError
[[autodoc]] huggingface_hub.errors.RevisionNotFoundError

#### EntryNotFoundError[[huggingface_hub.utils.EntryNotFoundError]]
#### BadRequestError[[huggingface_hub.errors.BadRequestError]]

[[autodoc]] huggingface_hub.utils.EntryNotFoundError
[[autodoc]] huggingface_hub.errors.BadRequestError

#### BadRequestError[[huggingface_hub.utils.BadRequestError]]
#### EntryNotFoundError[[huggingface_hub.errors.EntryNotFoundError]]

[[autodoc]] huggingface_hub.utils.BadRequestError
[[autodoc]] huggingface_hub.errors.EntryNotFoundError

#### LocalEntryNotFoundError[[huggingface_hub.utils.LocalEntryNotFoundError]]
#### RemoteEntryNotFoundError[[huggingface_hub.errors.RemoteEntryNotFoundError]]

[[autodoc]] huggingface_hub.utils.LocalEntryNotFoundError
[[autodoc]] huggingface_hub.errors.RemoteEntryNotFoundError

#### OfflineModeIsEnabledd[[huggingface_hub.utils.OfflineModeIsEnabled]]
#### LocalEntryNotFoundError[[huggingface_hub.errors.LocalEntryNotFoundError]]

[[autodoc]] huggingface_hub.utils.OfflineModeIsEnabled
[[autodoc]] huggingface_hub.errors.LocalEntryNotFoundError

#### OfflineModeIsEnabledd[[huggingface_hub.errors.OfflineModeIsEnabled]]

[[autodoc]] huggingface_hub.errors.OfflineModeIsEnabled

## 원격 측정[[huggingface_hub.utils.send_telemetry]]

Expand Down
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ def get_version() -> str:
"hf-xet>=1.1.3,<2.0.0; platform_machine=='x86_64' or platform_machine=='amd64' or platform_machine=='arm64' or platform_machine=='aarch64'",
"packaging>=20.9",
"pyyaml>=5.1",
"requests",
"httpx>=0.23.0, <1",
"tqdm>=4.42.1",
"typing-extensions>=3.7.4.3", # to be able to import TypeAlias
]
Expand Down Expand Up @@ -89,6 +89,7 @@ def get_version() -> str:
"soundfile",
"Pillow",
"gradio>=4.0.0", # to test webhooks # pin to avoid issue on Python3.12
"requests", # for gradio
"numpy", # for embeddings
"fastapi", # To build the documentation
]
Expand All @@ -99,7 +100,6 @@ def get_version() -> str:
extras["typing"] = [
"typing-extensions>=4.8.0",
"types-PyYAML",
"types-requests",
"types-simplejson",
"types-toml",
"types-tqdm",
Expand Down
30 changes: 27 additions & 3 deletions src/huggingface_hub/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -520,6 +520,8 @@
"read_dduf_file",
],
"utils": [
"ASYNC_CLIENT_FACTORY_T",
"CLIENT_FACTORY_T",
"CacheNotFound",
"CachedFileInfo",
"CachedRepoInfo",
Expand All @@ -528,13 +530,19 @@
"DeleteCacheStrategy",
"HFCacheInfo",
"HfFolder",
"HfHubAsyncTransport",
"HfHubTransport",
"cached_assets_path",
"configure_http_backend",
"close_client",
"dump_environment_info",
"get_async_session",
"get_session",
"get_token",
"hf_raise_for_status",
"logging",
"scan_cache_dir",
"set_async_client_factory",
"set_client_factory",
],
}

Expand All @@ -550,6 +558,7 @@
# ```

__all__ = [
"ASYNC_CLIENT_FACTORY_T",
"Agent",
"AsyncInferenceClient",
"AudioClassificationInput",
Expand All @@ -564,6 +573,7 @@
"AutomaticSpeechRecognitionOutput",
"AutomaticSpeechRecognitionOutputChunk",
"AutomaticSpeechRecognitionParameters",
"CLIENT_FACTORY_T",
"CONFIG_NAME",
"CacheNotFound",
"CachedFileInfo",
Expand Down Expand Up @@ -653,6 +663,8 @@
"HfFileSystemResolvedPath",
"HfFileSystemStreamFile",
"HfFolder",
"HfHubAsyncTransport",
"HfHubTransport",
"ImageClassificationInput",
"ImageClassificationOutputElement",
"ImageClassificationOutputTransform",
Expand Down Expand Up @@ -824,8 +836,8 @@
"cancel_access_request",
"cancel_job",
"change_discussion_status",
"close_client",
"comment_discussion",
"configure_http_backend",
"create_branch",
"create_collection",
"create_commit",
Expand Down Expand Up @@ -863,6 +875,7 @@
"file_exists",
"from_pretrained_fastai",
"from_pretrained_keras",
"get_async_session",
"get_collection",
"get_dataset_tags",
"get_discussion_details",
Expand All @@ -886,6 +899,7 @@
"grant_access",
"hf_hub_download",
"hf_hub_url",
"hf_raise_for_status",
"inspect_job",
"inspect_scheduled_job",
"interpreter_login",
Expand Down Expand Up @@ -953,6 +967,8 @@
"save_torch_state_dict",
"scale_to_zero_inference_endpoint",
"scan_cache_dir",
"set_async_client_factory",
"set_client_factory",
"set_space_sleep_time",
"snapshot_download",
"space_info",
Expand Down Expand Up @@ -1530,6 +1546,8 @@ def __dir__():
read_dduf_file, # noqa: F401
)
from .utils import (
ASYNC_CLIENT_FACTORY_T, # noqa: F401
CLIENT_FACTORY_T, # noqa: F401
CachedFileInfo, # noqa: F401
CachedRepoInfo, # noqa: F401
CachedRevisionInfo, # noqa: F401
Expand All @@ -1538,11 +1556,17 @@ def __dir__():
DeleteCacheStrategy, # noqa: F401
HFCacheInfo, # noqa: F401
HfFolder, # noqa: F401
HfHubAsyncTransport, # noqa: F401
HfHubTransport, # noqa: F401
cached_assets_path, # noqa: F401
configure_http_backend, # noqa: F401
close_client, # noqa: F401
dump_environment_info, # noqa: F401
get_async_session, # noqa: F401
get_session, # noqa: F401
get_token, # noqa: F401
hf_raise_for_status, # noqa: F401
logging, # noqa: F401
scan_cache_dir, # noqa: F401
set_async_client_factory, # noqa: F401
set_client_factory, # noqa: F401
)
6 changes: 3 additions & 3 deletions src/huggingface_hub/_commit_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ def as_file(self, with_tqdm: bool = False) -> Iterator[BinaryIO]:
config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s]

>>> with operation.as_file(with_tqdm=True) as file:
... requests.put(..., data=file)
... httpx.put(..., data=file)
config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s]
```
"""
Expand Down Expand Up @@ -389,7 +389,7 @@ def _upload_lfs_files(
If an upload failed for any reason
[`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
If the server returns malformed responses
[`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError)
[`HfHubHTTPError`]
If the LFS batch endpoint returned an HTTP error.
"""
# Step 1: retrieve upload instructions from the LFS batch endpoint.
Expand Down Expand Up @@ -500,7 +500,7 @@ def _upload_xet_files(
If an upload failed for any reason.
[`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
If the server returns malformed responses or if the user is unauthorized to upload to xet storage.
[`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError)
[`HfHubHTTPError`]
If the LFS batch endpoint returned an HTTP error.

**How it works:**
Expand Down
5 changes: 4 additions & 1 deletion src/huggingface_hub/_commit_scheduler.py
Original file line number Diff line number Diff line change
Expand Up @@ -315,10 +315,13 @@ def __len__(self) -> int:
return self._size_limit

def __getattribute__(self, name: str):
if name.startswith("_") or name in ("read", "tell", "seek"): # only 3 public methods supported
if name.startswith("_") or name in ("read", "tell", "seek", "fileno"): # only 4 public methods supported
return super().__getattribute__(name)
raise NotImplementedError(f"PartialFileIO does not support '{name}'.")

def fileno(self):
raise AttributeError("PartialFileIO does not have a fileno.")

def tell(self) -> int:
"""Return the current file position."""
return self._file.tell()
Expand Down
21 changes: 6 additions & 15 deletions src/huggingface_hub/_snapshot_download.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from pathlib import Path
from typing import Dict, Iterable, List, Literal, Optional, Type, Union

import requests
import httpx
from tqdm.auto import tqdm as base_tqdm
from tqdm.contrib.concurrent import thread_map

Expand Down Expand Up @@ -36,7 +36,6 @@ def snapshot_download(
library_name: Optional[str] = None,
library_version: Optional[str] = None,
user_agent: Optional[Union[Dict, str]] = None,
proxies: Optional[Dict] = None,
etag_timeout: float = constants.DEFAULT_ETAG_TIMEOUT,
force_download: bool = False,
token: Optional[Union[bool, str]] = None,
Expand Down Expand Up @@ -85,12 +84,9 @@ def snapshot_download(
The version of the library.
user_agent (`str`, `dict`, *optional*):
The user-agent info in the form of a dictionary or a string.
proxies (`dict`, *optional*):
Dictionary mapping protocol to the URL of the proxy passed to
`requests.request`.
etag_timeout (`float`, *optional*, defaults to `10`):
When fetching ETag, how many seconds to wait for the server to send
data before giving up which is passed to `requests.request`.
data before giving up which is passed to `httpx.request`.
force_download (`bool`, *optional*, defaults to `False`):
Whether the file should be downloaded even if it already exists in the local cache.
token (`str`, `bool`, *optional*):
Expand Down Expand Up @@ -163,22 +159,18 @@ def snapshot_download(
try:
# if we have internet connection we want to list files to download
repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision)
except (requests.exceptions.SSLError, requests.exceptions.ProxyError):
# Actually raise for those subclasses of ConnectionError
except httpx.ProxyError:
# Actually raise on proxy error
raise
except (
requests.exceptions.ConnectionError,
requests.exceptions.Timeout,
OfflineModeIsEnabled,
) as error:
except (httpx.ConnectError, httpx.TimeoutException, OfflineModeIsEnabled) as error:
# Internet connection is down
# => will try to use local files only
api_call_error = error
pass
except RevisionNotFoundError:
# The repo was found but the revision doesn't exist on the Hub (never existed or got deleted)
raise
except requests.HTTPError as error:
except HfHubHTTPError as error:
# Multiple reasons for an http error:
# - Repository is private and invalid/missing token sent
# - Repository is gated and invalid/missing token sent
Expand Down Expand Up @@ -315,7 +307,6 @@ def _inner_hf_hub_download(repo_file: str):
library_name=library_name,
library_version=library_version,
user_agent=user_agent,
proxies=proxies,
etag_timeout=etag_timeout,
resume_download=resume_download,
force_download=force_download,
Expand Down
Loading
Loading