Crate downlowd 

downlowd

Downloading a file is easy. Just make an HTTP request, and write the results to a file, right? That works, but it doesn’t cover a lot of corner cases. downlowd supports:

• Streaming the file to disk, instead of downloading it to memory and then writing it to disk. This is both faster and far more memory efficient for large files.
• Progress callback for displaying progress bar.
• Resuming file downloads.
• Files are written to disk as “filename.part” and then renamed to “filename” on completion, to make it obvious the file isn’t complete.
• Automatic retries for flakey network connections and servers, with exponential backoff.
• Uses content-disposition header to retrieve the name of the file.
• Support for bandwidth restrictions.
• Blocking client which is based on ureq so the blocking case doesn’t depend on reqwest or tokio, making a smaller executable and faster build times.

Documentation

See the documentation at docs.rs.

Crate Features

• async - Enabled by default. Provides Client for downloading files. This uses reqwest and tokio under the hood.
• blocking - Provides the [blocking::Client] for downloading files, which is based on ureq. To use the blocking client, install will cargo add downlowd --no-default-features -F blocking.

Usage

This is the simplest example:

use downlowd::Client;

let client = Client::new();
let result = client
    .get("http://localhost:8089/hello.txt")
    .destination(dirname)
    .send()
    .await?;

assert_eq!(&result.path, &dirname.join("hello.txt"));
let file_contents = tokio::fs::read_to_string(&result.path).await?;
assert_eq!(file_contents, "hello world");

This is a short example, but it has a lot packed into it. First, since we’ve passed in a directory as the destination, this will work out what filename the file should be saved as In this case, the filename is derived from the URL, but if the server responds to a HEAD request with a Content-Disposition header with a filename, we’ll use that filename. Note here we could also specify a filename instead of a directory name, and then downlowd would write our file to the specified filename.

If the file already exists, and has the correct length, then downlowd will just report success right away! If not, then we’ll start downloading the file into a file named hello.txt.part. The file will be renamed to hello.txt once the download is complete. While the download is in progress, a “sidecar” file named hello.txt.downloadinfo will be written alongside the file which will contain cache information about the file (the etag header, the last-modified header, etc…). The sidecar file is used to help determine whether or not a file has changed on the server if the download is interrupted and needs to be resumed.

If there’s an error during the download, such as a network error, or the transfer is interrupted, or the server returns a 5xx error, then downlowd will automatically retry the file, with an exponential backoff between retries. By default, downlowd will retry forever. Calling max_retries() will set a maximum number of tries, but note that downlowd will reset the retry counter if any progress is made downloading the file.

Reporting Progress

There are a couple of ways you can hook into downlowd to report on progress. The on_progress handler is called once at the start of the download, and then whenever bytes are downloaded.

let client = Client::new();
let result = client
    .get("http://localhost:8089/hello.txt")
    .destination(dirname.join("file.txt"))
    .on_progress(|progress| {
        println!(
            "Downloaded {} of {} bytes",
            progress.bytes(),
            progress.remote_length().unwrap()
        );
    })
    .send()
    .await?;

assert_eq!(&result.path, &dirname.join("file.txt"));

The progress handle can also be used to cancel a download via progress.cancel(). There’s quite a bit of data about the download that can be retrieved from the progress handle. You can see an example in the examples folder which uses indicatif to render a pretty download progress bar. If you’ve cloned the repo, you can run it with something like:

cargo run https://releases.ubuntu.com/24.04.3/ubuntu-24.04.3-desktop-amd64.iso .

Customizing Retries

You can also use the on_retry() method to register a handler that will be run immediately prior to a retry. This can be used to customize the backoff, or cancel the download:

use downlowd::{Client, Error};

let client = Client::new();
let result = client
    .get("http://localhost:8089/hello.txt")
    .destination(dirname)
    .on_retry(|r| {
        if matches!(r.error(), Error::FileChanged { .. }) {
            // No delay if the file changed.
            r.set_delay(Duration::ZERO);
        } else {
            r.set_delay(downlowd::exponential_backoff(
                Duration::from_secs(1),
                Duration::from_secs(30),
                r.retries(),
            ));
        }
    })
    .send()
    .await?;

Again, you can call r.cancel() here to not retry at all, and instead fail the entire download.

Client Options

You can create a custom client using the ClientBuilder:

use downlowd::{ClientBuilder, Client};

let client = ClientBuilder::new()
    .user_agent("my-cool-app")
    .header("Authorization", "Bearer secret-token")
    .build()?;

let result = client
    .get("http://localhost:8089/hello.txt")
    // Can set headers at the request level, too.
    .header("x-my-custom-header", "canon")
    .destination(dirname.join("file.txt"))
    .send()
    .await?;

Structs

Client

A client for downloading files over HTTP. A Client uses an internal connection pool to manage HTTP connections, and has a shared rate limiter, so it is recommended to create a single Client and reuse it for multiple downloads. Clients are cheap to clone.

ClientBuilder

Builder for creating a Client with custom configuration.

Download

Represents a file about to be downloaded.

DownloadResult

The result of a download operation.

HeaderMap

A set of HTTP headers

HeaderValue

Represents an HTTP header field value.

ProgressHandle

RetryHandle

Handle passed to the on_retry callback.

Enums

Error

Errors from the downlowd crate. Some errors are “retryable”, meaning that downlowd will automatically retry the download if the error occurs. Retryable errors are marked as such.

Traits

IntoHeaderName

A marker trait used to identify values that can be used as insert keys to a HeaderMap.

IntoUri

A trait for converting a type into a url::Url, heavily inspired by reqwest::IntoUri.

Progress

A trait for reporting progress of a download.

Functions

exponential_backoff

Function which calculates an exponential backoff duration based on the number of failures.

Type Aliases

RetryHandler

Callback function to invoke when trying a download.