HTTP content-encoding: zstd

[Malvoz]

Stumbled upon a relatively new content-encoding called zstd (Zstandard).

Defined in:
https://tools.ietf.org/id/draft-kucherawy-dispatch-zstd-03.html
https://datatracker.ietf.org/doc/html/rfc8878

Repo:
https://github.com/facebook/zstd

Blogpost:
https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/

[felixhandte]

To clarify, it's a body content encoding. The content-coding identifier is zstd. It's been standardized in RFC 8478. It's supported by some non-browser clients and servers (which I guess is outside your purview), and we're working towards browser support.

[andrew-aladev]

It's supported by some non-browser clients and servers (which I guess is outside your purview), and we're working towards browser support.

I saw that implementations, saw zstd_dict_file option and tried to search around all internet about an example and failed. There are no public dictionaries. I think Facebook don't want to give money on that research. You should make a research yourself and create dictionary for your private application (based on production database).

So zstd is the option for private client + server applications like... facebook! So why people are creating tasks about integrating zstd into common purpose applications like browsers?

I can't understand a reason for integrating zstd even in caniuse today.

[felixhandte]

Hi @andrew-aladev,

In general, using a dictionary with Zstd is entirely optional, you don't need one. And in fact, the HTTP extension specified in the above-linked RFC, the zstd content-coding, does not use a dictionary. The dictionary support in the nginx extension is non-standard (see tokers/zstd-nginx-module#2 for context). So none of that is really relevant to this topic.

I'll comment on your other concerns in a reply on facebook/zstd#1669.

[andrew-aladev]

using a dictionary with Zstd is entirely optional

You can use zstd without dictionary or with your own dictionary.

Caniuse is related to zstd support in common purpose applications like web browsers. I think that nobody will start to integrate zstd into web browsers without clear dictionary registry. Zstd RFC doesn't declear a way of synchronizing and protecting dictionaries. It will take many years to do it.

[tuxayo]

@felixhandte

It's supported by some non-browser clients and servers (which I guess is outside your purview), and we're working towards browser support.

This confirms that it's too early for inclusion in Can I use, right?

[xorgy]

So zstd is the option for private client + server applications like... facebook! So why people are creating tasks about integrating zstd into common purpose applications like browsers?

This is an extremely narrow way to read the situation. I use zstd at work, it has excellent technical properties that make sense for basically any user of general-purpose lossless compression. I've seen it used in the wild with HTTP APIs; yes it's internal for now, but it not being included in browsers is a matter of browsers picking up support for it... just like everything on caniuse.

[andrew-aladev]

This is an extremely narrow way to read the situation. I use zstd at work, it has excellent technical properties that make sense for basically any user of general-purpose lossless compression. I've seen it used in the wild with HTTP APIs; yes it's internal for now, but it not being included in browsers is a matter of browsers picking up support for it... just like everything on caniuse.

I am using it too, it is very effective in private applications where you can train dictionary based on real world data. General-purpose lossless compression using zstd without dictionary is almost meaningless.

Caniuse is related to web browsers. zstd without dictionary is not able to compete with brotli that uses single static dictionary optimized for web data. This question can be resolved by registration of zstd ditionaries optimized for web data in iana registries. I am waiting when facebook will make that step.

[felixhandte]

@andrew-aladev, I would hesitate to generalize a specific experience you've had into a global statement about one compressor being universally better than another. Compression is notoriously tricky that way--the specifics of the use case matter. "using zstd without dictionary is almost meaningless" is an almost meaningless statement. I can offer a counter-example: we chose zstd over brotli for HTTP traffic at Facebook.

Even without a dictionary, we found that zstd achieved equivalent compression, compared to brotli, at our target compression speed and performed much better on the client side (as a result of its much higher decompression speed). On mobile chipsets, decompression efficiency is actually very important, and that produced meaningful improvements in end-to-end / top-level metrics.

[andrew-aladev]

I would hesitate to generalize a specific experience you've had into a global statement about one compressor being universally better than another. Compression is notoriously tricky that way--the specifics of the use case matter. "using zstd without dictionary is almost meaningless" is an almost meaningless statement. I can offer a counter-example: we chose zstd over brotli for HTTP traffic at Facebook.

I was sure that this question will appear: I am making that research, will provide results later.

Please provide your research results based on HTTP traffic on official website. Existing results has no sense: you have re-compressed books, x-ray pictures, etc. Please provide clear results that web developer wants (we are talking about caniuse): graphs for xhtml, graphs for css, graphs for js, graphs for fonts, etc.

Even without a dictionary, we found that zstd achieved equivalent compression, compared to brotli, at our target compression speed and performed much better on the client side (as a result of its much higher decompression speed). On mobile chipsets, decompression efficiency is actually very important, and that produced meaningful improvements in end-to-end / top-level metrics.

I've already reproduced similar results (using small amounts of web data): zstd compression ratio was significantly lower than brotli, but compression and decompression speed was faster. Now just to be clear I want to provide reproducable results.

[xorgy]

I'll second @felixhandte 's point here, and go a little further. Decompression efficiency is extremely good with zstd.

Zstd can often produce better ratios and better decompression efficiency than lz4 at the same time, even in lz4's sweetspot. In my use case (being careful not to prematurely end frames) it is a massive difference. Far as I can tell, zstd decompresses more efficiently than any common compressor at equivalent ratios, for both my plain text and binary use cases. In my experience, zstd decompresses plain text an integer multiple of times faster than lz4 and gzip do, with dramatically better ratios; and for similar ratios between Brotli and zstd, (zstd can often do better), Brotli takes several times longer to compress. There are a lot of use cases where on-the-fly Brotli simply doesn't make sense because it will increase transfer times at any ratio that makes it worthwhile; for zstd, that list of use cases is several times smaller.

The data's not mine to give, but my two cents are.

In any event, it seems to me like the thing that should decide whether Caniuse shows something is whether any of the tracked web browsers have a release with support. Debating the merits of zstd without dictionaries is neither here nor there, once it's something available in browsers.