Serve static gz versions in download files

[JosePineiro]

When this function is called requesting a file (e.g., image.jpg), it looks for an image.jpg.gz file and, if found, sends it using the Content_Encoding = gzip header.
This behavior is disabled when download=true.
Is there a reason for this?
I suggest removing this limitation, as files can also be sent with Content_Encoding = gzip encoding.

[mathieucarbou]

Good catch, I never noticed that!
Maybe @me-no-dev knows why... ?

(update: verified in history that this behavior is there since the beginning..)

[mathieucarbou]

@JosePineiro : maybe because originally, ESPASyncWS was shipped with a file browser / upload UI, so let's say the UI lists the content of the folder foo which contains:

• image.jpg.gz
• image.jpg

with download == true:

• clicking image.jpg.gz:

  • would work with current code
  • would work with proposal because image.jpg.gz.gz is not found and falls back to image.jpg.gz

• clicking image.jpg:

  • would work with current code
  • would not work with proposal because will download image.jpg.gz instead of image.jpg

with download == false (file serving), if we happen to have both files on disk, image.jpg will be ignored when it is requested since the .gz version will be picked up.

What you are asking for is to align these 2 behaviors (file serving and file download) so that any gz version will hide and be picked up when the non-gzipped version is requested.

I don't know how much it breaks backward compatibility....

[JosePineiro]

The current condition is: if (!download && !fs.exists(path) && fs.exists(gzPath))
The proposed condition is: if (!fs.exists(path) && fs.exists(gzPath))

• If an image.jpg is requested and image.jpg and image.jpg.gz exist, image.jpg would be download without a Content-Encoding header.
• If an image.jpg is requested and not image.jpg exist and image.jpg.gz exist, image.jpg.gz would be download with Content-Encoding=gzip header
• If an image.jpg.gz is requested and image.jpg and image.jpg.gz exist, image.jpg.gz would be download without a Content-Encoding header.
  Whenever a GZ file is served, with or without Content-Encoding, eTag will be used.

Do you see any flaw, or breaks backward compatibility, in this behavior?

[mathieucarbou]

Yes, there is a behavior change. The new code would change the behavior when download == true.

when download == true: there is no "virtual" file switch: the requested file is the one downloaded if it exists. So if image.jpg is requested, image.jpg is downloaded, not image.jpg.gz.

What if image.jpg and image.jpg.gz do not have the same content ? For example, what if image.jpg.gz is an old version of image.jpg ? I am speaking in the case of a file system used to download and upload files, like a file server here. I am not speaking about serving web content.

when download == false: same in both cases.

[mathieucarbou]

ESPAsyncWebServer is not only used to serve http resources... It can be used in many other cases... and a file manager is also a typical usage.

[JosePineiro]

Sorry I didn't explain myself well. English isn't my native language.
With the proposed change:
If image.jpg is requested, image.jpg is downloaded, not image.jpg.gz.
Image.jpg.gz is only downloaded if image.jpg doesn't exist.

The behavior change:
If the user requests image.jpg and it doesn't exist, a 404 error will currently be returned. With proposed change, image.jpg.gz will be searched for and, if so, will be sent as image.jpg encoded with Content-Encoding=gzip.

[mathieucarbou]

The behavior change:
If the user requests image.jpg and it doesn't exist, a 404 error will currently be returned. With proposed change, image.jpg.gz will be searched for and, if so, will be sent as image.jpg encoded with Content-Encoding=gzip.

Exactly.
Note: in the case where download == true.

so now the question is: if a user is asking to download image.png, is it normal to serve the gzip version instead ?

this breaks the principle of serving what’s on disk - like for the file manager use case which should represent a view on what is available.

There might be some use cases where it might be wrong to serve (download) a file from disk when the non gzip version is asked for. And that’s what I would like to be certain of.

we are only speaking about the download case of course.

I am not against the change.

But I will be against it if this behavior change leads to some breakage for some valid use cases.

[JosePineiro]

At this point, I think we're both clear on what the change is and what it entails.
My intention is to add the ability to send a pre-compressed file with Content-Encoding=gzip.
The possibilities I can think of are:
1 - Send image.jpg.gz whenever image.jpg doesn't exist (this is the same behavior that occurs when no download is made). It is clear what the implications are.
2 - Change three functions to:
void send(FS &fs, const String &path, const char *contentType, bool download, AwsTemplateProcessor callback, bool sendGz=false)
AsyncWebServerResponse *beginResponse(FS &fs, const String &path, const char *contentType, bool download, AwsTemplateProcessor callback, bool sendGz=false)
AsyncFileResponse(FS &fs, const String &path, const char *contentType, bool download, AwsTemplateProcessor callback, bool sendGz=false): AsyncAbstractResponse(callback)
This adds the new functionality without changing the default behavior, but
But it adds one more parameter to the functions, and they're no longer similar to the rest. The function's behavior would be different depending on whether there's a download=true, which isn't very clear.
3 - Not implementing the functionality

Which option do you think is better?

[mathieucarbou]

I agree there is no best answer here.

I would prefer to change the default behavior to what you proposed first. It would be consistent also both for resource serving and file download.

We have to wait for @me-no-dev to read this discussion and comment because he is the original author who introduced this different behavior.

Maybe there are other reasons too…

I would like to have this discussion sorted out and if a PR is required, wait for it to be merged before cutting a new release with all the latest changes.

[me-no-dev]

If an image.jpg is requested and image.jpg and image.jpg.gz exist, image.jpg would be download without a Content-Encoding header.
If an image.jpg is requested and not image.jpg exist and image.jpg.gz exist, image.jpg.gz would be download with Content-Encoding=gzip header
If an image.jpg.gz is requested and image.jpg and image.jpg.gz exist, image.jpg.gz would be download without a Content-Encoding header.
Whenever a GZ file is served, with or without Content-Encoding, eTag will be used.

I agree to this. As long as the requested for download file is returned if it exists first (without content encoding gzip), then it should be OK. This should still work with the FS Browser.

[mathieucarbou]

@JosePineiro : did you see @me-no-dev answer ? Should I wait for a PR from your side before cutting anew release ?

[JosePineiro]

I just submitted a new pull request implementation as indicated by me-not-dev

[JosePineiro]

Merged pull request whit this change