Generate WebP image on-the-fly with Rust! Support all main platforms, macOS, Linux and Windows.
Speaking of switching to WebP image, at the first glance, I just did it with a very naive approach.
Then @Nova wrote a Node.JS server that can serve JPG/PNGs as WebP format on-the-fly. You can find that at n0vad3v/webp_server.
A few days ago, @Nova and @Benny rewrite the WebP Server in Golang, webp-sh/webp_server_go.
And that looks really promising, the size of the webp server, according to its description, had reduced from 43 MB to 15 MB, and it is a single binary instead of webp_server with node_modules.
I cloned that project and added a tiny feature. However, I just found that although it is absolutely easy to implement the tiny feature, there is a potential design issue with the fasthttp
module. In order to get everything work, it took me about 4 hours to debug on it.
Finally, it turned out to be a pointer of an internal variable (ctx.Request.uri
, or so) was directly returned from ctx.Path()
, and if users invoke ctx.SendFile(filepath)
, the ctx.Request.uri
will be set to filepath
, which will also propagate to all variables that hold the shared value of ctx.Path()
. You may visit my blog post for details.
Well, in aforementioned blog post, I said that it would be better if it was written in Rust. Now, let's make it come true and push the webp server even further.
webp_server
withnode_modules
: 43M
Darwin amd64 | Linux amd64 | Linux arm64 | Windows amd64 | |
---|---|---|---|---|
webp-server-go | 10.6 MB | 9.52 MB | 8.83 MB | 9.31 MB |
webp-server-rs | 2.0 MB | 2.5 MB | 2.07 MB | 2.41 MB |
- webp_server: Clone the repo -> npm install -> run with pm2
- webp-server-go: Download a single binary -> Run
- webp-server-rs: Download a single binary -> Run
Not really tested. But IMHO it should be as fast as golang version.
webp-server-rust allows you set directory-level config and all libwebp parameters are available to you.
webp-server-rust supports more image formats than webp-server-go.
Also GIF support is under consideration, currently webp-server-rust can only output the first frame of a GIF file.
Format | Converting |
---|---|
PNG | All supported color types |
JPEG | Baseline and progressive |
BMP | Yes |
ICO | Yes |
TIFF | Baseline(no fax support) + LZW + PackBits |
PNM | PBM, PGM, PPM, standard PAM |
DDS | DXT1, DXT3, DXT5 |
Please set proxy rules in Nginx / Apache configuration file to match specific types of files. example
Shamefully copy and paste most of the usage guidelines from webp-sh/webp_server_go, given that they are basically identical.
Regarding the img_path
section in config.json. If you are serving images at https://example.com/images/aya.jpg and your files are at /var/www/site/images/aya.jpg, then img_path
shall be /var/www/site
.
Download the webp-server from release page.
The webp-server-rs-${ver}-linux-amd64.deb
package will ONLY INSTALL the binary to /usr/local/bin/webp-server-rs
, the config file needs to be edited following the guideline below.
Wanna build your own binary? Check out build section
Create a config.json as follows to face your need.
{
"host": "127.0.0.1",
"port": 3333,
"img_path": "./images",
"webp_path": "./cache",
"global_config": {
"quality": 80
}
}
Under global_config
key, you can overwrite all parameters available from libwebp, which provides precise control to you. (also available in directory-level config)
int lossless; // Lossless encoding (0=lossy(default), 1=lossless).
float quality; // between 0 and 100. For lossy, 0 gives the smallest
// size and 100 the largest. For lossless, this
// parameter is the amount of effort put into the
// compression: 0 is the fastest but gives larger
// files compared to the slowest, but best, 100.
int method; // quality/speed trade-off (0=fast, 6=slower-better)
string image_hint; // Hint for image type (lossless only for now).
// - default: default preset.
// - picture: digital picture, like portrait, inner shot
// - photo: outdoor photograph, with natural lighting
// - graph: Discrete tone image (graph, map-tile etc).
int target_size; // if non-zero, set the desired target size in bytes.
// Takes precedence over the 'compression' parameter.
float target_psnr; // if non-zero, specifies the minimal distortion to
// try to achieve. Takes precedence over target_size.
int segments; // maximum number of segments to use, in [1..4]
int sns_strength; // Spatial Noise Shaping. 0=off, 100=maximum.
int filter_strength; // range: [0 = off .. 100 = strongest]
int filter_sharpness; // range: [0 = off .. 7 = least sharp]
int filter_type; // filtering type: 0 = simple, 1 = strong (only used
// if filter_strength > 0 or autofilter > 0)
int autofilter; // Auto adjust filter's strength [0 = off, 1 = on]
int alpha_compression; // Algorithm for encoding the alpha plane (0 = none,
// 1 = compressed with WebP lossless). Default is 1.
int alpha_filtering; // Predictive filtering method for alpha plane.
// 0: none, 1: fast, 2: best. Default if 1.
int alpha_quality; // Between 0 (smallest size) and 100 (lossless).
// Default is 100.
int pass; // number of entropy-analysis passes (in [1..10]).
int preprocessing; // preprocessing filter:
// 0=none, 1=segment-smooth, 2=pseudo-random dithering
int partitions; // log2(number of token partitions) in [0..3]. Default
// is set to 0 for easier progressive decoding.
int partition_limit; // quality degradation allowed to fit the 512k limit
// on prediction modes coding (0: no degradation,
// 100: maximum possible degradation).
int emulate_jpeg_size; // If true, compression parameters will be remapped
// to better match the expected output size from
// JPEG compression. Generally, the output size will
// be similar but the degradation will be lower.
int thread_level; // If non-zero, try and use multi-threaded encoding.
int low_memory; // If set, reduce memory usage (but increase CPU use).
int near_lossless; // Near lossless encoding [0 = max loss .. 100 = off(default)].
int exact; // if non-zero, preserve the exact RGB values under
// transparent area. Otherwise, discard this invisible
// RGB information for better compression. The default
// value is 0.
int use_delta_palette; // reserved for future lossless feature
int use_sharp_yuv; // if needed, use sharp (and slow) RGB->YUV conversion
By placing a .webp-conf
in intented directories, you can control the encoding mode
and quality
applied on the images inside that directory (the directory-level config will NOT propagate to its subdirectories).
If there is no directory level config file (.webp-conf
) in the directory, then parameters in config.json
will be used.
For example, we have such file layout
images
├── lossless
│ ├── .webp-conf
│ └── webp-server.jpeg (480911 bytes)
├── lossy
│ ├── .webp-conf
│ └── webp-server.jpeg (480911 bytes)
├── nearlossless
│ ├── .webp-conf
│ └── webp-server.jpeg (480911 bytes)
└── webp-server.jpeg (480911 bytes)
And the config files,
config.json
, lossy encoding, quality will be used to reduce the size of output image
{
"host": "127.0.0.1",
"port": 3333,
"img_path": "./images",
"webp_path": "./cache",
"global_config": {
"quality": 80
}
}
images/lossless/.webp-conf
, lossless encoding
{
"lossless": 1
}
images/nearlossless/.webp-conf
, near lossless encoding, quality
and near_lossless
will be used to reduce the size of output image
{
"quality": 60,
"near_lossless": 80
}
images/lossy/.webp-conf
, lossy encoding, quality will be used to reduce the size of output image
{
"quality": 40
}
And corresponding WebP images will be generated based on aforementioned rules,
cache
├── lossless
│ └── webp-server.jpeg.1579413991.webp (1938270 bytes)
├── lossy
│ └── webp-server.jpeg.1579413991.webp (160502 bytes)
├── nearlossless
│ └── webp-server.jpeg.1579413991.webp (212022 bytes)
└── webp-server.jpeg.1579413991.webp (317612 bytes)
Run the binary like this:
./webp-server-rs -c /path/to/config.json
# or
./webp-server-rs --config /path/to/config.json
To enable prefetch feature, using -p
.
Prefetch will be ran in background, WebP image service will operate normally.
./webp-server-rs -c /path/to/config.json -p
# or
./webp-server-rs --config /path/to/config.json --prefetch
By default, this will use all logical CPUs available in the system.
To set max allowed number of threads that prefetch can use, using -j
.
./webp-server-rs -c /path/to/config.json -p -j 4
# or
./webp-server-rs --config /path/to/config.json --prefetch --jobs 4
Use screen or tmux to avoid being terminated. Let's take screen for example
screen -S webp
./webp-server-rs --config /path/to/config.json
Don't worry, we've got you covered!
cp webp-image.service /lib/systemd/system/
systemctl daemon-reload
systemctl enable webp-image.service
systemctl start webp-image.service
This systemd service script will assume that the binary is located at /usr/local/bin/webp-server-rs
and the config file is located at /etc/webp-server-rs/config.json
. It also uses /var/cache/webps
as working directory.
Let Nginx to proxy_pass http://localhost:3333/;
, and your webp-server-rs
is on-the-fly
location ~* \.(png|jpg|jpeg)$ {
proxy_pass http://127.0.0.1:3333;
}
Install latest version of Rust, clone the repo, and then...
# debian
apt install cmake curl
# macOS / Linux
make libwebp
# windows
curl https://codeload.github.com/webmproject/libwebp/tar.gz/v1.1.0 -o v1.1.0.tar.gz
tar -xzf v1.1.0.tar.gz
mkdir -p libwebp-1.1.0/build && cd libwebp-1.1.0/build
## for VS 9.0 to VS 15.0
## please set generator that fits your system, e.g., "Visual Studio 15 2017 Win64"
## for VS 16.0
## you may use "Visual Studio 15 2017 Win64", or -G "Visual Studio 16 2019" -A x64
cmake -G "Visual Studio 15 2017 Win64" \
-DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../../deps \
-DWEBP_BUILD_CWEBP=OFF -DWEBP_BUILD_DWEBP=OFF -DWEBP_BUILD_GIF2WEBP=OFF \
-DWEBP_BUILD_IMG2WEBP=OFF -DWEBP_BUILD_VWEBP=OFF -DWEBP_BUILD_WEBPINFO=OFF \
-DWEBP_BUILD_WEBPMUX=OFF -DWEBP_BUILD_EXTRAS=OFF -DWEBP_BUILD_WEBP_JS=OFF \
-DWEBP_BUILD_ANIM_UTILS=OFF -DWEBP_NEAR_LOSSLESS=ON ..
cmake --build . --config Release
cmake --install .
cd ../..
# macOS / Linux, and windows user who has `make`
make libwebpwrapper
# windows
mkdir -p webpwrapper/build
cd webpwrapper/build
## for VS 9.0 to VS 15.0
## please set generator that fits your system, e.g., "Visual Studio 15 2017 Win64"
## for VS 16.0
## you may use "Visual Studio 15 2017 Win64", or -G "Visual Studio 16 2019" -A x64
cmake -G "Visual Studio 15 2017 Win64" ..
cmake --build . --config Release
cmake --install .
# binary will be located at `target/release/webp-server-rs`
cargo build --release
# test
cargo test --release