feat(zip): Support deflate compression (bring your own)

[kriskowal]

Preface

Endo has its own implementation of ZIP because a survey of the options available on npm, I found that it would be easier to create a library that was pure JavaScript, would work in the XS engine, would operate directly on Uint8Arrays, and would have received sufficient scrutiny to be sure it did not inadvertently invite pathological decompression. Many legacy libraries entrained vestigial code for modeling ZIP files as byte strings (which would have invited pathological performance on XS), entrained multiple compression systems regardless of whether there were needed, and had unnecessary layers of abstraction. It is a safe bet that they were generally designed to tolerate the widest variety of valid ZIP files rather than strictly those that were well-formed. Many were quite old and missed opportunities to use modern idioms. In short, our need was for a more compact, stricter ZIP library.

It proved to be straightforward to implement the subset of ZIP we needed, allowing folks to use off-the-shelf ZIP utilities to open the archives, but obligating them to use our bundler to create them. This also gave us an opportunity to scrutinize the implementation and to take the strict road instead of the permissive road when presented the option, in order to obviate hazards another implementation might have missed like overlapping content regions and explosive decompression. We omitted compression entirely.

In the intervening years, the web and Node.js beginning with version 20 provided native support for asynchronous raw deflate compression and decompression. So, adding support for DEFLATE has become a low-hanging fruit for those environments. And, reflecting on our own qualms with other libraries, gave us an opportunity to add compression without forcing all dependent projects to entrain the gamut of compression concerns.

One of our central library design principles is to avoid barrel modules. Or, generally, a dependent package should not need to entrain any modules it is not using. So, if you are reading a zip file, you should not be obligated to retain the code for writing a zip file, and vice versa. if you do not need compression, you should not entrain a compression library.

We are generally also fans of the principle of least authority, which leads us to generally inject dependencies.

So, we use dependency injection to both select and enable compression for ZIP archives. We also leave open the option of adding support for other compression systems with additional dependency injection.

We continue to omit entirely the concern of being able to read or write ZIP files that do not fit in a processes’s memory, for which we would want to rely on an asynchronous storage with seek, read, and write capabilities. This we leave as an exercise for the event of a future need.

Description

This change adds support for DEFLATE compression and decompression to the @endo/zip package. This feature is enabled by injecting the deflate or inflate capability to the ZipReader or ZipWriter constructor. The web and Node.js 18 provide the necessary facilities without any additional dependencies.

So, we reserve the option to package @endo/deflate in the future, using browser and node conditions to use the native support in those environments, and falling through to a JavaScript implementation by default.)

Security Considerations

None, to my knowledge.

Scaling Considerations

None, to my knowledge.

Documentation Considerations

This change includes updates for README and NEWS.

Testing Considerations

This change minimally tests the new asynchronous DEFLATE support.

Compatibility Considerations

This change deprecates the read and write functions because they were previously synchronous and when providing both synchronous and asynchronous functions, we prefer to qualify the synchronous variant. To that end, we’ve introduced get, getNow, set, and setNow. The deprecated function remains in place.

Although introducing asynchrony is necessary to embrace the platform’s asynchronous DEFLATE, we manage to preserve backward compatibility by ensuring that the ZipReader and ZipWriter continue to lack compression by default. Injecting DEFLATE features obligates the user to change usage. This is a minimal hindrance in practice since the primary user is CompartmentMapper which uses adapters to treat the ZIP archive as an async file system, since it must assume that some storage systems are async-only.

Upgrade Considerations

None.