Vendor chunkhash changes when app code changes

[kevinrenskers]

I'm trying to split my vendor code from my app code and got it all working. However, every time I change some code in my app and create a new build, the vendor file gets a new chunkhash and thus is downloaded by the clients again. The contents are exactly the same though.

Here is a fully working small example showing the problem: https://github.com/kevinrenskers/chunkhash-problem

And as you can see, the contents of these files are 100% identical even though the chunkhash keeps changing:

[alexhancock]

I am also experiencing this issue

[sokra]

Looks like a conflict between the plugin and the lastest changes. You could try a previous webpack version (thay had a bug that [chunkhash] didn't change when the content change).

[kevinrenskers]

Any idea which webpack version I should try?

[kevinrenskers]

Seems that going back to 1.10.1 works. 1.10.2 and up break the chunkhash as described above.

[sokra]

It doesn't "break" it, it fixes it. The chunk hash should change when the hashes of the child chunks change... ;)

[kevinrenskers]

But if the contents of the vendor file don't change then its hash shouldn't change right? How else are we supposed to have long term caching if the filename changes every time I make a change in my app code?

[sokra]

That's caused by the chunk-manifest-webpack-plugin. It move the hashes of the child chunks out of the entry chunk. I propably need to rewrite it...

[kevinrenskers]

I don't really understand. Isn't the whole point of chunk-manifest-webpack-plugin to remove the child chunk hashes exactly so that the contents of the entry chunk don't change and thus its hash should stay identical? It does work like that with 1.10.1.

Yet, you're saying that 1.10.2 have actually fixed this rather then breaking it. So, is the chunk-manifest-webpack-plugin doing something wrong? Or webpack? Or in other words, how am I supposed to have a working setup where changing the app code does not change the vendor's file name? :) Like I said it seems to work with 1.10.1 but your comments confuse me..

Are you able to get such a setup working with the latest webpack, as I've tried in the example project (linked above)?

[Zagitta]

I'm attempting the same setup as @kevinrenskers and naturally have run into the same issue. If we possibly can come up with a workaround I'd appreciate it 👍

[kevinrenskers]

Even going back to 1.10.1 only fixes the problem some of the time. Other times I create a build, the vendor file still gets a new file name. A fix would be extremely welcome.

[lsanwick]

We're seeing this issue as well, any consistent fix would be great.

[airwin]

same problem. I remove the chunkHash and use my own hash now :(.

[lsanwick]

If you're willing to share, what method are you using to produce a hash?

[airwin]

@lsanwick I just add my release date after the script path in the html file (eg: lib.js?v=20150818). It's a little stupid and all of this is unnecessary if webpack's chunkHash works corretly. :(

[Foxandxss]

Makes sense, @bebraw and I started to investigate different solutions for this and it is just a bug. We hope this can be fixed :)

[adrian-e]

+1 for this issue !

[michael-wolfenden]

👍 for this issue as well

[bitfrost]

Adding the webpack-md5-hash plugin in addition to the chunk-manifest plugin allowed me to work around this bug and get my non-changing vendor bundle to stay the same.

[kevinrenskers]

Great solution! Works like a charm.

[dubrowgn]

This all seems a little broken to me. Shouldn't the hash be applied at the file level instead of the chunk level? If a chunk produces an app.js and an app.css, I don't care if the chunk changed. I care if app.js and/or app.css files changed, because this ultimately determines if the client needs a new copy.

[kevinrenskers]

Sadly adding webpack-md5-hash doesn't work either, because in the vendor files some references to modules in the main file change yet the hash stays the same. This completely breaks the entire app.

[bitfrost]

Hmm, not seeing this behavior in my toy example. are you using OccurenceOrderPlugin ?
My plugin list:
new webpack.optimize.CommonsChunkPlugin({
name: 'vendor',
minChunks: Infinity
}),
new webpack.optimize.DedupePlugin(),
new ManifestPlugin(),
new ChunkManifestPlugin({
filename: 'chunk-manifest.json',
manifestVariable: 'webpackManifest'
}),
new ChunkManifestPluginWriter(),
new WebpackMd5Hash(),
new ManifestPlugin(),
new webpack.optimize.OccurenceOrderPlugin(),
new webpack.optimize.UglifyJsPlugin({
compress: {
warnings: false
}})

Hack to get around watch not triggering :
// Force output chunk-manifest after each build.
// TODO: figure out bug chunk-manifest-webpack-plugin that makes it miss
// webpack watches
class ChunkManifestPluginWriter {
apply(compiler) {
compiler.plugin('emit', (compilation, compileCallback) => {
const stats = compilation.getStats().toJson();
const result = {};
stats.assets.filter((asset) => {
return asset.name.endsWith('.js') & (asset.chunks[0] > 0);
}).forEach((asset) => {
result[String(asset.chunks[0])] = asset.name;
});
compilation.assets['chunk-manifest.json'] = new RawSource(JSON.stringify(result));
compilation.mainTemplate.plugin('require', (_, chunk, hash, chunkIdVar) => {
return _.replace('"CHUNK_MANIFEST"',
'window["webpackManifest"][' + chunkIdVar + ']');
});
compileCallback();
});
}
}

[kevinrenskers]

I'm using webpack.optimize.CommonsChunkPlugin, chunk-manifest-webpack-plugin, webpack-md5-hash, webpack.optimize.DedupePlugin, webpack.optimize.OccurenceOrderPlugin and webpack.optimize.UglifyJsPlugin.

Not sure what your ChunkManifestPluginWriter and ManifestPlugin plugins are?

[dmitry]

Looks like related issue: #1479

[at0g]

👎 for webpack-md5-hash + '[chunkhash]'

[danny-andrews]

@gabn88 I think your issue with the hashes changing when you add a new entry can be fixed by using NamedChunksPlugin. If that works, I'll add it to my solution. Also, we should add an example for solving this problem to in order to put this ticket to rest once and for all. :)

Also, what do you mean by "adding an entry"?

[danny-andrews]

@zaguini Agreed, it is very confusing. Have you tried the solution I posted? It's the minimal config I needed to get it working.

[dennis-johnson-dev]

For those interested in using Webpack records to maintain consistent file hashes between builds, I wrote a bit about that here - https://medium.com/@songawee/long-term-caching-using-webpack-records-9ed9737d96f2.

[earslap]

The saga continues...

I followed the official documentation on caching and it (thankfully) works fine. Then I added UglifyJS for tree shaking and compression, the output files themselves changed but the hashes didn't. That was surprising to me, since it can lead to complications (you minify but clients don't receive the new files since hashes don't change.)

Found out that chunkhash with the recommended method does the hashing before any other optimisations (like the ones done by UglifyJS.)

Then I found this: #4659

The proposed webpack-plugin-hash-output plugin seems to work, it does the hashing on top of the processed output (as far as I can understand) and seems to solve the issue... For now.

It's plugins on top of plugins on top of plugins at this point and I've been burned by this so many times over the years. Just when you think it all works, you change one config and it all crashes and burns until you add yet another plugin. The process is so so brittle.

Maybe this is not the right place but after all these years I'm convinced that this won't get any better, and the design is fundamentally flawed for this type of thing. Are there any commonly used solutions for long term caching outside of webpack? Like using webpack just for the vanilla bundling, and then something else goes through the HTML and linked resources to hash them separately and change the references in files?

In any case, if webpack-plugin-hash-output is the recommended solution now, is it worth adding to the documentation? The current documentation is leading people down a dangerous path (i.e. if they add anything to the plugin chain that processes / changes the files, the hashes won't change and at best people will waste their time diagnosing the issue like I did.)

[jjinux]

earslap, I hear what you're saying, but it just reminds me of (https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-an-open-source-maintainer/).

To answer your question, perhaps the best way to minimize the danger of getting burned by this type of thing, you should cache for a shorter amount of time. That way, if they accidentally cache the non-minimized version, it won't stay cached forever.

Sorry, I can't offer you more, but at least it's better than nothing. Patches are certainly welcome.

[earslap]

Oh no, I wasn't trying to put a blame on anyone involved with this project, sorry if my frustration made it sound that way.

I still think this is a fundamental design issue with the way webpack is architected. Brightest minds working on this deceptively simple issue for 2.5+ years, coming up with various solutions but none sticks. I don't know much about the internals of webpack but I assume something in there makes this extremely tricky.

If you strip all my frustrations from my earlier post, the meat should have been this: The documented method of cache busting (which I think summarises a happy path gathered from the discussions in this thread the past 2.5 years) will not work if any other plugin modifies the files along the way due to a config change, the hashes won't change - and that is dangerous (worse than hashes changing when they shouldn't as this can actually break things.) If my understanding is correct, there is a proposed solution that tries to fix this (here: #4659). Is this recommended, and if so should that be added to the documentation?

[chen86860]

Just named the vendor can solve this problem 😆

code:

···
new webpack.optimize.CommonsChunkPlugin({
      names: 'vendor',
      minChunks: Infinity,
      filename: 'vendor.bundle.js',
 })
···

[michaelBenin]

Anyone else coming here needing a solution, just create your own md5 hash.

const crypto = require('crypto');
const vendorList = [/* list of vendor files */];
const vendorHash = crypto.createHash('md5').update(vendorList.join('')).digest('hex');

Use vendorHash in the vendor filename options:

filename: `vendor.${vendorHash}.js

[michaelBenin]

Regarding my comment above - that won't work as the bundle is referenced in the vendor file.

[NeoVance]

@earslap if your cache does not invalidate when the contents of a file changes than perhaps you should look into fixing that.

Personally, I don't understand the obsession with using hashes in file names. There are far more simple, and easy to use methods of versioning files, and cache busting. This really isn't even an issue in webpack's wheel house to solve for you.

I feel like this is not something the webpack maintainers should worry about. This is not really webpack's role to fill.

[michael-wolfenden]

@NeoVance curious to hear what alternatives there are to file hashing

[michaelBenin]

@michael-wolfenden prior to webpack hashes and manifest, we versioned static files using the git sha or build number, as well as setting Cache-Control: max-age=31536000

[michael-wolfenden]

@NeoVance @michaelBenin the issue with using the git sha or version number is that if you doing continuous deployment and depoying multiple times a day, the vendor bundle version constantly changes with changes to the app bundle version thus invalidating it's cache. This is the problem that file hashes are intended to solve.

[dalimian]

@earslap had the same issue, @sokra wonder if this issue is gonna be addressed with webpack4 since you are rewriting treeshaking. I hope wp4 gets long term caching right, we been burned with it 3 times now with webpack 2-3

I am willing to help with the work

[maksimsemenov]

@michaelBenin can you please explain a little bit more why custom vendor hash will not work. I mean what will happen if bundle had changed, but vendor file still referencing the old bundle. Will it cause any errors?
I was thinking of using it because all other approaches end with different hashes after CI builds, even if vendors were not changed. So want to know what caveats are here.

[michaelBenin]

@maksimsemenov ~

First off we didn't get this working as intended, this issue is still open.

Basically we found that the vendor file has a reference of the app.js and that caused an error breaking JS.

[malliapi]

is this still happening for webpack 4?

[edvardchen]

To a certain extent, webpack 4 improved this situation, at least reducing the frequency of vendor hash changes, but the problem still exists.
See full code here
First,npm run build, output

If you just modify src/client.js, the vendor file's hash will remain unchanged unless your modified part changes the count of split chunks.

Like, comment one async import statement in src/client.js:

import React from 'react';

import { render } from 'react-dom';

// import('./App');

Then npm run build would output

Why? Because the vendors~app file still has a chunkId.

// head line in vendor-app file. This chunk Id is 1.
(window["webpackJsonp"] = window["webpackJsonp"] || []).push([[1],{

Maybe we need a special way to treat initial chunk. Treat that they are already installed without checking ?

[alexander-akait]

I close this issue due with a very large number of questions/posts with very difference problems (include deprecated webpack versions). It's also hard to understand exactly what the problem - in the configuration or it's a real bug. Also please read https://medium.com/webpack/predictable-long-term-caching-with-webpack-d3eee1d3fa31 (some steps no needs because it is already fixed in webpack). Feel free create new issue with minimum reproducible test repo.

In short ways:

webpackConfig.output = {
    chunkFilename: '[name]-[contenthash].chunk.js',
    filename: '[name]-[contenthash].js'
};

webpackConfig.plugins = [
   // ------------------------------------
    // Long Term Caching
    // ------------------------------------
    // More information https://medium.com/webpack/predictable-long-term-caching-with-webpack-d3eee1d3fa31
    new webpack.NamedChunksPlugin(chunk => {
      if (chunk.name) {
        return chunk.name;
      }

      // eslint-disable-next-line no-underscore-dangle
      return [...chunk._modules]
        .map(m =>
          path.relative(
            m.context,
            m.userRequest.substring(0, m.userRequest.lastIndexOf("."))
          )
        )
        .join("_");
    }),
    new webpack.HashedModuleIdsPlugin()
];

Webpack@5 will have better long term caching out of box. Thanks!