What's the intended workflow of the tarball cache?

[wycats]

I understand how the feature works, but as I'm not using it, I'm not quite sure what the exact intended workflow is.

@bestander @skevy @kittens

[bestander]

You mean the offline-mirror setting?

[wycats]

@bestander yeah

[bestander]

So we use it for offline installs: CI and internal projects.

1. .npmrc contains yarn-offline-mirror key with path to a folder with .tar.gz files
2. When you add dependencies via yarn add, they are downloaded from npm repo but yarn.lock stores local path to the .tar.gz file instead of "https://registry.npmjs.org/...
3. The mirror folder is checked in (that is ~1000 files and 30 MB for React Native which is fine)
4. When you do yarn install, node_modules are installed without going to network

[bestander]

Personally, this is the single reason why I joined the effort on this project :)
BTW a related topic to discuss #394

[skevy]

This is particularly useful in a monorepo setting (we have one at Exponent and then of course FB had theirs). All the packages and applications in the repo can share this same module cache. It makes CI/CD (especially monorepo CI/CD) great again. :-)

[wycats]

Personally, this is the single reason why I joined the effort on this project :)

Seems legit. 😄

Just so I'm clear, the primary goal is to make it possible to take a package.json that references packages from the registry. You would like to check the tarballs of those packages into version control, together with a yarn.lock, and you would like yarn install to use those checked in tarballs instead of the registry. You would also like this to work for indirect dependencies.

Is that correct?

[bestander]

That is correct

[bestander]

As my colleague, @kentaromiura, pointed out.
We don't have to check in the .tar.gz files, originally we considered using a shared storage for a project but a source control system was as good

[conartist6]

So I know the NPM guys intentionally made life difficult because they used the resolved field (when present) as the complete and only identity for a package. Their reasoning had to do with private registries. Somebody not targeting a public registry at all may have complete module name overlap. Even more ominously, a package caching server (so far the only sane way to deal with trying to use npm in a high dependability environment) could have been configured to selectively shadow some packages (or provide them past the point in time when the original source had deleted them). To complete the chaos, the npm registry owners proved that they are willing, under appropriate pressure, to themselves reassign a module name to a different project, as happened with kik prompting the famed left-pad disaster.

This makes the logic around upgrading a previously locked package awfully tricky.

Npm basically doesn't allow it. Either you keep everything locked exactly as it is, or you lose all your locked down versions at the same time. Or you manually chop out a big chunk of shrinkwrap and splicing in an updated version. o_O

How are we handling this? Do we see ourselves as having a certain contract with the user like, say, never accidentally at upgrade time replacing an installed package with something that is from a completely different codebase?

[wycats]

The reason I asked this question is that I want to propose a slightly different workflow that I think satisfies the requirements that people had when designing it in the first place, but with fewer rough edges.

The basic idea is that the yarn.lock would continue to store the original name, version and sha from the original npm registry, and that the offline mirror configuration would instruct the registry resolver to use the mirror instead.

(In Cargo, the mirroring configuration happens at a level above the individual sources, as I describe below)

This is how we designed the mirror feature in Cargo, and it has a few nice properties:

1. If you just need the offline mirror for production deploys (this is true of some, but not all people), you can continue to use the regular workflow in development.
2. It scales nicely to other kinds of mirrors. The way Cargo structures this is that you can specify, in the equivalent of .npmrc, that you would like a mirror to replace a particular source. This makes it pretty easy for build bots and other environments along those lines to strictly enforce whatever requirements they need (using whichever resolver strategy they want) without imposing the restriction unnecessarily on developers.
3. On the other hand, if the tarballs are checked into version control, it's still possible to use them in development by specifying the local mirror.
4. It also has the nice property of tracking the distinction between "version locked from a particular source" and the physical location of a mirror, which could change based on configuration.

@conartist6 I'm trying to understand the problem you're describing.

How are we handling this? Do we see ourselves as having a certain contract with the user like, say, never accidentally at upgrade time replacing an installed package with something that is from a completely different codebase?

The way bundler and Cargo handle (what I believe you mean by) this problem is by using a "precise" version for every dependency in the lockfile that includes enough information to precisely identify it (and its source) but not including mirror information (which is supplied by configuration).

In Cargo, mirrors are required to share precisely the same sha as the original upstream source, and any replacements that change the source code are specified in Cargo.toml (Cargo's equivalent of package.json) using [replace] sections:

[replace]
"foo:0.1.0" = { git = 'https://github.com/example/foo' }
"bar:1.0.2" = { path = 'my/local/bar' }

This means: "if you see foo v0.1.0 in the dependency graph, replace it with the code located at this git repo, and if you see bar v1.0.2 in the dependency graph, replace it with this local code I checked in. It works with any kind of resolver that normally works in Cargo, so it's pretty flexible. The rationale is that mirrors can be largely transparent to development if they share a SHA (and are largely operational concerns), while changing the code is a development concern and should be specified in the manifest.

Aside: Bundler works a little bit differently, but along the same lines: because bundler only allows a single name/version dependency in the entire dependency graph, a specified dependency in the top-level Gemfile always supersedes the registry. In other words, if you specify a dependency in a Gemfile, it's as if you had said [replace] in Cargo.

In Cargo (and bundler's) case, we also require that replacements share a name and version number with the original package they're replacing, and the feature is largely used for emergency patches or things like "the bug is fixed on master but the author hasn't gotten around to publishing it yet".

I'm not entirely sure whether any of that directly targets the issue you're talking about. Can you clarify it a bit?

[conartist6]

Yes, yes it definitely does target the issue I'm describing. Npm lacks hashes, and with that restriction they were forced to treat source URLs as the best guarantee of authenticity.

The setup that you are describing sounds quite attractive because it understands (on multiple levels) the difference between a cached copy and an override. Npm, infuriatingly, can't, which is why upgrading a cached package is such a nightmare.

[bestander]

@wycats do you propose moving resolved lines from yarn.lock into a registry file that would map left-pad:1.1.0 to a remote or local location of a tarball?

[bestander]

For large single-repo projects the experience looks like this.
A developer wants to add left-pad to a project.
There is an .npmrc at the root that haskpm-offline-mirror=./npm-offline-packages.

The developer writes

yarn add left-pad@~1.1.1

The new dependency is added to package.json and yarn.lock file and the tarball is downloaded to npm-offline-packages.

The nice thing about this approach is in simplicity, it is easy to review and easy to connect the dots.

If there is another project in the repository and another developer does

yarn add left-pad@~1.1.1

Then existing tarball will be reused and yarn.lock will refer it.

What would be different with the proposed approach?

[conartist6]

If I understand what @wycats is saying, nothing in the workflow your describe is different for the user. The major difference is what data is stored in yarn.lock.

I understand your earlier response to suggest that yarn.lock would contain something concrete like:

{
    location: "${npm-offline-packages}/leftpad-1.2.1.tgz",
    dependencies: structure recurses...
}

This is the npm approach. The suggestion here is to, in yarn.lock, store:

{
    descriptor: "leftpad-1.2.1",
    hash: "d672jef2",
    sourceRepository: "https://registry.npmjs.org",
    dependencies: structure recurses...
}

This way the cache directory is searched instead of being directly referenced, which means it is trivial to change the cache directory configuration, either as a one-off or between dev/prod/test.

It also means that the program can easily know that if the user says yarn upgrade left-pad, this means that it should go out to the npm central registry, fetch the newest left-pad available there, store it in the local cache, and update the descriptor and hash in yarn.lock.