Updater: Way to disable hash prefix for consistent_snapshot

[jku]

Currently if the repository is consistent_snapshot, Updater will prefix the target filename with the hash when constructing the download URL. In the Warehouse case this is not wanted (warehouse target file paths are "consistent", even if the filenames are not).

This is an example of a working URL:

https://files.pythonhosted.org/packages/ca/ab/5e004afa025a6fb640c6e983d4983e6507421ff01be224da79ab7de7a21f/Django-3.0.8-py3-none-any.whl

But currently Updater will try to download

https://files.pythonhosted.org/packages/ca/ab/5e004afa025a6fb640c6e983d4983e6507421ff01be224da79ab7de7a21f/caab5e004afa025a6fb640c6e983d4983e6507421ff01be224da79ab7de7a21f.Django-3.0.8-py3-none-any.whl

If there is no way to figure this out automatically, then there should probably be a way to tell Updater that I Know What I'm Doing, don't prefix filenames.

[MVrachev]

I am not sure I understand that:

warehouse target file paths are "consistent", even if the filenames are not).

if the filenames are not consistent, then the file path won't be too?

Also, in the spec in section 5.2 it's written:

If consistent snapshots are not used (see Section 7), 
then the filename used to download the target file is of the fixed form FILENAME.EXT (e.g., foobar.tar.gz).
Otherwise, the filename is of the form HASH.FILENAME.EXT 
(e.g., c14aeb4ac9f4a8fc0d83d12482b9197452f6adf3eb710e3b1e2b79e8d14cb681.foobar.tar.gz), 
where HASH is one of the hashes of the targets file listed in the targets metadata file found earlier in step 4. 
In either case, the client MUST write the file to non-volatile storage as FILENAME.EXT.

so, it seems that they didn't think of a case when you will use consistent_snapshot and the naming FILENAME.EXT

[joshuagl]

I am not sure I understand that:

warehouse target file paths are "consistent", even if the filenames are not).

if the filenames are not consistent, then the file path won't be too?

Warehouse doesn't follow what tuf (the reference implementation and specification) advice for naming consistent filenames, which is to prefix the filename with the hash of the files contents. However the target filenames it does use are consistent, only the hash is part of the target's file path not the target's file name. You can see in the example above that the filename prefix the updater is looking for matches the path components beneath the packages directory:

/ca/ab/5e004afa025a6fb640c6e983d4983e6507421ff01be224da79ab7de7a21f/Django-3.0.8-py3-none-any.whl ->
caab5e004afa025a6fb640c6e983d4983e6507421ff01be224da79ab7de7a21f.Django-3.0.8-py3-none-any.whl

This feature request is for callers to be able to tell an updater not to prefix the targets filename with the hash, even when the repository is using consistent_snapshot.

It is somewhat analogous to how repository_tool can be told to use_existing_fileinfo when writing metadata, rather than trying to compute the fileinfo from the target files on disk (#1004, #1007 and #1020 for more details on how that was implemented).

[MVrachev]

My research so far is the following:

In the tuf.client.updater the following functions are called in this order:

when the CLI arguments are parsed, then

1. tuf.client.scripts.update_client()
2. tuf.client.updater.download_target()
3. tuf.client.updater._get_target_file()

and in tuf.client.updater._get_target_file() the verify_target_file() function is defined which adds the preffix @jku had described.

So, my questions are:

where should I provide a flag for our users that will allow them to control the usage of the hash prefix when using consistent snapshot?

Where will be appropriate so that it could be easily integrated by adopters like Warehouse?

I will be PTO the upcoming week, so I will be able to look again at this issue after the 3-rd of August.

[joshuagl]

I would think this would have to be a Boolean that's configured on an Updater instance, probably as an argument to the constructor. I'd like to hear from @jku how he would like to configure this, though.

[jku]

The optimal solution would be that this is supported in tuf spec (I don't expect this to happen now, just mentioning to ensure we're on the same page): The server should be able to say that it operates this way.

More practical options that come to mind:

• Without changing the spec, is there still an option of using non-standard metadata? As in Warehouse would set a flag in one of the metadata files and updater would understand it? Maybe this is quite complex for little gain...
• Without modifying the actual metadata API at all, how about if Updater.download_target() accepted a non-standard target-info (one that included a boolean flag)? Then pip could inject that flag after calling get_one_valid_targetinfo() but before calling download_target(). This way the Updater API would not change in obtrusive ways
• The last option is to modify Updater public API -- the new boolean flag could be in download_target() but maybe init() makes more sense

[jku]

The optimal solution would be that this is supported in tuf spec

Joshua had questions about this and he may be right: this could be seen as an implementation detail and not something that needs to be exposed in the metadata. That would mean that a "straight up implementation of the spec" might not work with e.g. Warehouse: client has to have pre-shared information about the actual configuration before it can operate in a useful way. This sounds like a flaw but maybe it's not: running a random TUF client against warehouse is only useful in a debugging sense anyway. This is probably true for most TUF implementations.

The options listed should still be valid

[jku]

I guess modifying targetinfo between get_one_valid_targetinfo() and download_target() is not a great idea: the use of schemas for input data is actively trying to prevent this sort of thing.

That leaves us with the last option: Add a boolean argument to Updater API (either Updater __init__() or download_target()). It's ugly but I don't have better ideas ATM.

[jku]

Implemented in #1102