Unable to extract remote tgz

[eddieschoute]

General summary/comments

I am trying to add a remote dependency hosted at http://www.mathstat.dal.ca/~selinger/quipper/downloads/quipper-0.8.tgz but the extraction of the archive fails.

Steps to reproduce

1. Create a new project.
2. Add to extra-deps in stack.yaml:

extra-deps:
- http://www.mathstat.dal.ca/~selinger/quipper/downloads/quipper-0.8.tgz

3. Run stack build

stack.yaml.txt

Expected

I expect it to extract the archive for use as a dependecy. Downloading the archive manually and extracting using gunzip and tar does work.

Actual

I get an error message saying that the extraction failed because the file type is not supported.

$ stack build --verbose
Version 1.6.1, Git revision f25811329bbc40b0c21053a8160c56f923e1201b (5435 commits) x86_64 hpack-0.20.0
2017-12-13 12:02:22.545104: [debug] Checking for project config at: /Users/eddie/dev/quipper-dep/stack.yaml
@(Stack/Config.hs:842:9)
2017-12-13 12:02:22.547226: [debug] Loading project config file stack.yaml
@(Stack/Config.hs:868:13)
2017-12-13 12:02:22.548884: [debug] Decoding build plan from: /Users/eddie/.stack/build-plan/lts-9.18.yaml
@(Stack/Snapshot.hs:150:5)
2017-12-13 12:02:22.548967: [debug] Trying to decode /Users/eddie/.stack/build-plan-cache/lts-9.18.cache
@(Data/Store/VersionTagged.hs:66:5)
2017-12-13 12:02:22.553569: [debug] Success decoding /Users/eddie/.stack/build-plan-cache/lts-9.18.cache
@(Data/Store/VersionTagged.hs:70:13)
2017-12-13 12:02:22.553940: [debug] Using standard GHC build
@(Stack/Setup.hs:619:9)
2017-12-13 12:02:22.554253: [debug] Getting global package database location
@(Stack/GhcPkg.hs:46:5)
2017-12-13 12:02:22.554872: [debug] Asking GHC for its version
@(Stack/Setup/Installed.hs:98:13)
2017-12-13 12:02:22.555007: [debug] Getting Cabal package version
@(Stack/GhcPkg.hs:185:5)
2017-12-13 12:02:22.555089: [debug] Run process: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc-pkg --no-user-package-db list --global
@(System/Process/Log.hs:37:3)
2017-12-13 12:02:22.555196: [debug] Run process: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc --numeric-version
@(System/Process/Log.hs:37:3)
2017-12-13 12:02:22.555801: [debug] Run process: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version
@(System/Process/Log.hs:37:3)
2017-12-13 12:02:22.605355: [debug] Process finished in 50ms: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc-pkg --no-user-package-db list --global
@(System/Process/Log.hs:44:3)
2017-12-13 12:02:22.606124: [debug] Process finished in 49ms: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version
@(System/Process/Log.hs:44:3)
2017-12-13 12:02:22.638452: [debug] Process finished in 83ms: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc --numeric-version
@(System/Process/Log.hs:44:3)
2017-12-13 12:02:22.638623: [debug] GHC version is: ghc-8.0.2
@(Stack/Setup/Installed.hs:102:13)
2017-12-13 12:02:22.638728: [debug] Resolving package entries
@(Stack/Setup.hs:252:5)
2017-12-13 12:02:22.638954: [debug] Trying to decode /Users/eddie/.stack/loaded-snapshot-cache/x86_64-osx/ghc-8.0.2/lts-9.18.cache
@(Data/Store/VersionTagged.hs:66:5)
2017-12-13 12:02:22.673805: [debug] Success decoding /Users/eddie/.stack/loaded-snapshot-cache/x86_64-osx/ghc-8.0.2/lts-9.18.cache
@(Data/Store/VersionTagged.hs:70:13)
2017-12-13 12:02:22.674559: [debug] Run process: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc-pkg init /Users/eddie/dev/quipper-dep/.stack-work/install/x86_64-osx/lts-9.18/8.0.2/pkgdb/
@(System/Process/Log.hs:37:3)
2017-12-13 12:02:22.712381: [debug] Process finished in 37ms: /Users/eddie/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc-pkg init /Users/eddie/dev/quipper-dep/.stack-work/install/x86_64-osx/lts-9.18/8.0.2/pkgdb/
@(System/Process/Log.hs:44:3)
2017-12-13 12:02:22.712728: [debug] Starting to execute command inside EnvConfig
@(Stack/Runners.hs:170:18)
2017-12-13 12:02:22.712848: [debug] Parsing the targets
@(Stack/Build/Target.hs:460:3)
2017-12-13 12:02:22.713262: [debug] Running hpack on /Users/eddie/dev/quipper-dep/package.yaml
@(Stack/PrettyPrint.hs:63:22)
2017-12-13 12:02:22.718665: [debug] hpack output unchanged in /Users/eddie/dev/quipper-dep/quipper-dep.cabal
@(Stack/PrettyPrint.hs:63:22)
2017-12-13 12:02:22.720218: [debug] Downloading /~selinger/quipper/downloads/quipper-0.8.tgz
@(Network/HTTP/Download/Verified.hs:246:9)
2017-12-13 12:02:23.954034: [debug] Trying to untar /Users/eddie/dev/quipper-dep/.stack-work/downloaded/MsFEOwDSZ731.http-archive
@(Stack/PackageLocation.hs:113:17)
2017-12-13 12:02:23.954233: [debug] Got exception: Codec.Compression.Zlib: compressed data stream format error (incorrect header check)
@(Stack/PackageLocation.hs:127:21)
2017-12-13 12:02:23.954297: [debug] Trying to unzip /Users/eddie/dev/quipper-dep/.stack-work/downloaded/MsFEOwDSZ731.http-archive
@(Stack/PackageLocation.hs:119:17)
2017-12-13 12:02:23.968024: [debug] Got exception: Data.Binary.Get.runGet at position 4: Did not find end of central directory signature
CallStack (from HasCallStack):
  error, called at libraries/binary/src/Data/Binary/Get.hs:342:5 in binary-0.8.3.0:Data.Binary.Get
@(Stack/PackageLocation.hs:127:21)
Archive extraction failed. We support tarballs and zip, couldn't handle the following URL, http://www.mathstat.dal.ca/~selinger/quipper/downloads/quipper-0.8.tgz downloaded to the file MsFEOwDSZ731.http-archive

Stack version

$ stack --version
Version 1.6.1, Git revision f25811329bbc40b0c21053a8160c56f923e1201b (5435 commits) x86_64 hpack-0.20.0

Method of installation

Installed stack through OSX brew the upgraded through stack upgrade.

[mgsloan]

Very curious! Something very odd is happening, it seems like the file that stack downloads is just a tarball, with no gzipping. Whereas if I wget the url, I get a gzipped tar file.

Next step is to try uploading the file elsewhere and seeing if it works.

Would probably be good to accept plain tar files and ignore not being able to ungzip.

[snoyberg]

If I was to take a guess, the remote server is marking the stream as gzip-compressed, and http-client is therefore decompressing in transit. There's ambiguity in the HTTP spec (or at least in implementations) regarding how to handle this case, most servers do not mark a gzip-compressed file as a gzip-compressed stream. Having Stack either:

1. Tell servers it does not accept gzip encoding, or
2. Support plain tar files as mentioned

Should both work. A simpler workaround for now is hosting the file in question at a different location (a Github release, for example, would likely work fine).

[snoyberg]

I've opened up #3655 for support of .tar files, which seems to work in this case. However, note that your stack.yaml will still not work, since the tarball in question does not contain a .cabal file in its root directory.

[eddieschoute]

@snoyberg Is there a way to include dependencies on archives that do not have a .cabal descriptor?

P.S. I think your documentation of adding dependencies could use some expanding. Maybe especially the interactions between stack.yaml and the project .cabal file. And what are all the dependency descriptors there are (I see none, archive, git, location (what does that do?))

[snoyberg]

No, all packages must have a .cabal file, it's a pretty core requirement of the entire package system.

If you have ideas for expanding the documentation, a PR would definitely be appreciated, thank you. For now though, it looks like the issue here has been resolved, is that correct?

[eddieschoute]

Yes, I think so. I haven't been able to test but I will get back to you if it doesn't work.