Benchmarks fail due to "shadowed dependencies"

[crockeea]

This problem is described in a Stackage Server issue, but the problem should really be reported here. The problem is this:

library a
library b depends on a
benchmarks for library a depend on a and b

stack bench a results in the error:

> stack bench a
...
Preprocessing library a-0.0.1...
Preprocessing benchmark 'a-benches' for a-0.0.1...
<command line>: cannot satisfy -package-id b-0.0.1-2uppVa4SsB06R8hiHGD1I8: 
    b-0.0.1-2uppVa4SsB06R8hiHGD1I8 is unusable due to shadowed dependencies:
      a-0.0.1-B41UI81aQaCEjW3sk56eaB
    (use -v for more information)
Progress: 3/4
--  While building package a-0.0.1 using:
      /home/eric/.stack/setup-exe-cache/x86_64-linux/setup-Simple-Cabal-1.24.0.0-ghc-8.0.1 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.24.0.0 build bench:a-benches --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

Note that the benchmarks compile if a-benches only depends on a or b, but not both. I don't see any reason why the benchmarks should fail to compile when they depend on both a and b; it's clear what the compilation order should be.

As indicated in the Stackage Server issue, this problem affects many packages including

• text
• unordered-containers
• clock
• options
• attoparsec
• wai-logger

A minimal example is in shadowed-deps.tar.gz.

[snoyberg]

@lehins pointed this out to me. I'm looking at logs that look like this:

$ stack bench a
Downloaded lts-7.0 build plan.    
a-0.0.1: configure
a-0.0.1: build
a-0.0.1: copy/register
b-0.0.1: configure (lib)
b-0.0.1: build (lib)
b-0.0.1: copy/register
a-0.0.1: configure (bench)
Configuring a-0.0.1...
a-0.0.1: build (bench)
Preprocessing library a-0.0.1...
Preprocessing benchmark 'a-benches' for a-0.0.1...

And then the shadowing messages. From what I can see, there's nothing we can do in Stack to work around this. It seems like:

• We configure/build library a, and get package ID a-123
• We configure/build library b, and get package ID b-456
• Now we go back and configure a for building benchmarks, and explicitly tell Cabal to only build the benchmark component. However, Cabal builds both the library and benchmark, ends up with a different package ID, and shadowing occurs.

I could be mistaken, perhaps something else it happening. But it looks like the only solution will be switching to Cabal 2.0 and using the componentized configuration, which allows for not rebuilding a library.

[alexkazik]

I have a different problem the shadowing:

• I have 8 packages (within one project with only one stack.yaml for them all but each in a own subdirectory, 7 are only libraries and the 8th lib+exe, no test so far)
• They all have clear dependencies (nothing circular)
• When I compile from scratch it usually works
• If I change some sources it sometimes fails due to shadowing, stack clean usually does the trick
• Sometimes only removing the .stack-work directory helps

EDIT: with nightly-2017-09-03 (ghc 8.2)

[lehins]

@alexkazik I stumble upon exactly the same issue recently, which only happens for me with ghc-8.2 (current nightly snapshot), but not ghc-8.0. Naturally cabal versions used are also different.
The workaround that took care of this for me is touch package-name.cabal.

[lehins]

@snoyberg Just in case, here are the steps using massiv library that reliably replicate the issue:
https://github.com/lehins/massiv/tree/5f3d083a343b22e866656e22036d42d294116d4d

• Run benchmarks first time, which will compile everything and run fine:

~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'

• Change a file in the source library, say comment out a function zipWith3 in massiv/massiv/src/Data/Array/Massiv.hs

• Try running benchmarks again:

~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'
massiv-0.1.0.0: unregistering (local file changes: src/Data/Array/Massiv.hs)
massiv-bench-0.1.0.0: unregistering (missing dependencies: massiv)
massiv-0.1.0.0: build (lib)
massiv-0.1.0.0: copy/register
massiv-bench-0.1.0.0: build (lib + bench)
Preprocessing library for massiv-bench-0.1.0.0..
Building library for massiv-bench-0.1.0.0..
[3 of 6] Compiling Bench.Massiv.Auto ( src/Bench/Massiv/Auto.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/Bench/Massiv/Auto.o ) [Data.Array.Massiv changed]
Preprocessing benchmark 'fold-seq' for massiv-bench-0.1.0.0..
Building benchmark 'fold-seq' for massiv-bench-0.1.0.0..
<command line>: cannot satisfy -package-id massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr: 
    massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr is unusable due to shadowed dependencies:
      massiv-0.1.0.0-DkTbpETOx8EEaw3SIP44Bw
    (use -v for more information)
Progress: 2/3
--  While building package massiv-bench-0.1.0.0 using:
      /home/lehins/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.0.0.2_ghc-8.2.1 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.0.0.2 build lib:massiv-bench bench:fold-seq --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

• Change cabal file for benchmarks suite:

~/massiv/massiv-bench$ touch massiv-bench.cabal

• Everything works fine again:

~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'
massiv-bench-0.1.0.0: configure (lib + bench)
Configuring massiv-bench-0.1.0.0...
massiv-bench-0.1.0.0: build (lib + bench)
Preprocessing library for massiv-bench-0.1.0.0..
Building library for massiv-bench-0.1.0.0..
Preprocessing benchmark 'fold-seq' for massiv-bench-0.1.0.0..
Building benchmark 'fold-seq' for massiv-bench-0.1.0.0..
[1 of 1] Compiling Main             ( bench/FoldSeq.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/fold-seq/fold-seq-tmp/Main.o ) [Data.Array.Massiv changed]
Linking .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/fold-seq/fold-seq ...
massiv-bench-0.1.0.0: copy/register
Installing library in /home/lehins/github/massiv/massiv-bench/.stack-work/install/x86_64-linux/nightly-2017-07-31/8.2.1/lib/x86_64-linux-ghc-8.2.1/massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr
Registering library for massiv-bench-0.1.0.0..
massiv-bench-0.1.0.0: benchmarks
Running 1 benchmarks...
...

[ndmitchell]

FWIW, I'm now seeing this bug very regularly, with GHC 8.2 on Windows. Things like touching the cabal file sometimes work, but are far from reliable in my experience.

[snoyberg]

I received a similar bug report from a new Yesod users who was using Haskell Platform and cabal-install. I'm concerned that it may be a regression in Cabal 2.0. I'll try to repro this tomorrow. My goal is to get a minimal case and then test it with both cabal-install and plain old Setup.hs. If anyone else beats me to it, let me know.

[snoyberg]

I've created a minimal reproducing case for this:

https://github.com/snoyberg/stack-issue-2781

I'm going to see if I can further reproduce this using just the Setup.hs files.

[ndmitchell]

For info, I've had this happen with benchmarks and test suites disabled, so just running stack build. Not sure if that would make your test case any simpler.

[snoyberg]

I was able to reproduce this with just the Setup.hs files, so this is definitely not a bug in Stack. This bug is already being tracked upstream with haskell/cabal#4728.

[snoyberg]

@ndmitchell I just tried simplifying my repro to not use the benchmark and was unable to reproduce. If you have thoughts on how to simplify it, please do take a crack at it, I'm sure it will help the debugging.