Build-tools not detected when not using Stackage snapshot

[conklech]

This shows up with the gtk libraries; unsure how general it is.

With a dummy project depending on cairo, a normal stack.yaml depending on a snapshot correctly (somehow?) infers that gtk2hs-buildtools must be installed to build cairo:

$ cat stack.yaml
resolver: lts-2.18
packages:
- '.'
$ stack build --dry-run
Nothing to unregister

Would build:
alex-3.1.4: database=snapshot, source=package index
buildtools-test-0.1.0.0: database=local, source=D:\Projects\buildtools-test\, after: cairo-0.13.1.0
cairo-0.13.1.0: database=snapshot, source=package index, after: gtk2hs-buildtools-0.13.0.4,text-1.2.0.6
gtk2hs-buildtools-0.13.0.4: database=snapshot, source=package index, after: alex-3.1.4,hashtables-1.2.0.2
[..]

But with resolver: ghc-7.10 and manually specified dependencies, it it doesn't figure that out:

$ cat stack.yaml
resolver: ghc-7.10
flags:
  text:
    integer-simple: false
extra-deps: 
  - cairo-0.13.1.0
  - mtl-2.2.1
  - text-1.2.1.1
  - utf8-string-1
packages:
- '.'
$ stack build --dry-run
Nothing to unregister

Would build:
buildtools-test-0.1.0.0: database=local, source=D:\Projects\buildtools-test\, after: cairo-0.13.1.0
cairo-0.13.1.0: database=local, source=package index, after: mtl-2.2.1,text-1.2.1.1,utf8-string-1
[..]

That build would fail. Adding gtk2hs-buildtools-0.13.0.4 to the extra-deps field does not solve the problem.

How does stack determine that gtk2hs-buildtools is required, and why doesn't that work with the ghc-* resolver?

[snoyberg]

The build tool discovery is fragile at best, because of the way Cabal specifies these things. Currently, the tool map is only capable of finding things in the snapshot build plan. I can't think of an efficient way of making this work well, but if you (or someone else) is interested in taking a stab at this, I can work through the issue with you.

grep the codebase for getToolMap to get started with this.

[conklech]

It's documented in Stack.Build.ConstructPlan that only the snapshot is used to find build tools.

This is so, if I understand correctly, basically for circularity reasons. The mapping from tool names to providing packages is needed in the Ctx structure that's the Reader component of the RWST monad used in plan construction. But when Ctx is constructed, the non-snapshot packages (extraToBuild0) are just a set of package names; they're added to the build plan as the monad runs. In other words, snapshot packages start off in the build plan and are available to satisfy build-tool dependencies; extra packages are not.

It looks like our behavior is to silently accept the inability to satisfy a built-tool requirement.

The general solution would be to make two passes over the build plan, directly or indirectly, so that extra packages are available in the second pass.

To work around this problem, you can manually stack build the required packages (alex, happy, gtk2hs-buildtools). Doing so requires manually adding those packages' dependencies to stack.yaml. stack install is not necessary and would pollute the global $PATH.

This problem is particularly pernicuous because a package maintainer may have the required tools available on the global $PATH and therefore never encounter this problem.

[borsboom]

Another use case described in #1427.

[mgsloan]

Considering that another instance of this has come up, bumping to P2.

[mgsloan]

Another detail of this is that if you have a dependency of a build-tool in extra-deps, then it will not be considered as a candidate tool. Argh! This really ought to get fixed.

In 4fb0b06, I added some warnings that link to this issue. Once the issue is fixed, the suggested cause should be removed.