Skip to content

Missing dependency on a foreign library #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
noughtmare opened this issue Nov 4, 2019 · 38 comments
Closed

Missing dependency on a foreign library #10

noughtmare opened this issue Nov 4, 2019 · 38 comments
Labels
bug Something isn't working documentation Related to documentation ops Related to ops

Comments

@noughtmare
Copy link

noughtmare commented Nov 4, 2019

I have installed arrayfire 3.6.4 from https://arrayfire.com/download/ to /opt/arrayfire as described on http://arrayfire.org/docs/installing.htm#Linux.

Running cabal install arrayfire yields the following error message:

Build profile: -w ghc-8.8.1 -O1
In order, the following will be built (use -v for more details):
 - arrayfire-0.1.0.0 (lib:arrayfire, exe:gen, exe:main) (requires build)
Starting     arrayfire-0.1.0.0 (all, legacy fallback)

Failed to build arrayfire-0.1.0.0. The failure occurred during the configure
step.
Build log (
/home/jaro/.cabal/logs/ghc-8.8.1/arrayfire-0.1.0.0-5e7527530ca25ab12a051fd4d0f7055fae7a4dba9f99268656d18fb873a6c8ce.log
):
[1 of 1] Compiling Main             ( /tmp/cabal-install.-22655/dist-newstyle/tmp/src-22655/arrayfire-0.1.0.0/dist/setup/setup.hs, /tmp/cabal-install.-22655/dist-newstyle/tmp/src-22655/arrayfire-0.1.0.0/dist/setup/Main.o )
Linking /tmp/cabal-install.-22655/dist-newstyle/tmp/src-22655/arrayfire-0.1.0.0/dist/setup/setup ...
Configuring arrayfire-0.1.0.0...
setup: Missing dependency on a foreign library:
* Missing (or bad) C library: af
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.If the
library file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

cabal: Failed to build arrayfire-0.1.0.0. See the build log above for details.
@lehins
Copy link

lehins commented Nov 4, 2019

@noughtmare Looks like you need to build from source in order to be able to use the bindings:

For general usage of ArrayFire, we recommend the installer binaries. However, if you wish to dev on ArrayFire, you will need to build the source code.

https://github.com/arrayfire/arrayfire/wiki#build-instructions

@noughtmare
Copy link
Author

I think that "dev on ArrayFire" means modifying the ArrayFire library itself. I just want to use it, so the binaries should be enough (the binaries include the /opt/arrayfire/lib64 and /opt/arrayfire/include directories). But I will try compiling from source to be sure.

@noughtmare
Copy link
Author

I have finished building from source and I still get the same error.

@chessai
Copy link
Collaborator

chessai commented Nov 4, 2019

Have you tried using the flags it tells you to? --extra-lib-dirs and/or --extra-include-dirs? You shouldnt need to build from source at all.

@noughtmare
Copy link
Author

Yes

@noughtmare
Copy link
Author

This fixes it for me:

diff --git a/arrayfire.cabal b/arrayfire.cabal
index 1f53125..8c4e1f9 100644
--- a/arrayfire.cabal
+++ b/arrayfire.cabal
@@ -96,7 +96,7 @@ library
     extra-lib-dirs:
       /opt/arrayfire/lib
     ld-options:
-      -Wl -rpath /opt/arrayfire/lib
+      -Wl,-rpath /opt/arrayfire/lib

@chessai
Copy link
Collaborator

chessai commented Nov 4, 2019

Glad that works. Instead of a patch, it should be passed to cabal as --ghc-options (since we can't assume how arrayfire was installed on the system).

@noughtmare
Copy link
Author

I think it is just an error in your .cabal file. AFAIK there should always be a comma after -Wl and not a space.

@chessai
Copy link
Collaborator

chessai commented Nov 4, 2019

Ah, makes sense. Looking back I just realised that those linker flags are already there. Perhaps they shouldn't be.

@chessai
Copy link
Collaborator

chessai commented Nov 4, 2019

I think @dmjio and myself haven't noticed because we use nix. I think we should add a travis setup for Ubuntu

@chessai
Copy link
Collaborator

chessai commented Nov 4, 2019

Actually, opt is a standard enough place to let that be for non-NixOS. We should just document that, and apply the fix you gave @noughtmare. I'm on mobile right now so I can't do that.

@dmjio
Copy link
Member

dmjio commented Nov 4, 2019

@noughtmare nice catch, strange that error didn't manifest on OSX for me (when I wasn't using nix). I can make a PR for this.

@9prady9
Copy link
Member

9prady9 commented Nov 5, 2019

I had a similar problem with ArrayFire installed to a custom location. We (me and @dmjio ) are working on figuring out the cause of it. Passing custom location using --extra-lib-dirs and --extra-include-dirs didn't resolve the issue.

@chessai
Copy link
Collaborator

chessai commented Nov 5, 2019

I had a similar problem with ArrayFire installed to a custom location. We (me and @dmjio ) are working on figuring out the cause of it. Passing custom location using --extra-lib-dirs and --extra-include-dirs didn't resolve the issue.

what was the actual error?

@9prady9
Copy link
Member

9prady9 commented Nov 5, 2019

stack install --extra-lib-dirs=$AF_PATH/lib64 --extra-include-dirs=$AF_PATH/include arrayfire
arrayfire> configure
arrayfire> [1 of 2] Compiling Main             ( /tmp/stack-e44f53084a6bc144/arrayfire-0.1.0.0/Setup.hs, /tmp/stack-e44f53084a6bc144/arrayfire-0.1.0.0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/setup/Main.o )
arrayfire> [2 of 2] Compiling StackSetupShim   ( /home/pradeep/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /tmp/stack-e44f53084a6bc144/arrayfire-0.1.0.0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/setup/StackSetupShim.o )
arrayfire> Linking /tmp/stack-e44f53084a6bc144/arrayfire-0.1.0.0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/setup/setup ...
arrayfire> Configuring arrayfire-0.1.0.0...
arrayfire> setup: Missing dependency on a foreign library:
arrayfire> * Missing (or bad) C library: af
arrayfire> This problem can usually be solved by installing the system package that
arrayfire> provides this library (you may need the "-dev" version). If the library is
arrayfire> already installed but in a non-standard location then you can use the flags
arrayfire> --extra-include-dirs= and --extra-lib-dirs= to specify where it is.If the
arrayfire> library file does exist, it may contain errors that are caught by the C
arrayfire> compiler at the preprocessing stage. In this case you can re-run configure
arrayfire> with the verbosity flag -v3 to see the error messages.
arrayfire> 

--  While building package arrayfire-0.1.0.0 using:
      /tmp/stack-e44f53084a6bc144/arrayfire-0.1.0.0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/setup/setup --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 configure --user --package-db=clear --package-db=global --package-db=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/pkgdb --libdir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/lib --bindir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/bin --datadir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/share --libexecdir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/libexec --sysconfdir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/etc --docdir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/doc/arrayfire-0.1.0.0 --htmldir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/doc/arrayfire-0.1.0.0 --haddockdir=/home/pradeep/.stack/snapshots/x86_64-linux-tinfo6/8096d5c493ca1014d2da8250c11e63731bfe2fe6565556a6aa8a501eadfda3c2/8.6.5/doc/arrayfire-0.1.0.0 --dependency=Cabal=Cabal-2.4.1.0-3JC906oWQ73Ah7bYkIcI4J --dependency=base=base-4.12.0.0 --dependency=cabal-doctest=cabal-doctest-1.0.8-40Ym5zAcLNVCMmwzciX9Qk --dependency=directory=directory-1.3.3.0 --dependency=filepath=filepath-1.4.2.1 --dependency=parsec=parsec-3.1.14.0-8t8UKeFouAo5hdmomuIjPj --dependency=text=text-1.2.3.1 --dependency=vector=vector-0.12.0.3-LfvlcMFJAcY18uD1Y2O5Ig -f-disable-default-paths --extra-include-dirs=/home/pradeep/gitroot/ArrayFireWorkspace/worktrees/v3.6/build/pkg/include --extra-lib-dirs=/home/pradeep/gitroot/ArrayFireWorkspace/worktrees/v3.6/build/pkg/lib64 --exact-configuration --ghc-option=-fhide-source-paths
    Process exited with code: ExitFailure 1

@noughtmare
Copy link
Author

noughtmare commented Nov 5, 2019

@9prady9 I think your problem is that the line:

-Wl,-rpath /opt/arrayfire/lib

Still links to the wrong directory. So it will fail with something like (if you pass -v3 to cabal somehow):

/usr/bin/gcc returned ExitFailure 1 with error message:
gcc: error: /opt/arrayfire/lib: No such file or directory

I have tried building with that line removed and that still works, so maybe that line should just be removed completely.

Another way to fix it for your case is to pass the disable-default-paths flag to arrayfire.

@9prady9
Copy link
Member

9prady9 commented Nov 5, 2019

@9prady9 I think your problem is that the line:

-Wl,-rpath /opt/arrayfire/lib

Still links to the wrong directory. So it will fail with something like (if you pass -v3 to cabal somehow):

/usr/bin/gcc returned ExitFailure 1 with error message:
gcc: error: /opt/arrayfire/lib: No such file or directory

I have tried building with that line removed and that still works, so maybe that line should just be removed completely.

Another way to fix it for your case is to pass the disable-default-paths flag to arrayfire.

If that is the case, why I do I still see the paths I am passing via command line in the command output I shared earlier ?

Also, when I pass extra options via command line, the following appears in the command output

-f-disable-default-paths --extra-include-dirs=/home/pradeep/gitroot/ArrayFireWorkspace/worktrees/v3.6/build/pkg/include --extra-lib-dirs=/home/pradeep/gitroot/ArrayFireWorkspace/worktrees/v3.6/build/pkg/lib64

no default paths are being disabled if I am not wrong. Please correct me if I am understanding it incorrectly.

@noughtmare
Copy link
Author

noughtmare commented Nov 5, 2019

If that is the case, why I do I still see the paths I am passing via command line in the command output I shared earlier ?

The correct paths do get passed to gcc, but it will also pass the wrong paths which do not exist.

-f-disable-default-paths

I think this means that the flag is disabled (I think this is explaned here in the cabal documentation), but you want to enable it. In the stack documentation I read that that is done using:

--flag arrayfire:disable-default-paths

@9prady9
Copy link
Member

9prady9 commented Nov 5, 2019

Adding that additional flag did carry the build process much further, the following command did finish successfully. @noughtmare thank you.

stack install --flag arrayfire:disable-default-paths \
--extra-lib-dirs=$AF_PATH/lib64 --extra-include-dirs=$AF_PATH/include arrayfire

I think it worked as it copied couple of binaries into $HOME/.local/bin, gen and main, I was able to run main. Is it possible to place them somewhere local in a build sort of directory ? Or is this the normal to place haskell binaries in .local/bin ?

@dmjio however nix-shell --run test-runner command fails. It's probably the same issue as above but with nix-shell or whatever it does. I am not familiar with nix-shell. I think the README guide could definitely use more improvements on the set of instructions required to use arrayfire-haskell via all three routes (as per my understanding) : cabal, stack , nix-shell.

@9prady9
Copy link
Member

9prady9 commented Nov 5, 2019

@dmjio what does gen executable do ? It is writing some files in the location where I ran the executable.

@noughtmare
Copy link
Author

noughtmare commented Nov 5, 2019

@9prady9

Is it possible to place them somewhere local in a build sort of directory ? Or is this the normal to place haskell binaries in .local/bin ?

stack build will put the executables in an obfuscated local (I mean local to the package not literally .local) folder somewhere in .stack-work (I think it prints the exact location to the command line). The executables can then be run with stack exec main for example.

@9prady9
Copy link
Member

9prady9 commented Nov 5, 2019

true, it is printing the path as $HOME/.local/bin.

I was concerned that this path doesn't always necessarily is present in PATH environment variable and the user needs to add it (although a one time thing) to run the executable via stack exec command.

Is there a way to control where stack generates the executable and lookup for executable ? something like a configuration option ?

@noughtmare
Copy link
Author

noughtmare commented Nov 5, 2019

There is a local bin path flag: https://docs.haskellstack.org/en/stable/yaml_configuration/#local-bin-path

stack install --local-bin-path="~/.bin" should work

@dmjio
Copy link
Member

dmjio commented Nov 6, 2019

@9prady9 hmmm, nix-shell --run test-runner works for me on OSX and NixOS. Can you paste the error you're receiving and I can diagnose.

@dmjio
Copy link
Member

dmjio commented Nov 6, 2019

@dmjio what does gen executable do ? It is writing some files in the location where I ran the executable.

@9prady9 gen is used to lex / parse the C headers into Haskell FFI declarations. Should only be ran when upgrading versions.

@9prady9
Copy link
Member

9prady9 commented Nov 6, 2019 via email

@9prady9
Copy link
Member

9prady9 commented Nov 6, 2019 via email

@dmjio
Copy link
Member

dmjio commented Nov 6, 2019

when I run on OSX, I get:

λ Davids-MacBook-Pro arrayfire-haskell → λ git master* →       nix-shell --run test-runner
Resolving dependencies...
Configuring arrayfire-0.2.0.0...
Preprocessing library for arrayfire-0.2.0.0..
Building library for arrayfire-0.2.0.0..
Preprocessing test suite 'test' for arrayfire-0.2.0.0..
Building test suite 'test' for arrayfire-0.2.0.0..

ArrayFire.Algorithm
  Algorithm tests
    Should sum a scalar
    Should sum a vector
    Should sum a default value to replace NaN
    Should product a scalar
    Should product a vector
    Should product a default value to replace NaN
    Should take the minimum element of a vector
    Should find if all elements are true along dimension
    Should find if any elements are true along dimension
    Should get count of all elements
    Should get sum all elements
    Should get sum all elements
    Should product all elements in an Array
    Should product all elements in an Array
    Should find minimum value of an Array
    Should find maximum value of an Array
ArrayFire.Arith
  Arith tests
    Should negate scalar value
    Should negate a vector
    Should add two scalar arrays
    Should add two scalar bool arrays
    Should subtract two scalar arrays
    Should multiply two scalar arrays
    Should divide two scalar arrays
    Should add two matrices
    Should take cubed root
    Should take square root
    Should lt Array
    Should lte Array
    Should gte Array
    Should gt Array
    Should eq Array
    Should and Array
    Should and Array
    Should or Array
    Should not Array
    Should bitwise and array
    Should bitwise or array
    Should bitwise xor array
    Should bitwise shift left an array
    Should cast an array
    Should find the minimum of two arrays
    Should find the max of two arrays
    Should take the clamp of 3 arrays
    Should check if an array has positive or negative infinities
    Should check if an array has any NaN values
    Should check if an array has any Zero values
ArrayFire.Array
  Array tests
    Should perform Array tests
...
ArrayFire.Signal
  Signal spec
    Should do FFT in place
    Should do FFT
ArrayFire.Sparse
  Sparse spec
    Should create a sparse array
ArrayFire.Statistics
  Statistics spec
    Should find the mean
    Should find the weighted-mean
    Should find the variance
    Should find the weighted variance
    Should find the standard deviation
    Should find the covariance
    Should find the median
    Should find the mean of all elements across all dimensions
    Should find the weighted mean of all elements across all dimensions
    Should find the variance of all elements across all dimensions
    Should find the weighted variance of all elements across all dimensions
    Should find the stdev of all elements across all dimensions
    Should find the median of all elements across all dimensions
    Should find the correlation coefficient
    Should find the top k elements
ArrayFire.Util
  Util spec
    Should get size of
    Should get version
    Should get revision
    Should save / read array
ArrayFire.Vision
  Vision spec
    Should construct Features for fast feature detection

Finished in 6.2213 seconds
122 examples, 0 failures

what OS are you on?

@9prady9
Copy link
Member

9prady9 commented Nov 6, 2019 via email

@dmjio dmjio added the bug Something isn't working label Nov 10, 2019
@dmjio dmjio added documentation Related to documentation ops Related to ops labels Nov 10, 2019
@dmjio
Copy link
Member

dmjio commented Nov 11, 2019

I think the issue is that this ld-option should be removed.

-Wl,-rpath /opt/arrayfire/lib

It's presumptuous to assume we can always link with that directory even when the user is using the binary distribution (since it could be lib or lib64), and the failure always ends in compilation failure when its not found. Instead, we can add both

extra-lib-dirs:
   /opt/arrayfire/lib64 
   /opt/arrayfire/lib

And it will simply warn if one of the directories is missing w/o compilation failure. This should make a much better experience on initial install.

@9prady9
Copy link
Member

9prady9 commented Nov 11, 2019

What about the custom installation paths with nix-shell ? Please check my earlier messages for details.

@lehins
Copy link

lehins commented Nov 12, 2019

Just in case anyone is looking for a simple workaround for this problem you can just use a symbolic link:

$ sudo ln -s /opt/arrayfire/lib64 /opt/arrayfire/lib
$ sudo bash -c 'echo /opt/arrayfire/lib > /etc/ld.so.conf.d/arrayfire.conf'
$ sudo ldconfig

@dmjio
Copy link
Member

dmjio commented Nov 14, 2019

@lehins can you confirm if you install arrayfire with the binary installer on Linux that it creates /opt/arrayfire/lib64 and not /opt/arrayfire/lib (related to #46). Don't have ubuntu handy, and trying to make assumptions out here.

@lehins
Copy link

lehins commented Nov 14, 2019

@dmjio That is correct, I can confirm this on Ubuntu.

@dmjio
Copy link
Member

dmjio commented Nov 14, 2019

@lehins awesome! And not to be redundant, but can you also confirm that you did not install from source, but from the binary installer script ?

@lehins
Copy link

lehins commented Nov 14, 2019

@dmjio lol. No worries. I did indeed install it from the downloadable Linux installer

@dmjio
Copy link
Member

dmjio commented Nov 14, 2019

@lehins perfect ! then I'm going to go ahead and merge #46, since I think the default locations are accounted for on each platform. People who build from source will have to -f+disable-default-paths and --extra-lib-dirs=/path/to/arrayfire/lib and --extra-include-dirs=/path/to/arrayfire/include

@dmjio
Copy link
Member

dmjio commented Nov 15, 2019

@noughtmare @9prady9 @chessai @lehins this issue should be solved in 0.6.0.0. (no more symbolic link required)
If not, please re-open.

@9prady9 this issue is solved only in the case end-users are using the binary distribution of ArrayFire w/ the Haskell wrapper (not installing from source). I have an issue (and solution) to provide documentation for this case. #48

@dmjio dmjio closed this as completed Nov 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working documentation Related to documentation ops Related to ops
Projects
None yet
Development

No branches or pull requests

5 participants