clang-offload-bundler crashes when linked to shared libLLVM with Wild

[mati865]

Split out from #752

When the resulting binary is run with Wild and LD:

❯ ./bin -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/mateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.tgt1 -input=/home/m
ateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.tgt2 -output=/home/mateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.hip.bundle.bc -compress -verbose
Compressed bundle format version: 2
Total file size (including headers): 137 bytes
Compression method used: zstd
Compression level: 3
Binary size before compression: 188 bytes
Binary size after compression: 113 bytes
Compression rate: fish: Job 1, './bin -type=bc -targets=hip-amd…' terminated by signal SIGSEGV (Address boundary error)

❯ ./bin.ref-linker -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/mateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.tgt1 -input=/home/mateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.tgt2 -output=/home/mateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.hip.bundle.bc -compress -verbose
Compressed bundle format version: 2
Total file size (including headers): 137 bytes
Compression method used: zstd
Compression level: 3
Binary size before compression: 188 bytes
Binary size after compression: 113 bytes
Compression rate: 1.66
Compression ratio: 60.11%
Compression speed: 5.01 MB/s
Truncated MD5 hash: 0x2bcafb899a29dc94

Linker-diff finds the problem:

❯ WILD_REFERENCE_LINKER=ld.bfd ./run-with ~/Projects/wild/target/debug/wild
WARNING: wild: --plugin /usr/lib/gcc/x86_64-pc-linux-gnu/15.1.1/liblto_plugin.so is not yet supported
wild: ./bin
ref: ./bin.ref-linker
.dynamic.DT_FLAGS_1.NOW
  wild 1
  ref

dynsym._ZTVN4llvm13format_objectIJdEEE.section
  wild
  ref .data.rel.ro

rel.R_X86_64_PC32.R_X86_64_PC32
  `./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a` @ `OffloadBundler.cpp.o` .text._ZN5clang23CompressedOffloadBundle10decompressERKN4llvm12MemoryBufferEb _ZN5clang23CompressedOffloadBundle10decompressERKN4llvm12MemoryBufferEb
  ORIG 0x005a8: [ 48 8b 05 00 00 00 00 ] mov 0x5AF,%rax
                           ^^^^^^^^^^^ R_X86_64_REX_GOTPCRELX
  ORIG {vtable(llvm::format_object<double>)} -4
  wild 0x1fbb8: [ 48 8d 05 41 04 fe ff ] lea 0,%rax
                           ^^^^^^^^^^^ R_X86_64_PC32 MovIndirectToLea
  wild 0x0 (symbol is undefined)
  wild TRACE: relaxation applied relaxation=MovIndirectToLea, value_flags=DYNAMIC | NON_INTERPOSABLE,
  wild TRACE: resolution_flags=GOT, rel_kind=Relative,
  wild TRACE: value=0xfffffffffffe0441, symbol_name={vtable(llvm::format_object<double>)}
  ref  0x17278: [ 48 8d 05 51 ea 00 00 ] lea 0x25CD0,%rax
                           ^^^^^^^^^^^ R_X86_64_PC32 MovIndirectToLea
  ref  {vtable(llvm::format_object<double>)}

rel.R_X86_64_PC32.R_X86_64_PC32
  `./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a` @ `OffloadBundler.cpp.o` .text._ZN5clang23CompressedOffloadBundle8compressEN4llvm11compression6ParamsERKNS1_12MemoryBufferEtb _ZN5clang23CompressedOffloadBundle8compressEN4llvm11compression6ParamsERKNS1_12MemoryBufferEtb
  ORIG 0x004b2: [ 48 8b 05 00 00 00 00 ] mov 0x4B9,%rax
                           ^^^^^^^^^^^ R_X86_64_REX_GOTPCRELX
  ORIG {vtable(llvm::format_object<double>)} -4
  wild 0x1e582: [ 48 8d 05 77 1a fe ff ] lea 0,%rax
                           ^^^^^^^^^^^ R_X86_64_PC32 MovIndirectToLea
  wild 0x0 (symbol is undefined)
  wild TRACE: relaxation applied relaxation=MovIndirectToLea, value_flags=DYNAMIC | NON_INTERPOSABLE,
  wild TRACE: resolution_flags=GOT, rel_kind=Relative,
  wild TRACE: value=0xfffffffffffe1a77, symbol_name={vtable(llvm::format_object<double>)}
  ref  0x15c42: [ 48 8d 05 87 00 01 00 ] lea 0x25CD0,%rax
                           ^^^^^^^^^^^ R_X86_64_PC32 MovIndirectToLea
  ref  {vtable(llvm::format_object<double>)}


Error: linker-diff reported errors. To rerun, execute:
/home/mateusz/Projects/wild/target/debug/linker-diff --wild-defaults --display-names wild,ref --ref ./bin.ref-linker ./bin

If I disable MovIndirectToLea relaxation on R_X86_64_REX_GOTPCRELX relocations the binary works fine, but I think the root issue is that undefined symbol. Which really shouldn't be undefined.
That symbol is located in:

❯ rg -al _ZTVN4llvm13format_objectIJdEEE .
./bin
./bin.ref-linker
./usr/lib/libclangDriver.a
./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a
./home/mateusz/Projects/llvm-project/build/lib/libclangBasic.a
./usr/lib/libclangBasic.a
./usr/lib/libLLVM.so.21.0git
./home/mateusz/Projects/llvm-project/build/lib/libLLVM.so.21.0git

and linked with:

❯ rg 'libclang|LLVM' run-with
40:  $D/home/mateusz/Projects/llvm-project/build/lib/libclangBasic.a \
41:  $D/home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a \
42:  $D/home/mateusz/Projects/llvm-project/build/lib/libclangBasic.a \
43:  $D/home/mateusz/Projects/llvm-project/build/lib/libLLVM.so.21.0git \

They all define that symbol as a weak symbol:

❯ readelf -Ws ./home/mateusz/Projects/llvm-project/build/lib/libclangBasic.a | rg '_ZTVN4llvm13format_objectIJdEEE|Symbol table|File:' | rg _ZTVN4llvm13format_objectIJdEEE -B2
File: ./home/mateusz/Projects/llvm-project/build/lib/libclangBasic.a(Sanitizers.cpp.o)
Symbol table '.symtab' contains 333 entries:
   217: 0000000000000000    32 OBJECT  WEAK   DEFAULT  151 _ZTVN4llvm13format_objectIJdEEE

❯ readelf -Ws ./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a | rg '_ZTVN4llvm13format_objectIJdEEE|Symbol table|File:' | rg _ZTVN4llvm13format_objectIJdEEE -B2
File: ./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a(Driver.cpp.o)
Symbol table '.symtab' contains 1326 entries:
  1258: 0000000000000000    32 OBJECT  WEAK   DEFAULT  697 _ZTVN4llvm13format_objectIJdEEE
--
File: ./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a(OffloadBundler.cpp.o)
Symbol table '.symtab' contains 506 entries:
   448: 0000000000000000    32 OBJECT  WEAK   DEFAULT  204 _ZTVN4llvm13format_objectIJdEEE

❯ readelf -Ws ./home/mateusz/Projects/llvm-project/build/lib/libLLVM.so.21.0git | rg '_ZTVN4llvm13format_objectIJdEEE|Symbol table|File:' | rg _ZTVN4llvm13format_objectIJdEEE -B2
Symbol table '.dynsym' contains 38190 entries:
 11152: 0000000009535750    32 OBJECT  WEAK   DEFAULT   23 _ZTVN4llvm13format_objectIJdEEE@@LLVM_21.0
Symbol table '.symtab' contains 163685 entries:
 83382: 0000000009535750    32 OBJECT  WEAK   DEFAULT   23 _ZTVN4llvm13format_objectIJdEEE

And also Wild output:

❯ ./run-with ~/Projects/wild/target/debug/wild --sym-info _ZTVN4llvm13format_objectIJdEEE
WARNING: wild: --plugin /usr/lib/gcc/x86_64-pc-linux-gnu/15.1.1/liblto_plugin.so is not yet supported
Global name `_ZTVN4llvm13format_objectIJdEEE` refers to: Some(SymbolId(4031))
Definitions / references with name `_ZTVN4llvm13format_objectIJdEEE`:
  Weak Data: symbol_id=4031 -> 58203 ADDRESS | NON_INTERPOSABLE res=[]
    #217 in File #6144 (24/0) ./home/mateusz/Projects/llvm-project/build/lib/libclangBasic.a @ Sanitizers.cpp.o (NOT LOADED)
  Weak Data: symbol_id=24214 -> 58203 ADDRESS | NON_INTERPOSABLE res=[]
    #1258 in File #17152 (67/0) ./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a @ Driver.cpp.o (NOT LOADED)
  Weak Data: symbol_id=25272 -> 58203 ADDRESS | NON_INTERPOSABLE res=[]
    #448 in File #18432 (72/0) ./home/mateusz/Projects/llvm-project/build/lib/libclangDriver.a @ OffloadBundler.cpp.o (LOADED)
  Weak Data: symbol_id=58203 -> 58203 DYNAMIC res=[GOT]
    #11152 in File #35328 (138/0) ./home/mateusz/Projects/llvm-project/build/lib/libLLVM.so.21.0git (LOADED)

Lastly, the linkers outputs:

❯ readelf -Ws bin{,.ref-linker} | rg '_ZTVN4llvm13format_objectIJdEEE|Symbol table|File:' | rg _ZTVN4llvm13format_objectIJdEEE -B2
File: bin
Symbol table '.dynsym' contains 172 entries:
    39: 0000000000000000     0 OBJECT  WEAK   DEFAULT  UND _ZTVN4llvm13format_objectIJdEEE@LLVM_21.0 (2)
--
File: bin.ref-linker
Symbol table '.dynsym' contains 174 entries:
   166: 0000000000025cd0    32 OBJECT  WEAK   DEFAULT   21 _ZTVN4llvm13format_objectIJdEEE
Symbol table '.symtab' contains 372 entries:
   155: 0000000000025cd0    32 OBJECT  WEAK   DEFAULT   21 _ZTVN4llvm13format_objectIJdEEE

Reproducible steps:

❯ mkdir build && cd build
# Be sure to enable Wild in your prefered way
❯ cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE=Release -G Ninja -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON ../llvm
❯ ninja clang-offload-bundler

# adjust paths
bin/clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/mateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.tgt1 -input=/home/mateusz/Projects/llvm-project/build/tools/clang/test/Driver/Output/clang-offload-bundler-zstd.c.tmp.tgt2
# or do `ninja check` instead

[mati865]

I was a bit busy since reporting it, but I should be able to have a deeper look into it now.

[mati865]

I think I have nailed the problem, despite not understanding all the details.

The problematic symbol is present in .symtab of each libclangBasic.a, libclangDriver.a, libclangDriver.a and libLLVM.so.21.0git (iterated in this order in the loop described below), the shared object also has _ZTVN4llvm13format_objectIJdEEE@@LLVM_21.0 in .dynsym.
When iterating inside select_symbol, the symbol in the two archives (value flags: ADDRESS | NON_INTERPOSABLE) is undefined, but in the last archive this symbol is weak. Similarly, this symbol is weak in the shared object.

Given the iteration order and the rev() here:

wild/libwild/src/symbol_db.rs

Line 900 in b8f9b40

for &alt in alternatives.iter().rev() {

Wild selects the weak symbol from the shared object, unlike LD which seems to pick the definition from an archive (just my guess because it's unversioned in the executable).

I think this itself shouldn't break the binaries but sometime later in Wild this symbol becomes undefined:

❯ readelf -Ws bin | rg '_ZTVN4llvm13format_objectIJdEEE|Symbol table'
Symbol table '.dynsym' contains 172 entries:
    39: 0000000000000000     0 OBJECT  WEAK   DEFAULT  UND _ZTVN4llvm13format_objectIJdEEE@LLVM_21.0 (2)
Symbol table '.symtab' contains 206 entries:

That lead me to making this change:

diff --git a/libwild/src/symbol_db.rs b/libwild/src/symbol_db.rs
index 3cdf861605..26f38a0d72 100644
--- a/libwild/src/symbol_db.rs
+++ b/libwild/src/symbol_db.rs
@@ -899,7 +899,9 @@

     for &alt in alternatives.iter().rev() {
         let strength = symbol_db.symbol_strength(alt, resolved);
-        if strength != SymbolStrength::Undefined {
+        if strength != SymbolStrength::Undefined
+            && symbol_db.local_symbol_value_flags(alt) != ValueFlags::DYNAMIC
+        {
             return Ok(alt);
         }
     }

Linker-diff is happy now, finally all LLVM tests pass when linked to the shared object, and the binary looks more like the one created by LD:

 readelf -Ws bin | rg '_ZTVN4llvm13format_objectIJdEEE|Symbol table'
Symbol table '.dynsym' contains 174 entries:
   169: 00000000000275a8    32 OBJECT  WEAK   DEFAULT   22 _ZTVN4llvm13format_objectIJdEEE
Symbol table '.symtab' contains 208 entries:
   197: 00000000000275a8    32 OBJECT  WEAK   DEFAULT   22 _ZTVN4llvm13format_objectIJdEEE

Creating a test for it shouldn't be hard, but I'll get to it tomorrow, but I have no idea if my change is correct.

[davidlattimore]

Hopefully the test will reveal more exactly what the rule is here.

I've just had a look through the history of this code, since there were a few things that didn't make sense to me.

The .rev() call(s) came from aae2d83. At the time, they made sense because the symbol IDs being stored were stored in the reverse order of when they were encountered, so the .rev() calls returned us to the actual order. I'm not sure they're actually needed (or correct) now. One of the .rev calls appears to have been removed, but the other remains.

There also used to be a comment in that function that said "Dynamic symbols, even strong ones, don't override non-dynamic weak symbols". It looks like that comment got lost in 632b109 which was also where one of the .rev calls disappeared.

The change that made the .rev calls unneeded (and incorrect) was 517e157.

I just tried removing the remaining .rev call and none of the tests failed. I guess to the extent that ordering is important, there should perhaps be a test that fails when the .rev call is present.

[mati865]

Oh, I tried removing .rev first, but then thought there must be a reason for it and created the hack from my previous comment.

As far as LLVM's test suite is concerned, both, the hack, and removal of .rev allow it to fully pass.

Creating a test for it shouldn't be hard, but I'll get to it tomorrow

Well, easier said than done. I have only found a difference that is not related to this issue (each commit should be considered as standalone): main...mati865:wild:push-knqrqvouwurp
I have another idea, to use the weak symbol from a function defined in the shared object, but I didn't have time for that.

[mati865]

My C++ skills are too low to make a reproducer from their code, and I couldn't recreate the problem without vtables.

[davidlattimore]

I've merged a possible fix. The main issue fixed as that weak symbols in regular objects weren't taking priority over strong symbols in shared objects. I haven't tried to reproduce the original problem though, so I've left this issue open for now.

[mati865]

Thanks, works good!

Well, easier said than done. I have only found a difference that is not related to this issue (each commit should be considered as standalone): main...mati865:wild:push-knqrqvouwurp

These synthetic test cases (not based on the real code) are still failing, are they worth fixing?