Compile bug: SYCL backend build fail on debug config

[senhui2intel]

Git commit

f17a3bb

Operating systems

Linux

GGML backends

SYCL

Problem description & steps to reproduce

SYCL backend build fail on debug config with error: SYCL kernel cannot call an undefined function without SYCL_EXTERNAL attribute

cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx
cmake --build build

Release config works without error:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx
cmake --build build

First Bad Commit

No response

Compile command

cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx
cmake --build build

Relevant log output

[  3%] Building CXX object ggml/src/ggml-sycl/CMakeFiles/ggml-sycl.dir/ggml-sycl.cpp.o
/data/senhui/Projects/LearnLlamacpp/llama.cpp/ggml/src/ggml-sycl/ggml-sycl.cpp:2331:63: warning: unused parameter 'main_device' [-Wunused-parameter]
 2331 | static void ggml_sycl_set_peer_access(const int n_tokens, int main_device) {
      |                                                               ^
In file included from /data/senhui/Projects/LearnLlamacpp/llama.cpp/ggml/src/ggml-sycl/ggml-sycl.cpp:32:
In file included from /opt/intel/oneapi/compiler/2025.1/bin/compiler/../../include/sycl/sycl.hpp:25:
In file included from /opt/intel/oneapi/compiler/2025.1/bin/compiler/../../include/sycl/detail/core.hpp:21:
In file included from /opt/intel/oneapi/compiler/2025.1/bin/compiler/../../include/sycl/accessor.hpp:12:
In file included from /opt/intel/oneapi/compiler/2025.1/bin/compiler/../../include/sycl/atomic.hpp:16:
In file included from /opt/intel/oneapi/compiler/2025.1/bin/compiler/../../include/sycl/multi_ptr.hpp:15:
In file included from /opt/intel/oneapi/compiler/2025.1/bin/compiler/../../include/sycl/detail/type_traits.hpp:29:
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/array:208:2: error: SYCL kernel cannot call an undefined function without SYCL_EXTERNAL attribute
  208 |         __glibcxx_requires_subscript(__n);
      |         ^

[ShanoToni]

Hello @senhui2intel, this is something I have encountered as well in the past. Unfortunately I do not have a "good" fix, but I wanted to provide you with a workaround which I hope might unblock you.

The reason for the failure is there are some assertions inside sycl kernels which cause this failure, since assertions are only enabled in debug, the error is not reproducible in Release.

The workaround I have is to disable assertions so you can change your CmakeCache cxx flags toCMAKE_CXX_FLAGS_DEBUG:STRING=-g -U_GLIBCXX_ASSERTIONS or pass it to your cmake command cmake -DCMAKE_CXX_FLAGS="-U_GLIBCXX_ASSERTIONS" .. (I haven't tried the second approach)

Again, this is definitely not a good fix but it should be enough to unblock you.

[NeoZhangJianyu]

@senhui2intel
In fact, build with "Debug" is not recommended.
For SYCL kernel, if it runs on GPU, it's impossible to pending the code for debug.

If you want to debug the code flow, use gdb after source oneAPI.

[senhui2intel]

Hello @senhui2intel, this is something I have encountered as well in the past. Unfortunately I do not have a "good" fix, but I wanted to provide you with a workaround which I hope might unblock you.

The reason for the failure is there are some assertions inside sycl kernels which cause this failure, since assertions are only enabled in debug, the error is not reproducible in Release.

The workaround I have is to disable assertions so you can change your CmakeCache cxx flags toCMAKE_CXX_FLAGS_DEBUG:STRING=-g -U_GLIBCXX_ASSERTIONS or pass it to your cmake command cmake -DCMAKE_CXX_FLAGS="-U_GLIBCXX_ASSERTIONS" .. (I haven't tried the second approach)

Again, this is definitely not a good fix but it should be enough to unblock you.

Thanks, I tried the second approach as below cmake cmd and it works:
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx

Although it's not a "good" fix, it does make help to debug llama.cpp host code with sycl backend.

[NeoZhangJianyu]

Follow cmd is work:

mkdir build
cd build
cmake .. -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx
cmake --build . --config Debug -j -v

Please refer to examples/sycl/build.sh. Replace "Release" by "Debug".

[senhui2intel]

Follow cmd is work:

mkdir build
cd build
cmake .. -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx
cmake --build . --config Debug -j -v

Please refer to examples/sycl/build.sh. Replace "Release" by "Debug".

gdb could not found debug symlbols with --config Debug.
(No debugging symbols found in ./build/bin/llama-cli)

[ShanoToni]

Hello @senhui2intel, this is something I have encountered as well in the past. Unfortunately I do not have a "good" fix, but I wanted to provide you with a workaround which I hope might unblock you.
The reason for the failure is there are some assertions inside sycl kernels which cause this failure, since assertions are only enabled in debug, the error is not reproducible in Release.
The workaround I have is to disable assertions so you can change your CmakeCache cxx flags toCMAKE_CXX_FLAGS_DEBUG:STRING=-g -U_GLIBCXX_ASSERTIONS or pass it to your cmake command cmake -DCMAKE_CXX_FLAGS="-U_GLIBCXX_ASSERTIONS" .. (I haven't tried the second approach)
Again, this is definitely not a good fix but it should be enough to unblock you.

Thanks, I tried the second approach as below cmake cmd and it works: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx

Although it's not a "good" fix, it does make help to debug llama.cpp host code with sycl backend.

I'm happy the workaround helped, please share if you find a better solution for this problem. If/When I get a chance to look into it more I will update.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.