sycl : reviewing the backend documentation #13544
+94
−39
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,9 @@ Additionally, there the following images, similar to the above: | |
| - `ghcr.io/ggml-org/llama.cpp:full-musa`: Same as `full` but compiled with MUSA support. (platforms: `linux/amd64`) | ||
| - `ghcr.io/ggml-org/llama.cpp:light-musa`: Same as `light` but compiled with MUSA support. (platforms: `linux/amd64`) | ||
| - `ghcr.io/ggml-org/llama.cpp:server-musa`: Same as `server` but compiled with MUSA support. (platforms: `linux/amd64`) | ||
| - `ghcr.io/ggml-org/llama.cpp:full-intel`: Same as `full` but compiled with SYCL support. (platforms: `linux/amd64`) | ||
| - `ghcr.io/ggml-org/llama.cpp:light-intel`: Same as `light` but compiled with SYCL support. (platforms: `linux/amd64`) | ||
| - `ghcr.io/ggml-org/llama.cpp:server-intel`: Same as `server` but compiled with SYCL support. (platforms: `linux/amd64`) | ||
|
|
||
| The GPU enabled images are not currently tested by CI beyond being built. They are not built with any variation from the ones in the Dockerfiles defined in [.devops/](../.devops/) and the GitHub Action defined in [.github/workflows/docker.yml](../.github/workflows/docker.yml). If you need different settings (for example, a different CUDA, ROCm or MUSA library, you'll need to build the images locally for now). | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,16 +12,16 @@ source /opt/intel/oneapi/setvars.sh | |
|
|
||
| INPUT_PROMPT="Building a website can be done in 10 simple steps:\nStep 1:" | ||
| MODEL_FILE=models/llama-2-7b.Q4_0.gguf | ||
| NGL=99 | ||
| CONTEXT=4096 | ||
|
|
||
| if [ $# -gt 0 ]; then | ||
| GGML_SYCL_DEVICE=$1 | ||
| echo "use $GGML_SYCL_DEVICE as main GPU" | ||
| #use signle GPU only | ||
| ZES_ENABLE_SYSMAN=1 ./build/bin/llama-cli -m ${MODEL_FILE} -p "${INPUT_PROMPT}" -n 400 -e -ngl ${NGL} -s 0 -c ${CONTEXT} -mg $GGML_SYCL_DEVICE -sm none | ||
|
|
||
| else | ||
| #use multiple GPUs with same max compute units | ||
| ZES_ENABLE_SYSMAN=1 ./build/bin/llama-cli -m ${MODEL_FILE} -p "${INPUT_PROMPT}" -n 400 -e -ngl ${NGL} -s 0 -c ${CONTEXT} | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| #!/bin/bash | ||
|
|
||
| # MIT license | ||
| # Copyright (C) 2025 Intel Corporation | ||
| # SPDX-License-Identifier: MIT | ||
|
|
||
| export ONEAPI_DEVICE_SELECTOR="level_zero:0" | ||
|
There was a problem hiding this comment. any reason we are selecting this particularly ? Is this required ? There was a problem hiding this comment. The code will choose all Intel GPUs in the PC. |
||
| source /opt/intel/oneapi/setvars.sh | ||
|
|
||
| #export GGML_SYCL_DEBUG=1 | ||
|
|
||
| #ZES_ENABLE_SYSMAN=1, Support to get free memory of GPU by sycl::aspect::ext_intel_free_memory. Recommended to use when --split-mode = layer. | ||
|
|
||
| INPUT_PROMPT="Building a website can be done in 10 simple steps:\nStep 1:" | ||
| MODEL_FILE=models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf | ||
| NGL=99 # Layers offloaded to the GPU. If the device runs out of memory, reduce this value according to the model you are using. | ||
| CONTEXT=4096 | ||
|
|
||
| if [ $# -gt 0 ]; then | ||
| GGML_SYCL_DEVICE=$1 | ||
| echo "Using $GGML_SYCL_DEVICE as the main GPU" | ||
| ZES_ENABLE_SYSMAN=1 ./build/bin/llama-cli -m ${MODEL_FILE} -p "${INPUT_PROMPT}" -n 400 -e -ngl ${NGL} -c ${CONTEXT} -mg $GGML_SYCL_DEVICE -sm none | ||
| else | ||
| #use multiple GPUs with same max compute units | ||
| ZES_ENABLE_SYSMAN=1 ./build/bin/llama-cli -m ${MODEL_FILE} -p "${INPUT_PROMPT}" -n 400 -e -ngl ${NGL} -c ${CONTEXT} | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,4 +6,4 @@ set INPUT2="Building a website can be done in 10 simple steps:\nStep 1:" | |
| @call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64 --force | ||
|
|
||
|
|
||
| .\build\bin\llama-cli.exe -m models\llama-2-7b.Q4_0.gguf -p %INPUT2% -n 400 -e -ngl 99 -s 0 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,9 @@ | ||||||
| :: MIT license | ||||||
| :: Copyright (C) 2024 Intel Corporation | ||||||
| :: SPDX-License-Identifier: MIT | ||||||
|
|
||||||
| set INPUT2="Building a website can be done in 10 simple steps:\nStep 1:" | ||||||
| @call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64 --force | ||||||
|
|
||||||
|
|
||||||
| .\build\bin\llama-cli.exe -m models\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p %INPUT2% -n 400 -e -ngl 33 | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These two lines convey more or less the same thing ?
Maybe let's remove the below one ? so that we needn't keep appending it time and again.
Also it's better to just mention the architecture than a very specific CPU model, as it will cover all the cases for that series.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would also remove the word "Arc" from above, as technically B580 and Lunar lake are not a part of the Arc series.
Lets rename that column from "Intel Arc Series" to say "Intel Discrete GPUs" ?
Similarly for the one below, "Intel Built-in Arc GPU" to just "Intel iGPUs"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
List the detailed models will help use to check easily.
First line are the new iGPU based on Arc GPU. They are powerful than next line.
Next line is the old arch iGPU. Some user still hope run LLM on them. SYCL backend support it will make the ecosystem better on Intel GPU.
"Arc" is the common commercial name of Intel GPU.
B580's full name should be: Intel Arc B580.
So, here is correct.