Skip to content

IPEX support FP8 kvcache/softcap/slidingwindow #3144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
May 6, 2025
Merged

IPEX support FP8 kvcache/softcap/slidingwindow #3144

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
065f87a
IPEX support FP8 kvcache
sywangyi Mar 29, 2025
102e299
add kvcache dtype
sywangyi Apr 3, 2025
d9e47b6
add softcap and slidingwindow
sywangyi Apr 8, 2025
ad15a9c
kv scale in pageattn
sywangyi Apr 8, 2025
68ec760
remove triton installation, will be installed with torch
sywangyi Apr 8, 2025
8d36856
install xelink lib
sywangyi Apr 9, 2025
f8c8c3d
softcap default -1.0
sywangyi Apr 9, 2025
ce8548f
softcap default -1.0
sywangyi Apr 14, 2025
dac5a29
Merge branch 'main' into ipex_fp8_kv_cache
sywangyi Apr 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
install xelink lib 
Signed-off-by: Wang, Yi A <[email protected]>
  • Loading branch information
@sywangyi
sywangyi committed Apr 9, 2025
commit 8d36856d5789ab69c5aecd0855dbb41d4eb6f90d
4 changes: 2 additions & 2 deletions Dockerfile_intel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ RUN echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https:/

RUN mv /tmp/intel-for-pytorch-gpu-dev.list /etc/apt/sources.list.d

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt install -y xpu-smi cmake ninja-build pciutils intel-ocloc
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt install -y xpu-smi cmake ninja-build pciutils intel-ocloc libnl-genl-3-200

# Text Generation Inference base env
ENV HF_HOME=/data \
Expand All @@ -112,7 +112,7 @@ RUN cd server && \

ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/conda/lib
ENV CCL_ZE_IPC_EXCHANGE=sockets
#ENV TORCH_LLM_ALLREDUCE=1
ENV TORCH_LLM_ALLREDUCE=1
ENV CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0
ENV TORCH_DEVICE_BACKEND_AUTOLOAD=0

Expand Down