Search results

Using the GPU is slower than the CPU

bugSomething isn't workingSomething isn't working

performance

Status: Open.

#1123 In abetlen/llama-cpp-python;

· AreckOVO opened on Jan 24, 2024

Performance degradation running with half a socket in CPU system

bugSomething isn't workingSomething isn't working

performance

Status: Open.

#1098 In abetlen/llama-cpp-python;

· azhuvath opened on Jan 18, 2024

Investigate / improve streaming speed

bugSomething isn't workingSomething isn't working

performance

Status: Open.

#1039 In abetlen/llama-cpp-python;

· abetlen opened on Dec 22, 2023

Mixtral A100/H100 performance

bugSomething isn't workingSomething isn't working

performance

Status: Open.

#1017 In abetlen/llama-cpp-python;

· generalsvr opened on Dec 16, 2023

Inference speed on windows server 2022 which has Nvidia RTX A4000

bugSomething isn't workingSomething isn't working

performance

Status: Open.

#973 In abetlen/llama-cpp-python;

· adeelhasan19 opened on Dec 5, 2023

generation hangs forever

bugSomething isn't workingSomething isn't working

performance

Status: Open.

#659 In abetlen/llama-cpp-python;

· YerongLi opened on Sep 2, 2023

Is there any way to reduce the inference time for this part

llama.cppProblem with llama.cpp shared libProblem with llama.cpp shared lib

performance

Status: Open.

#554 In abetlen/llama-cpp-python;

· BugO0 opened on Aug 2, 2023

I have enabled CACHE=1, there is a long wait in the middle of the conversation

performance

Status: Open.

#447 In abetlen/llama-cpp-python;

· abc20220327 opened on Jul 6, 2023

Huge performance discrepency between llama-cpp-python and llama.cpp

performance

Status: Open.

#398 In abetlen/llama-cpp-python;

· shouyiwang opened on Jun 19, 2023

Use pygtrie for efficient prefix matching to speed up cache lookups

enhancementNew feature or requestNew feature or request

performance

Status: Open.

#280 In abetlen/llama-cpp-python;

· abetlen opened on May 26, 2023