Simple tutorial for beginers

[srem1]

I need run a llama localy on my computer.
How starting? I use linux.
Where is models, how download it, where put and how starting using it.

Anybody can help

[CRD716]

Check the Readme.
You can find models on huggingface. Here's an example: https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/tree/main

[srem1]

How find in huggingface compatible files? sometimes in one repositories is flie-1 and file-2 how use it?
Is any script to find (or meybe convert) and download ?

for example i search bard, I wrote $ downloaddata.rb -F bard and get all list of finding bard version compatible with llama

[KerfuffleV2]

Generally, we can't really help you find LLaMA models (there's a rule against linking them directly, as mentioned in the main README). This is because LLaMA models aren't actually free and the license doesn't allow redistribution. So the people that do distribute them are counting on the fact that Meta probably isn't going to go after them as random individuals.

Actual organizations or larger project need to be more careful though. A hint would be to search for models that have "ggml" in the name and were published fairly recently (in the last couple months). Here's a link to an OpenLLaMA model to get you started: https://huggingface.co/TheBloke/open-llama-7b-open-instruct-GGML (maybe you'll find some other useful models published by the same user? Who knows, anything is possible!)

for example i search bard, I wrote $ downloaddata.rb -F bard and get all list of finding bard version compatible with llama

You can't download Bard (or ChatGPT). These are large models that have never been released (or even leaked). They're also likely too large to run on a PC.

Where is models, how download it, where put and how starting using it.

You should probably start by reading the main project README. There's various helpful information there (although no links to models).

[johnwick123f]

It's pretty simple.

First, you need to clone the repository with git and change the directory to llama cpp
git clone https://github.com/ggerganov/llama.cpp cd llama.cpp

2nd, make the llama cpp with the command
make

and 3rd download the model(just search huggingface for any model you want but this code does it for vicuna 7b)
wget -c https://huggingface.co/CRD716/ggml-vicuna-1.1-quantized/resolve/main/ggml-vicuna-7b-1.1-q5_1.bin

4th, chat with the model
./main -m models/ggml-vicuna-13B-1.1-q5_1.bin --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-vicuna-v1.txt

So the full code should be
git clone https://github.com/ggerganov/llama.cpp cd llama.cpp wget -c https://huggingface.co/CRD716/ggml-vicuna-1.1-quantized/resolve/main/ggml-vicuna-7b-1.1-q5_1.bin ./main -m models/ggml-vicuna-13B-1.1-q5_1.bin --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-vicuna-v1.txt

[eguar11011]

The fourth step does not work for me. I have this error:
main: build = 800 (481f793) main: seed = 1688758643 error loading model: failed to open models/ggml-vicuna-13B-1.1-q5_1.bin: No existe el fichero o el directorio llama_load_model_from_file: failed to load model llama_init_from_gpt_params: error: failed to load model 'models/ggml-vicuna-13B-1.1-q5_1.bin' main: error: unable to load model

[BarfingLemurs]

@eguar11011 try this:

git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make -j4 && wget -c https://huggingface.co/SlyEcho/open_llama_3b_ggml/resolve/main/open-llama-3b-q4_0.bin && ./main -m open-llama-3b-q4_0.bin

Now its all setup. For future runs, do:

./server -m open-llama-3b-q4_0.bin

go to http://127.0.0.1:8080

you can chat with it, say: "what is your name?"

[johnwick123f]

Ok instead of wget the model link, do url = (vicuña model url) and filename = wget.download(url), and replace it with filename.

[eguar11011]

@BarfingLemurs
Thank you, it works.

[VOM7HC]

Hello @johnwick123f, I have LoRA file after using fine tune by using tool https://github.com/zetavg/LLaMA-LoRA-Tuner. So how to use LoRA file in LLaMA.cpp? Will I use python script to convert LoRA to GGML?

[jasonkaplan79]

These "Getting started" instructions no longer seem to be accurate, at least, they dont work from a Mac Terminal.

wget -c https://huggingface.co/SlyEcho/open_llama_3b_ggml/resolve/main/open-llama-3b-q4_0.bin
basename: illegal option -- c
usage: basename string [suffix]
       basename [-a] [-s suffix] string [...]
curl: (2) no URL specified
curl: try 'curl --help' or 'curl --manual' for more information

./main -m open-llama-3b-q4_0.bin
bash: ./main: No such file or directory

I started another discussion here for beginners, hoping that it will help me (and others down the line): #10631