Run Llama.cpp in silent mode

[OsaCode]

Is there an option to run the executable using a saved prompt without printing anything but the prediction by the selected model ? Some kind of silent mode in opposition to the --verbose.

[j-f1]

If you pipe stderr to /dev/null (e.g. with ./main 2> /dev/null on Unix-like systems), you should only see the model output being printed out. (I don’t know if there’s a way to stop printing the prompt as well, that could be a good option to add!)

[OsaCode]

I tested it with a prompt containing:

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello world!")
Hello world!
>>>

This prompt works as a python interpreter.

Here is the result:

./main -m ./models/13B/ggml-model-q4_0.bin -n 50 -f ./prompts/python.txt --color --temp 0.2 2> /dev/null
 Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello world!")
Hello world!
>>> print(42)
42
>>> print(42, end=" ")
42
>>> print(42, end=" ", sep="")
42
>>> print(42, end=" ")

Your solution indeed works for the main output, but the prompt is still printed as you predicted.

[heroofhyla]

If you don't need the output displayed in real time as it's generated, you could use tail to trim the prompt off the front.

Something like:

./main -m ./models/13B/ggml-model-q4_0.bin -n 50 -f ./prompts/python.txt | tail -c+$(wc -c < ./prompts/python.txt) 2>/dev/null

[MathiasSchindler]

This really sounds like a useful flag that llama.cpp should have.

Either as --silent or some --verbosity change.

[lee-b]

A proper logging system with a logging levels configuration file would be the proper solution here.

[MathiasSchindler]

I opened up a bug report at #8507 for a feature request.

[NeoZhangJianyu]

llama.cpp is on hot development.
In default, it will print logs with more debug info for trouble shooting.
When report issue, user just need copy/paste the log. It incudes necessary info for basic check.

I agree to improve the log print to support be closed by methods: parameter or >null.

[bandoti]

There are currently several log levels set up in the GGML library. Llama hooks into this via the llama_log_set function below. The various examples right now use separate logging macros within the common/log.h file. What you can do to have more control is set the macro #define LOG_DISABLE_LOGS and redefine the LOG/LOGTEE &co to also reroute to a custom log handler.

https://github.com/ggerganov/llama.cpp/blob/5e116e8dd51775f8f1c090570be148d5d7eea6c3/ggml/include/ggml.h#L548-L553

https://github.com/ggerganov/llama.cpp/blob/5e116e8dd51775f8f1c090570be148d5d7eea6c3/include/llama.h#L1158

[leoshimo]

hey @OsaCode - I accomplish this with two flags: --log-disable and --no-display-prompt:

$ ./llama-cli -m <model> -f input --log-disable --no-display-prompt

[MYusufY]

This somehow didnt work for me, but combining it with the /dev/null method worked:

./llama-cli -m model.gguf -p "Hi" --no-display-prompt 2> /dev/null