Misc. bug: llama-cli stopped starting in release b4191 (c9b00a7)

[Xitsa]

Name and Version

llama-cli
release b4191 (c9b00a7) and later

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-cli

Command line

llama-cli.exe

Problem description & steps to reproduce

OS and CPU

Windows 10 Pro (22H2)
AMD Ryzen 7 3700X 8-Core Processor

Summary

Llama-cli stopped to start after upgrading to version of release b4191.
Tried llama-cli from llama-b4191-bin-win-avx-x64.zip, llama-b4191-bin-win-avx2-x64.zip, llama-b4191-bin-win-noavx-x64.zip, llama-b4191-bin-win-vulkan-x64.zip

When run under windbg I see an access violation here:

[0x0]   MSVCP140!mtx_do_lock+0x9c   0x3bb45bc960   0x7ffca179c8d0   
[0x1]   ggml_base!ggml_critical_section_start+0x10   0x3bb45bc9c0   0x7ffca1787ba2   
[0x2]   ggml_base!ggml_init+0x12   0x3bb45bc9f0   0x7ffca171c79a   
[0x3]   ggml_cpu!ggml_cpu_init+0x3a   0x3bb45bca50   0x7ffca17228a9   
[0x4]   ggml_cpu!ggml_backend_cpu_reg+0x9   0x3bb45bcb70   0x7ffd2556181c   
[0x5]   ggml!ggml_backend_registry::ggml_backend_registry+0xcc   0x3bb45bcba0   0x7ffd255621ea   
[0x6]   ggml!std::allocator<ggml_backend_reg_entry>::deallocate+0xaa   0x3bb45bcbf0   0x7ffd25562968   
[0x7]   ggml!ggml_backend_load_all+0x38   0x3bb45bcc20   0x7ff783f5b211   
[0x8]   llama_cli+0x3b211   0x3bb45bccb0   0x7ff783f59b05   
[0x9]   llama_cli+0x39b05   0x3bb45cc830   0x7ff783f29b54   
[0xa]   llama_cli+0x9b54   0x3bb45cdab0   0x7ff783f8c1dc   
[0xb]   llama_cli+0x6c1dc   0x3bb45cf8e0   0x7ffd34c47374   
[0xc]   KERNEL32!BaseThreadInitThunk+0x14   0x3bb45cf920   0x7ffd3561cc91   
[0xd]   ntdll!RtlUserThreadStart+0x21   0x3bb45cf950   0x0   

First Bad Commit

Last working binary release b4179
First failing binary release b4191

Relevant log output

CommandLine: C:\WorkProg\llama\bin_vulkan\llama-cli.exe

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
ModLoad: 00007ff7`83f20000 00007ff7`83fbc000   image00007ff7`83f20000
ModLoad: 00007ffd`355d0000 00007ffd`357c8000   ntdll.dll
ModLoad: 00007ffd`34c30000 00007ffd`34cf2000   C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffd`32c60000 00007ffd`32f56000   C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffd`33430000 00007ffd`33530000   C:\WINDOWS\System32\ucrtbase.dll
ModLoad: 00007ffc`a1830000 00007ffc`a19a3000   C:\WorkProg\llama\bin_vulkan\llama.dll
ModLoad: 00007ffd`25560000 00007ffd`2556c000   C:\WorkProg\llama\bin_vulkan\ggml.dll
ModLoad: 00007ffc`a1780000 00007ffc`a1829000   C:\WorkProg\llama\bin_vulkan\ggml-base.dll
ModLoad: 00007ffd`02140000 00007ffd`021ce000   C:\WINDOWS\SYSTEM32\MSVCP140.dll
ModLoad: 00007ffd`06360000 00007ffd`0637b000   C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll
ModLoad: 00007ffd`290a0000 00007ffd`290ac000   C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll
ModLoad: 00007ffc`a16f0000 00007ffc`a177e000   C:\WorkProg\llama\bin_vulkan\ggml-cpu.dll
ModLoad: 00007ffd`33710000 00007ffd`337bf000   C:\WINDOWS\System32\ADVAPI32.dll
ModLoad: 00007ffd`337c0000 00007ffd`3385e000   C:\WINDOWS\System32\msvcrt.dll
ModLoad: 00007ffd`335f0000 00007ffd`3368f000   C:\WINDOWS\System32\sechost.dll
ModLoad: 00007ffd`34380000 00007ffd`344a3000   C:\WINDOWS\System32\RPCRT4.dll
ModLoad: 00007ffd`33530000 00007ffd`33557000   C:\WINDOWS\System32\bcrypt.dll
ModLoad: 00007ffc`dfe70000 00007ffc`dfe8b000   C:\WorkProg\llama\bin_vulkan\ggml-rpc.dll
ModLoad: 00007ffd`342f0000 00007ffd`3435b000   C:\WINDOWS\System32\WS2_32.dll
ModLoad: 00007ffc`a5fd0000 00007ffc`a6001000   C:\WINDOWS\SYSTEM32\VCOMP140.DLL
(365c.16ec): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ffd`356a07a0 cc              int     3
0:000> g
ModLoad: 00007ffd`1e0c0000 00007ffd`1e1c5000   C:\Program Files (x86)\Citrix\ICA Client\epclient64.dll
ModLoad: 00007ffd`34a90000 00007ffd`34c2d000   C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffd`33300000 00007ffd`33322000   C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffd`35560000 00007ffd`3558b000   C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffd`331e0000 00007ffd`332fa000   C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffd`33140000 00007ffd`331dd000   C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffd`2ada0000 00007ffd`2adaa000   C:\WINDOWS\SYSTEM32\VERSION.dll
ModLoad: 00007ffd`35270000 00007ffd`3529f000   C:\WINDOWS\System32\IMM32.DLL
ModLoad: 00007ffd`31ef0000 00007ffd`31f23000   C:\WINDOWS\SYSTEM32\ntmarta.dll
(365c.16ec): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\WorkProg\llama\bin_vulkan\ggml-base.dll
MSVCP140!mtx_do_lock+0x9c:
00007ffd`02152f58 488b00          mov     rax,qword ptr [rax] ds:00000000`00000000=????????????????
0:000> g
(365c.16ec): Access violation - code c0000005 (!!! second chance !!!)
MSVCP140!mtx_do_lock+0x9c:
00007ffd`02152f58 488b00          mov     rax,qword ptr [rax] ds:00000000`00000000=????????????????
0:000> g
(365c.16ec): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
MSVCP140!mtx_do_lock+0x9c:
00007ffd`02152f58 488b00          mov     rax,qword ptr [rax] ds:00000000`00000000=????????????????
0:000> gn
(365c.16ec): Access violation - code c0000005 (!!! second chance !!!)
MSVCP140!mtx_do_lock+0x9c:
00007ffd`02152f58 488b00          mov     rax,qword ptr [rax] ds:00000000`00000000=????????????????

[Xitsa]

I removed C:\Program Files (x86)\Citrix\ICA Client\epclient64.dll and retried llama-cli.exe from llama-b4191-bin-win-noavx-x64.zip:

CommandLine: C:\WorkProg\llama\bin_vulkan\llama-cli.exe

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
ModLoad: 00007ff7`5f030000 00007ff7`5f0cc000   image00007ff7`5f030000
ModLoad: 00007ffc`e6190000 00007ffc`e6388000   ntdll.dll
ModLoad: 00007ffc`e4a70000 00007ffc`e4b32000   C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffc`e3da0000 00007ffc`e4096000   C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffc`e39b0000 00007ffc`e3ab0000   C:\WINDOWS\System32\ucrtbase.dll
ModLoad: 00007ffc`dd010000 00007ffc`dd01c000   C:\WorkProg\llama\bin_vulkan\ggml.dll
ModLoad: 00007ffc`61870000 00007ffc`61919000   C:\WorkProg\llama\bin_vulkan\ggml-base.dll
ModLoad: 00007ffc`61920000 00007ffc`61a93000   C:\WorkProg\llama\bin_vulkan\llama.dll
ModLoad: 00007ffc`617e0000 00007ffc`6186e000   C:\WINDOWS\SYSTEM32\MSVCP140.dll
ModLoad: 00007ffc`882d0000 00007ffc`882eb000   C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll
ModLoad: 00007ffc`c3d50000 00007ffc`c3d5c000   C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll
ModLoad: 00007ffc`61750000 00007ffc`617de000   C:\WorkProg\llama\bin_vulkan\ggml-cpu.dll
ModLoad: 00007ffc`6fe70000 00007ffc`6fe8b000   C:\WorkProg\llama\bin_vulkan\ggml-rpc.dll
ModLoad: 00007ffc`e6020000 00007ffc`e60cf000   C:\WINDOWS\System32\ADVAPI32.dll
ModLoad: 00007ffc`e60e0000 00007ffc`e614b000   C:\WINDOWS\System32\WS2_32.dll
ModLoad: 00007ffc`e4500000 00007ffc`e459e000   C:\WINDOWS\System32\msvcrt.dll
ModLoad: 00007ffc`e5380000 00007ffc`e541f000   C:\WINDOWS\System32\sechost.dll
ModLoad: 00007ffc`e4910000 00007ffc`e4a33000   C:\WINDOWS\System32\RPCRT4.dll
ModLoad: 00007ffc`e3820000 00007ffc`e3847000   C:\WINDOWS\System32\bcrypt.dll
ModLoad: 00007ffc`6fe30000 00007ffc`6fe61000   C:\WINDOWS\SYSTEM32\VCOMP140.DLL
(31c4.31c8): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ffc`e62607a0 cc              int     3
0:000> g
(31c4.31c8): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\WorkProg\llama\bin_vulkan\ggml-base.dll
MSVCP140!mtx_do_lock+0x9c:
00007ffc`617f2f58 488b00          mov     rax,qword ptr [rax] ds:00000000`00000000=????????????????

Callstack:

[0x0]   MSVCP140!mtx_do_lock+0x9c   0xe6164eca20   0x7ffc6188c8d0   
[0x1]   ggml_base!ggml_critical_section_start+0x10   0xe6164eca80   0x7ffc61877ba2   
[0x2]   ggml_base!ggml_init+0x12   0xe6164ecab0   0x7ffc6177c79a   
[0x3]   ggml_cpu!ggml_cpu_init+0x3a   0xe6164ecb10   0x7ffc617828a9   
[0x4]   ggml_cpu!ggml_backend_cpu_reg+0x9   0xe6164ecc30   0x7ffcdd01181c   
[0x5]   ggml!ggml_backend_registry::ggml_backend_registry+0xcc   0xe6164ecc60   0x7ffcdd0121ea   
[0x6]   ggml!std::allocator<ggml_backend_reg_entry>::deallocate+0xaa   0xe6164eccb0   0x7ffcdd012968   
[0x7]   ggml!ggml_backend_load_all+0x38   0xe6164ecce0   0x7ff75f06b211   
[0x8]   llama_cli+0x3b211   0xe6164ecd70   0x7ff75f069b05   
[0x9]   llama_cli+0x39b05   0xe6164fc8f0   0x7ff75f039b54   
[0xa]   llama_cli+0x9b54   0xe6164fdb70   0x7ff75f09c1dc   
[0xb]   llama_cli+0x6c1dc   0xe6164ff9a0   0x7ffce4a87374   
[0xc]   KERNEL32!BaseThreadInitThunk+0x14   0xe6164ff9e0   0x7ffce61dcc91   
[0xd]   ntdll!RtlUserThreadStart+0x21   0xe6164ffa10   0x0   

[Xitsa]

Found a SO answer First std::mutex::lock() crashes in application built with latest Visual Studio 2022:
It looks like in the case when llama.cpp distributed without msvc redistributable, it is better to define _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR preprocessor macro.
The problem had gone after I installed latest version of msvc redistributable.