You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
./llama-cli -m ${dir}/meta-llama-405b-inst-q8_0-00001-of-00010.gguf -p "I believe the meaning of life is" -n 128 -v
First Bad Commit
No response
Relevant log output
root@e6a6e86bad41:/app# ./llama-cli -m ${dir}/meta-llama-405b-inst-q8_0-00001-of-00010.gguf -p "I believe the meaning of life is" -n 128 -v
build: 5280 (27aa2595) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: additional 9 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 31 key-value pairs and 1137 tensors from ${dir}/meta-llama-405b-inst-q8_0-00001-of-00010.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Meta-Llama-3.1-405B-Instruct
llama_model_loader: - kv 3: general.basename str = meta
llama_model_loader: - kv 4: general.size_label str = 405B
llama_model_loader: - kv 5: general.license str = llama3.1
llama_model_loader: - kv 6: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam...llama_model_loader: - kv 7: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ...llama_model_loader: - kv 8: llama.block_count u32 = 126llama_model_loader: - kv 9: llama.context_length u32 = 131072llama_model_loader: - kv 10: llama.embedding_length u32 = 16384llama_model_loader: - kv 11: llama.feed_forward_length u32 = 53248llama_model_loader: - kv 12: llama.attention.head_count u32 = 128llama_model_loader: - kv 13: llama.attention.head_count_kv u32 = 16llama_model_loader: - kv 14: llama.rope.freq_base f32 = 500000.000000llama_model_loader: - kv 15: llama.attention.layer_norm_rms_epsilon f32 = 0.000010llama_model_loader: - kv 16: general.file_type u32 = 7llama_model_loader: - kv 17: llama.vocab_size u32 = 128256llama_model_loader: - kv 18: llama.rope.dimension_count u32 = 128llama_model_loader: - kv 19: tokenizer.ggml.model str = gpt2llama_model_loader: - kv 20: tokenizer.ggml.pre str = smaug-bpellama_model_loader: - kv 21: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 22: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 23: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...llama_model_loader: - kv 24: tokenizer.ggml.bos_token_id u32 = 128000llama_model_loader: - kv 25: tokenizer.ggml.eos_token_id u32 = 128009llama_model_loader: - kv 26: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...llama_model_loader: - kv 27: general.quantization_version u32 = 2llama_model_loader: - kv 28: split.no u16 = 0llama_model_loader: - kv 29: split.count u16 = 10llama_model_loader: - kv 30: split.tensors.count i32 = 1137llama_model_loader: - type f32: 253 tensorsllama_model_loader: - type q8_0: 884 tensorsprint_info: file format = GGUF V3 (latest)print_info: file type = Q8_0print_info: file size = 405.80 GiB (8.50 BPW)init_tokenizer: initializing tokenizer for type 2load: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOGload: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOGload: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOGload: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOGload: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOGload: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOGload: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOGload: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOGload: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOGload: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOGload: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOGload: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOGload: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOGload: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOGload: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOGload: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOGload: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOGload: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOGload: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOGload: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOGload: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOGload: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOGload: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOGload: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOGload: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOGload: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOGload: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOGload: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOGload: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOGload: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOGload: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOGload: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOGload: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOGload: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOGload: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOGload: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOGload: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOGload: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOGload: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOGload: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOGload: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOGload: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOGload: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOGload: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOGload: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOGload: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOGload: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOGload: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOGload: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOGload: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOGload: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOGload: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOGload: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOGload: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOGload: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOGload: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOGload: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOGload: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOGload: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOGload: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOGload: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOGload: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOGload: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOGload: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOGload: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOGload: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOGload: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOGload: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOGload: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOGload: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOGload: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOGload: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOGload: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOGload: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOGload: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOGload: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOGload: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOGload: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOGload: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOGload: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOGload: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOGload: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOGload: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOGload: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOGload: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOGload: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOGload: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOGload: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOGload: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOGload: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOGload: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOGload: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOGload: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOGload: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOGload: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOGload: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOGload: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOGload: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOGload: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOGload: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOGload: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOGload: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOGload: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOGload: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOGload: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOGload: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOGload: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOGload: control token: 128015 '<|reserved_special_token_7|>' is not marked as EOGload: control token: 128013 '<|reserved_special_token_5|>' is not marked as EOGload: control token: 128011 '<|reserved_special_token_3|>' is not marked as EOGload: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOGload: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOGload: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOGload: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOGload: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOGload: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOGload: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOGload: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOGload: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOGload: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOGload: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOGload: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOGload: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOGload: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOGload: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOGload: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOGload: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOGload: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOGload: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOGload: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOGload: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOGload: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOGload: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOGload: control token: 128007 '<|end_header_id|>' is not marked as EOGload: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOGload: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOGload: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOGload: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOGload: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOGload: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOGload: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOGload: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOGload: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOGload: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOGload: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOGload: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOGload: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOGload: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOGload: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOGload: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOGload: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOGload: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOGload: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOGload: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOGload: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOGload: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOGload: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOGload: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOGload: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOGload: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOGload: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOGload: control token: 128012 '<|reserved_special_token_4|>' is not marked as EOGload: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOGload: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOGload: control token: 128001 '<|end_of_text|>' is not marked as EOGload: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOGload: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOGload: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOGload: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOGload: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOGload: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOGload: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOGload: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOGload: control token: 128006 '<|start_header_id|>' is not marked as EOGload: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOGload: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOGload: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOGload: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOGload: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOGload: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOGload: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOGload: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOGload: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOGload: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOGload: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOGload: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOGload: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOGload: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOGload: control token: 128000 '<|begin_of_text|>' is not marked as EOGload: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOGload: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOGload: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOGload: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOGload: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOGload: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOGload: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOGload: control token: 128010 '<|python_tag|>' is not marked as EOGload: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOGload: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOGload: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOGload: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOGload: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOGload: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOGload: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOGload: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOGload: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOGload: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOGload: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOGload: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOGload: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOGload: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOGload: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOGload: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOGload: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOGload: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOGload: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOGload: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOGload: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOGload: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOGload: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOGload: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOGload: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOGload: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOGload: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOGload: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOGload: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOGload: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOGload: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOGload: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOGload: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOGload: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOGload: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOGload: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOGload: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOGload: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOGload: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOGload: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOGload: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOGload: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOGload: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOGload: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOGload: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOGload: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOGload: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOGload: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOGload: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOGload: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOGload: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOGload: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOGload: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOGload: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOGload: control token: 128014 '<|reserved_special_token_6|>' is not marked as EOGload: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOGload: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOGload: special tokens cache size = 256load: token to piece cache size = 0.7999 MBprint_info: arch = llamaprint_info: vocab_only = 0print_info: n_ctx_train = 131072print_info: n_embd = 16384print_info: n_layer = 126print_info: n_head = 128print_info: n_head_kv = 16print_info: n_rot = 128print_info: n_swa = 0print_info: n_swa_pattern = 1print_info: n_embd_head_k = 128print_info: n_embd_head_v = 128print_info: n_gqa = 8print_info: n_embd_k_gqa = 2048print_info: n_embd_v_gqa = 2048print_info: f_norm_eps = 0.0e+00print_info: f_norm_rms_eps = 1.0e-05print_info: f_clamp_kqv = 0.0e+00print_info: f_max_alibi_bias = 0.0e+00print_info: f_logit_scale = 0.0e+00print_info: f_attn_scale = 0.0e+00print_info: n_ff = 53248print_info: n_expert = 0print_info: n_expert_used = 0print_info: causal attn = 1print_info: pooling type = 0print_info: rope type = 0print_info: rope scaling = linearprint_info: freq_base_train = 500000.0print_info: freq_scale_train = 1print_info: n_ctx_orig_yarn = 131072print_info: rope_finetuned = unknownprint_info: ssm_d_conv = 0print_info: ssm_d_inner = 0print_info: ssm_d_state = 0print_info: ssm_dt_rank = 0print_info: ssm_dt_b_c_rms = 0print_info: model type = ?Bprint_info: model params = 410.08 Bprint_info: general.name = Meta-Llama-3.1-405B-Instructprint_info: vocab type = BPEprint_info: n_vocab = 128256print_info: n_merges = 280147print_info: BOS token = 128000 '<|begin_of_text|>'print_info: EOS token = 128009 '<|eot_id|>'print_info: EOT token = 128009 '<|eot_id|>'print_info: EOM token = 128008 '<|eom_id|>'print_info: LF token = 198 'Ċ'print_info: EOG token = 128008 '<|eom_id|>'print_info: EOG token = 128009 '<|eot_id|>'print_info: max token length = 256load_tensors: loading model tensors, this can take a while... (mmap = true)load_tensors: layer 0 assigned to device CPU, is_swa = 0load_tensors: layer 1 assigned to device CPU, is_swa = 0load_tensors: layer 2 assigned to device CPU, is_swa = 0load_tensors: layer 3 assigned to device CPU, is_swa = 0load_tensors: layer 4 assigned to device CPU, is_swa = 0load_tensors: layer 5 assigned to device CPU, is_swa = 0load_tensors: layer 6 assigned to device CPU, is_swa = 0load_tensors: layer 7 assigned to device CPU, is_swa = 0load_tensors: layer 8 assigned to device CPU, is_swa = 0load_tensors: layer 9 assigned to device CPU, is_swa = 0load_tensors: layer 10 assigned to device CPU, is_swa = 0load_tensors: layer 11 assigned to device CPU, is_swa = 0load_tensors: layer 12 assigned to device CPU, is_swa = 0load_tensors: layer 13 assigned to device CPU, is_swa = 0load_tensors: layer 14 assigned to device CPU, is_swa = 0load_tensors: layer 15 assigned to device CPU, is_swa = 0load_tensors: layer 16 assigned to device CPU, is_swa = 0load_tensors: layer 17 assigned to device CPU, is_swa = 0load_tensors: layer 18 assigned to device CPU, is_swa = 0load_tensors: layer 19 assigned to device CPU, is_swa = 0load_tensors: layer 20 assigned to device CPU, is_swa = 0load_tensors: layer 21 assigned to device CPU, is_swa = 0load_tensors: layer 22 assigned to device CPU, is_swa = 0load_tensors: layer 23 assigned to device CPU, is_swa = 0load_tensors: layer 24 assigned to device CPU, is_swa = 0load_tensors: layer 25 assigned to device CPU, is_swa = 0load_tensors: layer 26 assigned to device CPU, is_swa = 0load_tensors: layer 27 assigned to device CPU, is_swa = 0load_tensors: layer 28 assigned to device CPU, is_swa = 0load_tensors: layer 29 assigned to device CPU, is_swa = 0load_tensors: layer 30 assigned to device CPU, is_swa = 0load_tensors: layer 31 assigned to device CPU, is_swa = 0load_tensors: layer 32 assigned to device CPU, is_swa = 0load_tensors: layer 33 assigned to device CPU, is_swa = 0load_tensors: layer 34 assigned to device CPU, is_swa = 0load_tensors: layer 35 assigned to device CPU, is_swa = 0load_tensors: layer 36 assigned to device CPU, is_swa = 0load_tensors: layer 37 assigned to device CPU, is_swa = 0load_tensors: layer 38 assigned to device CPU, is_swa = 0load_tensors: layer 39 assigned to device CPU, is_swa = 0load_tensors: layer 40 assigned to device CPU, is_swa = 0load_tensors: layer 41 assigned to device CPU, is_swa = 0load_tensors: layer 42 assigned to device CPU, is_swa = 0load_tensors: layer 43 assigned to device CPU, is_swa = 0load_tensors: layer 44 assigned to device CPU, is_swa = 0load_tensors: layer 45 assigned to device CPU, is_swa = 0load_tensors: layer 46 assigned to device CPU, is_swa = 0load_tensors: layer 47 assigned to device CPU, is_swa = 0load_tensors: layer 48 assigned to device CPU, is_swa = 0load_tensors: layer 49 assigned to device CPU, is_swa = 0load_tensors: layer 50 assigned to device CPU, is_swa = 0load_tensors: layer 51 assigned to device CPU, is_swa = 0load_tensors: layer 52 assigned to device CPU, is_swa = 0load_tensors: layer 53 assigned to device CPU, is_swa = 0load_tensors: layer 54 assigned to device CPU, is_swa = 0load_tensors: layer 55 assigned to device CPU, is_swa = 0load_tensors: layer 56 assigned to device CPU, is_swa = 0load_tensors: layer 57 assigned to device CPU, is_swa = 0load_tensors: layer 58 assigned to device CPU, is_swa = 0load_tensors: layer 59 assigned to device CPU, is_swa = 0load_tensors: layer 60 assigned to device CPU, is_swa = 0load_tensors: layer 61 assigned to device CPU, is_swa = 0load_tensors: layer 62 assigned to device CPU, is_swa = 0load_tensors: layer 63 assigned to device CPU, is_swa = 0load_tensors: layer 64 assigned to device CPU, is_swa = 0load_tensors: layer 65 assigned to device CPU, is_swa = 0load_tensors: layer 66 assigned to device CPU, is_swa = 0load_tensors: layer 67 assigned to device CPU, is_swa = 0load_tensors: layer 68 assigned to device CPU, is_swa = 0load_tensors: layer 69 assigned to device CPU, is_swa = 0load_tensors: layer 70 assigned to device CPU, is_swa = 0load_tensors: layer 71 assigned to device CPU, is_swa = 0load_tensors: layer 72 assigned to device CPU, is_swa = 0load_tensors: layer 73 assigned to device CPU, is_swa = 0load_tensors: layer 74 assigned to device CPU, is_swa = 0load_tensors: layer 75 assigned to device CPU, is_swa = 0load_tensors: layer 76 assigned to device CPU, is_swa = 0load_tensors: layer 77 assigned to device CPU, is_swa = 0load_tensors: layer 78 assigned to device CPU, is_swa = 0load_tensors: layer 79 assigned to device CPU, is_swa = 0load_tensors: layer 80 assigned to device CPU, is_swa = 0load_tensors: layer 81 assigned to device CPU, is_swa = 0load_tensors: layer 82 assigned to device CPU, is_swa = 0load_tensors: layer 83 assigned to device CPU, is_swa = 0load_tensors: layer 84 assigned to device CPU, is_swa = 0load_tensors: layer 85 assigned to device CPU, is_swa = 0load_tensors: layer 86 assigned to device CPU, is_swa = 0load_tensors: layer 87 assigned to device CPU, is_swa = 0load_tensors: layer 88 assigned to device CPU, is_swa = 0load_tensors: layer 89 assigned to device CPU, is_swa = 0load_tensors: layer 90 assigned to device CPU, is_swa = 0load_tensors: layer 91 assigned to device CPU, is_swa = 0load_tensors: layer 92 assigned to device CPU, is_swa = 0load_tensors: layer 93 assigned to device CPU, is_swa = 0load_tensors: layer 94 assigned to device CPU, is_swa = 0load_tensors: layer 95 assigned to device CPU, is_swa = 0load_tensors: layer 96 assigned to device CPU, is_swa = 0load_tensors: layer 97 assigned to device CPU, is_swa = 0load_tensors: layer 98 assigned to device CPU, is_swa = 0load_tensors: layer 99 assigned to device CPU, is_swa = 0load_tensors: layer 100 assigned to device CPU, is_swa = 0load_tensors: layer 101 assigned to device CPU, is_swa = 0load_tensors: layer 102 assigned to device CPU, is_swa = 0load_tensors: layer 103 assigned to device CPU, is_swa = 0load_tensors: layer 104 assigned to device CPU, is_swa = 0load_tensors: layer 105 assigned to device CPU, is_swa = 0load_tensors: layer 106 assigned to device CPU, is_swa = 0load_tensors: layer 107 assigned to device CPU, is_swa = 0load_tensors: layer 108 assigned to device CPU, is_swa = 0load_tensors: layer 109 assigned to device CPU, is_swa = 0load_tensors: layer 110 assigned to device CPU, is_swa = 0load_tensors: layer 111 assigned to device CPU, is_swa = 0load_tensors: layer 112 assigned to device CPU, is_swa = 0load_tensors: layer 113 assigned to device CPU, is_swa = 0load_tensors: layer 114 assigned to device CPU, is_swa = 0load_tensors: layer 115 assigned to device CPU, is_swa = 0load_tensors: layer 116 assigned to device CPU, is_swa = 0load_tensors: layer 117 assigned to device CPU, is_swa = 0load_tensors: layer 118 assigned to device CPU, is_swa = 0load_tensors: layer 119 assigned to device CPU, is_swa = 0load_tensors: layer 120 assigned to device CPU, is_swa = 0load_tensors: layer 121 assigned to device CPU, is_swa = 0load_tensors: layer 122 assigned to device CPU, is_swa = 0load_tensors: layer 123 assigned to device CPU, is_swa = 0load_tensors: layer 124 assigned to device CPU, is_swa = 0load_tensors: layer 125 assigned to device CPU, is_swa = 0load_tensors: layer 126 assigned to device CPU, is_swa = 0load_tensors: tensor 'token_embd.weight' (q8_0) (and 253 others) cannot be used with preferred buffer type AMX, using CPU insteadload_tensors: AMX model buffer size = 462027.75 MiBload_tensors: CPU_Mapped model buffer size = 43678.81 MiBload_tensors: CPU_Mapped model buffer size = 42433.69 MiBload_tensors: CPU_Mapped model buffer size = 41549.69 MiBload_tensors: CPU_Mapped model buffer size = 41549.62 MiBload_tensors: CPU_Mapped model buffer size = 42433.69 MiBload_tensors: CPU_Mapped model buffer size = 41549.69 MiBload_tensors: CPU_Mapped model buffer size = 41549.62 MiBload_tensors: CPU_Mapped model buffer size = 42433.69 MiBload_tensors: CPU_Mapped model buffer size = 41549.69 MiBload_tensors: CPU_Mapped model buffer size = 19584.88 MiBggml_backend_amx_buffer_set_tensor: amx repack tensor output.weight of type q8_0Segmentation fault (core dumped)
The text was updated successfully, but these errors were encountered:
Name and Version
version: 5280 (27aa259)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
I'm using latest llama-cpp container: ghcr.io/ggml-org/llama.cpp:full. Image ID is 780e2c05d696.
Operating systems
Linux
GGML backends
CPU, AMX
Hardware
Intel EMR 8570 dual socket
1TB memory
Models
LLAMA 405B quant to int8 with convert_hf_to_gguf.py
Also tried checkpoint from https://huggingface.co/nisten/meta-405b-instruct-cpu-optimized-gguf
Problem description & steps to reproduce
I met a Segmentation fault but not able to find out why it happens. (Llama3.1 70B works good for me)
First Bad Commit
No response
Relevant log output
The text was updated successfully, but these errors were encountered: