Segementation fault in prefill phase using Griffin model

[ufownl]

Backtrace of GDB:

Thread 1 "P0 X00 C005" received signal SIGSEGV, Segmentation fault.
gcpp::N_AVX2::SfpCodec::DecompressAndZeroPad<hwy::N_AVX2::Simd<hwy::bfloat16_t, 16ul, 0>, (void*)0> (num=1280, raw_bf=0x7ffffffe8c00, packed_ofs=<optimized out>, packed=..., dbf=...)
    at /home/ufownl/Softwares/gemma.cpp/./compression/sfp-inl.h:354
354	        const V8 packed = hn::LoadU(d8, base + i);
(gdb) bt
#0  gcpp::N_AVX2::SfpCodec::DecompressAndZeroPad<hwy::N_AVX2::Simd<hwy::bfloat16_t, 16ul, 0>, (void*)0> (num=1280, raw_bf=0x7ffffffe8c00, packed_ofs=<optimized out>, packed=..., dbf=...)
    at /home/ufownl/Softwares/gemma.cpp/./compression/sfp-inl.h:354
#1  gcpp::N_AVX2::CompressTraits<gcpp::SfpStream>::DecompressAndZeroPad<hwy::N_AVX2::Simd<hwy::bfloat16_t, 16ul, 0>, hwy::bfloat16_t> (num=1280, raw=0x7ffffffe8c00, packed_ofs=<optimized out>, 
    packed=..., d=...) at /home/ufownl/Softwares/gemma.cpp/./compression/compress-inl.h:421
#2  gcpp::N_AVX2::DecompressAndZeroPad<hwy::N_AVX2::Simd<hwy::bfloat16_t, 16ul, 0>, gcpp::SfpStream const, hwy::bfloat16_t> (num=1280, raw=0x7ffffffe8c00, packed_ofs=<optimized out>, 
    packed=<synthetic pointer>..., d=...) at /home/ufownl/Softwares/gemma.cpp/./compression/compress-inl.h:569
#3  gcpp::N_AVX2::MMPerPackage::DecompressB<gcpp::SfpStream> (B_view=<error reading variable: access outside bounds of object referenced via synthetic pointer>, range_kc=..., row_b=6400, 
    num_packed_B=<optimized out>, B=..., this=<optimized out>) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:1226
#4  gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(hwy::bfloat16_t*, gcpp::IndexRange const&, gcpp::IndexRange const&, auto:1)#1}::operator()<gcpp::N_AVX2::MMSetPartial>(hwy::bfloat16_t*, gcpp::IndexRange const&, gcpp::IndexRange const&, gcpp::N_AVX2::MMSetPartial) const (this=<optimized out>, 
    B_storage=B_storage@entry=0x7ffffffe8c00, range_kc=..., range_nc=..., out_tag=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:980
#5  0x0000555555c1edff in gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const::{lambda(gcpp::IndexRange const&)#1}::operator()(gcpp::IndexRange const&) const (range_kc=..., this=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:994
#6  gcpp::IndexRangePartition::VisitFirst<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const::{lambda(gcpp::IndexRange const&)#1}>(gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const::{lambda(gcpp::IndexRange const&)#1} const&) const (func=..., 
    this=<optimized out>) at /home/ufownl/Softwares/gemma.cpp/./util/threading.h:246
#7  gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const (range_nc=..., this=<optimized out>, this=<optimized out>) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:994
#8  0x0000555555d3421d in gcpp::MMParallel::ForNP<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}>(gcpp::IndexRange const&, unsigned long, unsigned long, unsigned long, gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2} const&)::{lambda(gcpp::IndexRange const&, unsigned long)#1}::operator()(gcpp::IndexRange const&, unsigned long) const
    (this=<optimized out>, worker_range=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul.h:94
#9  _ZZN4gcpp19ParallelizeOneRangeIZNS_10MMParallel5ForNPIZNKS_6N_AVX212MMPerPackage6DoNT_KINS_9SfpStreamEfEEvRKNS_8ConstMatIT_EEmRKNS_6RowPtrIT0_EEEUlRKNS_10IndexRangeEE0_EEvSJ_mmmRKS8_EUlSJ_mE_EEvRKNS_19IndexRangePartitionERN3hwy10ThreadPoolESM_ENKUlmmE_clEmm (this=<optimized out>, this=<optimized out>, thread=<optimized out>, task=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/./util/threading.h:296
#10 _ZN3hwy4pool5Tasks11CallClosureIZN4gcpp19ParallelizeOneRangeIZNS3_10MMParallel5ForNPIZNKS3_6N_AVX212MMPerPackage6DoNT_KINS3_9SfpStreamEfEEvRKNS3_8ConstMatIT_EEmRKNS3_6RowPtrIT0_EEEUlRKNS3_10IndexRangeEE0_EEvSN_mmmRKSC_EUlSN_mE_EEvRKNS3_19IndexRangePartitionERNS_10ThreadPoolESQ_EUlmmE_EEvPKvmm (opaque=<optimized out>, task=<optimized out>, worker=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:540
#11 0x0000555555c1c2b0 in hwy::pool::Tasks::WorkerRunSingle (worker=<optimized out>, this=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:485
#12 hwy::pool::Tasks::WorkerRun (worker=<optimized out>, this=<optimized out>) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:469
#13 hwy::pool::MainAdapter::operator()<hwy::SpinPause, hwy::pool::WaitSpinSeparate, hwy::pool::BarrierOrdered> (this=<optimized out>, barrier=..., wait=..., spin=...)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:834
#14 0x0000555555da50eb in hwy::pool::FunctorAddBarrier<hwy::pool::MainAdapter&>::operator()<hwy::SpinPause, hwy::pool::WaitSpinSeparate> (wait=..., spin=..., this=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:765
#15 hwy::pool::FunctorAddWait<hwy::pool::FunctorAddBarrier<hwy::pool::MainAdapter&> >::operator()<hwy::SpinPause> (spin=..., this=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:732
#16 hwy::CallWithSpin<hwy::pool::FunctorAddWait<hwy::pool::FunctorAddBarrier<hwy::pool::MainAdapter&> > > (func=..., spin_type=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/spin.h:321
#17 hwy::pool::CallWithConfig<hwy::pool::MainAdapter&> (func=..., config=...) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:788
#18 _ZN3hwy10ThreadPool3RunIZN4gcpp19ParallelizeOneRangeIZNS2_10MMParallel5ForNPIZNKS2_6N_AVX212MMPerPackage6DoNT_KINS2_9SfpStreamEfEEvRKNS2_8ConstMatIT_EEmRKNS2_6RowPtrIT0_EEEUlRKNS2_10IndexRangeEE0_EEvSM_mmmRKSB_EUlSM_mE_EEvRKNS2_19IndexRangePartitionERS0_SP_EUlmmE_EEvmmSP_ (this=0x555556beb500, begin=begin@entry=0, end=<optimized out>, closure=...)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:1065
#19 0x0000555555dae84e in gcpp::ParallelizeOneRange<gcpp::MMParallel::ForNP<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}>(gcpp::IndexRange const&, unsigned long, unsigned long, unsigned long, gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2} const&)::{lambda(gcpp::IndexRange const&, unsigned long)#1}> (func=..., pool=..., get1=...)
    at /home/ufownl/Softwares/gemma.cpp/./util/threading.h:291
#20 gcpp::MMParallel::ForNP<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}>(gcpp::IndexRange const&, unsigned long, unsigned long, unsigned long, gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2} const&) (func=..., pkg_idx=<optimized out>, inner_tasks=<optimized out>, nx_multiple=<optimized out>, range_np=..., this=<optimized out>)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul.h:92
#21 gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float> (C=..., num_packed_B=<optimized out>, B=..., this=0x7fffffff97b0) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:987
#22 gcpp::N_AVX2::MMPerPackage::operator()<gcpp::SfpStream, float> (this=this@entry=0x7fffffff97b0, B=..., C=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:894
#23 0x0000555555df3dfb in gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float>(gcpp::MatPtrT<hwy::bfloat16_t> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const (this=0x7fffffff9940, pkg_idx=pkg_idx@entry=0) at /home/ufownl/Softwares/gemma.cpp/./util/mat.h:124
#24 0x0000555555df4aef in gcpp::MMParallel::ForPkg<gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float>(gcpp::MatPtrT<hwy::bfloat16_t> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}>(unsigned long, gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float>(gcpp::MatPtrT<hwy::bfloat16_t> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1} const&)::{lambda(unsigned long, unsigned long)#1}::operator()(unsigned long, unsigned long) const (this=0x7fffffff9938, task=0, pkg_idx=0) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul.h:72
#25 hwy::ThreadPool::Run<gcpp::MMParallel::ForPkg<gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float>(gcpp::MatPtrT<hwy::bfloat16_t> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}>(unsigned long, gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float>(gcpp::MatPtrT<hwy::bfloat16_t> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1} const&)::{lambda(unsigned long, unsigned long)#1}> (this=<optimized out>, begin=<optimized out>, end=1, closure=...) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:1044
#26 0x0000555555df5446 in gcpp::MMParallel::ForPkg<gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float>(gcpp::MatPtrT<hwy::bfloat16_t> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}>(unsigned long, gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float>(gcpp::MatPtrT<hwy::bfloat16_t> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1} const&) (this=<optimized out>, func=..., max_packages=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#27 gcpp::N_AVX2::MMImpl::DoMatMul<hwy::bfloat16_t, gcpp::SfpStream, float> (A=..., B=..., C=..., args=..., config=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:1274
#28 0x0000555555df58a6 in gcpp::N_AVX2::MatMul<hwy::bfloat16_t, gcpp::SfpStream, float> (A=..., B=..., add=<optimized out>, env=..., C=...) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/timer.h:197
#29 0x0000555555df61ea in gcpp::N_AVX2::MatMul<hwy::bfloat16_t, gcpp::SfpStream, float> (A=..., B=..., add=<optimized out>, env=..., C=...) at /home/ufownl/Softwares/gemma.cpp/./util/mat.h:147
#30 0x0000555555df62a0 in gcpp::N_AVX2::FFWNoVit<gcpp::SfpStream> (activations=..., layer_weights=layer_weights@entry=0x55577e0dc000) at /home/ufownl/Softwares/gemma.cpp/./util/mat.h:440
#31 0x0000555555df667d in gcpp::N_AVX2::TransformerLayer<gcpp::SfpStream> (queries_pos=..., queries_prefix_end=..., num_tokens=num_tokens@entry=12, cache_layer_idx=cache_layer_idx@entry=0, layer_weights=0x55577e0dc000, activations=..., div_seq_len=..., kv_caches=...) at /home/ufownl/Softwares/gemma.cpp/./ops/ops-inl.h:493
#32 0x0000555555df6b63 in gcpp::N_AVX2::Prefill<gcpp::SfpStream> (queries_prompt=..., queries_pos=..., queries_prefix_end=..., query_idx_start=query_idx_start@entry=0, weights=..., activations=..., runtime_config=..., div_seq_len=..., kv_caches=...) at /usr/include/c++/9/bits/stl_vector.h:1058
#33 0x0000555555df7635 in gcpp::N_AVX2::GenerateT<gcpp::SfpStream> (model=..., weights=..., activations=..., runtime_config=..., queries_prompt=..., queries_pos_in=..., queries_prefix_end=..., query_idx_start=0, kv_caches=..., timing_info=...) at /home/ufownl/Softwares/gemma.cpp/./util/mat.h:122
#34 0x0000555555df8302 in gcpp::N_AVX2::GenerateSingleT<gcpp::SfpStream> (model=..., weights=..., runtime_config=..., prompt=..., pos=<optimized out>, prefix_end=<optimized out>, kv_cache=..., env=0x7fffffffd5d0, timing_info=...) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/aligned_allocator.h:273
#35 0x00005555555a34fb in gcpp::Gemma::<lambda(auto:1&)>::operator()<const std::unique_ptr<gcpp::ModelWeightsPtrs<gcpp::SfpStream> > > (weights=std::unique_ptr<gcpp::ModelWeightsPtrs<gcpp::SfpStream>> = {...}, __closure=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360
#36 gcpp::WeightsOwner::CallT<gcpp::Gemma::Generate(const gcpp::RuntimeConfig&, const PromptTokens&, size_t, size_t, gcpp::KVCache&, gcpp::TimingInfo&) const::<lambda(auto:1&)> > (func=..., this=0x7fffffffd9c8) at /home/ufownl/Softwares/gemma.cpp/./gemma/weights.h:621
#37 gcpp::Gemma::Generate (this=0x7fffffffd7f0, runtime_config=..., prompt=..., pos=0, prefix_end=0, kv_cache=..., timing_info=...) at /home/ufownl/Softwares/gemma.cpp/gemma/gemma.cc:116
#38 0x0000555555576703 in gcpp::ReplGemma (threading=..., inference=..., gemma=..., kv_cache=...) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/aligned_allocator.h:273
#39 0x0000555555576ff7 in gcpp::Run (loader=..., threading=..., inference=...) at /home/ufownl/Softwares/gemma.cpp/gemma/run.cc:288
#40 0x0000555555574a56 in main (argc=<optimized out>, argv=<optimized out>) at /home/ufownl/Softwares/gemma.cpp/gemma/run.cc:307

[jan-wassenberg]

Thank you for making us aware. This is not good, we are now investing more in test infra to prevent such regressions.
I found one obvious problem/mistake so far: in weights.h SplitW1 should be:

  void SplitW1() {
    // Used for Gemma and Griffin layers; FFWVit uses different tensors.
    if (layer_config.type == LayerAttentionType::kVit) return;

While debugging I started updating Griffin to add batch support. This is still in progress and I might not finish it before going out of office next week.

[ufownl]

@jan-wassenberg I tried to apply the fix you said locally, and it no longer crashed, but kept outputting meaningless tokens. It looks like the weights are not being used correctly, and the ViT part of gemma3 may have a similar issue.

[jan-wassenberg]

Thanks for testing. For Griffin, a large part of the problem is the incorrect usage of RowT in mat.cc. I am sending a CL to fix this before leaving. Unfortunately this is not sufficient to fix Griffin but it's certainly an improvement.
I think there might still be one msan uninitialized access remaining.