Attempt to use data in destroyed transaction pool

[AlexPeshkoff]

Bug was reported privately by firebird user. As a visible result there were core dump and an error in firebird.log:

servername  Tue Dec 27 19:01:14 2022
        Operating system call pthread_mutex_lock failed. Error code 22

Stack trace:

#0  0x00007fa3276c3a9f in raise () from /opt/lib64/libc.so.6
#1  0x00007fa327696e05 in abort () from /opt/lib64/libc.so.6
#2  0x00007fa325d6756d in __gnu_cxx::__verbose_terminate_handler () at ../../../../gcc-6.4.0/libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007fa325ce7406 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../gcc-6.4.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007fa325ce7fb9 in __cxa_call_terminate (ue_header=0x7f9974012630) at ../../../../gcc-6.4.0/libstdc++-v3/libsupc++/eh_call.cc:54
#5  0x00007fa325ce7b9d in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=<optimized out>, 
    exception_class=5138137972254386944, ue_header=<optimized out>, context=0x7f9a3d44e5d0)
    at ../../../../gcc-6.4.0/libstdc++-v3/libsupc++/eh_personality.cc:676
#6  0x00007fa327c6b583 in _Unwind_RaiseException_Phase2 () from /opt/lib64/libgcc_s.so.1
#7  0x00007fa327c6baf1 in _Unwind_RaiseException () from /opt/lib64/libgcc_s.so.1
#8  0x00007fa325ce616b in __cxxabiv1::__cxa_throw (obj=0x7f9974012650, tinfo=0x7fa32623df88 <typeinfo for Firebird::system_call_failed>, 
    dest=0x7fa325c210a0 <Firebird::system_call_failed::~system_call_failed()>) at ../../../../gcc-6.4.0/libstdc++-v3/libsupc++/eh_throw.cc:82
#9  0x00007fa325c20696 in Firebird::system_call_failed::raise (syscall=0x7fa325d8186c "pthread_mutex_lock", error_code=22)
    at /usr/home/firebird/B4.0-release/src/common/fb_exception.cpp:252
#10 0x00007fa325c5c52d in Firebird::Mutex::enter (aReason=0x0, this=<optimized out>)
    at /usr/home/firebird/B4.0-release/src/include/../common/classes/locks.h:211
#11 Firebird::EnsureUnlock<Firebird::Mutex, Firebird::NotRefCounted>::enter (this=<synthetic pointer>)
    at /usr/home/firebird/B4.0-release/src/include/../common/classes/RefMutex.h:129
#12 Firebird::MemPool::releaseBlock (this=0x7f9513f8d940, block=0x7f9a915f4e60, decrUsage=<optimized out>)
    at /usr/home/firebird/B4.0-release/src/common/classes/alloc.cpp:2423
#13 0x00007fa32587ecce in Jrd::Savepoint::destroy (savepoint=<optimized out>)
    at /usr/home/firebird/B4.0-release/src/jrd/../jrd/../jrd/Savepoint.h:243
#14 EXE_unwind (tdbb=tdbb@entry=0x7f9a3d44ef20, request=request@entry=0x7f9d9448cdc0) at /usr/home/firebird/B4.0-release/src/jrd/exe.cpp:999
#15 0x00007fa32587f111 in EXE_release (tdbb=tdbb@entry=0x7f9a3d44ef20, request=0x7f9d9448cdc0)
    at /usr/home/firebird/B4.0-release/src/jrd/exe.cpp:780
#16 0x00007fa325793bf8 in Jrd::JrdStatement::release (this=0x7f9e6390b910, tdbb=0x7f9a3d44ef20)
    at /usr/home/firebird/B4.0-release/src/jrd/JrdStatement.cpp:643
#17 0x00007fa3257ddc5a in Jrd::Routine::releaseStatement (this=this@entry=0x7f931f16d430, tdbb=tdbb@entry=0x7f9a3d44ef20)
    at /usr/home/firebird/B4.0-release/src/jrd/Routine.cpp:303
#18 0x00007fa3258f190b in MET_clear_cache (tdbb=tdbb@entry=0x7f9a3d44ef20) at /usr/home/firebird/B4.0-release/temp/Release/jrd/met.cpp:4588
#19 0x00007fa3258c648d in release_attachment (tdbb=tdbb@entry=0x7f9a3d44ef20, attachment=attachment@entry=0x7f98d2fd3ec0, 
    dropGuard=dropGuard@entry=0x0) at /usr/home/firebird/B4.0-release/src/jrd/jrd.cpp:7608
#20 0x00007fa3258d1ba3 in purge_attachment (tdbb=tdbb@entry=0x7f9a3d44ef20, sAtt=0x7f9850dfea60, flags=flags@entry=2)
    at /usr/home/firebird/B4.0-release/src/jrd/jrd.cpp:8345
#21 0x00007fa3258d2366 in Jrd::JAttachment::freeEngineData (this=this@entry=0x7f9d8bfd40c0, user_status=user_status@entry=0x7f9a3d44f0c0, 
    forceFree=forceFree@entry=false) at /usr/home/firebird/B4.0-release/src/jrd/jrd.cpp:3322
#22 0x00007fa3258d5547 in Jrd::JAttachment::internalDetach (this=this@entry=0x7f9d8bfd40c0, user_status=user_status@entry=0x7f9a3d44f0c0)
    at /usr/home/firebird/B4.0-release/src/jrd/jrd.cpp:3259
#23 0x00007fa3258d5582 in Jrd::JAttachment::detach (this=this@entry=0x7f9d8bfd40c0, user_status=user_status@entry=0x7f9a3d44f0c0)
    at /usr/home/firebird/B4.0-release/src/jrd/jrd.cpp:3271
#24 0x00007fa3258e3e6f in Firebird::IAttachmentBaseImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher (self=0x7f9d8bfd40c8, status=0x7f9a3d44f298)
    at /usr/home/firebird/B4.0-release/src/include/firebird/IdlFbInterfaces.h:11380
#25 0x00007fa3288fa313 in Firebird::IAttachment::detach<Firebird::CheckStatusWrapper> (status=0x7f9a3d44f290, this=0x7f9d8bfd40c8)
    at /usr/home/firebird/B4.0-release/src/include/firebird/IdlFbInterfaces.h:2685
#26 <lambda()>::operator() (__closure=0x7f9a3d44f1e0) at /usr/home/firebird/B4.0-release/src/yvalve/why.cpp:5930
#27 std::_Function_handler<void(), Why::YAttachment::detach(Firebird::CheckStatusWrapper*)::<lambda()> >::_M_invoke(const std::_Any_data &) (
    __functor=...) at /usr/include/c++/6.4.0/functional:1731
#28 0x00007fa32892a1bf in std::function<void ()>::operator()() const (this=0x7f9a3d44f200) at /usr/include/c++/6.4.0/functional:2127
#29 Why::done<Why::YAttachment>(Firebird::CheckStatusWrapper*, Why::YEntry<Why::YAttachment>&, Why::YAttachment*, std::function<void ()>, std::function<void ()>) (status=0x7f9a3d44f290, entry=..., y=y@entry=0x7fa31ef03d90, newClose=..., oldClose=...)
    at /usr/home/firebird/B4.0-release/src/yvalve/why.cpp:1340
#30 0x00007fa328910ec1 in Why::YAttachment::detach (this=this@entry=0x7fa31ef03d90, status=<optimized out>, status@entry=0x7f9a3d44f290)
    at /usr/home/firebird/B4.0-release/src/yvalve/why.cpp:5933
#31 0x00007fa32892a38f in Firebird::IAttachmentBaseImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher (self=0x7fa31ef03d98, status=0x7f9a3d44f338)
    at /usr/home/firebird/B4.0-release/src/include/firebird/IdlFbInterfaces.h:11380
#32 0x0000000000433298 in Firebird::IAttachment::detach<Firebird::CheckStatusWrapper> (status=0x7f9a3d44f330, this=0x7fa31ef03d98)
    at /usr/home/firebird/B4.0-release/src/include/firebird/IdlFbInterfaces.h:2685
#33 rem_port::disconnect (this=this@entry=0x7f9df489b8c0, sendL=sendL@entry=0x7f9d9e581758, receiveL=receiveL@entry=0x7f9d9e581cf8)
    at /usr/home/firebird/B4.0-release/src/remote/server/server.cpp:2990
#34 0x000000000044763a in process_packet (port=<optimized out>, sendL=sendL@entry=0x7f9d9e581758, receive=receive@entry=0x7f9d9e581cf8, 
    result=result@entry=0x7f9a3d450da8) at /usr/home/firebird/B4.0-release/src/remote/server/server.cpp:4838
#35 0x00000000004498ab in loopThread () at /usr/home/firebird/B4.0-release/src/remote/server/server.cpp:6539
#36 0x0000000000467580 in (anonymous namespace)::ThreadArgs::run (this=<synthetic pointer>)
    at /usr/home/firebird/B4.0-release/src/common/ThreadStart.cpp:78
#37 (anonymous namespace)::threadStart (arg=0x7f96f51e6b30) at /usr/home/firebird/B4.0-release/src/common/ThreadStart.cpp:94

clearly shows that an error happens due to an attempt to cleanup (too late attempt - when processing disconnect in CS) savepoint stored in metadata cache. Put attention that savepoints are always created in transaction memory pool and keeping them after commit/rollback in metadata cache is bad idea.

[dyemanov]

Alex, do you plan to backport it into v3/v4?

[AlexPeshkoff]

On 1/12/23 15:10, Dmitry Yemanov wrote: Alex, do you plan to backport it into v3/v4?

Yes, certainly. But sooner of all it needs one more change to make code cleaner. I wanted to wait for Pavel to fix tests for master but looks like it can take too long. Will have to use just comparison of failed list after that change. PS. This has nothing to do with a bug that was observed in HQBird for a long time.