[BUG]: Fatal SIGSEGV in `ddtrace` profiler (Python 3.11, Gunicorn) – unsafe use of `sys._current_frames()` / `_PyThread_CurrentFrames` causes use-after-free

[lattwood]

Tracer Version(s)

2.14.4, 2.21.8

Python Version(s)

Python 3.11.12

Pip Version(s)

pip 25.1.1

Bug Report

Hi!

We're seeing regular WORKER TIMEOUTs under gunicorn, and have tracked the cause to workers being killed by SIGSEGV originating from Datadog's ddtrace profiler. (See GDB backtrace and detailed analysis at the end.)

---

1 Summary

Our production Gunicorn workers (Python 3.11, Debian Bookworm base, ddtrace profiler enabled) routinely exit with WORKER TIMEOUT. Post-mortem analysis shows the workers are dying from a segmentation fault raised inside the Datadog in-process profiler, specifically in the Cython function that walks thread frame chains (pyframe_to_frames).
The crash is deterministic under load, reproducible on fresh containers, and is not an application error: the profiler’s sampling thread dereferences a stale PyFrameObject * obtained from sys._current_frames() / _PyThread_CurrentFrames, an API explicitly documented by CPython as unsafe for concurrent inspection.

2 Observed behaviour

1. Gunicorn logs a WORKER TIMEOUT.
2. systemd-coredump captures SIGSEGV (11) in the worker process.
3. Every core shows the same stack: segfault originates in
   ddtrace/profiling/collector/_traceback.cpython-311-x86_64-linux-gnu.so : pyframe_to_frames.

3 Back-trace (abbreviated)

#0  __pthread_kill_implementation               ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal                     ./nptl/pthread_kill.c:78
#2  __GI_raise                                  ../sysdeps/posix/raise.c:26
#3  libdd_wrapper-glibc-x86_64.so               close_stderr_chainer()
#4  libdd_wrapper-glibc-x86_64.so               (signal handler wrapper)
#5  <signal handler>
#6  libpython3.11.so.1.0                        (generic getattr path)
#7  libpython3.11.so.1.0                        _PyObject_GenericGetAttrWithDict
#8  _traceback.cpython-311-x86_64-linux-gnu.so  __pyx_pw...pyframe_to_frames
#9  stack.cpython-311-x86_64-linux-gnu.so       StackCollector.collect
#10 libpython3.11.so.1.0                        PyObject_Vectorcall
#16 ddtrace/internal/_threads.so                PeriodicThread::_M_run
#18 start_thread                                ./nptl/pthread_create.c:442
#19 clone                                       ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

4 Signal info

(gdb) print $_siginfo
{si_signo = 11, si_errno = 0, si_code = -6, _sifields = {_pad = {921, 0 <repeats 27 times>}, _kill = {si_pid = 921, si_uid = 0}, _timer = {si_tid = 921, si_overrun = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _rt = {si_pid = 921, si_uid = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _sigchld = {si_pid = 921, si_uid = 0, si_status = 0, si_utime = 0, si_stime = 0}, _sigfault = {si_addr = 0x399, _addr_lsb = 0, _addr_bnd = {_lower = 0x0, _upper = 0x0}}, _sigpoll = {si_band = 921, si_fd = 0}, _sigsys = {_call_addr = 0x399, _syscall = 0, _arch = 0}}}

5 Registers and disassembly at crash site

rip = 0x7f8693026761  (__pyx_pw...pyframe_to_frames+1281)
rax = 0x0             (function pointer resolved to NULL)
rbp = 0x7f85ff4ae5c0  (supposed PyFrameObject *)
...
       mov    0x8(%rbp),%rax        ; frame->f_code
       mov    0x90(%rax),%rax       ; slot from code/type object
       call   *%rax                 ; ---> segfault (NULL / garbage)

6 Memory inspection

(gdb) x/gx $rbp
0x7f85ff4ae5c0: 0x0000000000000002       # suspiciously low refcount

(gdb) x/gx $rbp+0x8                      # supposed f_code
0x7f85ff4ae5c8: 0x00007f869405ad60

(gdb) x/32gx 0x7f869405ad60
0x7f869405ad60 <PyFrame_Type>: ...       # pointer lands on the *type* struct,

7 Profiler source responsible

The Cython collector calls CPython’s internal API every sampling tick:

cdef dict running_threads = <dict>_PyThread_CurrentFrames()
...
frames, nframes = _traceback.pyframe_to_frames(frame, max_nframes)

CPython implementation of that API:

PyObject *_PyThread_CurrentFrames(void) { ... }   // exposes raw frame ptrs

8 Why this must crash

• sys._current_frames() / _PyThread_CurrentFrames returns raw PyFrameObject * for every thread.

• CPython docs (quoted verbatim):

  "The frame returned for a non-deadlocked thread may bear no relationship to that thread’s current activity by the time calling code examines the frame."

• Frames are reference-counted; another thread can return from a function and free its frame immediately after the mapping is created.

• The profiler thread, still holding those pointers, dereferences them -> use-after-free.

• This is unaffected by the GIL (a context switch may occur after the mapping is built but before each frame is examined).

9 Impact

Any high-concurrency workload with the profiler enabled will eventually dereference freed memory and bring down the entire worker process. In our ECS service this manifests as 502/504s, disrupted sessions, and cascading retries.

10 Expected behaviour

Profiling should never terminate the target process. If the design cannot be made safe, the risk must be explicitly documented and the feature disabled (or opt-in) for Python ≥3.11.

Error Logs

No response

Libraries in Use

No response

Operating System

python:3.11-slim-bookworm

[rumblefrog]

I encountered a similar case with Datadog profiler triggering SIGSEGV. The crashes stopped as soon as Datadog profiler was disabled.

I don't have a proper stack trace, but in my case—an application using a non-thread-safe native library (Z3)—the profiler appears to be dereferencing stale references freed by Z3, resulting in Gunicorn worker being killed with exit code 139.

[2025-06-11 18:03:26 +0000] [1] [ERROR] Worker (pid:284) was sent code 139

This was on Python 3.12 on ARM (AWS Graviton) w/ tracer 2.16

[MehdiZonjy]

same issue as others. i'm running a fully async fastapi application. disabling the profiler stopped the crashes. We have another sync fastapi application and it is perfectly fine with the profiler enabled. Here are the dependencies I'm using if that helps.

    "fastapi>=0.115.12",
    "uvicorn>=0.34.2",
    "sqlalchemy>=2.0.40",
    "alembic>=1.15.2",
    "pydantic>=2.11.3",
    "pydantic[email]>=2.6.1",
    "pydantic-settings>=2.1.0",
    "python-jose>=3.3.0",
    "python-dotenv>=1.0.1",
    "celery>=5.5.1",
    "celery-redbeat>=2.3.2",
    "pytz>=2025.1",
    "httpx>=0.28.1",
    "PyJWT>=2.9.0",
    "cryptography>=44.0.2",
    "structlog>=25.2.0",
    "greenlet>=3.1.1",
    "twilio>=9.5.2",
    "python-multipart>=0.0.20",
    "aioboto3>=14.1.0",
    "types-aiobotocore[essential,sns]>=2.22.0",
    "aiocsv>=1.3.2",
    "websockets>=15.0.1",
    "pyjwt-key-fetcher>=0.8.0",
    "aiomonitor>=0.7.1",
    "psycopg2-binary>=2.9.10",
    "asyncpg>=0.30.0",
    "redis==5.2.1",
    "types-cachetools>=6.0.0.20250525",
    "phonenumbers>=9.0.8",
    "tzlocal>=5.3.1",
    "ddtrace>=3.9.4",
    "gevent>=25.5.1",
    "arq>=0.26.3",

[andreaimprovised]

can't be entirely certain its the same issue (haven't been able to repro under controlled conditions), but also experience very frequent SIGSEGV in a fully async fastapi application with ddtrace 3.9.3 installed. we only started experiencing this when we upgraded from python 3.12 to 3.13. so, dunno what is up with that

disabling profiling, as mentioned in this thread, stopped the behavior. i also tried upgrading to 3.10.x and also replacing asyncpg with psycopg3 to no effect. actually the seg faulting was worse with psycopg3, which perhaps uses threads differently than asyncpg

this is the faulthandler output for two failures we've seen among many:

• https://gist.github.com/andreaimprovised/ce7572c0125943219cfd74a342f47393
• https://gist.github.com/andreaimprovised/4399232e4a61db309f8ecae12541e274

we do a bunch of our logging in a separate thread, so under high load, that thread can be fairly busy

here is pip freeze:

annotated-types==0.7.0
anyio==4.9.0
asttokens==3.0.0
azure-core==1.34.0
azure-cosmos==4.3.1
azure-storage-blob==12.25.1
azure-storage-file-datalake==12.20.0
blinker==1.9.0
boto3==1.38.28
boto3-stubs==1.35.99
botocore==1.38.28
botocore-stubs==1.35.99
bytecode==0.16.2
cachetools==5.5.2
certifi==2025.4.26
cffi==1.17.1
charset-normalizer==3.4.2
click==8.2.1
cloudpickle==3.1.1
cryptography==45.0.3
databricks==0.2
databricks-feature-engineering==0.11.0
databricks-feature-store==0.17.0
databricks-sdk==0.55.0
datadog==0.51.0
dbl-tempo==0.1.29
ddtrace==3.9.3
decorator==5.2.1
deprecated==1.2.18
docker==7.1.0
envier==0.6.1
execnet==2.1.1
executing==2.2.0
fastapi==0.115.12
flask==2.3.3
furl==2.1.4
gitdb==4.0.12
gitpython==3.1.44
google-api-core==2.24.2
google-auth==2.40.2
google-cloud-core==2.4.3
google-cloud-storage==3.1.0
google-crc32c==1.7.1
google-resumable-media==2.7.2
googleapis-common-protos==1.70.0
greenlet==3.2.2
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
idna==3.10
importlib-metadata==8.6.1
iniconfig==2.1.0
ipython==9.4.0
ipython-pygments-lexers==1.1.1
isodate==0.7.2
itsdangerous==2.2.0
jedi==0.19.2
jinja2==3.1.6
jmespath==1.0.1
legacy-cgi==2.6.3
loguru==0.7.3
markupsafe==3.0.2
matplotlib-inline==0.1.7
mlflow-skinny==2.22.0
mypy-boto3-s3==1.35.93
numpy==2.2.6
opentelemetry-api==1.33.1
opentelemetry-sdk==1.33.1
opentelemetry-semantic-conventions==0.54b1
orderedmultidict==1.0.1
packaging==24.2
pandas==2.2.3
parso==0.8.4
pexpect==4.9.0
pluggy==1.6.0
prompt-toolkit==3.0.51
proto-plus==1.26.1
protobuf==5.29.5
psutil==7.0.0
psycopg==3.2.9
psycopg-binary==3.2.9
psycopg2-binary==2.9.10
ptyprocess==0.7.0
pure-eval==0.2.3
py4j==0.10.9.9
pyasn1==0.6.1
pyasn1-modules==0.4.2
pycparser==2.22
pydantic==2.11.5
pydantic-core==2.33.2
pygments==2.19.1
pyspark==4.0.0
pytest==8.3.5
pytest-asyncio==1.0.0
pytest-mock==3.14.1
pytest-timeout==2.4.0
pytest-xdist==3.7.0
python-dateutil==2.9.0.post0
python-json-logger==3.3.0
pytz==2025.2
pyyaml==6.0.2
requests==2.32.3
rsa==4.9.1
s3transfer==0.13.0
scipy==1.15.3
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
sqlalchemy==2.0.41
sqlparse==0.5.3
stack-data==0.6.3
starlette==0.46.2
tenacity==9.1.2
toolz==1.0.0
tqdm==4.67.1
traitlets==5.14.3
types-awscrt==0.27.2
types-s3transfer==0.13.0
typing-extensions==4.13.2
typing-inspection==0.4.1
tzdata==2025.2
urllib3==2.4.0
uvicorn==0.34.2
wcwidth==0.2.13
werkzeug==3.1.3
wrapt==1.17.2
xmltodict==0.14.2
zipp==3.22.0

[Apakottur]

I'm having a similar issue in my project, getting Python Segmentation Fault when enabling the profiler (Python 3.13 and most recent ddtrace version).
I opened a ticket with Datadog and they said that while they work on resolving the issue, it's possible to use the old profiler by setting the environment variable

DD_PROFILING_STACK_V2_ENABLED=false

The old profiler is very CPU heavy (at least for us), but doesn't crash, so it's a temporary solution untll they fix the SEGFAULT.

[lucasrothman]

Any updates from the datadog team here?

We're seeing similar SIGSEGV issues running an async FastAPI server on Python 3.13

[taegyunkim]

Hi I work on continuous profiler for Python and have been actively fixing crashes that stem from stack v2.

@lattwood The crash you're seeing is actually not from stack v2, but from stack v1, our old implementation. It is coming from _traceback.so and stack.so. Stack v1 is the default on ddtrace<2.20.0 and can be used with DD_PROFILING_STACK_V2_ENABLED=false on ddtrace>=2.20.0. We plan to deprecate/remove stack v1 from ddtrace 4.x release. We hope you can try using stack v2 before 4.x release and migrate to it. Happy to help on another GitHub issue or via appropriate support channels we provide if you see problems upgrading and migrating.

@MehdiZonjy, @andreaimprovised, @lucasrothman, @Apakottur
For async heavy applications, there were a few fixes related to asyncio.

1. Fixed in fix: undefined behavior on python_stack by using unsigned types with underflow check P403n1x87/echion#124, and released in dd-trace-py 3.14.0, this commit

A typical native stack backtrace looks like the following

_stack_v2.cpython-313-x86_64-linux-gnu.so                        ThreadInfo::unwind_tasks()
_stack_v2.cpython-313-x86_64-linux-gnu.so                          ThreadInfo::unwind(_ts*) 
_stack_v2.cpython-313-x86_64-linux-gnu.so     ThreadInfo::sample(long, _ts*, unsigned long) 
_stack_v2.cpython-313-x86_64-linux-gnu.so  Datadog::Sampler::sampling_thread(unsigned long) 
_stack_v2.cpython-313-x86_64-linux-gnu.so                       call_sampling_thread(void*)

2. Fixed in fix: copy ob_digit field in PyLongObject before dereferencing P403n1x87/echion#133, and to be released in dd-trace-py 3.15.0, merged to main in 9c89921

Stack backtrace

Version: {'3.14.0'}
Runtime_version: {'3.12.11'}
       off                                       file                                 name
0   0xda58  _stack_v2.cpython-312-x86_64-linux-gnu.so                     StringTable::key
1   0xe886  _stack_v2.cpython-312-x86_64-linux-gnu.so                         Frame::Frame 
2   0xec60  _stack_v2.cpython-312-x86_64-linux-gnu.so                           Frame::get
3   0xee4f  _stack_v2.cpython-312-x86_64-linux-gnu.so                          Frame::read
4   0xef18  _stack_v2.cpython-312-x86_64-linux-gnu.so                         unwind_frame 
5   0xf02b  _stack_v2.cpython-312-x86_64-linux-gnu.so                  unwind_python_stack
6  0x119b6  _stack_v2.cpython-312-x86_64-linux-gnu.so                   ThreadInfo::unwind
7  0x11b3d  _stack_v2.cpython-312-x86_64-linux-gnu.so                   ThreadInfo::sample
8  0x10518  _stack_v2.cpython-312-x86_64-linux-gnu.so                      for_each_thread

@rumblefrog
I don't think I have seen that kind of crash yet. Would you be willing to provide stack backtrace from the crash? You'd need to do the following.

ulimit -c unlimited
<run your application til crash>
gdb <path to python binary you used to run your application> <path to core dump>
gdb> bt

On ddtrace>=3.14.x top 3 common crashes that are related to or caused by ddtrace are

1. uwsgi<2.0.30 with (--lazy or --lazy-apps) and without --skip-atexit. Fixed in uwsgi>=2.0.30 fix: install atexit handlers after apps are initialized unbit/uwsgi#2726. We updated our docs a few months ago, docs(uwsgi): update docs on --lazy-apps for uwsgi and tests #13550. The profiler won't load if such configs are detected starting from ddtrace==3.15.0, with fix(profiling): require --skip-atexit for uWSGI<2.0.30, --lazy or --lazy-apps #14526
2. The second fix I mentioned above, fix: copy ob_digit field in PyLongObject before dereferencing P403n1x87/echion#133
3. fix(profiling): avoid use-after-free when profiling realloc #14550 fixed a memory profiler related one, and it will be released in 3.15.0

Again, all of these will be resolved in 3.15.0 release, which will happen next week.

[taegyunkim]

3.15.0 is available on PyPI https://pypi.org/project/ddtrace/3.15.0/