3.12+: segfault tokenizing multiline fstring placeholder

[asottile]

Bug report

f'{
    hello
}:{world}'

output:

$ python3.12 -m tokenize t.py 
free(): invalid pointer
Aborted (core dumped)

here's a backtrace:

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352824640)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352824640)
    at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352824640, signo=signo@entry=6)
    at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c896f6 in __libc_message (action=action@entry=do_abort, 
    fmt=fmt@entry=0x7ffff7ddbb8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff7ca0d7c in malloc_printerr (
    str=str@entry=0x7ffff7dd9764 "free(): invalid pointer")
    at ./malloc/malloc.c:5664
#7  0x00007ffff7ca2ac4 in _int_free (av=<optimized out>, p=<optimized out>, 
    have_lock=0) at ./malloc/malloc.c:4439
#8  0x00007ffff7ca54d3 in __GI___libc_free (mem=<optimized out>)
    at ./malloc/malloc.c:3391
#9  0x000055555572430d in PyMem_RawFree (ptr=<optimized out>)
    at Objects/obmalloc.c:685
#10 _PyObject_Free (ctx=<optimized out>, p=<optimized out>)
    at Objects/obmalloc.c:1853
#11 _PyObject_Free (ctx=<optimized out>, p=<optimized out>)
    at Objects/obmalloc.c:1843
#12 0x000055555569540c in _PyTokenizer_Free (tok=0x555555d113a0)
    at Parser/tokenizer.c:1007
#13 0x00005555557cddeb in tokenizeriter_dealloc (it=0x7ffff7427f10)
    at Python/Python-tokenize.c:280
#14 0x00005555558065aa in Py_DECREF (op=<optimized out>)
    at ./Include/object.h:692
#15 Py_XDECREF (op=<optimized out>) at ./Include/object.h:788
#16 _PyFrame_ClearExceptCode (frame=0x7ffff74b38d8) at Python/frame.c:140
#17 0x00005555557de92f in clear_gen_frame (frame=0x7ffff74b38d8, 
    tstate=0x555555c17c68 <_PyRuntime+459432>) at Python/ceval.c:1492
#18 _PyEvalFrameClearAndPop (frame=0x7ffff74b38d8, 
    tstate=0x555555c17c68 <_PyRuntime+459432>) at Python/ceval.c:1504
#19 _PyEvalFrameClearAndPop (tstate=0x555555c17c68 <_PyRuntime+459432>, 
    frame=0x7ffff74b38d8) at Python/ceval.c:1498
#20 0x0000555555655e4f in _PyEval_EvalFrameDefault (tstate=<optimized out>, 
--Type <RET> for more, q to quit, c to continue without paging--c
    frame=0x7ffff74c2c88, throwflag=<optimized out>) at Python/bytecodes.c:724
#21 0x00005555556dabfd in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff74c2c88, tstate=<optimized out>) at ./Include/internal/pycore_ceval.h:87
#22 gen_send_ex2 (closing=0, exc=0, presult=<synthetic pointer>, arg=0x0, gen=0x7ffff74c2c40) at Objects/genobject.c:230
#23 gen_iternext (gen=0x7ffff74c2c40) at Objects/genobject.c:608
#24 0x00005555556f08e9 in list_extend (self=self@entry=0x7ffff7313400, iterable=iterable@entry=0x7ffff74c2c40) at Objects/listobject.c:944
#25 0x00005555556f0f68 in list___init___impl (iterable=0x7ffff74c2c40, self=0x7ffff7313400) at Objects/listobject.c:2792
#26 list___init___impl (iterable=0x7ffff74c2c40, self=0x7ffff7313400) at Objects/listobject.c:2778
#27 list_vectorcall (type=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/listobject.c:2817
#28 0x00005555556bcce3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7ffff74c7b40, callable=0x555555abe780 <PyList_Type>, tstate=0x555555c17c68 <_PyRuntime+459432>) at ./Include/internal/pycore_call.h:92
#29 PyObject_Vectorcall (callable=callable@entry=0x555555abe780 <PyList_Type>, args=args@entry=0x7ffff7fb0318, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:325
#30 0x0000555555657503 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb0248, throwflag=<optimized out>) at Python/bytecodes.c:2744
#31 0x00005555557def2f in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb01b8, tstate=0x555555c17c68 <_PyRuntime+459432>) at ./Include/internal/pycore_ceval.h:87
#32 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=0x7ffff75fd4c0, func=0x7ffff75a04a0, tstate=0x555555c17c68 <_PyRuntime+459432>) at Python/ceval.c:1610
#33 PyEval_EvalCode (co=co@entry=0x555555ca11f0, globals=globals@entry=0x7ffff75fd4c0, locals=locals@entry=0x7ffff75fd4c0) at Python/ceval.c:567
#34 0x00005555557dc5b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffff75fd4c0, globals=0x7ffff75fd4c0, source=0x555555ca11f0) at Python/bltinmodule.c:1079
#35 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:583
#36 0x000055555571a943 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7ffff7595d00, args=0x7ffff7fb0180, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:438
#37 0x00005555556bcce3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7ffff75fd4c0, callable=0x7ffff7595d00, tstate=0x555555c17c68 <_PyRuntime+459432>) at ./Include/internal/pycore_call.h:92
#38 PyObject_Vectorcall (callable=callable@entry=0x7ffff7595d00, args=args@entry=0x7ffff7fb0180, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:325
#39 0x0000555555657503 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb00d8, throwflag=<optimized out>) at Python/bytecodes.c:2744
#40 0x0000555555866e39 in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at Modules/main.c:300
#41 0x00005555558674a7 in pymain_run_python (exitcode=0x7fffffffde40) at Modules/main.c:604
#42 0x0000555555867f70 in Py_RunMain () at Modules/main.c:688
#43 pymain_main (args=0x7fffffffde00) at Modules/main.c:718
#44 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:742
#45 0x00007ffff7c29d90 in __libc_start_call_main (main=main@entry=0x5555556519e0 <main>, argc=argc@entry=4, argv=argv@entry=0x7fffffffdf88) at ../sysdeps/nptl/libc_start_call_main.h:58
#46 0x00007ffff7c29e40 in __libc_start_main_impl (main=0x5555556519e0 <main>, argc=4, argv=0x7fffffffdf88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdf78) at ../csu/libc-start.c:392
#47 0x000055555565e585 in _start ()

Your environment

• CPython versions tested on: 8da9d1b
• Operating system and architecture: ubuntu 22.04 x86_64

Linked PRs

• gh-105718: Fix buffer allocation in tokenizer with readline #105728
• [3.12] gh-105718: Fix buffer allocation in tokenizer with readline (GH-105728) #105729

[lysnikolaou]

Looking into this.

[lysnikolaou]

Fixed.