Skip to content

[subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression) #90228

Open
@graysky

Description

@graysky
mannequin
BPO 46070
Nosy @vstinner, @encukou, @ericsnowcurrently, @ndjensen, @hroncok, @uckelman, @corona10, @miss-islington, @shihai1991, @erlend-aasland, @bsteffensmeier, @M-Reimer, @prahal
PRs
  • bpo-46070: Fix asyncio initialisation guard #30423
  • [3.10] bpo-46070: Fix asyncio initialisation guard (GH-30423) #30453
  • [3.9] bpo-46070: Fix asyncio initialisation guard (GH-30423) #30454
  • bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30564
  • [3.10] bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30565
  • [3.9] bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30566
  • bpo-46070: _PyGC_Fini() untracks objects #30577
  • [3.10] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30578
  • [3.9] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30579
  • [3.9] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30580
  • Files
  • win_py399_crash_reproducer.py: Reproducer that intermittently crashes on windows
  • bug.py
  • pyobject_ob_interp.patch
  • sqlite3_crash.py: Reproducer that crashes 90% of the time
  • bug.py_asyncio_cpustressed-crash.log: asyncio bug.py crash while running stress -c nproc --all
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-12-14.10:57:50.149>
    labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11', 'expert-subinterpreters']
    title = '[subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)'
    updated_at = <Date 2022-03-24.21:16:20.771>
    user = 'https://bugs.python.org/graysky'

    bugs.python.org fields:

    activity = <Date 2022-03-24.21:16:20.771>
    actor = 'prahal'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Interpreter Core', 'Subinterpreters']
    creation = <Date 2021-12-14.10:57:50.149>
    creator = 'graysky'
    dependencies = []
    files = ['50494', '50497', '50560', '50699', '50700']
    hgrepos = []
    issue_num = 46070
    keywords = ['patch']
    message_count = 54.0
    messages = ['408520', '408539', '408540', '408541', '408558', '408568', '408633', '408662', '408664', '408665', '409245', '409255', '409461', '409573', '409686', '409772', '409778', '409780', '409795', '409798', '409802', '409962', '409965', '409966', '409970', '409980', '409983', '409984', '409991', '410010', '410014', '410442', '410444', '410446', '410447', '410493', '410497', '410498', '410500', '410505', '410507', '410509', '410510', '410513', '410517', '410518', '410520', '415954', '415955', '415963', '415964', '415970', '415975', '415977']
    nosy_count = 15.0
    nosy_names = ['vstinner', 'petr.viktorin', 'eric.snow', 'ndjensen', 'hroncok', 'uckelman', 'corona10', 'miss-islington', 'shihai1991', 'erlendaasland', 'graysky', 'bsteffensmeier', 'M-Reimer', 'jokot3', 'prahal']
    pr_nums = ['30423', '30453', '30454', '30564', '30565', '30566', '30577', '30578', '30579', '30580']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue46070'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    graysky

    graysky commented on Dec 14, 2021

    @graysky
    MannequinAuthor

    Seems as though cpython is broken when working with subinterpreters. The problematic change could be (d0d2965) affecting import.c.[1] Reverting this commit and rebuilding python fixes the issues on my system with some scripts that import sqlite, for example, the Kodi plugin YouTube[2] and the IMDB Trailers plugin. Others have reported similar breakage with other python code[3].

    Example output when the bug manifests:

    ERROR <general>: Traceback (most recent call last):
    ERROR <general>:   File "/usr/lib/python3.10/sqlite3/dbapi2.py", line 83, in register_adapters_and_converters
    ERROR <general>:     
    ERROR <general>: register_converter("timestamp", convert_timestamp)
    ERROR <general>: 
    ERROR <general>: KeyError
    ERROR <general>: : 
    ERROR <general>: 'timepart_full'
    ERROR <general>:                                              
    ERROR <general>: Exception ignored deletion of interned string failed
    ERROR <general>: :

    References:

    1. d0d2965
    2. Plugin incompatible with python 3.10.1 anxdpanic/plugin.video.youtube#255
    3. https://bbs.archlinux.org/viewtopic.php?id=272121
    added
    3.10only security fixes
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Dec 14, 2021
    ericsnowcurrently

    ericsnowcurrently commented on Dec 14, 2021

    @ericsnowcurrently
    Member

    (related: bpo-44059)

    Presumably the problem relates to global state used in different interpreters leading to an inconsistent state in the crashing extension (or its dependencies).

    @graysky, do you know if this was a problem before Python 3.8?

    encukou

    encukou commented on Dec 14, 2021

    @encukou
    Member

    Interned strings were broken in #64257, see bpo-46006. Maybe that's also the issue here?

    75 remaining items

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10only security fixes3.11only security fixes3.9only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-regextopic-subinterpreterstype-bugAn unexpected behavior, bug, or error

      Projects

      Status

      Todo

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

        [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression) · Issue #90228 · python/cpython