Is the "tbody" specificity required for LaTeXFootnoteVisitor.depart_table?

[munozejm]

Describe the bug

Sphinx Version: 7.1.2
Command Issued: sphinx-build -b latex -d build/doctrees source build/latex (to generate a .tex file which is to be later converted to a PDF)

In my project's documentation, we have custom "longtable" class tables that don't include "tbody" tags.

When Sphinx runs the LaTeXFootnoteTransform and subsequently employs the LaTeXFootnoteVisitor.depart_table method to end the visit, it performs the following: tbody = next(node.findall(nodes.tbody)). Given that there are no "tbody" tags for our custom tables, the next function within the LaTeXFootnoteVisitor.depart_table attempts to operate on an empty iterator, issues a StopIteration error, and all Sphinx document processing dies. Our custom node processing function, process_required_equipment_list_nodes, is never invoked because emitting event: 'doctree-resolved' is never reached.

If I comment out app.add_post_transform(LaTeXFootnoteTransform) from the setup function in builders/latex/transforms.py, I'm able to bypass that transform. Our custom node processing function, process_required_equipment_list_nodes, is invoked since emitting event: 'doctree-resolved' is reached in this case. Sphinx processing is then able to complete, and the desired LaTeX file is generated with the correct content in the custom tables. The resulting PDF file that is converted from the output .tex file looks as expected.

Given the above information, is the "tbody" specificity required?

An example of the LaTeX for one of those custom tables is provided below. It was copied directly from the .tex file generated by Sphinx.

\begin{savenotes}
\sphinxatlongtablestart
\sphinxthistablewithglobalstyle
\makeatletter
\LTleft @totalleftmargin plus1fill
\LTright\dimexpr\columnwidth-@totalleftmargin-\linewidth\relax plus1fill
\makeatother
\begin{longtable}{{3}{\X{1}{3}}}
\sphinxthelongtablecaptionisattop
\caption{Required Equipment\strut}\label{\detokenize{front_matter/required_materials:id3}}\[\sphinxlongtablecapskipadjust]
\sphinxtoprule
\sphinxstyletheadfamily \sphinxstylestrong{Part Number}&\sphinxstyletheadfamily \sphinxstylestrong{Title}&\sphinxstyletheadfamily \sphinxstylestrong{Referenced Sections}\
\sphinxmidrule
\endfirsthead

\multicolumn{3}{c}{\sphinxnorowcolor
\makebox[0pt]{\sphinxtablecontinued{\tablename\ \thetable{} \textendash{} continued from previous page}}%
}\
\sphinxtoprule
\sphinxstyletheadfamily \sphinxstylestrong{Part Number}&\sphinxstyletheadfamily \sphinxstylestrong{Title}&\sphinxstyletheadfamily \sphinxstylestrong{Referenced Sections}\
\sphinxmidrule
\endhead

\sphinxbottomrule
\multicolumn{3}{r}{\sphinxnorowcolor
\makebox[0pt][r]{\sphinxtablecontinued{continues on next page}}%
}\
\endfoot

\endlastfoot
\sphinxtableatstartofbodyhook

\sphinxAtStartPar

< ...Rows are filled in here...>

\
\sphinxbottomrule
\end{longtable}
\sphinxtableafterendhook
\sphinxatlongtableend
\end{savenotes}

How to Reproduce

Below is the definition (within one of our .rst files) of the custom table from above.

.. required_equipment_list::
:widths: 30 30 30
:include_references: true
:name: Required Equipment Table

Below is some of the pertinent Python code for that custom table.

def setup(app):

app.add_config_value('required_equipment_list', 'configuration_files/ifc_baseline/equipment.yml', 'env')
app.add_node(required_equipment_list,
             html=(visit_pass, depart_pass),
             latex=(visit_pass, depart_pass),
             text=(visit_pass, depart_pass))
app.add_directive('required_equipment_list', RequiredEquipmentListDirective)
app.add_role('equipment', equipment_role)
app.connect('doctree-resolved', process_required_equipment_list_nodes)

class RequiredEquipmentListDirective(Table):

required_arguments = 0
optional_arguments = 1
has_content = False

option_spec = {'include_references': directives.unchanged,
               'include_attributes': directives.unchanged,
               'exact_match_only': directives.unchanged,
               'class': directives.class_option,
               'name': directives.unchanged,
               'align': align,
               'widths': directives.value_or(('auto', 'grid'),
                                             directives.positive_int_list)}

def run(self):
    env = self.state.document.settings.env
    app = env.app
    config = app.config

    if not self.arguments:
        self.arguments = ['Required Equipment']
    title, messages = self.make_title()
    env.required_equipment_title_node = title

    rel = required_equipment_list()
    rel['options'] = self.options
    table = nodes.table('', classes=['longtable'])
    table.insert(0, title)
    rel['header'] = [nodes.strong('Part Number', 'Part Number'),
                     nodes.strong('Title', 'Title')]
    num_cols = len(rel['header'])

    rel['attribute_cols'] = []
    if 'include_attributes' in self.options:
        for attribute in self.options.get('include_attributes').split(','):
            attr_pair = attribute.strip().split(':')
            if len(attr_pair) == 2:
                attr_title = attr_pair[0].strip()
                attr_name = attr_pair[1].strip()
            else:
                attr_title = attr_name = attr_pair[0].strip()

            rel['header'] += [nodes.strong(str(attr_title), str(attr_title))]
            rel['attribute_cols'] += [attr_name]
            num_cols += 1

    if self.options.get('include_references', '').lower() in ('yes', '1', 'true'):
        rel['include_references'] = True
        num_cols += 1
        rel['header'] += [nodes.strong('Referenced Sections', 'Referenced Sections')]
    else:
        rel['include_references'] = False

    rel['num_cols'] = num_cols

    table += rel
    return [table]

def process_required_equipment_list_nodes(app, doctree, fromdocname):

global relations
global included_docs

env = app.builder.env

if not relations:
    relations = get_doc_relations(env)
    included_docs = relations['included_docs']

for table in doctree.traverse(required_equipment_list):
    content = []
    header = table['header']
    tgroup = nodes.tgroup(cols=len(table['header']))
    table += tgroup

    col_widths = get_column_widths(table, table['num_cols'], env, fromdocname)
    for col_width in col_widths:
        colspec = nodes.colspec()
        if col_width is not None:
            colspec.attributes['colwidth'] = col_width
        tgroup += colspec

    thead = nodes.thead()
    tgroup += thead
    thead += create_table_row(header)
    tbody = nodes.tbody()
    tgroup += tbody

    count = 0

    required_equipment = {}

    for docname in included_docs:
        for entry in [i for i in env.all_equipment if i['docname'] == docname]:
            if entry['pn'] not in required_equipment:
                required_equipment[entry['pn']] = {
                    'referenced_sections': [],
                    'info': entry['equipment_info'],
                }
            section_info = get_section_containing_node(entry['target'])
            section_number = get_section_number(env, docname, section_info['section_id'])
            section_info['section_number'] = section_number
            section_info['docname'] = docname
            section_info['targetid'] = entry['targetid']
            required_equipment[entry['pn']]['referenced_sections'].append(section_info)
    count = 0
    for pn in sorted(required_equipment):
        row = [pn, required_equipment[pn]['info']['title']]
        for attr in table['attribute_cols']:
            row += [str(env.equipment_info[pn].get(attr, ''))]
        if table['include_references']:
            paragraphs = []
            for reference in sorted(required_equipment[pn]['referenced_sections'], key=lambda x: x['section_number']['section_num_list']):
                paragraph = nodes.paragraph()
                new_link = create_link(env, app, fromdocname, reference['docname'], reference['targetid'], reference['section_number']['section_num_str'])
                paragraph += new_link
                paragraphs.append(paragraph)
            row.append(paragraphs)
        tbody.append(create_table_row(row))
        count += 1
    if count == 0:
        tbody.append(create_table_row(['N/A'] * table['num_cols']))
    table.replace_self([table])

Environment Information

We are running Gitlab with a runner built using the "sphinxdoc/sphinx-latexpdf" image from Docker Hub.  When the StopIteration issue is encountered and Sphinx dies, Gitlab automatically performs cleanup and removes the container (i.e., the runner).  Attempting to execute "sphinx-build --bug-report" does not pan out in that situation.

Below is the stack trace that is communicated as part of Sphinx's processing output.

[app] emitting event: 'build-finished'(StopIteration(),)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sphinx/cmd/build.py", line 290, in build_main
    app.build(args.force_all, args.filenames)
  File "/usr/local/lib/python3.11/site-packages/sphinx/application.py", line 351, in build
    self.builder.build_update()
  File "/usr/local/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 287, in build_update
    self.build(['__all__'], to_build)
  File "/usr/local/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 360, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/local/lib/python3.11/site-packages/sphinx/builders/latex/__init__.py", line 294, in write
    doctree = self.assemble_doctree(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sphinx/builders/latex/__init__.py", line 360, in assemble_doctree
    self.env.resolve_references(largetree, indexfile, self)
  File "/usr/local/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 658, in resolve_references
    self.apply_post_transforms(doctree, fromdocname)
  File "/usr/local/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 670, in apply_post_transforms
    transformer.apply_transforms()
  File "/usr/local/lib/python3.11/site-packages/sphinx/transforms/__init__.py", line 80, in apply_transforms
    super().apply_transforms()
  File "/usr/local/lib/python3.11/site-packages/docutils/transforms/__init__.py", line 182, in apply_transforms
    transform.apply(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/sphinx/transforms/post_transforms/__init__.py", line 37, in apply
    self.run(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/sphinx/builders/latex/transforms.py", line 366, in run
    self.document.walkabout(visitor)
  File "/usr/local/lib/python3.11/site-packages/docutils/nodes.py", line 186, in walkabout
    if child.walkabout(visitor):
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docutils/nodes.py", line 186, in walkabout
    if child.walkabout(visitor):
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docutils/nodes.py", line 186, in walkabout
    if child.walkabout(visitor):
       ^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python3.11/site-packages/docutils/nodes.py", line 200, in walkabout
    visitor.dispatch_departure(self)
  File "/usr/local/lib/python3.11/site-packages/docutils/nodes.py", line 2021, in dispatch_departure
    return method(node)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sphinx/builders/latex/transforms.py", line 441, in depart_table
    tbody = next(node.findall(nodes.tbody))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration
Exception occurred:
  File "/usr/local/lib/python3.11/site-packages/sphinx/builders/latex/transforms.py", line 441, in depart_table
    tbody = next(node.findall(nodes.tbody))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration
The full traceback has been saved in /tmp/sphinx-err-h_ja2qut.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

Sphinx extensions

On the "sphinxdoc/sphinx-latexpdf" image's Debian OS, we perform an update, an upgrade, and add packages.

  - apt-get update
  - apt-get upgrade -y
  - apt-get install -y git
  - apt-get install -y unzip
  - apt-get install -y wget
  - apt-get install -y texlive
  - apt-get install -y texlive-bibtex-extra
  - apt-get install -y texlive-font-utils
  - apt-get install -y texlive-lang-english
  - kpsewhich -var-value=TEXMFLOCAL
  - kpsewhich -var-value=TEXMFDIST
  - unzip -qo acrotex.zip
  - cd acrotex
  - |+
    for acrofile in $(ls -1 *.ins | egrep -v 'exerquiz|acrotex')
    do
      latex $acrofile
    done
    mkdir -p /usr/share/texlive/texmf-dist/tex/latex/acrotex
    cp *.sty /usr/share/texlive/texmf-dist/tex/latex/acrotex
    cp *.cfg /usr/share/texlive/texmf-dist/tex/latex/acrotex
    cp *.def /usr/share/texlive/texmf-dist/tex/latex/acrotex
  - cd ..
  - mktexlsr /usr/share/texlive/texmf-dist
  - rm -rf acrotex*

We also need to pip install a number of Python packages that are required for our documentation to be generated.

  - python -m pip install sphinx-autobuild
  - python -m pip install sphinx-git
  - python -m pip install sphinxcontrib-actdiag
  - python -m pip install sphinxcontrib-ansibleautodoc
  - python -m pip install sphinxcontrib-autoprogram
  - python -m pip install sphinxcontrib-blockdiag
  - python -m pip install sphinxcontrib-confluencebuilder
  - python -m pip install sphinxcontrib-jsonschema
  - python -m pip install sphinxcontrib-jupyter
  - python -m pip install sphinxcontrib-nwdiag
  - python -m pip install sphinxcontrib-plantuml
  - python -m pip install sphinxcontrib-seqdiag
  - python -m pip install sphinxcontrib-websupport
  - python -m pip install ciscoconfparse
  - python -m pip install decorator
  - python -m pip install enum34
  - python -m pip install funcparserlib
  - python -m pip install gitdb
  - python -m pip install Jinja2==3.0.3
  - python -m pip install jupyter-core
  - python -m pip install netaddr
  - python -m pip install plantuml
  - python -m pip install python-dateutil
  - python -m pip install pyyaml
  - python -m pip install sets
  - python -m pip install tablib

Additional context

The version of Acrotex is 2021-10-03.