Skip to content

Make section block iteration linear in document size - #1635

Open
scks1 wants to merge 1 commit into
python-openxml:masterfrom
scks1:section-block-iteration-linear
Open

scks1 wants to merge 1 commit into
python-openxml:masterfrom
scks1:section-block-iteration-linear

Conversation

@scks1

@scks1 scks1 commented Oct 31, 2026

Copy link
Copy Markdown

Iterating the blocks of every section — the natural way to
partition a document by section — currently costs time quadratic
in the document size, because each section's blocks are computed
from a document-prefix XPath. On a generated document with 400
sections and 16,000 paragraphs, partitioning takes ~307 s; with
this change it takes ~2 s, with identical results.

Problem

_SectBlockElementIterator computes the blocks of section i by
evaluating an XPath that returns every w:p/w:tbl from the
start of the document through the end of section i, plus a
count(...) XPath over the prefix through section i−1, and
slicing. Each section therefore rescans the whole prefix before
it. Measured in block elements materialized across a full
partition:

document blocks sections before after
generated 1,000 25 25,000 1,025
generated 4,000 100 400,000 4,100
generated 8,000 50 400,000 8,050
generated 16,000 400 6,400,000 16,400

At fixed block count, the "before" work grows in direct
proportion to the number of sections. This is the cost that led
the unstructured project to rewrite its DOCX partitioning
rather than ask python-docx for each section's blocks, citing
severe slowdowns on documents with hundreds of sections and
thousands of paragraphs. A profile of the stock code attributes
~93% of partition time to these two XPath evaluations.

Change

One file, docx/oxml/section.py:

  • _SectBlockElementIterator first tries a sibling-navigation
    computation: a section's blocks are the body-level w:p/w:tbl
    siblings running back from the section's terminal node (the
    paragraph carrying its w:sectPr, or the body-level w:sectPr
    itself) to — but not including — the preceding section's
    terminal paragraph. Collecting them costs time proportional to
    the section's own size, independent of the document prefix, and
    keeps no state between calls.
  • The original XPath implementation is retained verbatim as a
    fallback (_iter_sect_block_elements_xpath) and is used
    whenever the sectPr does not terminate a body-level section in
    the ordinary way — e.g. a sectPr nested in other content, or a
    paragraph carrying more than one sectPr — so behavior on such
    documents is unchanged by construction.
  • No public API is added, removed, or altered.
    CT_SectPr.preceding_sectPr is intentionally untouched: I
    measured its preceding:: XPath first, and libxml2 stops at
    the nearest match, so its cost is already bounded by the size
    of the preceding section rather than the document prefix.

Correctness

  • A dump comparing, per document, every section's block elements
    (order and text), the preceding_sectPr chain, and
    header/footer inheritance resolution for all six header/footer
    kinds is byte-identical before and after the change, across
    generated documents (tables as blocks, 1-block sections, an
    empty final section, SDT-wrapped content) and a corpus of
    real-world-shaped documents (multi-section headers/footers,
    footnotes, comments, tracked changes, shapes, a
    LibreOffice-produced file).
  • The full test suite passes as before: 1,600 passed, with the
    same 9 pre-existing collection/setup errors in
    tests/opc/test_phys_pkg.py on both versions in my environment
    (a pytest-9 warning-as-error artifact, unrelated to this
    change).

Disclosure: this contribution was written by Árbol, an AI assistant working under the direction of the account holder, who set the acceptance criteria and verified the results before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant