Skip to content

Restore interrupted with-enter coverage with pending exception support - #9497

Merged
youknowone merged 2 commits into
RustPython:mainfrom
youknowdot:python315-with-async-exception-20261031
Oct 31, 2026
Merged

youknowone merged 2 commits into
RustPython:mainfrom
youknowdot:python315-with-async-exception-20261031

Conversation

@youknowdot

@youknowdot youknowdot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Restore the missing test_with.InterruptDuringEnter regression from pinned CPython d220409a13b77de7a6c30abe16ea90db01528edc through the real update_lib workflow. The canonical class/import are restored unchanged; existing test markers remain unchanged.

The fixture needs genuine deferred delivery rather than an immediate KeyboardInterrupt from __enter__. This adds:

  • A VM-owned pending exception-class reference, delivered at bytecode safepoints after existing signals and scheduled GC. Validation/normalization calls the class normally, including its metaclass and constructor, and preserves normalization failure notes.
  • Separate generic replacement/notification primitives so the native fixture preserves CPython's exchange → notify → release-old-reference order. The API is restricted to the active VM; this is not cross-thread PyThreadState_SetAsyncExc support.
  • The native SelfInterruptingContextManager fixture and explicit root clearing on supported terminal thread/interpreter paths. Reusable enter/run scopes do not clear pending state.

Lifecycle and API review points

  • Terminal owners close even an empty slot; the closed state is the last VM field. Existing safe atomic-reference replacement is retained, without new locks or unsafe blocks.
  • Normal owner cleanup and matched GILState release clear in an active VM. Unmatched GILState TLS teardown, foreign-thread finalization, already-shutting-down slots, and unavailable slot TLS suppress Python callbacks while releasing this new root instead of attempting an unsafe attachment. This defensive policy is not a guarantee for arbitrary teardown misuse or all existing VM roots.
  • Adding Drop to Interpreter restricts moving its public global_state field out directly. In-repository uses were checked; external consumers were not. ThreadedVirtualMachine.vm is not a public external field.
  • Pending/closed flags are thread-owned Cell<bool> state on the non-Sync VM. A future cross-thread API needs a separate synchronization/wakeup design. There is still an extra plain flag load per opcode; no performance-neutrality claim is made.

Validation

Local validation used base c2bb450316db3ce3dc4ddc6bcde9f3059ee4e90c. Main advanced to 0072412e9523b37f7c7254c9ceb3468d5041c454 (#9495) during publication; that newer merge composition is left to PR CI, not claimed as locally retested.

The local build/runtime checks below cover original implementation commit 770484ecfd47e6b3af502cc72ce7819fa16e594e: checks preceded that metadata-only commit, and normal hooks left all 3,779 build-bound source files unchanged.

Review follow-up 33de790f50417c21c6a67d3a5a3a0661a53d86ba only changes the class attribute to #[pyclass(name, module = "_testcapi")]. The shared macro name resolver returns the same string from the Rust identifier as from the previous literal; registration and type metadata are unchanged by source inspection. Normal hooks passed. Local compilation/runtime were not repeated for that one-line follow-up; its own PR CI is tracked separately.

On the original implementation source tree:

  • Normal commit hooks passed; no hooks bypassed.
  • Offline/locked cargo check, strict cargo clippy -- -D warnings, and CLI build passed with threading,stdlib,stdio,importlib,host_env,doc,ucd_3_2_0 and actual VM opt-level 3.
  • A separate offline/locked check passed with the same feature list minus threading; the actual VM compiler invocation had no threading cfg.
  • The unchanged canonical InterruptDuringEnter.test_exit_called_after_interrupt ran once against the candidate and once against a fresh pinned CPython build containing the real _testinternalcapi fixture. Both ran exactly one test with zero failures, errors, skips, expected failures, or unexpected successes. Inputs were hash-bound, output matched, caches stayed empty, and owned process groups were cleaned up.

This was a bounded Linux single-method comparison, not full test_with, workspace/C-API suites, cross-platform runtime testing, a benchmark, or validation of every generic normalization/finalizer/TLS corner case. No baseline crash probes, OS-signal injection, or cross-thread stress tests were run.

There is file-level overlap with #9143 in _testinternalcapi.rs and VM registration-area files, but this does not duplicate its _testcapi registration work. #9407's C-API tp_new work remains separate. Neither owner's branch was modified.

AI assistance

Implementation, source analysis, independent agent review, guarded validation, and this description were assisted by OpenAI Codex (model version unavailable).

Deliver VM-owned pending exception classes at bytecode safepoints and clear their roots before terminal owner teardown. Add the native SelfInterruptingContextManager fixture and restore the canonical test_with regression via update_lib.

Assisted-by: OpenAI Codex: model version unavailable
@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] test: cpython/Lib/test/test_with.py

dependencies:

dependent tests: (no tests depend on with)

[ ] test: cpython/Lib/test/test_cmd_line.py (TODO: 12)

dependencies:

dependent tests: (no tests depend on cmd_line)

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

@codspeed

codspeed Bot commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1


Comparing youknowdot:python315-with-async-exception-20261031 (770484e) with main (0072412)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

const SPECIALIZATION_THRESHOLD: usize = 2;

#[pyattr]
#[pyclass(name = "SelfInterruptingContextManager", module = "_testcapi")]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[pyclass(name = "SelfInterruptingContextManager", module = "_testcapi")]
#[pyclass(module = "_testcapi")]

you don't need to repeat name

Apply the review request to avoid repeating SelfInterruptingContextManager. Keep the required bare name marker: the current pyclass macro resolves it to the identical Rust identifier.

Assisted-by: OpenAI Codex: model version unavailable
@youknowone
youknowone marked this pull request as ready for review October 31, 2026 23:16
@youknowone
youknowone merged commit cae6a38 into RustPython:main Oct 31, 2026
29 checks passed
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T23:26:34.265295Z 33de790 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 33de790f50

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Lib/test/test_with.py
Comment on lines +856 to +858
def test_exit_called_after_interrupt(self):
cm = SelfInterruptingContextManager()
self.assertFalse(cm.within())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Obtain a policy exception before restoring this test body

The root AGENTS.md states that “the only acceptable modifications to test files” are the specified expected-failure decorator and TODO-comment changes. Adding this test body and its import exceeds that restriction, and the guidance documents no exemption for restoring canonical CPython tests. Obtain an explicit upstream-restoration exemption or keep the test-file edits within the permitted forms.

AGENTS.md reference: AGENTS.md:L280-L282

Useful? React with 👍 / 👎.


/// Called only at bytecode safepoints, after pending signals and GC.
pub(crate) fn raise_async_exception(&self) -> PyResult<()> {
let Some(exception) = self.async_exception.take() else {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Prevent profiling callbacks from consuming the pending interrupt

With a Python callback such as sys.setprofile(lambda *args: None) installed, the successful native __enter__ call invokes its c_return profiling callback before returning to the with frame (protocol/callable.rs:119–123). The callback's first bytecode safepoint consumes the pending exception here, but _trace_event_inner discards profiling callback errors (protocol/callable.rs:409–419). Consequently, the KeyboardInterrupt disappears, profiling is disabled, and the restored test reaches “No exception raised.” Defer delivery during instrumentation callbacks or propagate the callback error so profiling cannot silently cancel the interrupt.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants