Activity

1. moreal

   added

   RFCRequest for commentsRequest for comments

   on Jul 15, 2025
2. youknowone commented on Jul 15, 2025

   

   youknowone

   on Jul 15, 2025

   Member

   More actions

   Recently @arihant2math wrote a helpful script
   Could you try this script? https://github.com/RustPython/RustPython/blob/main/scripts/fix_test.py
3. moreal commented on Jul 15, 2025

   

   moreal

   on Jul 15, 2025

   ContributorAuthor

   More actions

   Yeah, I saw that. While it's not good to have work stalled by pursuing perfectionism, this issue shows traces of that exact concern. Initially, I thought simply marking the failed tests locally would suffice, but then considered that tests might fail on other operating systems too, and felt we'd need to cover those cases for true automated operation. Of course, even then it wouldn't cover all scenarios, so I might be getting unnecessarily bogged down in minor details... 😢
4. youknowone commented on Jul 15, 2025

   

   youknowone

   on Jul 15, 2025

   Member

   More actions

   To be honest, I haven’t thought very deeply about this issue, so I believe others who have considered it more seriously are better positioned to discuss it.

   First, I haven't given much thought to the problem of tracking the base version of each file, because in most cases, git log Lib/.py gives a reasonably accurate answer. Of course, this approach has its downsides. it doesn't work well when dealing with multiple files, and it's difficult to visualize. However, in practice, it hasn’t significantly interfered with the workflow of updating necessary libraries.

   The major downside is that the update process still requires a substantial amount of manual effort. Ideally, we could specify a target Python version and run a script to update all files accordingly. But in many cases, replacing a Python file is not isolated. it’s tied to other Python files or even to updates in Rust code. Even if we assume that it’s possible to switch the target Python version with a single large patch, given RustPython’s current level of compatibility, we can’t know when such a patch would be mergeable. For example, while most of the codebase was compatible with Python 3.9–3.11, features related to async remained stuck at around the 3.6 level for a long time.

   Realistically, to carry out automation at a mergeable scale, we need automation that works per module and includes dependency handling. A reasonable level of automation for now might involve combining fix_test.py with GitHub Actions to enable automated processing across all platforms.

   As for avoiding having to manually mark each function with @Skip or @expectedfailure, I’m not exactly sure what the best solution would be. But based on my experience maintaining the project, I believe two conditions must be met.

   First, we must avoid patching Lib modules like unittest directly, because we update these libraries about once a year, and custom patches make that harder.
   Second, we need to be able to detect regressions immediately. We're not running tests just for statistical analysis. we're running them to know when actual regressions happen.

   So, having a centralized list somewhere that controls test expectations and editing that list to change unittest results would be a possible approach. It may be slightly inconvenient when writing patches, but I think it's a reasonable trade-off between ease of library updates and the cost of writing proper bugfix patches. If the upgrade script can add new expectedFailure functions to the existing list but cannot remove them, then it should be acceptable to review regressions manually.

   👍React with 👍2Reacted by Lee Dogeon and Mr. Walls
5. reactive-firewall commented on Jul 17, 2025

   

   reactive-firewall

   on Jul 17, 2025

   Last edited by reactive-firewall

   More actions

   Here’s a comprehensive summary of the work on #5974 and the related discussion and prototype, tailored for other developers interested in the project:

   ---

   Background & Motivation

   RustPython aims to provide a Python 3 interpreter written in Rust, maintaining a clean codebase and closely tracking Python 3.x compatibility. Updating the Python standard library (Lib/) from CPython in RustPython has been labor-intensive and error-prone, especially for test files. Key challenges include:

   • Manual copying and patching of files from CPython.
   • Difficulty tracking which CPython version each file is based on.
   • Labor-intensive manual addition of markers like @unittest.expectedFailure and # TODO: RUSTPYTHON.
   • Complicated merging, especially when CPython upstream changes.

   ---

   RFC & Discussion Highlights (Issue #5974)

   @moreal proposed a more automated, robust, and reproducible workflow for updating CPython standard library files in RustPython, @youknowone then proposed narrowing the scope to be particularly focused on the test suite. Major discussion points:

   Their Proposal

   • Automated Scripted Updates: Use GitHub Actions to automate copying, patching, and testing CPython files (with OS matrix testing).
   • Version Tracking: Improve traceability of which CPython version each file is based on.
   • Automated Test Outcome Reporting: Collect test results (success/failure/panic) in JSON across OSes, enabling better planning for what needs patching.
   • Automated Decorator Management: Use tools like LibCST for programmatic insertion/removal of decorators and comments, reducing manual labor.
   • Separation of Reference and Local Changes: Propose keeping original CPython files unchanged where possible, tracking RustPython-specific modifications separately.

   Community Feedback & My Comments

   • #issuecomment-3082110076: Clarified the need for a more sustainable integration process and outlined the challenges with manual work.
   • #issuecomment-3097588786 & #issuecomment-3097659973: Further elaborated on technical details; discussed test runner design, version hints, and workflow robustness.
   • #issuecomment-3104387523: Provided an example of how integration and testing could be automated.
   • #issuecomment-3105052901: Addressed edge cases and discussed how to preserve helpful comments or markers that aid development.
   • #issuecomment-3105642894: Summarized lessons learned and suggested future improvements.

   ---

   Experimental Implementation (Prototype)

   I've designed and prototyped a set of GitHub Actions and supporting workflow files that implement much of your proposal: Demo Results

   Key Components

   • Custom Actions

     • .github/actions/CI-5974-Fetch-CPython/action.yaml: Checks out a specified CPython version, pulls the desired files (using glob patterns), and prepares them for integration.
     • .github/actions/CI-5974-Fetch-RustPython/action.yaml: Checks out and builds RustPython at the requested commit/ref.
     • .github/actions/CI-5974-Integrate-CPython/action.yaml: Integrates selected CPython files into the RustPython tree, handling errors and warnings gracefully.

   • Workflow

     • .github/workflows/Check_Tests.yml: Orchestrates my experiment of full integration and testing cycle across OSes and Python versions. It:
       • Fetches RustPython and CPython source files via custom actions.
       • Copies reference files via custom actions, and attempts to run them using RustPython.
       • Tries auto-fixes for failing tests, reverts broken integrations, and tracks outcomes.
       • Runs tests both directly (if unittest.main() present) and via the modular test runner for more complex cases.
       • All testing is done in sub-processes and hung/deadlocked tests are timed-out.

   Technical Highlights

   • Matrix Testing: Runs across multiple OSes and Python versions, including experimental future releases.
   • Timeout Handling: Detects and mitigates hanging tests, which is crucial for CI robustness.
   • Outcome Recording: Tracks whether direct copies or auto-fixes succeed, fail, or are reverted, and provides clear CI output.
   • Non-destructive Integration: Attempts to integrate new reference files and falls back gracefully if integration fails.
   • Extensible Design: Modular actions and workflow steps allow future enhancements (e.g., more granular test selection, better version tracking).

   ---

   Lessons Learned & Next Steps

   • Automation Reduces Manual Labor: Most integration work can now be scriptable, dramatically improving efficiency and reliability.
   • Edge Cases Remain: Not all files are directly runnable, and some require special handling. The workflow should log and skip these gracefully.
   • Collaboration Needed: Further refinement can benefit from input from maintainers and contributors (especially around handling edge cases and integrating with existing workflows).

   ---

   Summary for Developers

   If you’re interested in improving how RustPython keeps its standard library and tests up-to-date with CPython:

   • Review the RFC and discussion in Issue #5974.
   • Examine the prototype diff for a working example of automated integration and testing.
   • Try the workflow locally or in CI—it’s designed to be modular and extensible.
   • Contribute ideas for handling edge cases, improving version tracking, or making the workflow more maintainable.

   This experiment demonstrates a concrete way forward for sustainable, automated updates of RustPython’s CPython-derived files, with clear opportunities for collaboration and improvement.

   ---

   TL;DR My Outdated and Historical Comments and Edits

   Regarding @youknowone's suggestion

   A reasonable level of automation for now might involve combining fix_test.py with GitHub Actions to enable automated processing across all platforms.

   any objection to me taking on this work to get my feet wet in the project. I'm very familiar with GHA and python's unittest module and actually discoverd rustpython when looking to expand my cross-python tests to other python implementations.

   Plus, I'm wanting more test-run diversity information to go off before responding to the rest of the topic as a whole. I'm curious if we could tree-build with the AST module for traversing the python and then somehow modify the resulting tree when adding the rust specific stuff.

   EDIT: TL;DR - My idea is more an experiment to blind test the testing workflow
   regarding what I was thinking here (for clarity):

   • Work on a fork (to keep prototyping CI usage down) but still tag this GHI (for tracking)

   • a bare-basic GHA workflow (triggered by any push to "*.py" files to start / manual trigger for testing)

   • implement a separate reusable action (concluding only with pass/canceled/skipped/neutral results at first) to run per input tuple of (runner OS image (so we can handle diff arch/versions too), implementation (e.g., cpython ref versus rustpython), git-ref/version, env_overrides, and other optional variables with defaults as they make sense to the implementation)

     • this will be called as a reusable @uses: rust-python-ci-min-tests (name is WIP) etc.
     • will allow common input/output usage later on
     • will handle fetching/bootstrapping the given source implementation
     • performing the actual test run
     • evaluating the result as pass/neutral (e.g. non-blocking result, but with test metrics for us humans)
     • handling overhead of multi-platform quirks of (windows, macOS and Ubuntu to start)

   • these can then be used to gain more insight for this discussion

   ---

   EDIT (Continued):

   [!IMPORTANT]
   Disclaimer: I am just a fellow user of rustpython and have no official association with the project or its maintainers. Any feedback or comments I provide are based on my personal experience and are offered AS-IS. I cannot guarantee their accuracy or effectiveness, and they should not be considered official support or advice. Please reach out to the maintainers for authoritative guidance.

   This may be know to some already, however, I'm including it here to start some kind of documentation on it for this broader topic.

   TDIL

   So first thing I discovered, is that pulling the rustpython repo from it's tip (e.g. main) will be problomatic, it is quite possible to just fail to compile before it can run a single cpython test. I'm not sure on the "best" way to solve this, but one way is to use a staging branch (e.g., when the normal CI/CD is able to build or some low-bar definition of 'semi-stable' automatically tag or merge to some staging branch, for these kinds of integration updates from cpython)

   Second, we'll want test file (e.g., per Lib/test/*.py file) level control over integrating from Cpython, which is totally do-able. Beyond that looking at the individual tests will be out of my experiment's scope for now.

   Third, this experiment will take me a long time (especially because I'm only working on it in my spare time) so the curious may take a look at the experiment over there

   Fourth, the experiment's first milestone will be to have an automation with a single successful test run of the fix_test.py script in CI (not happening this week)

   Thanks for reading this far 🙇 I hope this comment helps in some small way 🙊

   ---

   In short

   What if we store the built binary from main in cache, and pull it via the dedicated workflow to run the workflow that you're working on? That way your workflow will stay in scope of what matters, as it's redundant to see if the program compiles, we already have a process for that

   I totally agree that "it's redundant to see if the program compiles." Currently, I'm only doing the redundant rebuilds on my development branch while prototyping, as I've disabled the normal build workflows during my experiments. This is one of the implementation details I plan to explore further.

   Initial thoughts in more detail

   TL;DR: Long-term, using cached builds is ideal, but during development, it has limitations.

   There’s a case to be made for building and caching each commit where core Rust files change, and once for each target (like OS, Rust compiler version, and architecture). Each target version of CPython could then use the same cached RustPython for testing. For now, I’m simplifying this by targeting a single Rust compiler version, but once we have a stable release for RustPython, it will be important to track regressions. This is crucial for identifying issues when a new Rust compiler version causes a regression. Regression testing like this limits our use of cached builds since we would already know the cached build works.

   We still need to rebuild sometimes to test our changes, which could lead to unexpected regressions. These are the cases I was referring to when I mentioned it could be "problematic." I meant to highlight the dependency of CPython’s tests on stable build branches. Here’s a simplified code snippet to illustrate:

     rust-python-ci-min-tests:  # (name is WIP)
       needs: [rust-python-build-job]
       if: ${{ needs.rust-python-build-job.outcome == success() }} # syntax simplified for clarity
       # rest of job

   ---

   My suggested solution:

   I believe it would be effective to implement logic that inputs the build outcomes from existing jobs (as @ShaharNaveh correctly suggested) and dynamically rebuilds only when necessary. This means if a development branch (e.g., not main) fails to build, we could still rebuild from main (a known working branch) and cache the binary. We could then integrate the 'pure' Python source and CPython's test source for comprehensive regression testing.

   I plan to handle this by adding optional inputs to the reusable workflow I'm developing. These inputs would allow us to bypass caching (which I intend to have as the default) and force rebuilds when needed. This approach aims to future-proof the workflow before integrating it with the core project, following the open-closed design principle.

   I’m still figuring out how to differentiate the cache hash and whether to use the existing actions/cache (which is more maintainable but less flexible) or to pass builds via artifacts (actions/upload-artifact and actions/download-artifact) (which is more flexible but slightly more complex). I’m eager to learn more about the RustPython project and its requirements to make the best choice here.

   Hopefully this helps! 🙇 Thanks for reading this far.

   ---

   any objection to me taking on this work

   @reactive-firewall you can comment the word take and this issue will be assigned to you

   @ShaharNaveh, Thanks, that is good to know! However, I don't want to over-commit on this, I only intend on looking into the mentioned automation sub-task, leaving the rest alone:

   Namely I'm focused on:

   A reasonable level of automation for now might involve combining fix_test.py with GitHub Actions to enable automated processing across all platforms.

   Namely Ignoring:

   ... and the other is that it's difficult to easily determine which Python version each file is based on.
   ... 하나는 각각의 파일들이 현재 Python 몇 버전 기준 파일인지 쉽게 알기 어렵다는 점입니다.

   ---

   I initially tried to create a new GH issue with this ( #5974 ) as the parent issue but lack the necessary permissions; If this is somehow an issue for project tracking, I apologise for that, and someone with the necessary permissions may create the sub-issue and assign that sub-issue to me for tracking as needed.

   ---

   You could as a first step have them in a "do not touch" list, so you won't update them, whether it's the entire directory or specific file inside those problematic directories.

   Instead of doing everything in one go, treat it as a migration process where files who are not in the "do not touch" list will forever(?) be updated, it will require a lot of tinkering but imo that's the way to go, would love to hear what other people think

   Thank you for your suggestion regarding the "do not touch" list. I appreciate your insights, and I am currently experimenting with a similar approach, starting with an empty list. I found some heuristics from the CPython dev-docs that may be useful as well.

   @ShaharNaveh, it seems we share a similar mindset on the need for granular control over the tooling. I am focused on the research phase of my experiment (e.g. all that "tinkering" you mentioned). I'd also wish to hear what other people think. To update the discussion on my existing design, here’s a summary of my experimental code:

   AI analysis so-far

   GitHub Actions Workflow Analysis: CI-5974

   Overview

   • Name and Description: The workflow is named CI-5974 and is described as a continuous integration workflow for GHI 5974.
   • Trigger Events: The workflow is triggered on:
     • Push events to any branch (branches: ["**"]).
     • Push events for tags that start with "v" (tags: ["v*"]).

   Environment Variables

   The workflow defines several environment variables related to Python versions:

   • PYTHON_DEFAULT: The default Python version.
   • PYTHON_OLD_MIN: The oldest supported Python version.
   • PYTHON_OLD_EXTRA: Additional older Python versions for extra coverage.
   • PYTHON_EXPERIMENTAL: Future Python versions for testing.
   • TEST_MATCH_PATTERN: A pattern to match test files in the Lib/test directory.

   Jobs

   Job: TEST-5974

   • Permissions: The job has specific permissions set for actions, contents, statuses, and pull requests, while denying access to packages and security events.
   • Conditions: The job will not run if it is cancelled.
   • Execution Environment: It runs on a matrix of operating systems (ubuntu-latest, macos-14, macos-15, windows-latest) and Python versions defined in the environment variables.
   • Timeout: The job has a timeout of 30 minutes.
   • Continue on Error: It can continue on error if the matrix variable experimental is set to true.

   Strategy

   • Matrix Strategy: The job uses a matrix strategy to run tests across different OS and Python versions, including experimental configurations.
   • Fail-Fast: The fail-fast option is set to false, meaning that all jobs will run even if one fails.

   Steps

   1. Checkout RustPython Repository: Uses the actions/checkout action to check out the RustPython repository.
   2. Pre-Test Build Check: Runs a build check to ensure the RustPython can be built successfully.
   3. Fetch CPython: Checks out the CPython repository to fetch the reference implementation.
   4. Enumerate Reference Lib Files: Lists the test files in the CPython repository and outputs them for further processing.
   5. Show CPython License: Displays the license of the CPython repository.
   6. Integrate CPython Test Files: Copies the reference implementation tests from CPython to RustPython.
   7. Bootstrap Python: Sets up the Python environment for testing.
   8. Try Smoke Testing: Runs smoke tests on the integrated test files, handling potential failures and attempting to auto-fix issues.
   9. Post-Clean: A cleanup step that always runs, ensuring that the workflow does not break on regression.

   Key Features

   • Error Handling: The workflow includes various error handling mechanisms, such as warnings for missing files and conditions to skip tests that cannot be run directly.
   • Experimental Features: The use of experimental Python versions indicates that the workflow is designed to test new features or changes in Python.
   • Modular Steps: Each step is modular, allowing for easier debugging and maintenance. The use of shell scripts within steps provides flexibility in executing commands.

   TODOs and Future Work

   The workflow contains several TODO comments indicating areas for improvement or features that are still in progress, such as:

   • Moving certain steps to separate actions for better organization.
   • Implementing more robust filtering for reference library usage.
   • Addressing specific issues with Windows timeouts during testing.

   Conclusion

   The CI-5974 workflow is a comprehensive CI setup for testing a project that integrates RustPython with CPython. It is well-structured, utilizing matrix builds to cover multiple environments and Python versions, while also incorporating error handling and modular steps for clarity and maintainability. The presence of TODOs suggests ongoing development and refinement of the workflow.

   ---

   [!NOTE]
   As my code is merely experimental, I recognize that parts may need refactored to be of any use to RustPython, especially the duplication of what is already in other CI workflows. I appreciate any early feedback! 🙇

   Regarding known files and directories that currently break the test suite if updated:

   • Lib/test/libregtest
   • Lib/unittest/
   • Lib/test/support
   • Lib/importlib
   • probably more 😁

   This insight is valuable! Additionally, could anyone clarify why Lib/test/libregtest and Lib/test/support cause issues? A reference to a relevant GHI would be helpful.

   There are indeed more 🙊 e.g., test_builtins.py will hang

   ---

   Feel free to try and open a GH issue about any errors that you encounter

   Understood, I'm only dealing with {RustPython,cpython}/Lib/test/*.py at the moment, as the RustPython docs mentioned already porting unittest over, and importlib makes sense with the C-lang and Rust backends potential for providing different binary interfaces.

   Second, we need to be able to detect regressions immediately. We're not running tests just for statistical analysis. we're running them to know when actual regressions happen.

   I'll continue with the blackbox testing approach for my experiment regarding {RustPython,cpython}/Lib/test/{support,libregtest} and open/update relevent issues if they actually error, inline with the rationale previously mentioned. This will allow me to build the comprehensive list empirically as well.

   EDIT: I've hidden my previous comments to help manage the long-length of the discussion.
6. reactive-firewall

   added 5 commits that reference this issue on Jul 17, 2025

   Create Check_Tests.yml

   ...

   8f5127c

   Prototyping 5974

   ...

   944b343

   First just-integrate Cpython into Rustpython Automation test for Rust…

   ...

   071496e

   Second-Attempt for GHI RustPython#5974

   ...

   d68673c

   Remove ulimit and re-test

   ...

   cd911c3
7. ShaharNaveh commented on Jul 18, 2025

   

   ShaharNaveh

   on Jul 18, 2025

   Contributor

   More actions

   any objection to me taking on this work

   @reactive-firewall you can comment the word take and this issue will be assigned to you

   👀React with 👀1Reacted by Mr. Walls
8. ShaharNaveh commented on Jul 18, 2025

   

   ShaharNaveh

   on Jul 18, 2025

   Contributor

   More actions

   So first thing I discovered, is that pulling the rustpython repo from it's tip (e.g. main) will be problomatic, it is quite possible to just fail to compile before it can run a single cpython test.

   What if we store the built binary from main in cache, and pull it via the dedicated workflow to run the workflow that you're working on? That way your workflow will stay in scope of what matters, as it's redundant to see if the program compiles, we already have a process for that

   👍React with 👍1Reacted by Mr. Walls
9. reactive-firewall commented on Jul 21, 2025

   

   reactive-firewall

   on Jul 21, 2025 · Hidden as resolved

   show commentMore actions
10. reactive-firewall commented on Jul 21, 2025

    

    reactive-firewall

    on Jul 21, 2025 · Hidden as off-topic

    show commentMore actions
11. ShaharNaveh commented on Jul 21, 2025

    

    ShaharNaveh

    on Jul 21, 2025

    Contributor

    More actions

    I plan to handle this by adding optional inputs to the reusable workflow I'm developing. These inputs would allow us to bypass caching (which I intend to have as the default) and force rebuilds when needed. This approach aims to future-proof the workflow before integrating it with the core project, following the open-closed design principle.

    Sounds good imo.

    I’m still figuring out how to differentiate the cache hash and whether to use the existing actions/cache (which is more maintainable but less flexible) or to pass builds via artifacts (actions/upload-artifact and actions/download-artifact) (which is more flexible but slightly more complex). I’m eager to learn more about the RustPython project and its requirements to make the best choice here.

    The general rule of thumb that I'm following is:

    • artifacts: share assets between jobs in the same workflow.
    • cache: share assets between different workflows & between workflow runs.

    ---

    For now if it simply things, assume that for the majority of files, if updated will not break RustPython. there are some known files & directories that are known to currently break the test suite if updated such as:

    • Lib/test/libregtest
    • Lib/unittest/
    • Lib/test/support
    • Lib/importlib
    • probably more 😁

    You could as a first step have them in a "do not touch" list, so you won't update them, whether it's the entire directory or specific file inside those problematic directories.

    Instead of doing everything in one go, treat it as a migration process where files who are not in the "do not touch" list will forever(?) be updated, it will require a lot of tinkering but imo that's the way to go, would love to hear what other people think
12. 29 remaining items

    Load more
13. reactive-firewall

    added 11 commits that reference this issue on Sep 28, 2025

    DEBUG sparse checkout for actions

    ...

    c5b556e

    [DEMO] Protoytpe from GHI RustPython#5974 with rpau from PR RustPytho…

    ...

    6534d1c

    [DEBUG] possible workaround for braindead windows runners

    ...

    cd3dc0a

    [DEBUG] Minor Regression fix for double entries. Oops.

    ...

    42254c7

    [STYLE] Cleaning up the report for CI runs a bit.

    ...

    8b8398b

    [DEBUG] Adjust output path to upload at end

    ...

    e79c13b

    [PATCH] workaround for elusive patched files

    ...

    b211032

    Oops! forgot to mkdir in setup phase re-try test.

    ...

    e20f6c5

    Test for new Lib_updater.py script in integration phase (for Lib/test…

    ...

    3dd549f

    [PATCH] Template Regression check in CI/CD

    ...

    64728df

    [PATCH] Cleanup from automated review in PR RustPython#6176

    ...

    1960425
14. youknowone commented on Jan 18, 2026

    

    youknowone

    on Jan 18, 2026

    Member

    More actions

    @moreal we now have scripts/lib_updater.py and scripts/auto_mark_test.py for each file. This is not full-automation but helpful for each file. Could the issue to be updated to current status? We now need to regard higher level automation.

    👍React with 👍3Reacted by Lee Dogeon, Jiseok CHOI and Mr. Walls
15. moreal commented on Jan 18, 2026

    

    moreal

    on Jan 18, 2026

    ContributorAuthor

    More actions

    @moreal we now have scripts/lib_updater.py and scripts/auto_mark_test.py for each file. This is not full-automation but helpful for each file. Could the issue to be updated to current status? We now need to regard higher level automation.

    Okay, I'll update until tomorrow and let you know by mention. 🙆🏻

    ❤️React with ❤️3Reacted by Jeong, YunWon, Jiseok CHOI and Mr. Walls
16. moreal commented on Jan 20, 2026

    

    moreal

    on Jan 20, 2026

    ContributorAuthor

    More actions

    Okay, I'll update until tomorrow and let you know by mention. 🙆🏻

    @youknowone I'm sorry to be late. I've finished updating it now!

    🚀React with 🚀1Reacted by Mr. Walls
17. reactive-firewall

    added a commit that references this issue on Feb 11, 2026

    [CI] Add primitive json format of results as uploaded artifact

    ...

    d39c6ec