Repository navigation
Interaction with PyO3 #3016
Description
Hi,
I'm a maintainer of PyO3 and have been vaguely aware of the RustPython project for some time. Unfortunately I'm woefully ill-informed as to your current progress and how much work is needed for RustPython to become a drop-in replacement for CPython. At the moment there's no collaboration between our projects (as far as I'm aware), however it feels like there could be. I wanted to reach out to welcome communication from you folks, if it's of interest.
PyO3's primary use case is to create Python extension modules in Rust (currently supporting CPython and PyPy). We also support embedding a Python interpreter in Rust (usually using dynamic linking to libpython, as statically linking the CPython interpreter is very awkward). Also the PyOxidizer project has recently been looking at PyO3 as a way to manage its own building and configuration, which is another place where using RustPython might be interesting (it could potentially help PyOxidizer avoid needing to build its own copy of libpython to link with).
Mostly I just wanted to reach out to see what you think the interaction between RustPython and PyO3 could be. For example, it might be really interesting to build a feature into PyO3 to embed a RustPython interpreter directly as a crate dependency.
We could also add support to build extension modules for a RustPython interpreter, which might range from easy to hard depending on how much of the CPython C API you support (or plan to support).
Finally, I think there's always potentially interesting things to learn from each other about ways to represent the GIL and other Python concepts in Rust - I think we do a reasonable job in PyO3, however there's definitely things we could do better, and IMO it's beneficial to the whole ecosystem if we share what we've learned.
Really, this issue doesn't contain anything concrete other than an invitation to open channels - feel free to ping me if you ever want my opinion on anything, and if you'd like us to add support for RustPython in any of the forms above (or perhaps ways I haven't considered yet) then I'd be interested to hear your opinions on what you think makes sense too.
Feel free to close this issue if it's not of interest right now.
👋
Activity
ShadowJonathan commentedon Sep 5, 2021
Would love to be able to just "turn on" rustpython as a dependency in pyo3 freestanding and run all python code in safe rust ❤️
youknowone commentedon Sep 5, 2021
welcome :) as a user of PyO3, I always hope the future like that.
For now, the only extension RustPython allows is adding
rustpython-vmas a dependency and loading custom module written in Rust from the project. It sounds similar like "embed a RustPython interpreter directly" as you said, but it will be very different to how PyO3 currently handle CPython or PyPy. Maybe wrapping RustPython objects for embedding will be approachable goal because rustpython-vm is exposing lots of python.h features.We are following up compatibility quickly and now it looks working great for basic use cases, but RustPython is yet far from drop-in-replacement in some reasons. One of the major thing for PyO3 will be C API support which we don't have yet. I don't know anyone realistically investigate how to do it, but I think we will implement hpy since CPython looks like to finally support it.
davidhewitt commentedon Sep 5, 2021
Ah great, interesting!
It sounds like there's definitely interest in coordinating between our projects, though of course this will probably be a slow process over time.
There's actually been a longstanding issue to explore HPy support for PyO3 - PyO3/pyo3#1113.
I think the main reason we haven't explored it is it's not clear how much effort is needed for users to be able to configure a build with HPy correctly. If this becomes the API which RustPython supports, then that becomes a good reason for us to understand and support it in PyO3!
coolreader18 commentedon Sep 5, 2021
Ooh, ok, so I've been thinking for a pretty long time about what the best way to allow subclassing in RustPython would be - currently, there's a 1:1 correspondence between payload type (marked by
TypeId) and object model type (type(x),PyObjectRef.class()). Subclasses ofobjectcan have a different payload type thanPyBaseObject, but stuff breaks if any subclasses of that use a different payload (basically cause we never do.payload::<PyBaseObject>(), but methods on custom classes do try to access the payload of their objects which only identify payloads byTypeId).One thing I've been inspired by when thinking about how to solve this is PyO3's model of subclassing, with
PyTypeInfoandPyClassInitializerand whatnot. I think it'd be really cool if we could somehow extract that and make it generic, and share PyO3's solution to recreating the Python object model in Rust between both PyO3 and RustPython.davidhewitt commentedon Sep 5, 2021
Oh cool! Note that I actually think there's long-term challenges with
PyTypeInfoin particular, because it assumes a global type object instantiation which is not really compatible with PEP 489.There's also some very interesting questions about python class inheritance - at the moment we have a
PyCellprimitive which handles both mutability and inheritence structure. However it has challenges, e.g. PyO3/pyo3#1068, PyO3/pyo3#1637 (comment), and PyO3/pyo3#991I'm expecting eventually to split out the PyClass layout / subclassing into a
PyClassObjectstructure, which will be a fairly general representation of a Python object with a data segment which originates from Rust. That can almost definitely be a general-use thing which RustPython might also be interested in.One gotcha with re-using this might be that it will probably depend heavily on definitions in
pyo3::ffi, and I'm not sure exactly how we'd untangle those for RustPython...marcpabst commentedon Jan 31, 2024
Sorry for reviving this, but is there any progress on this? I think it would be quite the killer feature!
For reference, I'm desinging a Rust library that compiles to desktop and browser/WASM, but also offers Python bindings via PyO3. It would be amazing if I could just ship RustPython so the same (altough somehat limited) Python scripts could run in the browser, calling into the same PyO3 bidings!
davidhewitt commentedon Feb 2, 2024
I haven't been able to put any time into this from the PyO3 side yet.
I think the most straightforward solution for the ecosystem would be for RustPython to support the C API. Then PyO3 can be configured to build with it without significant changes, and RustPython would be able to run a lot more Python software too.
youknowone commentedon Feb 9, 2024
I wish we could go more on this issue, but we are still far from C API support.
bschoenmaeckers commentedon Apr 10, 2026
I've been experimenting with a CPython api implementation for RustPython in #7562. Since recent Limited API (ABI3) versions most hurdles are gone (except for the static exported PyTypeObject's). I think we can implement the CPython Limited API without leaking to much implementation details.
smarcd commentedon Apr 18, 2026
Take a look at PyO3/pyo3#5986