pypy support in pythonnet

cpyext is mature enough (numpy, cython, scipy, pandas soon) to support pythonnet in pypy. In fact pypy used to have a branch which ran on top of CLR (similar to IronPython), but this branch is not maintained.

The use of pypy in .NET is quite popular request:

http://stackoverflow.com/questions/6084697/is-it-possible-to-embed-pypy-into-a-net-application

The only limitation of pypy is 32-bit on Windows and poor support for Python 3+, although the latter is improving with funding from Mozilla.

Also this might be quite nice collaboration with pypy developers due to their level of expertise in python C-API and cffi.

https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/

src/monoclr/pynetinit.c: In function ‘main_thread_handler’:
src/monoclr/pynetinit.c:106:5: error: unknown type name ‘Dl_info’
src/monoclr/pynetinit.c:107:5: warning: implicit declaration of function ‘dladdr’ [-Wimplicit-function-declaration]
src/monoclr/pynetinit.c:107:22: error: ‘Py_Initialize’ undeclared (first use in this function)
src/monoclr/pynetinit.c:107:22: note: each undeclared identifier is reported only once for each function it appears in
src/monoclr/pynetinit.c:109:23: error: request for member ‘dli_fname’ in something not a structure or union
src/monoclr/pynetinit.c:110:29: error: request for member ‘dli_fname’ in something not a structure or union

above is the traceback from building on pypi/travis

I tried on pypy with pythonnet on Win10. It builds fine, but then fails to find clr during import.

C:\Python\pypy2-v5.6.0-win32>bin\pip install pythonnet
Collecting pythonnet
  Downloading pythonnet-2.2.2.tar.gz (1.5MB)
    100% |################################| 1.5MB 480kB/s
Building wheels for collected packages: pythonnet
  Running setup.py bdist_wheel for pythonnet ... done
  Stored in directory: C:\Users\denis\AppData\Local\pip\Cache\wheels\14\da\65\1042e21167805ad08c6319a87c9cd6b2e6e97ff828aa37eb3d
Successfully built pythonnet
Installing collected packages: pythonnet
Successfully installed pythonnet-2.2.2

C:\Python\pypy2-v5.6.0-win32>pypy
Python 2.7.12 (aff251e54385, Nov 09 2016, 17:54:55)
[PyPy 5.6.0 with MSC v.1500 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>> import clr
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named clr
>>>>

interesting. you tried building using the debug configuration? Im curious if the clrmodule is the one failing to find python.runtime.dll or if its not even finding the clrmodule.

I've don't know if the error is the same or not in either case.

can I pass debug flag to pip/setuptools (not from VS)?

its on my todo list to add a cli option. currently you need to update setup.py and it will build on debug

still the same error, here is the full output from the build:

buildoutput.txt

http://stackoverflow.com/questions/42152122/pypy-import-clr-fails-on-windows

This may have something to do w it.
https://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html

@vmuriart

@vmuriart yes, that's possible that we are using some unsupported C-API features in cpyext. But it fails silently. I'm trying with nightly pypy builds:

http://buildbot.pypy.org/nightly/trunk/

https://bitbucket.org/pypy/pypy/issues/2474/import-clr-fails-on-windows

same issue on nightly pypy build:

c:\Python\pypy
λ bin\pip2 install pythonnet
Collecting pythonnet
  Using cached pythonnet-2.2.2.tar.gz
Building wheels for collected packages: pythonnet
  Running setup.py bdist_wheel for pythonnet ... done
  Stored in directory: C:\Users\denis\AppData\Local\pip\Cache\wheels\14\da\65\1042e21167805ad08c6319a87c9cd6b2e6e97ff828aa37eb3d
Successfully built pythonnet
Installing collected packages: pythonnet
Successfully installed pythonnet-2.2.2

c:\Python\pypy
λ pypy
Python 2.7.13 (4a7ae8e412f5, Feb 01 2017, 22:52:31)
[PyPy 5.7.0-alpha0 with MSC v.1500 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``poststructuralism etc.''
>>>> import clr
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named clr

I see that it creates and installs clr.pyd even on PyPy. This file name is wrong: it should be called clr.pypy-41.pyd. The normal distutils creates files of the correct name, but I guess a custom setup.py could hard-code clr.pyd.

After renaming it manually, it fails differently, which is probably another issue.

@arigo

@arigo thanks, so now clr is looking for python27.dll, which I guess is libpypy-c.dll:

>>>> import clr

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.DllNotFoundException: Unable to load DLL 'python27': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
   at Python.Runtime.Runtime.Py_IsInitialized()
   at Python.Runtime.Runtime.Initialize()
   at Python.Runtime.PythonEngine.Initialize(IEnumerable`1 args)
   at Python.Runtime.PythonEngine.Initialize()
   at Python.Runtime.PythonEngine.InitExt()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at clrModule.initclr()

Then I created a "soft" symlink mklink Python27.dll libpypy-c.dll, but now clr cannot find Py_IsInitialized from Python C-API in libpypy-c.dll:

Python 2.7.12 (aff251e54385, Nov 09 2016, 17:54:55)
[PyPy 5.6.0 with MSC v.1500 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>> import clr

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.EntryPointNotFoundException: Unable to find an entry point named 'Py_IsInitialized' in DLL 'python27'.
   at Python.Runtime.Runtime.Py_IsInitialized()
   at Python.Runtime.Runtime.Initialize()
   at Python.Runtime.PythonEngine.Initialize(IEnumerable`1 args)
   at Python.Runtime.PythonEngine.Initialize()
   at Python.Runtime.PythonEngine.InitExt()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at clrModule.initclr()

the loaded libraries look like expected:

libpypy-c.dll exposes the symbols under names PyPy_Xxx instead of Py_Xxx, to avoid conflicts in other situations. The normal header Python.h distributed with PyPy contains #define to do the renaming transparently. It seems that the CLR-compiled Python.Runtime.dll looks up the original names anyway, probably because it was not compiled with Python.h but instead with its own headers (rewritten in .NET maybe?).

@arigo

I have some progress on pypy debugging with pythonnet. The issue right now is setting tp_flags which crashes PyType_Ready:

https://bitbucket.org/pypy/pypy/issues/2486/pypytype_ready-with-tp_flags-crashes-pypy

some issues were resolved in pypy by @arigo, which enable parsing pyheaptypeobject without hacking the source:

https://bitbucket.org/pypy/pypy/issues/2484/pyheaptypeobject-difference-with-cpython

https://bitbucket.org/pypy/pypy/issues/2483/clang-complains-about-parsing-pypy-headers

https://bitbucket.org/pypy/pypy/issues/2478/sysconfigget_config_var-includepy-empty-on

The only difference in pyheaptypeobject is still ob_pypy_link like I pointed out above by using diff.

some progress in these issues:

https://bitbucket.org/pypy/pypy/issues/2486/pypytype_ready-with-tp_flags-crashes-pypy
https://bitbucket.org/pypy/pypy/issues/2504/pypy-pythonnet-call-fails-on-pypyclass_new

I can actually do import clr with pypy, but it can't do anything useful right now :)

One more issue to track in PyPy cpyext:

https://bitbucket.org/pypy/pypy/issues/2447/cpyext-_pyobject_getdictptr-is

any hope ?

I'm personally waiting for hpy to mature before working on this myself. If anyone wants to take another shot at cpyext (which should be really easy now!), I'll gladly help out.

Conda-forge recently started the support for PyPy on Windows and started a migration.

As I can read here I can close this Merge Request for now?

Yes. There is no support for Pypy in Python.NET.

Can it be achieved? I also need to use pythonnet in pypy now.
Python 3.10.12 (af44d0b8114cb82c40a07bb9ee9c1ca8a1b3688c, Jun 15 2023, 15:42:22)
[PyPy 7.3.12 with MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import clr
Failed to initialize pythonnet: System.TypeInitializationException: “Delegates”的类型初始值设定项引发异常。 ---> Pytho
n.Runtime.BadPythonDllException: Runtime.PythonDLL was not set or does not point to a supported Python runtime DLL. See
https://github.com/pythonnet/pythonnet#embedding-python-in-net ---> System.MissingMethodException: Failed to load symbol
Py_IncRef. ---> System.ComponentModel.Win32Exception: 找不到指定的程序。
--- 内部异常堆栈跟踪的结尾 ---
在 Python.Runtime.Platform.WindowsLoader.GetFunction(IntPtr hModule, String procedureName)
在 Python.Runtime.Runtime.Delegates.GetFunctionByName(String functionName, IntPtr libraryHandle)
--- 内部异常堆栈跟踪的结尾 ---
在 Python.Runtime.Runtime.Delegates.GetFunctionByName(String functionName, IntPtr libraryHandle)
在 Python.Runtime.Runtime.Delegates..cctor()
--- 内部异常堆栈跟踪的结尾 ---
在 Python.Runtime.Runtime.Delegates.get_PyGILState_Ensure()
在 Python.Runtime.Runtime.PyGILState_Ensure()
在 Python.Runtime.Py.GIL()
在 Python.Runtime.Loader.Initialize(IntPtr data, Int32 size)
在 Python.Runtime.Runtime.Delegates.get_PyGILState_Ensure()
在 Python.Runtime.Runtime.PyGILState_Ensure()
在 Python.Runtime.Py.GIL()
在 Python.Runtime.Loader.Initialize(IntPtr data, Int32 size)Traceback (most recent call last):
File "", line 1, in
File "D:\pypy3.10\lib\site-packages\clr.py", line 6, in
load()
File "D:\pypy3.10\lib\site-packages\pythonnet_init_.py", line 146, in load
raise RuntimeError("Failed to initialize Python.Runtime.dll")
RuntimeError: Failed to initialize Python.Runtime.dll

I also would like to see this. Has there been any progress?

It seems like there's no way.

I think the whole approach is doomed for PyPy or any non-CPython interpreter. A way forward might be to use CFFI embeddings, like mentioned above: you can build with CFFI a DLL that basically exports whatever (C-level) API you like, and then you can write C# code to call functions in that DLL, install callbacks, and so on. You don't get any of the pythonnet-like features, though; only a C API. Depending on your goal, it might be enough.

Alternatively, someone could then write a C API that gives Python some generic control over the .NET objects, with logic both on the C# and the Python side to give generic wrappers of C# objects in Python and/or vice-versa, using runtime introspection on both sides.

I think the most important reason for choosing Pypy is to consider runtime performance, but currently I think it may not be necessary because Python 3.13 has already removed GIL, so I may not prioritize choosing Pypy.
Even so, as some alternatives, the Mojo programming language may perform better than Pypy.
I have now shifted my energy to Rust, and many features are not slower to implement than Python.

Navigation Menu

Repository navigation

pypy support in pythonnet #330

19 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pypy support in pythonnet #330

Description

Activity

den-run-ai commented on Jan 23, 2017

vmuriart commented on Feb 9, 2017

den-run-ai commented on Feb 10, 2017

vmuriart commented on Feb 10, 2017

den-run-ai commented on Feb 10, 2017

vmuriart commented on Feb 10, 2017

den-run-ai commented on Feb 10, 2017

den-run-ai commented on Feb 10, 2017

vmuriart commented on Feb 10, 2017

den-run-ai commented on Feb 10, 2017

den-run-ai commented on Feb 10, 2017

den-run-ai commented on Feb 10, 2017

arigo commented on Feb 10, 2017

den-run-ai commented on Feb 10, 2017

arigo commented on Feb 10, 2017

19 remaining items

den-run-ai commented on Feb 26, 2017

den-run-ai commented on Mar 19, 2017

den-run-ai commented on Mar 20, 2017

stonebig commented on Jun 12, 2021

filmor commented on Jun 12, 2021

m-rossi commented on Jul 3, 2021

filmor commented on Jul 3, 2021

mzdk100 commented on Sep 16, 2023

ethindp commented on Feb 16, 2024

mzdk100 commented on Feb 16, 2024

arigo commented on Feb 17, 2024

mzdk100 commented on Feb 17, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions