PythonEngine.Shutdown() / Runtime.Shutdown() slowness

Environment

Pythonnet version: 3.0.1
Python version: 3.9.12
Operating System: Windows or Linux
.NET Runtime: net6.0

Details

Describe what you were trying to get done.

We are embedding Python in a .NET app using pythonnet.

The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.

Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().

Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.

Minimal, Complete, and Verifiable example

using Python.Runtime;

namespace Test;

static class Program
{
    static void Main()
    {
        var mem = new List<Array>();
        Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
        PythonEngine.Initialize();
        var threadState = PythonEngine.BeginAllowThreads();

        using (Py.GIL())
        {
            using (var scope = Py.CreateScope())
            {
                scope.Set("a", 1);
                scope.Set("b", 2);
                scope.Exec("result = a + b");
                var result = scope.Get("result").As<int>();
                Console.WriteLine($"Sum = {result}");
            }
        }

        // allocate dotnet-managed memory unrelated to pythonnet
        var rnd = new Random();
        for (var i = 0; i < 100_000_000; ++i)
        {
            mem.Add(new byte[rnd.Next(1, 10)]);
        }

        PythonEngine.EndAllowThreads(threadState);
        Console.WriteLine("Shutdown start");
        var start = DateTime.Now;
        PythonEngine.Shutdown();
        var diff = DateTime.Now - start;
        Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
    }
}

Typical output:

Sum = 3
Shutdown start
Shutdown stop 61.6796983

@lostmsu

This would probably be helped by handling #1781.

If we are finally shutting down, we don't actually have to stash anything, so we also shouldn't try to clean up garbage, right @lostmsu @BadSingleton ?

@filmor

Any updates on this @filmor ?

Apologies for the late reply. Yes, if we're shutting down for the last time, then there is no need to stash anything. If I recall correctly, it used to be that way before when there was still multiple modes. But I'm not sure about garbage collection, I've had some issues in the past (crashing on shutdown?) if it wasn't run correctly.

@lostmsu

@lostmsu @BadSingleton Now that issues like #2107 show up because BinaryFormatter.Serialize has been hard-deprecated in .NET 7, we need to do somethhing. How about making Shutdown() behave like it used to (no stashing) and adding a new ShutdownAllowRestart or something like that?

@filmor

@filmor @BadSingleton do we even need to support restart after shutdown on .NET 7?

[...] do we even need to support restart after shutdown on .NET 7?

I've been out of the .NET loop for a few months, why wouldn't we need to support it?

Otherwise, I don't think it has to be BinaryFormatter, as long as the behaviour is the same(?)

Unity is moving to .NET 7 next year supposedly (in large part to support Span). So, it would be best if it were still supported.
https://blog.unity.com/technology/unity-and-net-whats-next

For the original comment: do we actually need to call Shutdown on shutdown? If not, that would save a lot of time.

@benoithudson

@benoithudson what is Unity on now? I thought it relies on AppDomain, and AppDomain is not supported in .NET Core/6+. Without AppDomain, I do not think one needs to call Shutdown at all. What should be done instead perhaps is ability to reload individual types.

Now that issues like #2107 show up because BinaryFormatter.Serialize has been hard-deprecated in .NET 7, we need to do somethhing.

Hi, anyway I can work around this issue for now?

@dclaudeone11

I faced the same problem (Pythonnet 3.0.1, .NET Framework 4.7.2). In my app I have a window with the python code editor. I call PythonEngine.Initialize() when the user opens it, and the PythonEngine.Shotdown() when closing the window.

After executing simple scripts, the window closes with a small but noticeable delay. When executing scripts that create more complex objects, for example, matplayotlib figures, the delay increases to tens or even hundreds of seconds. I can call PythonEngine.Shotdown() not when closing the editor window, but when closing the application. But for now this only shifts the problem and does not solve it.

Unfortunately, I wasn’t able to find the official position on this subject in the documentation. The fact that many people use Pythonnet suggest that there is a way of bypass, because without it the issue looks serious. @dclaudeone11 Did you manage to find a temporary solution?

Without AppDomain, I do not think one needs to call Shutdown at all

@lostmsu First of all, thank you and other contributors for such a useful library.
Could you please explain what exactly is considered as a right way to stop Pythonnet (or app that uses it) for today? If I do not call PythonEngine.Shotdown() at all, after closing the main window of the program, a working process remains, which is obviously unacceptable. Given the popularity of this thread in the last days, it seems that many more people would be grateful for your official commentary, not only me.

@AlexGilmor

@AlexGilmor If you want to shut down Python, use Shutdown. The slowness is due to to multiple garbage collection rounds and (potentially contributing, but not the only factor) the runtime stashing that was introduced to allow for AppDomain switching.

@nlogozzo Since you are not actually using the stashing, you can try setting a no-op formatter:

public class NoopFormatter : IFormatter {
  public object Deserialize(Stream s) => throw new NotImplementedException();
  public void Serialize(Stream s, object o) {}

  public SerializationBinder Binder { get; set; }
  public StreamingContext Context { get; set; }
  public ISurrogateSelector SurrogateSelector { get; set; }
}

Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);

@nlogozzo

@nlogozzo Since you are not actually using the stashing, you can try setting a no-op formatter:

That worked...thanks!

Environment

Pythonnet version: 3.0.1
Python version: 3.9.12
Operating System: Windows or Linux
.NET Runtime: net6.0

Details

Describe what you were trying to get done.

We are embedding Python in a .NET app using pythonnet.

The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.

Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().

Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.

Minimal, Complete, and Verifiable example

using Python.Runtime;

namespace Test;

static class Program
{
    static void Main()
    {
        var mem = new List<Array>();
        Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
        PythonEngine.Initialize();
        var threadState = PythonEngine.BeginAllowThreads();

        using (Py.GIL())
        {
            using (var scope = Py.CreateScope())
            {
                scope.Set("a", 1);
                scope.Set("b", 2);
                scope.Exec("result = a + b");
                var result = scope.Get("result").As<int>();
                Console.WriteLine($"Sum = {result}");
            }
        }

        // allocate dotnet-managed memory unrelated to pythonnet
        var rnd = new Random();
        for (var i = 0; i < 100_000_000; ++i)
        {
            mem.Add(new byte[rnd.Next(1, 10)]);
        }

        PythonEngine.EndAllowThreads(threadState);
        Console.WriteLine("Shutdown start");
        var start = DateTime.Now;
        PythonEngine.Shutdown();
        var diff = DateTime.Now - start;
        Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
    }
}

Typical output:

Sum = 3
Shutdown start
Shutdown stop 61.6796983

Your issue does not occur when the project is ran in .NET 7 using a Release configuration. When ran in Debug configuration, the shutdown does indeed slow down significantly - from 0.08 seconds up to 63 seconds.

public class NoopFormatter : IFormatter {
public object Deserialize(Stream s) => throw new NotImplementedException();
public void Serialize(Stream s, object o) {}

public SerializationBinder Binder { get; set; }
public StreamingContext Context { get; set; }
public ISurrogateSelector SurrogateSelector { get; set; }
}

Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);

This does not fix the issue when the code is ran in Debug configuration under .NET 7.

v3.0.2 same question, NoopFormatter has no effact

Navigation Menu

Repository navigation

PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008

Environment

Details

Environment

Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008

Description

Environment

Details

Activity

filmor commented on Nov 22, 2022

fischja commented on Feb 16, 2023

BadSingleton commented on Feb 17, 2023

filmor commented on Feb 20, 2023

lostmsu commented on Feb 20, 2023

BadSingleton commented on Feb 20, 2023

benoithudson commented on Feb 20, 2023

lostmsu commented on Feb 20, 2023

nlogozzo commented on Feb 23, 2023

AlexGilmor commented on Feb 23, 2023

filmor commented on Feb 23, 2023

nlogozzo commented on Feb 23, 2023

kikaragyozov commented on Mar 30, 2023

Environment

Details

gitlsl commented on Aug 30, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions