NameError: name 'PathDistribution' is not defined #525

Activity

1. jaraco commented on Dec 29, 2025

   

   jaraco

   on Dec 29, 2025

   Member

   More actions

   I don't understand the context in which this failure occurs. This project is mechanically released only after tests pass and tests cannot pass if there's a NameError on import. How is it that it imports in CI and on my machine, but not in celery? Perhaps the bug is with Celery not supporting forward references to types?
2. nightcityblade commented on Aug 18, 2026

   

   nightcityblade

   on Aug 18, 2026

   More actions

   Hi, I'd like to work on this. I'll submit a focused PR with regression coverage shortly.
3. jaraco commented on Oct 31, 2026

   

   jaraco

   on Oct 31, 2026

   Member

   More actions

   @nightcityblade Thanks for offering to help. I don't think the problem is well defined enough yet. I don't want to have workarounds for problems that are poorly understood.

   I'd really like to see a more detailed explanation of why the code is failing and under what conditions, because so far, I have no way to reproduce the issue.
4. jaraco commented on Oct 31, 2026

   

   jaraco

   on Oct 31, 2026

   Member

   More actions

   I think I can account for this now, and it's reproducible. The line ordering @stivluc points at is indeed the bug, but not for the reason stated — a function body resolving a global at call time is normally fine. The problem is that @install publishes the finder into a process-global registry before the module has finished executing.

   The window

   install(MetadataPathFinder) runs sys.meta_path.append(cls()) while __init__.py is still executing — line 946 in 8.7.0, 36 lines before PathDistribution is bound at 982. For that span there is a globally reachable, callable finder belonging to a half-built module, and PathDistribution is its only forward reference: Prepared (897), FastPath (789) and DistributionFinder (727) are all already defined. That's why it's specifically this name that fails.

   It's worse than just a gap, because install() appends first and then calls disable_stdlib_finder(), which does del PathFinder.find_distributions. Inside the window the half-built finder isn't merely present — it is the only provider of distributions in the process.

   The stdlib copy is immune: install/NullFinder are stripped in the CPython port, and PathFinder.find_distributions imports MetadataPathFinder lazily. This is backport-only, which is part of why CI never sees it.

   Condition A: a permanently poisoned sys.meta_path

   If the import is aborted inside that window, Python removes sys.modules['importlib_metadata'] — but the half-built finder stays on sys.meta_path for the life of the process, holding the abandoned module dict alive via __globals__.

   import importlib.util
   import sys
   
   spec = importlib.util.find_spec('importlib_metadata')
   with open(spec.origin) as strm:
       lines = strm.read().splitlines()
   bind_line = next(
       num
       for num, line in enumerate(lines, 1)
       if line.startswith('class PathDistribution')
   )
   
   
   def tracer(frame, event, arg):
       return liner if frame.f_code.co_filename == spec.origin else None
   
   
   def liner(frame, event, arg):
       if event == 'line' and frame.f_lineno == bind_line:
           raise KeyboardInterrupt('async exception lands in the window')
       return liner
   
   
   sys.settrace(tracer)
   try:
       import importlib_metadata
   except KeyboardInterrupt as exc:
       print('import aborted:', exc)
   finally:
       sys.settrace(None)
   
   assert 'importlib_metadata' not in sys.modules
   print('orphan left on sys.meta_path:',
         [f for f in sys.meta_path if type(f).__name__ == 'MetadataPathFinder'])
   
   import importlib.metadata
   import importlib_metadata  # a *successful* import this time
   
   print('this is the call celery makes:')
   list(importlib.metadata.distributions())

   import aborted: async exception lands in the window
   orphan left on sys.meta_path: [<importlib_metadata.MetadataPathFinder object at 0x105a05940>]
   this is the call celery makes:
   Traceback (most recent call last):
     ...
     File ".../importlib/metadata/__init__.py", line 486, in <genexpr>
       resolver(context) for resolver in cls._discover_resolvers()
       ~~~~~~~~^^^^^^^^^
     File ".../importlib_metadata/__init__.py", line 1054, in find_distributions
       return map(PathDistribution, found)
                  ^^^^^^^^^^^^^^^^
   NameError: name 'PathDistribution' is not defined. Did you mean: 'IDistribution'?
   

   Note the second-to-last import: a subsequent, entirely successful import importlib_metadata does not heal it. The orphan is still earlier in sys.meta_path and still raises. That accounts for the parts that didn't add up:

   • The import that failed is not the import that reports. Celery calls stdlib importlib.metadata.entry_points(group='celery.result_backends'); the stdlib walks sys.meta_path; the backport's orphan raises. Celery need never import importlib_metadata itself — some other dependency put the finder there.
   • The original failure is typically invisible, since consumers of a backport conventionally write try: import importlib_metadata / except ImportError: ....
   • Nothing in a test suite interrupts a module body mid-execution, so CI can't catch it.

   The window is pure, cheap, import-free code, so the abort has to be an asynchronous exception. Celery is unusually rich in those: soft time limits (SIGUSR1 → SoftTimeLimitExceeded, delivered at an arbitrary bytecode boundary), billiard's Terminated, signal.alarm timeouts, KeyboardInterrupt during startup, MemoryError. With the prefork pool, a parent poisoned before forking hands the broken sys.meta_path to every child.

   Condition B: a plain thread race

   Another thread calling stdlib importlib.metadata.entry_points() while this thread is mid-import. No import lock protects this, because the other thread isn't importing importlib_metadata — it is only iterating sys.meta_path. Replacing the raise above with a time.sleep() to stand in for an unlucky thread switch yields the same NameError in the other thread.

   The untraced window is only a few microseconds, so on a GIL build you have to be unlucky. Two things change the odds considerably: a free-threaded build removes the need for a preemption point at all, and running under coverage, a profiler or a debugger inflates the window roughly 40x (43 µs measured under settrace). A __del__ or weakref callback that queries entry points, firing on an allocation inside the window, gets there single-threaded.

   Ruled out

   • Interpreter shutdown / module teardown. Globals survive via func.__globals__; dropping the module from sys.modules and collecting it still works. And _PyModule_Clear sets globals to None, which would surface as TypeError: 'NoneType' object is not callable, not NameError.
   • The cross-thread circular-import deadlock fallback (_lock_unlock_module swallowing _DeadlockError and handing out a partially initialized module — still present on main). The window contains no imports, so no deadlock can be detected inside it.

   @stivluc — two things would confirm which condition you hit

   1. Is it every subsequent call in that worker process, or one-shot? Persistent means Condition A.
   2. In the failing process:

   import sys
   print('importlib_metadata' in sys.modules,
         [f for f in sys.meta_path if type(f).__name__ == 'MetadataPathFinder'])

   A finder present while sys.modules reports False is the smoking gun. The full traceback would also help (to see whether the call entered via importlib.metadata or importlib_metadata), along with which pool you run and whether soft/hard time limits are configured.

   On a fix

   Installing the finder below PathDistribution closes the window under both conditions: an abort before it leaves no finder at all, and an abort after it leaves a complete one. I don't think that's a workaround for a poorly understood problem — "don't publish a half-constructed object into a process-global registry" holds regardless of which trigger is at play here. It's also cheap with respect to the CPython sync, since install and NullFinder are already stripped from that copy.