ctypes array cache can return distinct types when types are created concurrently #160978

Bug report

On the free-threaded build, this fails with TypeError: expected c_int_Array_1 instance instead of c_int_Array_1:

from concurrent.futures import ThreadPoolExecutor
from ctypes import c_int
from threading import Barrier

barrier = Barrier(8)

def make_type(n):
    barrier.wait()
    return c_int * n

with ThreadPoolExecutor(8) as pool:
    for n in range(1, 1001):
        types = list(pool.map(make_type, [n] * 8))
        for t in types:
            types[0].from_param(t())

It passes on the GIL-enabled build because there isn't an opportunity for a GIL thread switch.

If I instead create a type that has a Python hook, I can deterministically trigger the same race on the GIL-enabled build as well:

import ctypes
from concurrent.futures import ThreadPoolExecutor
from threading import Barrier

barrier = Barrier(2)

class Meta(type(ctypes.c_int)):
    def __set_name__(cls, owner, name):
        barrier.wait(timeout=5)

class C(ctypes.c_int, metaclass=Meta):
    pass

with ThreadPoolExecutor(2) as pool:
    a, b = pool.map(lambda _: C * 100, range(2))

print(a is b)
a.from_param(b())

This second script prints False and then TypeError: expected C_Array_100 instance instead of C_Array_100 on both builds, because defining __set_name__ makes python code run after the cache miss on the array type cache during creation of the array class.

I think the fix is to rewrite how PyCArrayType_from_ctype uses the array_cache:

The current implementation has a check-then-insert TOCTOU bug.

I found this looking at the root cause of numpy/numpy#29943. See also #127945.

CPython versions tested on:

3.14, 3.15, 3.16, main

Operating systems tested on:

macOS (M1)