Repository navigation
Conversation
…ent) The Windows test leg (CLANG64 x86_64, ASan+UBSan) has a nondeterministic crash that only shows on trees with mimalloc 3.5.3: 1-3 suites die in their first test with 0xC0000005 and no sanitizer report. mimalloc 3.5.3 is the first version whose loader hooks are live on MinGW: TLS callbacks in .CRT$XLB/.CRT$XLY run _mi_auto_process_init under the loader lock and _mi_thread_done on every thread detach. As an experiment, compile the MinGW TEST allocator with MI_WIN_INIT_USE_FLS=1. That build carries none of those hooks: process init runs once from the priority-101 constructor and thread-done goes through a fiber-local destructor. FLS also needs MI_DEBUG=1, because its internal assertion that the FLS key exists fires on the first thread init, which runs before the key is allocated, and that would abort every MI_DEBUG=2 process at startup. Level 1 keeps the fill patterns, padding and double-free checks. The production allocator flags (MIMALLOC_CFLAGS) are unchanged on every platform, and on macOS and Linux MIMALLOC_CFLAGS_TEST is byte-identical. Signed-off-by: Martin Vogel <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The fix for the nondeterministic Windows test-leg crash that came with the
mimalloc 3.5.3 update (#2731). It changes how the vendored mimalloc is compiled
for the Windows (MinGW) TEST build only. It started as a CI experiment; the
experiment concluded (see "Result" below), and the decision was to land it as
the test-build fix and to measure the shipped binary separately in #2865.
The crash
test / test-windows (windows-latest, CLANG64, x86_64)(ASan+UBSantest-runner, 4 suites in parallel).
(pass=0, secs=0, no sanitizer report), while the other ~9,600 tests pass.
repro_reference_precision, openapi. Which suites die varies between runs.
3.5.3, merge beddaaa), and 0 of 35 legs on
trees without it. The legs ran on the same runner image.
Hypothesis
mimalloc 3.5.3 is the first version whose Windows loader hooks are live in a MinGW
build. In 3.3.2 the hook tables were placed with MSVC-only
#pragma const_seg/data_seg, which clang ignores for MinGW targets. That left them as plain datathe loader never called. 3.5.3 adds a
__MINGW32__branch(
vendored/mimalloc/src/prim/windows/prim.c, around lines 902-907) that putsreal entries in
.CRT$XLB,.CRT$XLYand.CRT$XIB. Since then:mi_tls_attachruns_mi_auto_process_initfrom a TLS callback, under theloader lock, before the CRT is initialised;
mi_tls_detachruns_mi_thread_doneon every thread detach, also under theloader lock, next to our own
.CRT$XLBthread-detach callback(
src/foundation/compat_thread.c).The hypothesis is that these hooks cause the first-test 0xC0000005.
What this PR changes
On MinGW only,
MIMALLOC_CFLAGS_TESTgains-DMI_WIN_INIT_USE_FLS=1 -DMI_DEBUG=1.MI_WIN_INIT_USE_FLS=1: the test allocator carries no.CRT$XLB,.CRT$XLYor.CRT$XIBhooks. Process init runs once, from thepriority-101 constructor in
vendored/mimalloc/src/prim/prim.c, so thedouble-init / blank-stderr-line problem described at
MI_MINGW_UCRT64doesnot come back. Thread-done runs through a fiber-local (FLS) destructor.
MI_DEBUG=1: FLS cannot run without it. The first thread init(
src/init.c,mi_thread_initinsidemi_process_init_once) associates themain theap with FLS before the FLS key is allocated a few lines later. That
trips the internal assertion
mi_fls_key != (DWORD)(-1)inprim.candwould abort every MI_DEBUG=2 process at startup. Level 1 drops only the
internal assertions and the other
MI_DEBUG>1blocks. It keeps the0xD0/0xDF fill patterns, padding checks, double-free checks and statistics.
A cross-compile of the test allocator object for x86_64-w64-mingw32 (the
repository's pinned llvm-mingw image) shows the difference:
.CRT$hook sections, holdingmi_tls_attach,mi_tls_detachandmi_crt_init;.ctors.65434/.dtors.65434(the constructor) plus the FLS imports;
_mi_prim_thread_associate_default_theapcalls_mi_assert_fail; with MI_DEBUG=1 it does not.What this PR does NOT change
MIMALLOC_CFLAGSis unchanged on every platform,so the shipped binary and its
--versionoutput are unaffected.MIMALLOC_CFLAGSandMIMALLOC_CFLAGS_TESTare byte-identical to main, and so are the compile commands from
make -n -Bfor the test, TSan and production mimalloc objects.Experiment protocol
first test with 0xC0000005 and no sanitizer report).
rate is 1 in 3, 9 clean legs in a row would happen with probability
(2/3)^9 = ~2.6%.
outcome. A deterministic one, such as a mimalloc assertion at startup on
every leg, means the experiment build itself is broken, and the run stops
there.
Caveat: this run compares the FLS configuration (no loader hooks, internal
assertions off) with main as a whole. A clean result points at the loader
hooks, but on its own it cannot rule out the MI_DEBUG change. The follow-up
decides how far to take that.
Result
9 of 9 Windows test legs were clean (each 9722 passed, 0 failed, no
rc=3221225477; run 41147605534, attempts 1-9). Before the change, about 1 in 3
legs on trees with #2731 crashed; 9 clean legs in a row by chance would be about
2.6% likely. The shipped binary is unchanged by this PR; whether it is affected
by the 3.5.3 loader hooks is tracked in #2865.