Complete rustpython-unicode isolation: case mapping, casing predicates, and sre case/space parity #8236

Follow-up to #7560, continuing from #8211 (merged).

Background

#8211 extracted crates/unicode (rustpython-unicode) and routed the
unicodedata database, str classification predicates, casefold, identifier
predicates, \N{} name lookup, and the sre \d/\w tables through it.

The goal of #7560 — one authoritative Unicode path, usable outside the
Python runtime — is not finished yet. Case conversion and casing predicates
still live inside rustpython-vm as direct icu4x calls, so any consumer
other than the vm has to re-implement them (and will drift from CPython in
exactly the ways the shared crate is meant to prevent: titlecase mappings
for Lt characters, final-sigma context, case-ignorable segmentation).
The sre engine also still carries two unverified case/space quirks marked
TODO: check with cpython. This issue proposes the remaining work to make
the crate the single Unicode authority.

Current gaps (verified against main)

1. Case conversion and casing predicates are vm-private

crates/vm is the only crate besides crates/unicode that still depends on
icu_casemap / icu_locale / icu_properties directly:

• vm/src/builtins/str.rs:
  • title / capitalize — TitlecaseMapper::titlecase_segment plus a
    hand-written cased/case-ignorable segmentation loop
    (titlecase_string, titlecase_first).
  • lower (via str::to_lowercase), capitalize/title/swapcase —
    context-sensitive final-sigma handling (lowercase_or_sigma,
    handle_capital_sigma over Cased / CaseIgnorable).
  • istitle — GeneralCategoryGroup::TitlecaseLetter +
    char::is_uppercase/is_lowercase.
• vm/src/anystr.rs:
  • is_cased<VALID, INVALID> — the shared islower/isupper kernel,
    generic over the icu BinaryProperty trait (Lowercase / Uppercase),
    also consulting TitlecaseLetter.

Everything above is Unicode semantics (context-sensitive case mapping,
derived casing properties), not a string algorithm; per #7560's own
architecture ("crate::str calls into rustpython-unicode for all
Unicode-sensitive behavior") it belongs in the shared crate. The crate's
case module currently exposes only casefold_str / casefold_wtf8.

2. sre_engine case/space helpers are unverified against CPython

crates/sre_engine/src/string.rs:

• is_uni_space — SRE_UNI_IS_SPACE is Py_UNICODE_ISSPACE, i.e. exactly
  classify::is_space, but the current implementation is a hand-rolled BMP
  matches! list marked TODO: check with cpython.
• lower_unicode / upper_unicode — CPython's sre_lower_unicode /
  sre_upper_unicode use the simple one-to-one mappings
  (Py_UNICODE_TOLOWER / TOUPPER). The current implementation takes the
  first character of the full mapping
  (to_lowercase().next()), which diverges wherever a code point has a
  full mapping but no simple one — e.g. upper_unicode('ß') returns 'S'
  while CPython keeps 'ß' (U+00DF has no simple uppercase). Both are
  marked TODO: check with cpython.

Proposal

A. Expand unicode::case (move, no behavior change)

Code-point level:

• simple_lowercase / simple_uppercase / simple_titlecase /
  simple_casefold — direct wrappers over
  icu_casemap::CaseMapper's simple mappings (pure, borrowed compiled
  data, no_std-clean). Needed by sre (B) and by any consumer implementing
  per-code-point casing.
• Casing predicates: is_lowercase (Lowercase), is_uppercase
  (Uppercase), is_titlecase (Lt), is_cased (Cased),
  is_case_ignorable (Case_Ignorable). These retire the icu
  BinaryProperty generics in vm/anystr.rs and provide the
  islower/isupper/istitle kernels.

String level (&str and &Wtf8, following the casefold_str /
casefold_wtf8 precedent; lone surrogates pass through unchanged):

• lower_* / upper_* — full mappings incl. final-sigma context.
• title_* / capitalize_* / swapcase_* — move titlecase_string,
  titlecase_first, lowercase_or_sigma, handle_capital_sigma from
  vm/builtins/str.rs verbatim. The crate gains an icu_locale dependency
  (root-locale TitlecaseMapper), which is no_std-compatible.

This is a pure relocation: byte-for-byte behavior, gated by test_str.

B. Fix sre_engine parity (behavior changes, separate commits)

• is_uni_space → classify::is_space.
• lower_unicode / upper_unicode → case::simple_lowercase /
  case::simple_uppercase.
• Each change is a documented CPython-correctness fix, verified by a full
  scalar-range sweep (see Verification) — not folded into the move commit.

C. Drop icu from rustpython-vm

After A, vm/builtins/str.rs and vm/anystr.rs consume unicode::case,
and icu_casemap / icu_locale / icu_properties leave
crates/vm/Cargo.toml. Acceptance: crates/unicode is the only workspace
member with a direct icu dependency.

D. (Optional) CodePoint-level convenience layer

#7560 sketched a u32-first API; the crate today is char-first for
predicates and CodePoint/Wtf8 where non-scalars matter. If consumer
call sites show repeated CodePoint -> char boilerplate, add thin
CodePoint variants (predicates return false, mappings return identity
for lone surrogates). Decide from real usage, not up front.

Verification

• Extend crates/unicode/tests/differential.rs and the committed reference
  dataset with the new dimensions, reusing the existing version-skew
  allow-list mechanism:
  • casing predicates (islower-class, isupper-class, cased,
    case-ignorable, Lt) swept over 0..0x110000;
  • simple case mappings swept against CPython — _sre.getlower(ch, flags)
    covers the lowercase path directly; for upper/title, generate the
    reference from the vendored UnicodeData.txt simple-mapping columns
    and pin down CPython's exact table choice as part of the work.
• Context-sensitive string-level behavior (final sigma, titlecase
  segmentation) is covered by CPython's own test_str (title,
  capitalize, swapcase cases) plus a small generated corpus if gaps
  show up.
• Regression gate unchanged: test_unicodedata, test_str, test_re,
  test_pkgutil, extra_tests/snippets/stdlib_unicode_shared.py —
  identical or better only. cargo build -p rustpython-unicode --target thumbv7em-none-eabi stays green.

Out of scope

• Python str object behavior and non-case string algorithms (unchanged
  non-goals of rustpython-unicode #7560).
• Locale-dependent casing (Turkish/Azeri dotted-I etc.) — CPython's str
  does not perform it either; the root locale is the specified behavior.