Skip to content

Align Unicode lookup and decomposition with CPython 3.15 - #9476

Draft
youknowdot wants to merge 2 commits into
RustPython:mainfrom
youknowdot:python315-unicode-lookup-decomposition-20261030
Draft

Align Unicode lookup and decomposition with CPython 3.15#9476
youknowdot wants to merge 2 commits into
RustPython:mainfrom
youknowdot:python315-unicode-lookup-decomposition-20261030

Conversation

@youknowdot

@youknowdot youknowdot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Validate names returned by unicode_names2 against canonical names, while preserving exact case-insensitive alias and named-sequence lookup. Reject loose whitespace/hyphen matching and noncanonical CJK hexadecimal names.
  • Match CPython's distinction for the old UCD view: lookup() rejects aliases/sequences but accepts ordinary names from the shared database; name() and decomposition() retain historical membership checks.
  • Return full Hangul L/V/T decomposition and use current decomposition mappings in both views, removing the incorrect historical normalization-correction override. Normalization code is unchanged.
  • Use scripts/update_lib migrate to retire four related test_unicodedata markers while preserving the other nine adaptations.

Sources: pinned CPython lookup, decomposition, and canonical tests.

Validation and limits

  • Scoped locked/offline CLI check, strict Clippy (-D warnings) and build succeeded.
  • 120 bounded scratch observations matched pinned CPython exactly, including exception text, aliases/sequences, significant hyphens, both UCD views, Hangul and corrected decomposition mappings.
  • Four restored test_unicodedata methods passed in both interpreters. Of twelve selected canonical tests, CPython passed twelve; RustPython passed eleven and retained one pre-existing expected failure in test_ucn.test_tangut_ideographs. The strict wrapper therefore exited 1, not a full-suite pass.
  • A separately bounded single-test diagnostic confirmed that expected failure is the existing ast.literal_eval/compile Unicode-name literal SyntaxError, not unicodedata.lookup. Its decorator and test body remain unchanged; direct Tangut lookup is covered by the passing differential observations.

No full Unicode sweep, full workspace/CAPI suite, normalization corpus or cross-platform runtime validation was performed. This does not claim full Unicode or parser compatibility.

AI assistance: OpenAI Codex (model version unavailable), disclosed in the commit trailer. Automated validation is not human review.

@codex
Align Unicode lookup and decomposition with CPython 3.15
d4506d0 
Validate Unicode names instead of accepting loose dependency matching, preserve
old-UCD alias and sequence restrictions without filtering ordinary names, and
match CPython's Hangul and historical-membership decomposition behavior.
Remove the incorrect historical correction override and refresh four related
unicodedata test markers using update_lib. Leave normalization unchanged.

Assisted-by: OpenAI Codex: model version unavailable
@coderabbitai

coderabbitai Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[ ] test: cpython/Lib/test/test_unicodedata.py (TODO: 7)
[x] test: cpython/Lib/test/test_unicode_file.py
[x] test: cpython/Lib/test/test_unicode_file_functions.py
[x] test: cpython/Lib/test/test_unicode_identifiers.py
[x] test: cpython/Lib/test/test_ucn.py (TODO: 1)

dependencies:

dependent tests: (no tests depend on unicode)

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

@codspeed

codspeed Bot commented Oct 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1

Comparing youknowdot:python315-unicode-lookup-decomposition-20261030 (8cb31d7) with main (d24ca41)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@codex
Preserve Unicode 3.2 widths for still-unassigned characters
8cb31d7 
Match CPython's old-UCD width policy: retain modern width defaults for
characters still unassigned, and use the historical table for assigned
characters. Restore the bounded east Asian width tests and retire the
obsolete 9.0 override through update_lib.

Permit explicit exact-source retirement of unconditional skip markers in
the existing updater path, preserving its exact-match and consumed-once
guards. Add focused tool tests for successful and rejected retirements.

Assisted-by: OpenAI Codex: model version unavailable
youknowone
youknowone requested changes Oct 30, 2026
View reviewed changes

@youknowone youknowone left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review this again if it is aligned to 3.2.0 feature well

@youknowdot

youknowdot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

In response to the 3.2 feature review:

I rechecked the 3.2 feature against pinned CPython source and our actual old data. #9476 targets CPython's ucd_3_2_0 API, not a literal historical database for every operation:

  • Old lookup accepts current ordinary names but rejects aliases/sequences; unlike name, it does not check old membership. The lookup correction preserves this asymmetry.
  • Old decomposition rejects newly assigned characters, then uses current mappings/full Hangul decomposition. Thus U+2F868 gives 36FC; historical 2136A belongs to the separate old normalization corrections, not this API.
  • EAW distinguishes newly assigned from still-unassigned characters: old-view U+9FA6 is N, but still-unassigned U+3FFFD remains W. The generator leaves current width defaults for the latter. Our historical derived-width file explicitly omits unassigned points.

The earlier scoped comparisons covered 120 lookup/decomposition rows and 100 EAW/property rows across both views, on their respective implementation heads. This review additionally checked 27 finite source-table boundaries, not an exhaustive sweep or new runtime run.

Importantly, RustPython's pre-existing normalize/is_normalized still ignore the old view. Those functions and feature wiring are unchanged; this PR does not complete all Unicode 3.2 semantics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@youknowone youknowone youknowone requested changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@youknowdot @youknowone @codex