Repository navigation
fix: preserve Unicode RawJSON values with ensure_ascii=True - #258
Open
parijat1222q wants to merge 1 commit into
Open
parijat1222q wants to merge 1 commit into
parijat1222q wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preserve Unicode RawJSON values when ensure_ascii=True.
RawValue transcodes Unicode code points to an ASCII target without JSON escaping. Pass the ensure_ascii setting through the recursive encoder and escape non-ASCII RawJSON characters before that call, including UTF-16 surrogate pairs for astral characters. Existing ASCII raw spelling/whitespace and ensure_ascii=False literal behavior remain unchanged. Normal string encoding is unchanged. The shared stream-error check is retained.
Docs clarify that ensure_ascii=True escapes non-ASCII characters in RawJSON. This follows the encoder's ASCII-output guarantee rather than preserving corrupt output under a literal interpretation of 'verbatim'. RawJSON still accepts unvalidated content; this is not a general JSON validator or a fix for embedded-NUL input.
Validation on Linux CPython 3.14.8 with the native extension rebuilt from each source state:
Tests cover dumps/Encoder/text+binary dump, compact/pretty output, ASCII flag on/off, BMP/astral Unicode, unchanged raw whitespace, and stream exceptions. No speedup, hosted CI, sanitizer, debug-Python or cross-platform result is claimed.
Related but distinct: #169 discusses RawJSON performance. No competing corruption fix found in the current open PRs, targeted all-state searches, or relevant recent commit history.
Compatibility decision: the existing RawJSON docs promise verbatim insertion including whitespace, while dumps docs say all incoming non-ASCII characters are escaped under ensure_ascii=True. Current behavior satisfies neither for these inputs: it truncates Unicode or raises rather than inserting the original value. The proposal gives the global ASCII setting precedence for valid RawJSON string content. That deliberately changes literal non-ASCII spelling to escapes under the default; callers who require exact Unicode text must use ensure_ascii=False. An alternative would preserve Unicode verbatim regardless of ensure_ascii, but would weaken that documented global guarantee. ASCII content, existing escapes and whitespace are not normalized, and arbitrary raw content is still not validated. Maintainer preference on this compatibility tradeoff is welcome.
Closes #257