Skip to content

Fix source decoding diagnostics and normalize decoded newlines - #9429

Merged
youknowone merged 1 commit into
RustPython:mainfrom
youknowdot:youknowdot/source-encoding-315-20261028
Oct 28, 2026
Merged

Fix source decoding diagnostics and normalize decoded newlines#9429
youknowone merged 1 commit into
RustPython:mainfrom
youknowdot:youknowdot/source-encoding-315-20261028

Conversation

@youknowdot

@youknowdot youknowdot commented Oct 28, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Preserve the coding-cookie line for BOM conflicts, recognize CR line boundaries, and continue scanning after invalid coding candidates.
  • Convert codec initialization failures to SyntaxError while preserving the underlying reason, following CPython's tokenizer initialization error handling.
  • Normalize successfully decoded source before parsing so literal values, warning ranges, and diagnostic text use the same universal-newline input.
  • Add the unchanged CPython 3.15 test_source_encoding suite using scripts/update_lib migrate from d220409a13b77de7a6c30abe16ea90db01528edc.

The native changes are confined to the source decoder and its encoding-location record. They do not replace the compiler/parser diagnostic or type-comment work in #8954.

Validation

  • Focused CLI/VM cargo check and strict Clippy (-D warnings) passed.
  • Repository hooks passed, including cspell; 228 updater unit tests passed.
  • All 93 ordinary canonical cases passed, without new skips or assertion changes.
  • The separate canonical Linux CPU validator passed all 489 fixed subprocess cases within its 300-second wall-time limit.
  • A bounded CPython/RustPython comparison agrees on CR/CRLF literal values, AST locations, and syntax-error location attributes for compile(str) and compile(bytes).

Scope and remaining differences

This is not a claim of complete tokenizer compatibility. Successful decoder consumers are normalized together; direct string paths that bypass the decoder are not changed. CPython's pre-codec newline conversion and file-stream-relative codec error offsets remain separate work. Generic SyntaxError.args location tuples also remain different even where the location attributes match. No dependency pins or Ruff patches are changed.

AI assistance

OpenAI Codex assisted with the implementation, CPython source comparison, tests, and this pull request description; the model version is unavailable. Full workspace and C-API tests were not run locally; hosted CI remains pending.

Summary by CodeRabbit

  • Bug Fixes
    • Improved recognition of source-file encoding declarations, including declarations on either of the first two lines and files with a UTF-8 byte-order mark.
    • Source files using CRLF or CR line endings are now normalized consistently, helping prevent parsing issues.
    • Encoding-related errors now provide more useful location and filename information.

@codex
Fix source decoding diagnostics and normalize decoded newlines
1f77499 
Track coding-cookie locations, preserve codec initialization errors as SyntaxError, and normalize decoded source before parsing. Add unchanged CPython 3.15 source-encoding coverage via update_lib.

Assisted-by: OpenAI Codex: model version unavailable
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 3b0dd78 and 1f77499.

Files ignored due to path filters: 1
  • Lib/test/test_source_encoding.py is excluded by !Lib/**
Files selected for processing: 1
  • crates/vm/src/vm/compile.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

Source encoding detection now returns cookie metadata, including its line and byte range. The compiler uses that metadata for decoding and error reporting. Successfully decoded source normalizes CRLF and bare CR to LF.

Changes: Source encoding

Layer / File(s) Summary
Encoding cookie detection
crates/vm/src/vm/compile.rs		 Detection scans up to two physical lines, handles an optional UTF-8 BOM and CR/LF line endings, and returns the encoding name with the cookie’s line and byte range.
Decoding and error reporting
crates/vm/src/vm/compile.rs		 program_text passes the detected encoding name to decoding. BOM and codec errors use location data, and successfully decoded source normalizes CRLF and bare CR to LF.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Possibly related issues

  • RustPython issue 6839: This PR performs one item of the tracked CPython 3.15 standard-library migration work by adding the test_source_encoding suite.

Suggested reviewers: youknowone

Merge Risk: Minimal · up to 1f774

The decoding and newline changes appear mergeable with normal checks. No new merge-blocking defect was identified.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main changes: improved source-decoding diagnostics and normalized decoded newlines.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 1 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@youknowone
youknowone marked this pull request as ready for review October 28, 2026 13:00
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-28T13:02:05.535922Z
We couldn’t complete this review. Comment @codex review to try again.		 1f77499 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] test: cpython/Lib/test/test_startfile.py

dependencies:

dependent tests: (no tests depend on startfile)

[x] lib: cpython/Lib/zipimport.py
[x] test: cpython/Lib/test/test_zipimport.py (TODO: 2)
[ ] test: cpython/Lib/test/test_zipimport_support.py (TODO: 2)

dependencies:

  • zipimport

dependent tests: (10 tests)

  • zipimport: test_cmd_line_script test_importlib test_pkgutil test_zipimport test_zipimport_support
    • pkgutil: test_pydoc test_pyrepl test_runpy
      • pydoc: test_enum
      • site: test_site

[x] lib: cpython/Lib/pdb.py
[ ] test: cpython/Lib/test/test_pdb.py (TODO: 30)

dependencies:

  • pdb

dependent tests: (2 tests)

  • pdb: test_pdb test_remote_pdb

[x] test: cpython/Lib/test/test_source_encoding.py

dependencies:

dependent tests: (no tests depend on source_encoding)

[ ] test: cpython/Lib/test/test_setcomps.py

dependencies:

dependent tests: (no tests depend on setcomps)

[x] test: cpython/Lib/test/test_dict.py (TODO: 1)
[ ] test: cpython/Lib/test/test_dictcomps.py
[x] test: cpython/Lib/test/test_dictviews.py (TODO: 1)
[x] test: cpython/Lib/test/test_userdict.py
[x] test: cpython/Lib/test/mapping_tests.py

dependencies:

dependent tests: (no tests depend on dict)

[ ] test: cpython/Lib/test/test_list.py
[ ] test: cpython/Lib/test/test_listcomps.py
[ ] test: cpython/Lib/test/test_userlist.py

dependencies:

dependent tests: (no tests depend on list)

[x] test: cpython/Lib/test/test_generators.py
[ ] test: cpython/Lib/test/test_genexps.py (TODO: 2)
[x] test: cpython/Lib/test/test_generator_stop.py
[x] test: cpython/Lib/test/test_yield_from.py

dependencies:

dependent tests: (no tests depend on generator)

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

@youknowone
youknowone merged commit 2e84a07 into RustPython:main Oct 28, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@youknowone youknowone youknowone approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@youknowdot @youknowone @codex