Bug report

Bug description:

TextIOWrapper.tell() guesses where to start re-decoding its decoder snapshot as the bytes-per-character ratio times the number of characters to skip, and bounds that guess only with an assert. After seek() to a cookie that skips characters, the snapshot can be much shorter than the ratio of the previously decoded chunk implies, so the next tell() passes the decoder bytes from past the end of the snapshot.

import io
data = b"+AGEAYg-+" + b"A" * 8183
f = io.TextIOWrapper(io.BytesIO(data), encoding="utf-7")
f.read(1)
f.seek(f.tell())
f.tell()

On a debug build the last line fails with Assertion 'skip_bytes <= PyBytes_GET_SIZE(next_input)' failed. On a release build it reads about 4 KiB past the 8-byte snapshot and raises UnicodeDecodeError for a byte that is not in the input, where it should return the position passed to seek(). Codecs that decode one byte sequence to several characters (utf-7, big5hkscs) and errors="ignore" on mostly undecodable data can over-read by thousands of bytes this way.

Common codecs reach the same assertion in ordinary use, for example a CRLF file opened with newline="" as the csv module requires:

f = io.TextIOWrapper(io.BytesIO(b"a\r\n" * 10), encoding="ascii", newline="")
f.read(29)  # stops between "\r" and "\n"
f.seek(f.tell())
f.readline()
f.tell()    # assertion failure on a debug build

On release builds that case over-reads by one byte, into the bytes object's terminating NUL, so it goes unnoticed there.

gh-141314 fixed another way to fail the same assertion; gh-153662 is a separate open crash in the same cookie handling.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-161053: Fix out-of-bounds read in TextIOWrapper.tell() after seek() #161054