Speed up tarfile.calc_chksums()

Feature or enhancement

Proposal:

tarfile.calc_chksums() runs once per header block, every time a header is read
(TarInfo.frombuf) or written (TarInfo._create_header). It currently builds
two 504-element tuples with struct.unpack_from (one unsigned, one signed) and
sums each one in Python:

unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
return unsigned_chksum, signed_chksum

Both totals can be had from one C-level sum(buf): a byte b >= 128 adds b
to the unsigned sum and b - 256 to the signed one, so
signed == unsigned - 256 * (number of bytes >= 128). Almost every real header
is pure ASCII, in which case the two sums are equal and bytes.isascii()
detects that cheaply. Otherwise the high bytes are counted with
bytes.translate(None, ...).

The patch (+13 lines, no deletions) keeps the original code verbatim as the
fallback, and only takes the fast path when type(buf) is bytes and len(buf) == BLOCKSIZE, which is the only thing tarfile itself ever passes.
Any other input (memoryview, array, bytearray, other lengths) behaves exactly as
before, including raising struct.error below 512 bytes.

+_HIGH_BYTES = bytes(range(128, 256))
+
 def calc_chksums(buf):
     ...
+    if type(buf) is bytes and len(buf) == BLOCKSIZE:
+        # Fast path for the only input tarfile itself passes. The chksum
+        # field (buf[148:156]) counts as eight spaces, i.e. 256. A byte with
+        # the high bit set adds b to the unsigned sum and b - 256 to the
+        # signed one, so signed == unsigned - 256 * (number of such bytes).
+        unsigned_chksum = 256 + sum(buf) - sum(buf[148:156])
+        if buf.isascii():
+            return unsigned_chksum, unsigned_chksum
+        head = buf[:148] + buf[156:]
+        high = BLOCKSIZE - 8 - len(head.translate(None, _HIGH_BYTES))
+        return unsigned_chksum, unsigned_chksum - 256 * high
     unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
     signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
     return unsigned_chksum, signed_chksum

Benchmarks

The absolute gain per archive is small (milliseconds), so I'm not claiming
this fixes a bottleneck. It is a modest, free speedup on a function called
about once or twice per archive member (PAX archives carry an extra x
header per member).

calc_chksums alone, ASCII header, ns per call (stock / patched):

Python	stock	patched
3.11.10	4877	1027
3.14.10	3008	826
3.15.2	3474	827

Headers containing high bytes are about 2.6-3.3x faster.

End to end (tarfile.open + iterate + read every member, CPU time via
time.process_time, 25 ABBA-interleaved rounds, geometric mean of the
per-workload median speedups, stock / patched):

Python	read	list (getmembers)	stream (`r\|`)	write
3.9	1.35x	1.51x	1.07x	1.40x
3.11	1.52x	1.79x	1.08x	1.63x
3.12	1.34x	1.52x	1.05x	1.38x
3.13	1.36x	1.54x	1.05x	1.38x
3.14	1.32x	1.50x	1.05x	1.33x
3.15	1.40x	1.61x	1.39x	1.38x
3.16.0a1+dev (built from main, 4a9b981)	1.34x	1.55x	1.34x	1.35x

Each cell aggregates 20 synthetic workload/operation rows: many small members,
few large members, ustar/gnu/pax formats, high-bit names, and plain/gz/bz2/xz
variants. No row's 95% bootstrap confidence interval reaches 1.0; the smallest
gain is about 1.02x (bz2 stream, where decompression dominates) and the
largest about 2.0x.

For scale, on a synthetic archive of 20,000 x 100-byte members, reading every
member went from 199 ms to 134 ms on the 3.16 build (1.48x), and getmembers()
from 158 ms to 94 ms. Archives with only a few hundred members save a few
milliseconds. Peak memory (tracemalloc) is unchanged to within 3 KB on 3.11,
3.15 and 3.16.

Verification

Exhaustive equivalence against the original function on 1,118,530 headers
per interpreter (every byte value at every position of 7 base blocks, plus
random, clustered-high-byte and k-high-byte buffers): 0 mismatches on 3.9,
3.11, 3.12, 3.13, 3.14 and 3.15.
The full test_tarfile suite passes identically with and without the patch
(3.11.10: 591 tests, 3.12.15: 633, 3.13.16: 666, 3.14.10: 768, and 795 on a
3.16.0a1+dev build of main, with _lzma and _zstd available). test_shutil
also passes identically on that build.
Differential run of stock vs patched tarfile over about 4.1 million cases
(about 120 base archives including the test data, ustar/gnu/pax, Sun-style
signed-checksum headers; every single-bit flip in the first headers,
truncation at every length, header mutations, r:*, r:, r|,
gz/bz2/xz, extractall): 0 differences. Archives written by the patched
tarfile are byte-identical in every format. The corpus exercises the
unsigned-match, signed-only-match and bad-checksum branches
(about 2.09M, 3,238 and 200k headers in one run).
To check the harness could actually fail, I injected six deliberate bugs
into the patch (signed == unsigned, checksum field not excluded, off-by-one
high-byte count, wrong high-byte threshold, wrong head slice, inverted
isascii); all six were caught.

Hi, I'd like to take this issue. Could you assign it to me?

We don't assign issues and it's polite to ask the OP first. I believe they would be willing to submit a patch because they already have one in mind. Nonetheless, I want to know more ab out the benchmarks first:

Please share the benchmark script.
Please share the configuration for building Python.

Thank you for taking the time to open this issue. Python is a community-driven project, and we want it to be a welcoming place to learn and collaborate.

This issue appears to have been written with AI assistance. Please explain your issue in your own words and describe how you checked the information you provided. This helps us understand your issue and leaves more time to address other issues.

Please read our guidelines for using AI tools.

We don't assign issues and it's polite to ask the OP first. I believe they would be willing to submit a patch because they already have one in mind. Nonetheless, I want to know more ab out the benchmarks first:

Please share the benchmark script.

Please share the configuration for building Python.

Hello,

Benchmark scripts and results

All numbers below are for CPython 3.16.0a1+dev built from main at 4a9b98115f (-O3 -DNDEBUG, no PGO/LTO; full configuration below), on an Apple M5 Pro (arm64) macOS laptop that was not isolated (pyperf system tune is Linux-only). "stock" is main's unmodified Lib/tarfile.py; "patched" is the same file with the calc_chksums change from this issue. The archives are all synthetic. Earlier results for 3.9, 3.11-3.15 are in the issue description.

Build configuration

Out-of-tree build of main at 4a9b98115fec4d93c32c554aa541d6c3fc4c3b27, macOS 26.7.1 (arm64, Apple M5 Pro), Apple clang 21.0.0 (CC=gcc is the Apple clang shim on macOS):

mkdir build && cd build
../configure -q \
    LIBLZMA_CFLAGS=-I/opt/homebrew/include 'LIBLZMA_LIBS=-L/opt/homebrew/lib -llzma' \
    LIBZSTD_CFLAGS=-I/opt/homebrew/include 'LIBZSTD_LIBS=-L/opt/homebrew/lib -lzstd'
make

No --enable-optimizations, no --with-lto, no --with-pydebug, so this is a release build without profile-guided or link-time optimization. The LIBLZMA_*/LIBZSTD_* variables are only there because pkg-config is not installed on this machine; without them _lzma and _zstd are not built.

Resulting flags (sysconfig):

CONFIG_ARGS = 'LIBLZMA_CFLAGS=-I/opt/homebrew/include' 'LIBLZMA_LIBS=-L/opt/homebrew/lib -llzma' 'LIBZSTD_CFLAGS=-I/opt/homebrew/include' 'LIBZSTD_LIBS=-L/opt/homebrew/lib -lzstd'
OPT           = -DNDEBUG -g -O3 -Wall
PY_CFLAGS     = -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall

1. `calc_chksums` microbenchmark

Two versions: a plain time.perf_counter loop and a pyperf one.

bench_calc_chksums.py (median of 15 rounds x 20000 calls):

python 3.16.0a1+dev - ns per call, median of 15 rounds x 20000 calls
ascii header   stock   5820  patched   1676  3.47x
high-bit name  stock   5837  patched   2315  2.52x

bench_calc_chksums_pyperf.py (pyperf 2.10.0, --processes 30 --values 5, 31 runs each):

header	stock	patched	speedup
ASCII	5.81 us +- 0.33	1.68 us +- 0.13	3.46x
high-bit name	5.81 us +- 0.24	2.16 us +- 0.19	2.69x

2. End to end (`bench_tarfile.py`)

CPU time (time.process_time), 25 rounds, stock/patched interleaved ABBA, ratio is the median of the per-round paired ratios, CI is a bootstrap 95% interval on that median. read iterates the archive and reads every regular member, list is getmembers(), stream is the same as read in r| mode, write adds 3000 members.

workload	op	stock	patched	speedup	95% CI
synth 20k x 100B pax	read	250.07 ms	167.48 ms	1.492x	1.488 – 1.506
synth 20k x 100B pax	list	202.67 ms	120.33 ms	1.689x	1.678 – 1.713
synth 20k x 100B pax .gz	read	326.02 ms	243.28 ms	1.339x	1.332 – 1.347
synth 20k x 100B pax .gz	list	263.82 ms	181.50 ms	1.453x	1.442 – 1.461
synth 20k x 100B .bz2	read	377.55 ms	290.27 ms	1.302x	1.285 – 1.315
synth 20k x 100B .bz2	list	304.86 ms	216.04 ms	1.408x	1.392 – 1.419
synth 20k x 100B .xz	read	338.29 ms	257.86 ms	1.314x	1.308 – 1.319
synth 20k x 100B .xz	list	274.50 ms	191.06 ms	1.428x	1.420 – 1.443
synth 2000 x 20KB ustar	read	25.40 ms	17.28 ms	1.469x	1.463 – 1.479
synth 2000 x 20KB ustar	list	20.27 ms	11.68 ms	1.710x	1.697 – 1.744
synth 20 x 4MB gnu	read	1.92 ms	1.80 ms	1.048x	1.032 – 1.063
synth 20 x 4MB gnu	list	0.20 ms	0.12 ms	1.692x	1.689 – 1.697
synth 5k hi-bit names gnu/latin1	read	63.24 ms	44.64 ms	1.412x	1.393 – 1.417
synth 5k hi-bit names gnu/latin1	list	50.81 ms	32.59 ms	1.544x	1.532 – 1.574
synth 5k hi-bit names pax/utf8	read	120.43 ms	77.41 ms	1.549x	1.529 – 1.565
synth 5k hi-bit names pax/utf8	list	103.63 ms	63.46 ms	1.636x	1.627 – 1.641
synth 20k x 100B pax .gz	stream	308.88 ms	225.01 ms	1.375x	1.364 – 1.390
synth 20k x 100B .bz2	stream	337.85 ms	253.41 ms	1.342x	1.324 – 1.359
synth 20k x 100B .xz	stream	310.79 ms	227.37 ms	1.365x	1.353 – 1.374
write 3000 members pax	write	45.97 ms	33.38 ms	1.372x	1.363 – 1.385

Geometric mean of the median speedups: {'read': 1.357, 'list': 1.565, 'stream': 1.361, 'write': 1.372}

3. Scripts

Run everything with ./run.sh /path/to/python. It takes tarfile.py from that interpreter's own (unpatched) stdlib, so use a stock interpreter, or edit the build_variants.py call to point at git show main:Lib/tarfile.py. For a quick smoke test: BENCH_ARGS="-n 5 --quick" ./run.sh /path/to/python. The pyperf version needs pyperf installed (python -m pip install pyperf).

`run.sh`

#!/bin/sh
# Usage: ./run.sh /path/to/python [TAG]
# Builds stock/patched tarfile.py from that interpreter's own (unpatched) tarfile,
# then runs both benchmarks. Set BENCH_ARGS="-n 5 --quick" for a fast smoke test.
PY=${1:?usage: run.sh /path/to/python [TAG]}
cd "$(dirname "$0")"
TAG=${2:-$("$PY" -c 'import sys; print("%d.%d" % sys.version_info[:2])')}
"$PY" build_variants.py "$("$PY" -c 'import tarfile; print(tarfile.__file__)')" "$TAG" || exit 1
"$PY" -u bench_calc_chksums.py
"$PY" -u bench_tarfile.py "$TAG" ${BENCH_ARGS:--n 25}

`build_variants.py` (writes mods//stock|patched/tarfile.py)

"""Build stock/ and patched/ tarfile.py for the running interpreter (from its own stdlib, or argv[1]) into mods/<ver>/."""
import os, re, sys, sysconfig
src = sys.argv[1] if len(sys.argv) > 1 else os.path.join(sysconfig.get_paths()["stdlib"], "tarfile.py")
tag = sys.argv[2] if len(sys.argv) > 2 else "%d.%d" % sys.version_info[:2]
text = open(src).read()
OLD = '''    unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
    signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
    return unsigned_chksum, signed_chksum
'''
NEW = '''    if type(buf) is bytes and len(buf) == BLOCKSIZE:
        # Fast path for the only input tarfile itself passes. The chksum
        # field (buf[148:156]) counts as eight spaces, i.e. 256. A byte with
        # the high bit set adds b to the unsigned sum and b - 256 to the
        # signed one, so signed == unsigned - 256 * (number of such bytes).
        unsigned_chksum = 256 + sum(buf) - sum(buf[148:156])
        if buf.isascii():
            return unsigned_chksum, unsigned_chksum
        head = buf[:148] + buf[156:]
        high = BLOCKSIZE - 8 - len(head.translate(None, _HIGH_BYTES))
        return unsigned_chksum, unsigned_chksum - 256 * high
    unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
    signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
    return unsigned_chksum, signed_chksum
'''
assert text.count(OLD) == 1, "calc_chksums body not found exactly once in " + src
assert text.count("def calc_chksums(buf):") == 1
patched = text.replace(OLD, NEW).replace("def calc_chksums(buf):", "_HIGH_BYTES = bytes(range(128, 256))\n\ndef calc_chksums(buf):", 1)
d = os.path.join(os.path.dirname(os.path.abspath(__file__)), "mods", tag)
for kind, body in (("stock", text), ("patched", patched)):
    os.makedirs(os.path.join(d, kind), exist_ok=True)
    open(os.path.join(d, kind, "tarfile.py"), "w").write(body)
print(tag, "built from", src, "BLOCKSIZE defined before calc_chksums:", text.index("BLOCKSIZE =") < text.index("def calc_chksums"))

`bench_calc_chksums.py`

"""Microbenchmark: original calc_chksums vs the proposed fast path, ns per call.
usage: python bench_calc_chksums.py [-n ROUNDS]
Run it under each interpreter you care about."""
import argparse, statistics as st, struct, sys, time

BLOCKSIZE = 512
_HIGH_BYTES = bytes(range(128, 256))

def orig(buf):
    unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
    signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
    return unsigned_chksum, signed_chksum

def patched(buf):
    if type(buf) is bytes and len(buf) == BLOCKSIZE:
        unsigned_chksum = 256 + sum(buf) - sum(buf[148:156])
        if buf.isascii():
            return unsigned_chksum, unsigned_chksum
        head = buf[:148] + buf[156:]
        high = BLOCKSIZE - 8 - len(head.translate(None, _HIGH_BYTES))
        return unsigned_chksum, unsigned_chksum - 256 * high
    return orig(buf)

def header(name):
    b = bytearray(BLOCKSIZE)
    b[:len(name)] = name
    b[100:107] = b"0000644"; b[124:135] = b"00000000144"; b[148:156] = b"        "
    b[257:262] = b"ustar"
    return bytes(b)

CASES = {"ascii header": header(b"config/dir001/file00001.dat"),
         "high-bit name": header("config/d\xe9r001/f\xefle\xff.dat".encode("latin-1"))}

ap = argparse.ArgumentParser(); ap.add_argument("-n", type=int, default=15); a = ap.parse_args()
LOOPS = 20000
print("python", sys.version.split()[0], "- ns per call, median of %d rounds x %d calls" % (a.n, LOOPS))
for label, buf in CASES.items():
    assert orig(buf) == patched(buf)
    res = {}
    for name, f in (("stock", orig), ("patched", patched)):
        runs = []
        for _ in range(a.n):
            t = time.perf_counter()
            for _ in range(LOOPS): f(buf)
            runs.append((time.perf_counter() - t) / LOOPS * 1e9)
        res[name] = st.median(runs)
    print("%-14s stock %6.0f  patched %6.0f  %.2fx" % (label, res["stock"], res["patched"], res["stock"] / res["patched"]))

`bench_calc_chksums_pyperf.py`

"""pyperf version of bench_calc_chksums.py: original calc_chksums vs the proposed fast path.
usage: python bench_calc_chksums_pyperf.py -o result.json      (then: python -m pyperf compare_to ...)
Benchmarks are named "<case>/stock" and "<case>/patched"; pyperf groups them for compare_to."""
import pyperf, struct

BLOCKSIZE = 512
_HIGH_BYTES = bytes(range(128, 256))

def orig(buf):
    unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf))
    signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
    return unsigned_chksum, signed_chksum

def patched(buf):
    if type(buf) is bytes and len(buf) == BLOCKSIZE:
        unsigned_chksum = 256 + sum(buf) - sum(buf[148:156])
        if buf.isascii():
            return unsigned_chksum, unsigned_chksum
        head = buf[:148] + buf[156:]
        high = BLOCKSIZE - 8 - len(head.translate(None, _HIGH_BYTES))
        return unsigned_chksum, unsigned_chksum - 256 * high
    return orig(buf)

def header(name):
    b = bytearray(BLOCKSIZE)
    b[:len(name)] = name
    b[100:107] = b"0000644"; b[124:135] = b"00000000144"; b[148:156] = b"        "
    b[257:262] = b"ustar"
    return bytes(b)

CASES = {"ascii": header(b"config/dir001/file00001.dat"),
         "highbit": header("config/d\xe9r001/f\xefle\xff.dat".encode("latin-1"))}

def time_func(loops, func, buf):
    range_it = range(loops)
    t0 = pyperf.perf_counter()
    for _ in range_it:
        func(buf)
    return pyperf.perf_counter() - t0

if __name__ == "__main__":
    runner = pyperf.Runner()
    runner.metadata["description"] = "tarfile.calc_chksums: stock vs fast path"
    for case, buf in CASES.items():
        assert orig(buf) == patched(buf)
        for variant, func in (("stock", orig), ("patched", patched)):
            runner.bench_time_func("calc_chksums/%s/%s" % (case, variant), time_func, func, buf)

`bench_tarfile.py`

"""Stock vs patched tarfile, many workloads, paired + interleaved (ABBA), CPU and wall time, bootstrap CI.
usage: python bench_tarfile.py MODSVER [-n ROUNDS] [--quick]
MODSVER names a directory mods/<MODSVER>/{stock,patched}/tarfile.py (see build_variants.py)."""
import argparse, bz2, gzip, importlib.util, io, lzma, os, random, statistics as st, sys, time
HERE = os.path.dirname(os.path.abspath(__file__))
def load(name, path):
    spec = importlib.util.spec_from_file_location(name, path); m = importlib.util.module_from_spec(spec)
    sys.modules[name] = m; spec.loader.exec_module(m); return m
ap = argparse.ArgumentParser(); ap.add_argument("ver"); ap.add_argument("-n", type=int, default=25); ap.add_argument("--quick", action="store_true"); a = ap.parse_args()
S = load("tf_stock", "%s/mods/%s/stock/tarfile.py" % (HERE, a.ver)); P = load("tf_patched", "%s/mods/%s/patched/tarfile.py" % (HERE, a.ver))
rng = random.Random(1)
def synth(n, size, namefn, fmt=None, enc="utf-8"):
    bio = io.BytesIO(); body = bytes(rng.randrange(256) for _ in range(size)) if size < 5000 else os.urandom(size)
    with S.open(fileobj=bio, mode="w", format=fmt if fmt is not None else S.PAX_FORMAT, encoding=enc, errors="surrogateescape") as tf:
        for i in range(n):
            ti = S.TarInfo(namefn(i)); ti.size = size; ti.mtime = 1700000000 + i; ti.uid = ti.gid = 1000; ti.uname = ti.gname = "u"; ti.mode = 0o644
            tf.addfile(ti, io.BytesIO(body))
    return bio.getvalue()
ascii_name = lambda i: "config/dir%03d/file%05d.dat" % (i % 50, i)
hi_name = lambda i: "config/d\xe9r%03d/f\xefle\xff%05d.dat" % (i % 50, i)
W = []   # (label, data, mode)
raw = lambda b: b
many = synth(20000, 100, ascii_name); few = synth(20, 4_000_000, ascii_name, S.GNU_FORMAT); mid = synth(2000, 20000, ascii_name, S.USTAR_FORMAT)
hi = synth(5000, 100, hi_name, S.GNU_FORMAT, "latin-1"); hipax = synth(5000, 100, hi_name, S.PAX_FORMAT)
W += [("synth 20k x 100B pax", many, "r:"), ("synth 20k x 100B pax .gz", gzip.compress(many, 6), "r:gz"), ("synth 20k x 100B .bz2", bz2.compress(many), "r:bz2"), ("synth 20k x 100B .xz", lzma.compress(many, preset=1), "r:xz"),
      ("synth 2000 x 20KB ustar", mid, "r:"), ("synth 20 x 4MB gnu", few, "r:"), ("synth 5k hi-bit names gnu/latin1", hi, "r:"), ("synth 5k hi-bit names pax/utf8", hipax, "r:")]
if a.quick: W = W[:3] + W[-4:]
def op_read(mod, data, mode):    # mkreports-style: iterate + read every regular member
    with mod.open(fileobj=io.BytesIO(data), mode=mode) as tf:
        n = 0
        for m in tf:
            if m.isreg(): n += len(tf.extractfile(m).read())
    return n
def op_list(mod, data, mode):    # header parsing only
    with mod.open(fileobj=io.BytesIO(data), mode=mode) as tf: return len(tf.getmembers())
def op_stream(mod, data, mode):
    with mod.open(fileobj=io.BytesIO(data), mode="r|" + mode.split(":")[1] if mode != "r:" else "r|") as tf:
        n = 0
        for m in tf:
            if m.isreg(): n += len(tf.extractfile(m).read())
    return n
def op_write(mod, data, mode):   # write path: 3000 members
    bio = io.BytesIO(); body = b"x" * 100
    with mod.open(fileobj=bio, mode="w", format=mod.PAX_FORMAT) as tf:
        for i in range(3000):
            ti = mod.TarInfo(ascii_name(i)); ti.size = 100; ti.mtime = 1700000000; tf.addfile(ti, io.BytesIO(body))
    return len(bio.getvalue())
def boot(ratios, B=2000):
    r = random.Random(0); m = sorted(st.median([r.choice(ratios) for _ in ratios]) for _ in range(B)); return m[int(.025 * B)], m[int(.975 * B)]
def timeit(op, mod, data, mode, clock):
    t = clock(); op(mod, data, mode); return clock() - t
def row(label, opname, op, data, mode):
    assert op(S, data, mode) == op(P, data, mode)
    cs, cp, ws, wp = [], [], [], []
    for i in range(a.n):
        order = ((S, cs, ws), (P, cp, wp)) if i % 2 == 0 else ((P, cp, wp), (S, cs, ws))       # alternate who goes first (cancels drift/cache order)
        for mod, c, w in order:
            c.append(timeit(op, mod, data, mode, time.process_time)); w.append(timeit(op, mod, data, mode, time.perf_counter))
    pr = [x / y for x, y in zip(cs, cp)]; lo, hi_ = boot(pr)
    print("%-34s %-6s stock med %8.2fms (sd %5.2f)  patched med %8.2fms (sd %5.2f)  speedup med %.3fx  min/min %.3fx  95%%CI [%.3f, %.3f]  wall %.3fx  faster-in %d/%d" % (
        label[:34], opname, st.median(cs) * 1e3, st.pstdev(cs) * 1e3, st.median(cp) * 1e3, st.pstdev(cp) * 1e3, st.median(pr), min(cs) / min(cp), lo, hi_, st.median(ws) / st.median(wp), sum(r > 1 for r in pr), len(pr)))
    return st.median(pr)
print("python %s  modsver %s  rounds %d (CPU=process_time, paired ratios stock/patched, >1 = patched faster)" % (sys.version.split()[0], a.ver, a.n))
agg = {}
for label, data, mode in W:
    for opname, op in (("read", op_read), ("list", op_list)):
        agg.setdefault(opname, []).append(row(label, opname, op, data, mode))
for label, data, mode in W:
    if mode in ("r:gz", "r:bz2", "r:xz") and not a.quick: agg.setdefault("stream", []).append(row(label, "stream", op_stream, data, mode))
agg.setdefault("write", []).append(row("write 3000 members pax", "write", op_write, None, None))
print("geomean of median speedups:", {k: round(st.geometric_mean(v), 3) for k, v in agg.items()})

@encukou

For the sake of keep a simple code, I don't think I'm going to actually accept this change considering that, as you observed, the overall E2E operation isn't really affected.

What I find odd is that 3.15+ has a larger speed-up for the stream mode as if we were somehow slower. The speed-ups on previous versions are less pronounced.

That being said, I find it attractive as an overall speed-up but I'm not confident in accepting (I still need to convince myself that the checksum computation is equivalent which I hadn't check yet, even if it's valid with your assertions I want to convince myself of the equal arithmetic). I also wonder how it fares on real data (say, compressed Python sources) or when it comes to images.

Also, why limiting yourself to bytes of size BLOCKSIZE only? wouldn't it work for any bytes-like object (memoryviews, etc)? the only thing you need is len(head.translate(None, _HIGH_BYTES)) to count the number of occurrences of high bytes right? this can be achieved in a more generic way for sequences (or simply converting to bytes for the method).

Alternatively, since calc_chksums() is only used with a blocks of size 512, let's have a private _calc_chksums512() which would be your optimization. The public (albeit undocumented function) could still be left as is (the fast path looks weird when you check the code).

cc @encukou @StanFromIreland

@picnixz

@picnixz

I don't think speed up is indicitive that 3.15 is slower, what I measured was roughly a 55ms gain, however the baseline was different

r:gz stock → patched, 20k×100B	read	stream
3.14	208 → 158 ms	609 → 555 ms (saves 54)
3.15	196 → 141 ms	191 → 135 ms (saves 55)

On 3.14 and earlier, stream mode is about 3x slower than read, so a fixed ~55 ms saving shows as only 1.05–1.1x. On 3.15+ the stream path is fast, so the same saving shows as 1.39-1.41x.

I agree with you that we should try to keep it simple and extract it into a private function, here's the diff I ran with initially, as you'll see it introduced a regression in the reading of non-ASCII header, signed checksum and the corrupt checksum cases, which my original proposal did not. Below this section I'm leaning towards

Revised proposal: private `_calc_chksum512()`

calc_chksums() is left as is. The optimisation is a private helper used by _create_header and as a quick first check in _frombuf:

--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -232,6 +232,11 @@
     signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
     return unsigned_chksum, signed_chksum
 
+def _calc_chksum512(buf):
+    """Return the unsigned checksum of a 512-byte header block."""
+    # The checksum field (buf[148:156]) is summed as eight spaces (256).
+    return 256 + sum(buf) - sum(buf[148:156])
+
 def copyfileobj(src, dst, length=None, exception=OSError, bufsize=None):
     """Copy length bytes from fileobj src to fileobj dst.
        If length is None, copy the entire content.
@@ -1250,7 +1255,7 @@
         ]
 
         buf = struct.pack("%ds" % BLOCKSIZE, b"".join(parts))
-        chksum = calc_chksums(buf[-BLOCKSIZE:])[0]
+        chksum = _calc_chksum512(buf)
         buf = buf[:-364] + bytes("%06o\0" % chksum, "ascii") + buf[-357:]
         return buf
 
@@ -1358,7 +1363,7 @@
             raise EOFHeaderError("end of file header")
 
         chksum = nti(buf[148:156])
-        if chksum not in calc_chksums(buf):
+        if chksum != _calc_chksum512(buf) and chksum not in calc_chksums(buf):
             raise InvalidHeaderError("bad checksum")
 
         obj = cls()

`pyperf` comparison (plain build)

bench_chksum_check_pyperf.py, pyperf 2.10.0, --processes 20 --values 3, mean ± std dev. It times the checksum check _frombuf does per header (and the one _create_header does for writes) for stock vs the proposal:

case	stock	new	speedup
read, ASCII header, normal checksum	5.76 ± 0.19 µs	1.67 ± 0.14 µs	3.45x
read, non-ASCII header, normal checksum	5.85 ± 0.29 µs	1.61 ± 0.15 µs	3.63x
write (`_create_header`)	5.80 ± 0.26 µs	1.60 ± 0.18 µs	3.62x
read, non-ASCII header, signed checksum (Sun/NeXT)	5.78 ± 0.25 µs	7.46 ± 0.47 µs	0.77x
read, ASCII header, corrupt checksum	5.74 ± 0.19 µs	7.43 ± 0.48 µs	0.77x
read, non-ASCII header, corrupt checksum	5.80 ± 0.26 µs	7.40 ± 0.39 µs	0.78x

Normal archives, including non-ASCII headers, get about 3.3-3.6x faster. The cost is on the rarely used path: a header whose stored checksum is not the unsigned one (signed checksums written by old Sun/NeXT tar, or a corrupt header) pays for one extra sum() before the original function runs.

Real data (`bench_tarfile.py --lib-root`, 25 rounds, CPU time)

Every Lib/**/*.py file of the checkout (2082 files, 44.9 MB tar, 8.3 MB .gz) and the 65 image files in it (4.5 MB tar), read iterates and reads every member, list is getmembers(). stock → new, median of 25 paired rounds, the ratio is stock/new:

workload	op	plain `-O3`	PGO+LTO
Lib/*.py tar	read	50.41 → 33.66 ms (1.516x)	46.00 → 27.31 ms (1.690x)
Lib/*.py tar	list	43.92 → 27.29 ms (1.614x)	39.87 → 21.79 ms (1.831x)
Lib/*.py tar.gz	read	99.94 → 82.80 ms (1.215x)	93.43 → 74.02 ms (1.264x)
Lib/*.py tar.gz	list	91.54 → 74.57 ms (1.235x)	85.66 → 66.42 ms (1.280x)
Lib/*.py tar.xz	read	391.04 → 374.72 ms (1.044x)	384.58 → 364.85 ms (1.053x)
Lib/*.py tar.xz	list	381.49 → 364.73 ms (1.046x)	375.50 → 356.92 ms (1.053x)
images tar	read	1.62 → 1.09 ms (1.496x)	1.49 → 0.91 ms (1.636x)
images tar	list	1.38 → 0.85 ms (1.645x)	1.20 → 0.66 ms (1.805x)
images tar.gz	read	11.81 → 11.28 ms (1.047x)	11.55 → 10.97 ms (1.053x)
images tar.gz	list	11.42 → 10.91 ms (1.047x)	11.23 → 10.65 ms (1.054x)

The gain shrinks when decompression dominates (.xz, gzip of already-compressed images). Worst case, an archive where every header carries only the signed checksum:

workload	op	plain `-O3`	PGO+LTO
5k non-ASCII headers, all with signed checksums	read	62.74 → 71.27 ms (0.881x)	56.77 → 62.63 ms (0.908x)
5k non-ASCII headers, all with signed checksums	list	48.76 → 56.96 ms (0.860x)	45.78 → 51.09 ms (0.893x)

So I think best of both worlds is to keep the private function but with the original proposal for the checksum calculation

Revised proposal: private `_calc_chksums512()`

calc_chksums() is left as is. The optimisation is a private helper that returns both checksums for the 512-byte header block, and _create_header / _frombuf use it instead of calc_chksums():

--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -219,6 +219,8 @@
 
     return s
 
+_HIGH_BYTES = bytes(range(128, 256))
+
 def calc_chksums(buf):
     """Calculate the checksum for a member's header by summing up all
        characters except for the chksum field which is treated as if
@@ -232,6 +234,17 @@
     signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf))
     return unsigned_chksum, signed_chksum
 
+def _calc_chksums512(buf):
+    """Return the unsigned and signed checksums of a 512-byte header block."""
+    # The checksum field (buf[148:156]) is summed as eight spaces (256).
+    unsigned_chksum = 256 + sum(buf) - sum(buf[148:156])
+    if buf.isascii():
+        return unsigned_chksum, unsigned_chksum
+    # Each byte >= 128 adds b to the unsigned sum but b - 256 to the signed one.
+    head = buf[:148] + buf[156:]
+    high = BLOCKSIZE - 8 - len(head.translate(None, _HIGH_BYTES))
+    return unsigned_chksum, unsigned_chksum - 256 * high
+
 def copyfileobj(src, dst, length=None, exception=OSError, bufsize=None):
     """Copy length bytes from fileobj src to fileobj dst.
        If length is None, copy the entire content.
@@ -1250,7 +1263,7 @@
         ]
 
         buf = struct.pack("%ds" % BLOCKSIZE, b"".join(parts))
-        chksum = calc_chksums(buf[-BLOCKSIZE:])[0]
+        chksum = _calc_chksums512(buf)[0]
         buf = buf[:-364] + bytes("%06o\0" % chksum, "ascii") + buf[-357:]
         return buf
 
@@ -1358,7 +1371,7 @@
             raise EOFHeaderError("end of file header")
 
         chksum = nti(buf[148:156])
-        if chksum not in calc_chksums(buf):
+        if chksum not in _calc_chksums512(buf):
             raise InvalidHeaderError("bad checksum")
 
         obj = cls()

Both callers only ever pass exactly BLOCKSIZE bytes (_frombuf raises TruncatedHeaderError for any other length before it gets to the checksum, and _create_header builds its buffer with struct.pack("%ds" % BLOCKSIZE, ...)), so the helper has no type or length checks. There is no fallback path, so every header gets the fast computation: ASCII headers need one sum() and an isascii(), non-ASCII ones also count their high bytes. calc_chksums() is no longer called by tarfile itself and is left in place for compatibility.

Numbers are for CPython 3.16.0a1+dev built from main at 4a9b98115f on an Apple M5 Pro (arm64, macOS), not isolated, so compare the ratios and not the absolute times between tables. "plain" is the -O3 -DNDEBUG build, "PGO+LTO" is --enable-optimizations --with-lto (that tree also had unrelated uncommitted edits to Lib/csv.py and Lib/gettext.py, which don't touch tarfile).

`pyperf` comparison (plain build)

pyperf 2.10.0, --processes 20 --values 3, mean ± std dev. It times the checksum check _frombuf does per header (and the one _create_header does for writes), stock vs the proposal:

case	stock	new	speedup
read, ASCII header, normal checksum	3.85 ± 0.15 µs	1.12 ± 0.08 µs	3.44x
read, non-ASCII header, normal checksum	3.85 ± 0.13 µs	1.44 ± 0.05 µs	2.67x
write (`_create_header`)	3.84 ± 0.09 µs	1.09 ± 0.06 µs	3.52x
read, non-ASCII header, signed checksum (Sun/NeXT)	3.86 ± 0.15 µs	1.45 ± 0.05 µs	2.66x
read, ASCII header, corrupt checksum	3.84 ± 0.10 µs	1.12 ± 0.06 µs	3.43x
read, non-ASCII header, corrupt checksum	3.86 ± 0.11 µs	1.45 ± 0.07 µs	2.66x

Every case is faster (2.7x-3.5x), including non-ASCII headers carrying the signed checksum that old Sun/NeXT tar wrote and headers with a corrupt checksum. No case regresses.

Real data (`bench_tarfile.py --lib-root`, 25 rounds, CPU time)

Every Lib/**/*.py file of the checkout (2082 files, 44.9 MB tar, 8.3 MB .gz) and the 65 image files in it (4.5 MB tar). read iterates and reads every member, list is getmembers(). Stock → new, median of 25 paired rounds, the ratio is stock/new:

workload	op	plain `-O3`	PGO+LTO
Lib/*.py tar	read	35.06 → 23.19 ms (1.510x)	30.29 → 18.86 ms (1.598x)
Lib/*.py tar	list	31.04 → 18.85 ms (1.647x)	26.29 → 15.09 ms (1.737x)
Lib/*.py tar.gz	read	68.50 → 55.31 ms (1.239x)	61.82 → 49.81 ms (1.234x)
Lib/*.py tar.gz	list	61.84 → 49.87 ms (1.241x)	56.25 → 44.73 ms (1.254x)
Lib/*.py tar.xz	read	255.40 → 241.67 ms (1.053x)	254.17 → 241.61 ms (1.053x)
Lib/*.py tar.xz	list	249.44 → 235.26 ms (1.054x)	247.21 → 235.39 ms (1.049x)
images tar	read	1.16 → 0.78 ms (1.496x)	0.99 → 0.63 ms (1.538x)
images tar	list	0.93 → 0.57 ms (1.625x)	0.84 → 0.47 ms (1.777x)
images tar.gz	read	7.59 → 7.21 ms (1.052x)	7.51 → 7.16 ms (1.046x)
images tar.gz	list	7.34 → 6.95 ms (1.055x)	7.14 → 6.85 ms (1.043x)

The gain shrinks when decompression dominates (.xz, gzip of already-compressed images). An archive where every header carries only the signed checksum:

workload	op	plain `-O3`	PGO+LTO
5k non-ASCII headers, all with signed checksums	read	43.10 → 30.36 ms (1.420x)	38.01 → 24.95 ms (1.512x)
5k non-ASCII headers, all with signed checksums	list	34.59 → 22.05 ms (1.568x)	30.82 → 18.32 ms (1.681x)

Geometric mean of the median speedups over every row of the run (synthetic, real-data and signed-checksum archives, plus streaming and write): plain {read: 1.326, list: 1.468, stream: 1.224, write: 1.382}, PGO+LTO {read: 1.381, list: 1.525, stream: 1.245, write: 1.431}.

Checking equivalence

_calc_chksums512(buf) returns exactly calc_chksums(buf) on 1,118,530 buffers per interpreter (every byte value at every position of 7 base blocks, plus random, clustered-high-byte and k-high-byte buffers): 0 mismatches on 3.9, 3.11, 3.12, 3.13, 3.14, 3.15 and 3.16.0a1+dev.
The accept/reject decision chksum in _calc_chksums512(buf) equals the original chksum in calc_chksums(buf) for stored checksums in {unsigned, signed, unsigned+1, signed-1, unsigned+256, 0, random}: 1,808,590 decisions per interpreter, 0 mismatches.
A differential run of stock vs patched tarfile over about 667-671 thousand archive cases (ustar/gnu/pax, signed-checksum headers, bit flips, truncations, header mutations, r:*/r:/r|, gz/bz2/xz, extractall) found 0 differences on all seven interpreters, and archives written by the patched tarfile are byte-identical to stock.

if chksum != _calc_chksum512(buf) and chksum not in calc_chksums(buf):

I wasn't thinking of that. I was thinking of _calc_chksum512 doing exactly your fast path you suggested. And please, don't post very large LLM answers like that, or at loeast change the tone of the text because I'm really annoyed by how AI write in general.

_calc_chksums512(buf) returns exactly calc_chksums(buf) on 1,118,530 buffers per interpreter

I wanted a mathematical proof, not a check like that. Equivalence through examples, unless the entire space is checked is not releavnt.

For the E2E numbers: I wanted to know how long a single compress/decompress operation would take. If we are optimizing a cold path, it doesn't matter IMO. But if it's a hot path and the overall time decreases significantly and is visible, then it's ok to optimize.

Navigation Menu

Repository navigation

Speed up tarfile.calc_chksums() #160991

Feature or enhancement

Proposal:

Benchmarks

Verification

Benchmark scripts and results

Build configuration

1. `calc_chksums` microbenchmark

2. End to end (`bench_tarfile.py`)

3. Scripts

`run.sh`

`build_variants.py` (writes mods//stock|patched/tarfile.py)

`bench_calc_chksums.py`

`bench_calc_chksums_pyperf.py`

`bench_tarfile.py`

Revised proposal: private `_calc_chksum512()`

`pyperf` comparison (plain build)

Real data (`bench_tarfile.py --lib-root`, 25 rounds, CPU time)

Revised proposal: private `_calc_chksums512()`

`pyperf` comparison (plain build)

Real data (`bench_tarfile.py --lib-root`, 25 rounds, CPU time)

Checking equivalence

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Speed up tarfile.calc_chksums() #160991

Description

Feature or enhancement

Proposal:

Benchmarks

Verification

Activity

Anurag-S1ngh commented on Oct 31, 2026

picnixz commented on Oct 31, 2026

bedevere-app commented on Oct 31, 2026

turtle-byte-17 commented on Oct 31, 2026

Benchmark scripts and results

Build configuration

1. calc_chksums microbenchmark

2. End to end (bench_tarfile.py)

3. Scripts

run.sh

build_variants.py (writes mods//stock|patched/tarfile.py)

bench_calc_chksums.py

bench_calc_chksums_pyperf.py

bench_tarfile.py

picnixz commented on Oct 31, 2026

turtle-byte-17 commented on Oct 31, 2026

Revised proposal: private _calc_chksum512()

pyperf comparison (plain build)

Real data (bench_tarfile.py --lib-root, 25 rounds, CPU time)

Revised proposal: private _calc_chksums512()

pyperf comparison (plain build)

Real data (bench_tarfile.py --lib-root, 25 rounds, CPU time)

Checking equivalence

picnixz commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

1. `calc_chksums` microbenchmark

2. End to end (`bench_tarfile.py`)

`run.sh`

`build_variants.py` (writes mods//stock|patched/tarfile.py)

`bench_calc_chksums.py`

`bench_calc_chksums_pyperf.py`

`bench_tarfile.py`

Revised proposal: private `_calc_chksum512()`

`pyperf` comparison (plain build)

Real data (`bench_tarfile.py --lib-root`, 25 rounds, CPU time)

Revised proposal: private `_calc_chksums512()`

`pyperf` comparison (plain build)

Real data (`bench_tarfile.py --lib-root`, 25 rounds, CPU time)