Skip to content

Fix chop_threshold crashes in the pandas-free complex repr - #620

Open
eriknw wants to merge 3 commits into
25-pandas-free-reprfrom
39-repr-chop-complex-overflow
Open

eriknw wants to merge 3 commits into
25-pandas-free-reprfrom
39-repr-chop-complex-overflow

Conversation

@eriknw

@eriknw eriknw commented Aug 5, 2026 •

Copy link
Copy Markdown
Member

Stacked on #605. An overnight differential fuzz of #605's pandas-free renderer against the real pandas renderer (34,618 repr/HTML comparisons across 17,309 case pairs) found a crash family in the display.chop_threshold path for complex dtypes, reachable in the COO (long) display form:

  1. Tip-only crash, pandas renders fine: abs() of a python complex raises OverflowError for finite components near the float max, where numpy's abs (which pandas uses) returns inf and never chops. Fixed by treating an overflowing magnitude as never chopped; the result is byte-identical to the pandas renderer (verified, repr and HTML hashes equal against 6f1eb02).
import numpy as np, pandas as pd, graphblas as gb
big = float(np.finfo(np.float64).max)
v = gb.Vector.from_coo([10_000, 500_000], [complex(big, big), 1e-7], size=10**6, dtype="FC64")
with pd.option_context("display.chop_threshold", 1e-8):
    repr(v)   # OverflowError before this fix; byte-identical to pandas after
  1. Both-sides crash: a complex value that actually chops was replaced with float 0.0, putting a j-less string into the complex column, which _trim_zeros_complex cannot parse (IndexError). pandas 3.0.3 crashes the same way on the same input in its own chop path, so byte-identity would mean reproducing a pandas crash. This fix chops to the value's own type of zero, so the cell renders 0.000000e+00+0.000000e+00j and the object reprs where pandas cannot. A deliberate, documented divergence, reachable only with chop_threshold set on a complex dtype.

Gates: full pinned suite 1114 passed / 145 skipped; pre-commit all hooks pass. The broader fuzz findings (all in non-crash territory) are written up in the overnight stack report for the maintainer meeting.

@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 7ee751f to 44033e7 Compare August 5, 2026 17:44
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 44033e7 to 2fa3cb8 Compare August 5, 2026 18:03
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 2fa3cb8 to 376ee94 Compare August 5, 2026 18:05
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 376ee94 to e0eb986 Compare August 6, 2026 07:59
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from e0eb986 to 9d99244 Compare August 6, 2026 15:39
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 2 times, most recently from 25d557b to dbff5f6 Compare August 6, 2026 20:36
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from dbff5f6 to b022fa4 Compare August 6, 2026 20:41
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from b022fa4 to b66bc7c Compare August 7, 2026 02:49
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from b66bc7c to d09186f Compare August 7, 2026 05:09
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from d09186f to f9295a2 Compare August 26, 2026 17:30
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from f9295a2 to 9a142d0 Compare September 18, 2026 03:09
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 9a142d0 to 582a1d6 Compare September 18, 2026 03:46
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 582a1d6 to 5910201 Compare September 18, 2026 17:34
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 2 times, most recently from 363fb20 to 9fbecb9 Compare September 23, 2026 16:22
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 6 times, most recently from 7ea40c7 to 7929d00 Compare October 3, 2026 04:57
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 7929d00 to 9541cd1 Compare October 4, 2026 06:21
@eriknw
eriknw removed this pull request from stack #634 October 8, 2026 21:02
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 9541cd1 to ac94068 Compare October 8, 2026 21:40
@eriknw
eriknw added this pull request to stack #635 October 8, 2026 21:46
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from ac94068 to f0c372e Compare October 9, 2026 03:06
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 2 times, most recently from 440df4a to 887eca6 Compare October 14, 2026 18:45
@eriknw
eriknw removed this pull request from stack #635 October 19, 2026 07:45
@eriknw
eriknw changed the base branch from 38-monoid-guard-udt-fallback-builtins to main October 19, 2026 07:46
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 887eca6 to 1178c46 Compare October 19, 2026 07:47
@eriknw
eriknw changed the base branch from main to 25-pandas-free-repr October 19, 2026 07:49
@eriknw
eriknw added this pull request to stack #650 October 19, 2026 07:50
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 1178c46 to a4bdb8d Compare October 19, 2026 07:55
@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.97%. Comparing base (9d295e5) to head (92cdb6a).

Additional details and impacted files
@@                   Coverage Diff                   @@
##           25-pandas-free-repr     #620      +/-   ##
=======================================================
+ Coverage                98.77%   98.97%   +0.20%     
=======================================================
  Files                      108      108              
  Lines                    32603    32623      +20     
  Branches                  3730     3730              
=======================================================
+ Hits                     32202    32288      +86     
+ Misses                     269      213      -56     
+ Partials                   132      122      -10     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from a4bdb8d to 53ad8e3 Compare October 20, 2026 22:28
The COO form formats a complex column through _format_float_column, where
display.chop_threshold compares each value's magnitude. Two crashes hid
there, found by an overnight differential fuzz against the pandas
renderer:

- abs() of a python complex raises OverflowError for finite components
  near the float max, where numpy's abs returns inf. pandas therefore
  never chops such a value and renders the object fine; the pandas-free
  path crashed. Treat an overflowing magnitude as never chopped.
  Byte-identity for this case is restored, verified against the pandas
  renderer at 6f1eb02 (repr and HTML hashes equal).

- A value that actually chops was replaced with float 0.0, which puts a
  j-less string into the complex column, and _trim_zeros_complex cannot
  parse that (IndexError). pandas 3.0.3 has the same bug in its own chop
  path (pandas.io.formats.format._trim_zeros_complex raises the same
  IndexError on the same input), so byte-identity here means identity
  with a crash. Chop to the value's own type of zero instead: the
  chopped cell renders 0.000000e+00+0.000000e+00j and the object reprs
  where pandas cannot.

The second point is a deliberate, documented divergence from pandas,
reachable only with display.chop_threshold set on a complex dtype.
On Windows with python-suitesparse-graphblas older than 7.4.3.1 the CFFI
bindings are built without complex support (MSVC lacks C99 _Complex), so
dtypes._supports_complex is False, FC64 is never registered, and
Vector.from_coo(..., dtype="FC64") raises ValueError before the test
gets to the formatting code under test. Guard with the same
skipif("not dtypes._supports_complex") pattern test_scalar.py uses;
the module already imports dtypes, which the string condition needs.

Verified the test still runs and passes on a complex-capable host, and
that the skipif string evaluates to True in the module namespace when
_supports_complex is flipped to False.
test_chop_threshold_complex failed on the macos-latest job that drew numpy
1.24.4:

    RuntimeWarning: overflow encountered in absolute

The pandas-free float formatter takes np.abs of the column to choose between
fixed and scientific notation. For a complex value with components near the
float max the magnitude is inf, which those checks already handle, but numpy
1.24 on arm64 also raises the FP overflow flag computing it (1.26 and 2.4 do
not), so the warning escaped from repr and the test suite's filterwarnings
setting turned it into an error. Reproduced locally with numpy 1.24.4,
pandas 2.2.3 and psg 8.2.0.1, then green with the magnitudes computed under
np.errstate(over="ignore").

pandas' FloatArrayFormatter makes the same np.abs call unguarded (2.2 and
3.0), so this departs from it only in not warning.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant