Repository navigation
Conversation
eriknw
marked this pull request as ready for review
August 4, 2026 16:07
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 4, 2026 16:12
7d55403 to
d4c4481
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 5, 2026 00:06
d4c4481 to
0bd9607
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
2 times, most recently
from
August 5, 2026 17:44
479c26c to
563b80a
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 5, 2026 18:03
563b80a to
123b148
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
2 times, most recently
from
August 6, 2026 07:59
683dcc9 to
ec062f1
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 6, 2026 15:39
ec062f1 to
b8e797f
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 6, 2026 15:41
b8e797f to
17bc219
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 6, 2026 20:36
17bc219 to
da41e30
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 6, 2026 20:42
da41e30 to
15b9955
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 7, 2026 02:48
15b9955 to
02464a8
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 7, 2026 05:09
02464a8 to
2f6fb4d
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
August 26, 2026 17:30
2f6fb4d to
cdefa0b
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
September 18, 2026 03:09
cdefa0b to
dc5dbaf
Compare
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
September 18, 2026 03:46
dc5dbaf to
3b4489d
Compare
Code moves, plus import changes and a few small helpers, so scalar, vector, matrix and base read mostly as the built-in path. - core/udt.py (new): rules for UDT elements: layout, op typing, store casts, the ewise_add rule, equality without Numba, and element values as bytes. The type rules moved here from udt_utils, which now only generates code and is renamed udt_codegen. - core/literals.py (new): NEP 50 typing of literal operands, for built-in and UDT types (moved from scalar.py and udt_utils). - operator/udf.py (new): the Numba-only UDF-on-UDT block from operator/base.py, plus _bool_to_int8. - scalar._union_default replaces the ewise_union default block, which was copied six times across Scalar, Vector and Matrix. - New helpers pulled out of scalar.py's inline code: _scalar_ewise_add_converted, _udt_eq_without_numba, _element_bytes and _element_value. core/udt.py imports operator, base and scalar names inside functions, after the built-in early returns, so built-in calls pay nothing extra.
Codecov takes tokenless uploads only from forks. Every upload from this PR's CI runs was refused with "Token required because branch is protected", so the PR had no coverage report, and the PRs stacked on it had no base to compare against. The repository now has the secret.
Tests for the literal-typing paths CI never ran: - Beside built-in dtypes: a comparison op given as a string with an int out of range (ewise_union with "<"), an op string that names no BinaryOp (the op reports it), an int beyond INT64 beside a bool (OverflowError, as numpy's bools & 2**70), a user's BinaryOp with an int out of range, and a complex literal on a backend without complex types (vanilla only). - Beside UDTs: an infinite or NaN Decimal, literals not laid out like an element (ragged, too long or too short for a field, not broadcastable) or not numbers, which numpy then rejects, a number beside a bytes field, a nested record that is not the last field, and the conversions that lose a number (an imaginary part, a float32 overflow, a fraction in a record's array field) or keep a bytes field as it is. - Scalar.isequal with a ragged literal on an array UDT (ValueError).
Tests for the branches of operator/select.py's _value_select_in_own_type that CI never ran: a complex thunk beside integers and floats (only valueeq and valuene, which have an answer), a NaN thunk of another float type, an infinite thunk beside integers, and an empty thunk Scalar, which GraphBLAS rejects.
Tests for the paths in core/udt.py, Matrix and udt_codegen that CI never ran: - store casts between records whose field shapes differ or whose bytes fields differ in width (DomainMismatch saying why), a Scalar expression assigned to many elements of another UDT type, one element extracted from a Matrix into a Scalar of another type, and Matrix.power with a semiring whose result is another type; - without Numba no UDT cast compiles, so every such store raises (monkeypatched, so it runs everywhere); - Matrix.ewise_add with operands of another type than the result's (DomainMismatch, and DimensionMismatch first for other shapes), and Scalar.ewise_add with a Monoid and with a non-lifted op; - a fused store with a built-in Scalar operand, and fused stores that cannot be made (first, a user's op); - (x | y) + z of non-BOOL vectors, which reports the infix error.
Tests for operator/udf.py: a UDF on a UDT that returns None, or an array with no array UDT input, or a tuple beside a UDT that is neither a record nor an array, raises UdfParseError naming the type; built-in ops on such a UDT raise their KeyError. The one-line summary of a Numba error falls back to the first non-empty line, then to a fixed sentence.
A record with explicit offsets can have a C struct's size with a field elsewhere, as can such a record nested in one whose own offsets match, and a nested record with an array field has no C typedef. Each skips the JIT and still computes with the cfunc.
Behavior-neutral; each guard tests a state its callers never produce: - literals._check_literal_fits: every caller passes a UDT, never None. - select._value_select_in_own_type: an int too large for any float never reaches the float branch (beside a float type it is typed as that type, or float() raises first, as in numpy). - udt_codegen._c_aligned_version is only called with records. - udt_codegen._c_expr_unary: both callers pass the field's dtype. - udt_codegen._make_jit_c_comparison_definition: _udt_c_typedef has already returned None for a UDT that is neither array nor record. - udt_codegen.set_jit_c_on_op and set_jit_c_comparison_on_op: callers check _has_jit_set first, since the setter they pass does not exist without it.
- numba: _complex_times and _complex_truediv run only inside compiled code, as do two UDFs in test_op.py. - safety: guards no public call reaches: a record leaf lookup_dtype refuses (a void field fails to register first), a Numba error in the UDF probe (the wrapper has the same operand types), a tuple whose length differs from the record _resolve_udt_return_type matched, a C expression for an op or dtype no lifted op gives, and the frame walk in _warn_from_caller running off the stack. - only on failure: the udt_op_path fixture's demotion check, and the re-raise for a large UDT off Windows.
- test_dtype: the JIT name test is skipped unless _has_jit_set, which needs the suitesparse backend, so it always has gb.ss. - test_ssjit: the two complex JIT tests are skipped without Numba, so their own "numba is None" skips never ran. - test_op: test_compile_codegen_helper compiles plain Python, so its Numba skip was needless; it now runs without Numba too, which covers _compile_codegen's no-Numba namespace in CI's no-Numba jobs.
Behavior fix: wrong results before, on main too (not a #591 regression). SuiteSparse:GraphBLAS reads the thunk of a value select as the input's type when the op is typed for another type, and the workaround from 6917311 converted the thunk only for bool, integer and float inputs. On an FC32 vector of [1, 2+1j, 3, nan], select(valueeq, np.complex128(1)) kept nothing (now [0]), valueeq with np.float64(3) or np.int64(3) kept nothing (now [2]), and valuene with np.complex128(2+1j) kept everything (now [0, 2, 3]); apply with the same ops was right. (Main also got a Python 3 wrong there; #591's typing of a weak int as FC32 already fixed that.) A complex input now gets the thunk in its own type when that converts exactly; otherwise (or with a NaN part) the thunk equals no element, so valueeq keeps nothing and valuene keeps everything. "Exactly" is as for float inputs: the thunk in the input's type must equal the thunk, part by part, in Python's int and float comparison, so np.int64(2**53 + 1) equals no complex64 element, as it equals no float32 one, where apply compares in complex128 and matches 2**53. The ordered value ops still raise KeyError on complex values, which have no order. A real input with a complex thunk keeps its rule: an imaginary part of 0 converts, any other equals nothing.
Behavior fix. A UDT of bytes, dates or the like holds one value per
element. binary.eq and binary.ne on it raised TypeError ('NoneType'
object is not subscriptable), and so did Vector and Matrix isequal and
==: on main too. A Scalar's == raised it as well, where main compared
the values (a #591 regression, from 6917311's Scalar == through
binary.eq), as did == with a literal (b"ab"), which the literal checks
read as an array UDT. On the vanilla backend a complex literal is such
a UDT, so int8_vec == 1j raised the same TypeError.
eq and ne now compare the two values as numpy does, as a record field
of that type compares (bytes by value, NaT equal to nothing), and only
two elements of the one UDT compare: any other pair raises KeyError,
from _check_udt_pair, with or without Numba. So int8_vec == 1j on
vanilla is a KeyError, and so is a Scalar's == with another type
(another bytes UDT, or a number), where main compared the values (True
or False). A literal beside such a UDT is checked as one value.
A Scalar's == on such a UDT compares in numpy, as without Numba, so it
needs no Numba type for the element: a float16 UDT, which Numba cannot
compile, compares as on main. Vector and Matrix == on float16 still
need Numba and raise its NotImplementedError, as main raised there too.
Behavior fix of a #591 regression: main, which typed UDT ops otherwise, computed int8 results here. power types the semiring for A, and GraphBLAS casts a built-in operand of another type to it, but not a UDT. With a semiring whose result is another type, such as plus_truediv on an int8 array UDT (float64 results), power(2) worked, while from n = 3 on the products of float64 squares with A or with each other raised a bare GrB_DOMAIN_MISMATCH from GrB_mxm. Each product of UDT operands that are not of the types the semiring was typed for now takes the semiring typed for its own operands, as any mxm of them does. power(n) into a float64 array object now equals the power of a float64 copy of A, as for built-in INT8, for every n. Unchanged: A.power(n, plus_truediv).new() makes an object of A's int8 type, and an mxm cannot cast a UDT as it stores, so it raises DomainMismatch by #591's store rule, where main computed int8 results (built-in INT8 truncates as it stores).
Behavior fix, small: f32_vec.select(valueeq, np.float64(1e300)) gave the right result but numpy warned "overflow encountered in cast" from inside python-graphblas (an error under -W error), as the select workaround from 6917311 converted the thunk to float32. A #591 regression: main had no such conversion (it let SuiteSparse read the thunk's bytes as float32, which was wrong in another way). numpy's own f32_array == np.float64(1e300) does not warn. The conversion now ignores the overflow: the thunk is infinity in float32, which compares with every element as 1e300 does, so the selected elements are unchanged.
…keep Behavior fix. A literal that must become an element of a UDT and does not fit it by type converts, with a DeprecationWarning, only when its values convert exactly (#591's MAINT rule). The check skipped every leaf that holds no number, so whatever was given for a bytes, str, date or duration leaf counted as exact. Under a user's op, (1.5, 1) for a record of a bytes field and an int8 field became (b"1.5", 1), and 1/3 became b"0.33", with a warning that said "Its values convert exactly"; np.timedelta64(5, "ms") for a timedelta of seconds became 0 s, and a structured literal's six bytes became four, under the same warning. - main: converted silently (1.5 became b"1.5"). - 0e1ad03: a record field converted with that warning; a bare UDT of bytes or durations raised "does not fit" (the check failed on its layout). - the commit "Compare UDTs that are neither records nor arrays in eq and ne" let a bare UDT convert with the warning too. - now: such a leaf keeps a value only if it converts back to it, so a Python number, a numpy bool, a string cut short or a duration in a coarser unit raises ValueError "does not fit". bytes or str that fit still convert, as does np.timedelta64(5000, "ms") as 5 s (with the warning, since ms to s is not a safe cast). Only a literal whose type does not fit reaches this check. A numpy number that numpy casts safely to the leaf's type fits by type: np.uint8(5) beside a 3-byte bytes UDT is b"5" without a warning, by numpy's rule. bytes or str alone beside a bare bytes UDT has no type, so numpy still cuts b"abcdef" to b"abc" there. A record literal holding a numpy number, text or a date gets no type at all and so no check; the next commit types it.
Behavior fix, in #591's literal typing (older than its last commits; main had it too). A tuple or dict literal for a record UDT got no type at all when a leaf held anything but Python numbers: a numpy number, text or a date. Then nothing checked it, and numpy converted it into the record, losing value where a Python number alone raises. In an int8 record, under a user's op, (np.float64(1.5), 1) became (1, 1) while (1.5, 1) raised "does not fit"; under a lifted op, int8_rec + (np.float64(1.5), 1) computed with (1, 1), and int8_rec == (np.float64(1.5), 2) was True. A record of a bytes and an int8 field holding (b"cd", 1) had (b"cd", 1.5) == it True, and a user's op stored (b"cd", 1.5) as (b"cd", 1); a record with a date field the same. - main and 0e1ad03: all of these as above, silently. - now every leaf is typed. A numpy number or bool is strong, as alone or beside a built-in type: the leaf takes its dtype, promoted with any Python numbers in it, and a numpy number may stand for a whole nested record. Text or a date in a text or date field takes the field's own type when it converts there and back unchanged, so ops still pair the field with itself (rec + (b"ab", 1) is (b"abab", 2), and (np.timedelta64(5, "m"), 1) adds 300 s to a field of seconds, as on main); otherwise it keeps its own type, which does not fit: text at its own length, a numpy date or duration in its unit. Other dates (datetime, str) take the field's type. So the user's op raises "does not fit" for np.float64(1.5) or 1.5 in an int8 field next to text or a date, for text longer than its field (it was cut short) and for a date or duration in a finer unit that loses value (5 ms in seconds); np.int64(2) converts with the DeprecationWarning (exact, but not a safe cast) and np.int16(2) fits an int16 field; the lifted op gives a float64 field (2.5, 3), and + with 5 ms in a field of seconds raises KeyError (main added 0 s); and the comparisons are False. In a comparison, an int beyond its integer leaf's range now becomes infinity in that leaf only, not in the whole literal, so a text leaf beside it still pairs: rec == (b"ab", 300) is False (it raised). A text date is compared as a date in the exactness check. A sequence for an array UDT keeps its rule, the array numpy makes of it.
… Scalar == Behavior fix. Numba has a float16 type but no code for it, so every op compiled on a UDT with a float16 leaf (a field, an array's element, or a bare float16 UDT) failed with a bare NotImplementedError: float16. - A record with a float16 field, + (1.0,) or == (1.5,): NotImplementedError on main, at 0e1ad03 and before this commit; now KeyError. - Its Scalar == (1.5,): True on main, NotImplementedError at 0e1ad03 and before; now True again. - A bare float16 UDT, Vector == or isequal: TypeError on main and at 0e1ad03, NotImplementedError before; now KeyError. - An int8 record + (np.float16(1.5), 1) or == (np.float16(1.5), 2): main and 0e1ad03 cut 1.5 to 1 and answered (2, 3) and True, both wrong; before, KeyError ("no common type") and NotImplementedError; now KeyError. Every UDT op's _compile_udt (unary, binary, indexunary, select, indexbinary) now refuses a float16 leaf up front with the KeyError an op gives for a type it does not take: "binary.eq does not work with (...): Numba cannot compute on float16, which ... holds." A Scalar's == on a UDT with a float16 leaf compares in numpy, as for a UDT that is neither a record nor an array. A store that would cast to or from a UDT with a float16 leaf (another record or array UDT, or dup(dtype=...)), which raised the same NotImplementedError on main, at 0e1ad03 and before, now raises the DomainMismatch of a store that cannot cast, saying why: "... Numba cannot compute on float16, which ... holds." Widening float16 to float32 was not done: the result type would change, and results stored back as float16 would round.
Behavior fix. A sequence for an array UDT is typed by the rule #591 gives it: Python numbers alone are weak, each taking the element's type when its kind holds them, and a numpy array, or a sequence holding a numpy value, is the array numpy makes of it. The same sequence for an array field of a record was typed element by element, numpy values strong and Python numbers weak beside the field, and a numpy array was read as Python numbers. So, with an int8[3] array UDT and a record with an int8[3] field: - (np.int8(1), 300, 3): the array UDT gives int64[3] and computes [2, 302, 6]; the record raised OverflowError (main and 0e1ad03 too). - np.array([1, 300, 3]): the array UDT gives int64[3]; the record raised OverflowError (0e1ad03), and main wrapped 300 to 44 silently. - (np.float64(1.5), 2, 3): main and 0e1ad03 cut 1.5 to 1; the commit "Type every leaf of a record literal, numpy numbers, text and dates too" already gave a float64 field. A record's array field now takes the array UDT's rule: an array, or a sequence holding a numpy number or bool, is the array numpy makes of it, broadcast to the field's shape. So the record gives an int64[3] field and [2, 302, 6], as the array UDT does. Python numbers alone, and a numpy number for the whole field, are typed as before. A comparison keeps a Python int beyond the element's range exact, as it does everywhere (int8_vec == 300 is False), also inside such an array: the int is infinity in its place before numpy makes the array. So rec == ((np.int8(1), 2**70, 3), 4) and int8_array_udt == (np.int8(1), 2**70, 3) are False and != True. The array UDT raised numpy's OverflowError there (older than this commit), and the record field, False before, would have raised it under the array rule alone. A user's op with such a literal still raises OverflowError.
Behavior fix. binary.eq and ne compared each pair of record leaves with Numba, which cannot compare bytes with a number, so such a pair raised UdfParseError carrying Numba's typing error: - a record with a bytes and a float64 field, eq with a record of two float64 fields: GrB_DOMAIN_MISMATCH on main, UdfParseError at 0e1ad03 and before this commit; - the same record == 1 or != (1.0, 1.0), and its Scalar == (1.0, 1.0): main converted the number into the bytes field (1 became b"1") and compared the converted value; UdfParseError at 0e1ad03 and before; - a nested record == (b"x", np.float64(1)), whose number stands for a nested record holding a bytes field: False on main and at 0e1ad03, UdfParseError before this commit. Text, a date, a duration and a number now compare only with their own kind; any other pair of leaves raises the KeyError binary.eq gives for a pair it does not take: "binary.eq does not work with (...): elements of |S4 and float64 do not compare." Scalar == raises it with or without Numba. This follows the rules #591 already has: plus finds no common type for such leaves (a KeyError), and a UDT that is neither a record nor an array pairs only with its own type. Answering False instead (a number for a text leaf can never be equal) would turn the record pair, an error since main, into an answer, and would differ from main anyway where main compared b"1" with 1 as equal.
… typing
Tests only. Two branches of the record literal typing had no row:
- a date given as text shorter than numpy's own date string
("1970-01-01" for a field of seconds), which _converts_back compares
as a date: with a number that converts exactly, the literal converts
with the DeprecationWarning, and without the branch it raised "does
not fit";
- a numpy duration given for a date field, which keeps its own type:
numpy converts np.timedelta64(5, "s") to a date by value, so without
the guard it took the date field's type and failed in numpy's own
conversion; with it, it raises "does not fit".
…array Behavior fix. A user's op on a UDT of bytes, str or the like that returns an operand as it is (lambda x, y: x) failed to compile: Numba types the element as a char sequence, which no DataType is registered for, so the return type was not found. - main: UdfParseError "UDT UDF returned an unsupported type [char x 8]". - 0e1ad03: TypeError 'NoneType' object is not subscriptable (the literal beside it was read as an array UDT). - before this commit: the UdfParseError again. - now: the return takes the operand's UDT when its Numba type is that UDT's, so the op computes; [b"abc"] for x, the literal for y.
… text widths Behavior fix. Text given beside a UDT of bytes or str (not a record) had no type, so numpy cut it to the UDT's width: with an S4 element b"abcd", b"abcdef" became b"abcd". - v == b"abcdef": TypeError on main and at 0e1ad03; True before this commit; now False (!= is True). - Scalar == and Scalar.isequal with b"abcdef": True on main (isequal True at 0e1ad03 too), now False. - a user's op with b"abcdef": main and the commits before this one cut it silently (0e1ad03 raised TypeError); now ValueError "does not fit". Such text is now typed as a record's text field is: the UDT itself when it converts into it and back unchanged (b"ab", "ab" into S4, so bare_s4 == b"ab" works as before), else its own type at its own length, which does not fit. For eq and ne, that literal is text of the same kind at another width, so this commit also loosens, there and only there, the pairing rule of "Compare UDTs that are neither records nor arrays in eq and ne": two UDTs of text of the same kind (bytes with bytes, str with str) at different widths now pair under eq and ne and compare as text, as numpy compares them and as a record's S4 and S2 fields already do. An S4 and an S6 Vector compare element by element (they raised KeyError before; TypeError on main and at 0e1ad03). Bytes against str, a number, float16 or any other kind still raise the pairing KeyError, as does every other op on such a pair. Stores are unchanged: v[0] = b"abcdef" still stores b"abcd", as every literal store converts by numpy's cast (int8_vec[0] = 1.5 stores 1).
Behavior fix. numpy before 2.3 converts a NaT timedelta64 scalar into another unit as the integer NaT is stored as: NaT ms assigned into timedelta64[s] is 9223372036854774 s, and NaT minutes is 0 s. An array of durations (astype) and a date convert NaT correctly, and numpy 2.3 and later convert it correctly every way. On numpy 2.2 (numpy 1.26 converts NaT the same way), with a UDT of timedelta64[s] or a record with a timedelta64[s] field: - Scalar.from_value, v[0] = ..., Scalar.value = ..., Vector.from_coo, and the Scalar a literal becomes, given NaT ms or NaT minutes: stored 9223372036854774 s or 0 s, as on main; now NaT. - A NaT ms literal beside the bare UDT: ValueError "does not fit", since it converted to a number and so not exactly; now it converts with the DeprecationWarning, as on numpy 2.4. - (NaT ms, 1) beside the record: typed as timedelta64[ms], so binary.plus raised KeyError "no common type"; now it takes the field's type, as on numpy 2.4. The conversion into an element, and the checks of whether a literal converts exactly, now hand numpy a NaT of no unit in place of each NaT duration (alone, or in a tuple or list), which every numpy converts to NaT in any unit. numpy 2.4 gives the same results before and after.
SuiteSparse before 9 has no GrB_NAME setter, so registration names a UDT by its numpy repr, which must fit in 128 bytes, and warns when it does not; pytest makes the warning an error. test_record_layouts_c_cannot_declare_skip_jit registered its record with an array field with align=True, which gives the dict form of the repr: 143 bytes. Its float64 fields have the same offsets packed or aligned, so it is now registered packed (a 71-byte repr); a C struct still cannot declare it, and the test runs the same lines. Every other UDT the tests register with a repr over 128 bytes is in a test that ignores that warning.
"Keep a NaT duration NaT in another unit on numpy before 2.3" handed
numpy a NaT of no unit, np.timedelta64("NaT"), in place of each NaT
duration. numpy 2.5 deprecates that generic unit ("The 'generic' unit
for NumPy timedelta is deprecated"), so on numpy 2.5 every store or
literal of a NaT duration into a duration UDT or field warned with a
DeprecationWarning, which the test suite's filterwarnings makes an
error: test_udt_nat_duration_in_another_unit_stays_nat,
test_udt_sequence_and_array_literals_type_like_numbers and
test_udt_literal_converts_only_when_exact failed.
Each NaT duration is now converted into the unit of the leaf it fills
by converting it as an array (astype), which keeps NaT on every numpy,
before numpy assigns it. The value is walked beside the dtype: a tuple
of a record's length as its fields, a list (or a tuple for any other
dtype) as a sequence of elements, and an array UDT by its element type.
Nothing makes a generic unit.
numpy 2.5.4: DeprecationWarning before, NaT with no warning now. numpy
1.26, 2.2 and 2.4: NaT in the target unit before and after. New rows:
a nested record's duration field, and an array UDT of durations given
a list, where Numba is absent (Numba cannot type an array of durations,
so registering one fails with it).
test_udt_layouts_under_one_name_get_their_own_jit_c_names set jit_c_control to 'on' and compiled kernels for both layouts, so on a host whose JIT cannot compile (no C compiler, or a broken one) it failed with JitError instead of skipping. CI always installs a compiler, so CI never saw it. It now asks _enable_jit_for_udt, which probes with a real compile once per process, and skips the kernel part when the answer is no. The C name checks and the register_new check need no compiler, so they come first and run everywhere.
Behavior fix. SuiteSparse exports the iso value of an array UDT with shape (1, *subarray shape), and normalize_values broadcast it to shape + values.shape, keeping that axis of length 1. With the suitesparse backend, for an iso Vector or Matrix of an array UDT such as INT64[4] (as v[:] = [0, 1, 2, 3] makes): - Vector.to_dense: TypeError "unsupported operand type(s) for +: 'int' and 'tuple'" on main and before this commit (it passed its size as an int); now an array of shape (size, 4). - Matrix.to_dense: shape (nrows, ncols, 1, 4); now (nrows, ncols, 4). - Matrix.to_csr, to_csc, to_dcsr and to_dcsc: values of shape (nvals, 1, 4); now (nvals, 4). - Matrix.to_dicts, which uses to_dcsr or to_dcsc: each value as [[0, 1, 2, 3]]; now [0, 1, 2, 3]. Vector.to_coo, Matrix.to_coo, the vanilla backend and records were already right. normalize_values now broadcasts to shape + values.shape[1:], and Vector.to_dense passes (size,). CI found it through #652's test_udt_mixed_jit_c_name_is_its_own (17 jobs, e.g. job 122633175811): SuiteSparse before 10.5 makes a vector built from equal values iso, so that test's constant inputs gave an iso result, and to_dense raised. With 10.5 the vectors are not iso.
…rse 7
Behavior fix. register_anonymous passed a dtype's itemsize to
GrB_Type_new without a check. SuiteSparse 8 and later refuse a size of
0, but SuiteSparse 7 takes it as 1. lookup_dtype auto-registers what it
does not know, so on SuiteSparse 7 a str became the UDT "<U0", which
holds nothing.
- register_anonymous of str, bytes, np.dtype("U"), an empty record or
a subarray of length 0: InvalidValue with no message on SuiteSparse
8 and later, a UDT on 7; now ValueError "dtype must not have a size
of 0" on every version.
- lookup_dtype(str): ValueError "Unknown dtype" on 8 and later, the
"<U0" UDT on 7; now ValueError "Unknown dtype" on every version.
- Scalar.from_value("x"), Scalar + "x" and v.apply(binary.plus,
right="x"): on SuiteSparse 7, ValueError "got more than 0 values to
unpack" from cffi; now the TypeError they raise on 8 and later.
CI found it in wave 2 through #654's test_bad_ewise case (s1, "x"),
which expects the TypeError, in its one job with
python-suitesparse-graphblas 7.4.2 (122632718294).
SuiteSparse:GraphBLAS 10.5.2 fixed the bug _value_select_in_own_type works around (DrTimothyAldenDavis/GraphBLAS#463): select with a VALUE* op of another type than the input's now casts both to the op's type. Its MAINT note said to delete the function once every supported version has the fix, but that would change two results that test_select_value_thunk_of_another_type pins: with the function bypassed on 10.5.2, an INT64 thunk of 2**53 + 1 beside FP32 or FC32 values of 2**53 rounds to 2**53 in the op's type and keeps that element. The other 61 cases of the test match. The note now says so, and gives the versions with the bug as 8.0.2 through 10.5.1. Comments only.
eriknw
force-pushed
the
11-udt-record-leaf-count
branch
from
October 29, 2026 02:01
c045eea to
7efd40b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_check_udt_pairrejects two record operands that disagree on shape, but itcompared only top-level field names. Two records can share those and still
nest differently:
[("a", f8), ("b", f8)]against[("a", [("n1", f8), ("n2", f8)]), ("b", f8)]both report fields["a", "b"]while contributing two leaves and three. The codegen walks oneoperand's leaf paths and applies them to both, so applying the scalar path
["a"]to the nested side asks Numba to add a float to a record.Measured before this change, that pair gets through the checks and comes
back from Numba's typing pass as:
That reports a compile failure for what is the same shape disagreement the
three sibling checks in the same function already report as a KeyError.
Comparing leaf counts makes it the fourth of those checks.
The exception class changes, and that is visible in both directions:
UdfParseError derives from GraphblasException, not from KeyError, so code
catching KeyError to mean "no such op" now catches this pair, and code
catching UdfParseError no longer does.
OpBase.__contains__catches both,so
udt in binary.plusis unaffected either way.Added in the 2026-10-20 coverage round
CI: the workflow passes
secrets.CODECOV_TOKENto the codecov action. Codecovrefuses tokenless uploads from this repository's own branches with "Token required
because branch is protected", so this PR had no coverage report. (The input moved
here from #636, which now only pins the action's version in that step.)
The CI cap that keeps an unpinned pandas below 3.1 (conda-forge's pandas
3.1.0 fails three test_formatting.py tests) is now on main, from #658.
Coverage: tests for lines this PR adds that CI never ran (literal typing,
select's thunk rewrite, UDT store and ewise_add casts, UDF return-type errors,
JIT layout checks);
# pragma: no coverfor Numba-compiled bodies and unreachableguards; and removal of a few branches no caller can reach. Locally, coverage of
literals.py, udt.py, udt_codegen.py, udf.py and select.py went from 92-96% to
99-100%.
Test fix: test_dtype's test of two UDT layouts under one name compiled kernels with
the JIT forced on, so where the JIT cannot compile (no C compiler, or a broken one) it
failed with JitError instead of skipping; CI always has a compiler. It now asks the
one-compile probe the JIT tests use and skips only its kernel part, so its C name checks
still run everywhere.
Behavior fixes found along the way, each in its own commit with before and
after in its message:
selectwith a value op on a complex input and a thunk of another type keptthe wrong elements (main too). The thunk now compares exactly.
==,!=andisequalon a UDT that is neither a record nor an array(bytes, dates) raised TypeError (main too; Scalar
==was a regression inthis PR). They now compare values as numpy does. Such a UDT pairs only with
its own type, so a Scalar
==with a number, which main answered, raisesKeyError; the exception is two text UDTs of one kind (bytes or str) at
different widths, which compare under
==and!=.Matrix.powerwith a type-changing semiring on a UDT raisedGrB_DOMAIN_MISMATCH for n >= 3 (a regression in this PR).
selectwith a float64 thunk too large for FP32 warned about overflow (aregression in this PR).
one) would not keep (a number, a finer duration, text that is too long)
raises "does not fit" for a user op instead of converting with a loss (main
converted silently). Text too long for a bare text UDT no longer compares
equal after being cut short, as it did in main's Scalar
==.so
(np.float64(1.5), 1)no longer becomes(1, 1)in an int8 record, as itdid on main.
naming float16, and casting stores a DomainMismatch, where they raised
Numba's bare NotImplementedError (main too; a bare float16 UDT's
==raisedTypeError there). A
np.float16in an int8 record's literal, which main cutto an integer, gets the same KeyError. Scalar
==compares them in numpy, ason main.
==between leaves of different kinds (text and a number, for example)raises eq's KeyError. Visible against main: a record with a bytes field
== 1compared b"1" with 1, andnested == (b"x", np.float64(1))wasFalse; both now raise.
(bytes, str); it failed to compile (main too).
converted into a duration UDT or field:
np.timedelta64("NaT", "ms")intoseconds was 9223372036854774 s, and NaT minutes 0 s (main too). It now stays
NaT, as on numpy 2.3 and later, and a literal holding one is typed as there
rather than refused.
array UDT such as INT64[4] kept the extra length-1 axis SuiteSparse gives
its iso value (main too):
Matrix.to_densehad shape (nrows, ncols, 1, 4),to_csr,to_csc,to_dcsrandto_dcscvalues of shape (nvals, 1, 4),to_dictseach value as [[0, 1, 2, 3]], andVector.to_denseraisedTypeError. They now give (nrows, ncols, 4), (nvals, 4), [0, 1, 2, 3] and
(size, 4). CI found it through JIT-compile lifted UDT ops on mixed operand types, and find conda's compiler on Windows #652's JIT test, whose constant inputs
SuiteSparse before 10.5 builds as iso.
register_anonymousrefuses a dtype of size 0 (str, bytes, an emptyrecord, a subarray of length 0) with ValueError "dtype must not have a
size of 0" on every version. SuiteSparse 8 and later refused it with a
bare InvalidValue, but SuiteSparse 7 takes a size of 0 as 1, so there
lookup_dtype(str)registered the UDT "<U0" andScalar + "x"raised aValueError from cffi; it now raises the TypeError it raises on 8 and later.
CI found it in its one job with python-suitesparse-graphblas 7.4.2.
Element stores still use numpy's assignment cast.
Stack created with GitHub Stacks CLI • Give Feedback 💬