Skip to content

Reject record UDT pairs whose leaf counts differ - #591

Open
eriknw wants to merge 39 commits into
mainfrom
11-udt-record-leaf-count
Open

eriknw wants to merge 39 commits into
mainfrom
11-udt-record-leaf-count

Conversation

@eriknw

@eriknw eriknw commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

_check_udt_pair rejects two record operands that disagree on shape, but it
compared only top-level field names. Two records can share those and still
nest differently: [("a", f8), ("b", f8)] against
[("a", [("n1", f8), ("n2", f8)]), ("b", f8)] both report fields
["a", "b"] while contributing two leaves and three. The codegen walks one
operand's leaf paths and applies them to both, so applying the scalar path
["a"] to the nested side asks Numba to add a float to a record.

Measured before this change, that pair gets through the checks and comes
back from Numba's typing pass as:

UdfParseError: binary.plus does not work with (_NestFlat, _NestDeep):
No implementation of function Function(<built-in function add>) found
for signature: >>> add(float64, Record(nst_n1[type=float64;offset=0],
nst_n2[type=float64;offset=8];16;False))

That reports a compile failure for what is the same shape disagreement the
three sibling checks in the same function already report as a KeyError.
Comparing leaf counts makes it the fourth of those checks.

The exception class changes, and that is visible in both directions:
UdfParseError derives from GraphblasException, not from KeyError, so code
catching KeyError to mean "no such op" now catches this pair, and code
catching UdfParseError no longer does. OpBase.__contains__ catches both,
so udt in binary.plus is unaffected either way.

Added in the 2026-10-20 coverage round

CI: the workflow passes secrets.CODECOV_TOKEN to the codecov action. Codecov
refuses tokenless uploads from this repository's own branches with "Token required
because branch is protected", so this PR had no coverage report. (The input moved
here from #636, which now only pins the action's version in that step.)

The CI cap that keeps an unpinned pandas below 3.1 (conda-forge's pandas
3.1.0 fails three test_formatting.py tests) is now on main, from #658.

Coverage: tests for lines this PR adds that CI never ran (literal typing,
select's thunk rewrite, UDT store and ewise_add casts, UDF return-type errors,
JIT layout checks); # pragma: no cover for Numba-compiled bodies and unreachable
guards; and removal of a few branches no caller can reach. Locally, coverage of
literals.py, udt.py, udt_codegen.py, udf.py and select.py went from 92-96% to
99-100%.

Test fix: test_dtype's test of two UDT layouts under one name compiled kernels with
the JIT forced on, so where the JIT cannot compile (no C compiler, or a broken one) it
failed with JitError instead of skipping; CI always has a compiler. It now asks the
one-compile probe the JIT tests use and skips only its kernel part, so its C name checks
still run everywhere.

Behavior fixes found along the way, each in its own commit with before and
after in its message:

  • select with a value op on a complex input and a thunk of another type kept
    the wrong elements (main too). The thunk now compares exactly.
  • ==, != and isequal on a UDT that is neither a record nor an array
    (bytes, dates) raised TypeError (main too; Scalar == was a regression in
    this PR). They now compare values as numpy does. Such a UDT pairs only with
    its own type, so a Scalar == with a number, which main answered, raises
    KeyError; the exception is two text UDTs of one kind (bytes or str) at
    different widths, which compare under == and !=.
  • Matrix.power with a type-changing semiring on a UDT raised
    GrB_DOMAIN_MISMATCH for n >= 3 (a regression in this PR).
  • select with a float64 thunk too large for FP32 warned about overflow (a
    regression in this PR).
  • A literal value that a bytes, str, date or duration field (or a bare UDT of
    one) would not keep (a number, a finer duration, text that is too long)
    raises "does not fit" for a user op instead of converting with a loss (main
    converted silently). Text too long for a bare text UDT no longer compares
    equal after being cut short, as it did in main's Scalar ==.
  • Every leaf of a record literal is typed, numpy numbers, text and dates too,
    so (np.float64(1.5), 1) no longer becomes (1, 1) in an int8 record, as it
    did on main.
  • float16 leaves, which Numba cannot compute on: UDT ops raise a KeyError
    naming float16, and casting stores a DomainMismatch, where they raised
    Numba's bare NotImplementedError (main too; a bare float16 UDT's == raised
    TypeError there). A np.float16 in an int8 record's literal, which main cut
    to an integer, gets the same KeyError. Scalar == compares them in numpy, as
    on main.
  • A record's array field types its literal as an array UDT does.
  • == between leaves of different kinds (text and a number, for example)
    raises eq's KeyError. Visible against main: a record with a bytes field
    == 1 compared b"1" with 1, and nested == (b"x", np.float64(1)) was
    False; both now raise.
  • A UDF may return an operand of a UDT that is neither a record nor an array
    (bytes, str); it failed to compile (main too).
  • On numpy before 2.3, a NaT duration in another unit became a number when
    converted into a duration UDT or field: np.timedelta64("NaT", "ms") into
    seconds was 9223372036854774 s, and NaT minutes 0 s (main too). It now stays
    NaT, as on numpy 2.3 and later, and a literal holding one is typed as there
    rather than refused.
  • With the suitesparse backend, the exports of an iso Vector or Matrix of an
    array UDT such as INT64[4] kept the extra length-1 axis SuiteSparse gives
    its iso value (main too): Matrix.to_dense had shape (nrows, ncols, 1, 4),
    to_csr, to_csc, to_dcsr and to_dcsc values of shape (nvals, 1, 4),
    to_dicts each value as [[0, 1, 2, 3]], and Vector.to_dense raised
    TypeError. They now give (nrows, ncols, 4), (nvals, 4), [0, 1, 2, 3] and
    (size, 4). CI found it through JIT-compile lifted UDT ops on mixed operand types, and find conda's compiler on Windows #652's JIT test, whose constant inputs
    SuiteSparse before 10.5 builds as iso.
  • register_anonymous refuses a dtype of size 0 (str, bytes, an empty
    record, a subarray of length 0) with ValueError "dtype must not have a
    size of 0" on every version. SuiteSparse 8 and later refused it with a
    bare InvalidValue, but SuiteSparse 7 takes a size of 0 as 1, so there
    lookup_dtype(str) registered the UDT "<U0" and Scalar + "x" raised a
    ValueError from cffi; it now raises the TypeError it raises on 8 and later.
    CI found it in its one job with python-suitesparse-graphblas 7.4.2.

Element stores still use numpy's assignment cast.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@eriknw
eriknw marked this pull request as ready for review August 4, 2026 16:07
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from 7d55403 to d4c4481 Compare August 4, 2026 16:12
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from d4c4481 to 0bd9607 Compare August 5, 2026 00:06
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch 2 times, most recently from 479c26c to 563b80a Compare August 5, 2026 17:44
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from 563b80a to 123b148 Compare August 5, 2026 18:03
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch 2 times, most recently from 683dcc9 to ec062f1 Compare August 6, 2026 07:59
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from ec062f1 to b8e797f Compare August 6, 2026 15:39
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from b8e797f to 17bc219 Compare August 6, 2026 15:41
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from 17bc219 to da41e30 Compare August 6, 2026 20:36
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from da41e30 to 15b9955 Compare August 6, 2026 20:42
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from 15b9955 to 02464a8 Compare August 7, 2026 02:48
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from 02464a8 to 2f6fb4d Compare August 7, 2026 05:09
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from 2f6fb4d to cdefa0b Compare August 26, 2026 17:30
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from cdefa0b to dc5dbaf Compare September 18, 2026 03:09
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from dc5dbaf to 3b4489d Compare September 18, 2026 03:46
eriknw added 29 commits October 28, 2026 20:21
Code moves, plus import changes and a few small helpers, so scalar,
vector, matrix and base read mostly as the built-in path.

- core/udt.py (new): rules for UDT elements: layout, op typing,
  store casts, the ewise_add rule, equality without Numba, and
  element values as bytes. The type rules moved here from udt_utils,
  which now only generates code and is renamed udt_codegen.
- core/literals.py (new): NEP 50 typing of literal operands, for
  built-in and UDT types (moved from scalar.py and udt_utils).
- operator/udf.py (new): the Numba-only UDF-on-UDT block from
  operator/base.py, plus _bool_to_int8.
- scalar._union_default replaces the ewise_union default block, which
  was copied six times across Scalar, Vector and Matrix.
- New helpers pulled out of scalar.py's inline code:
  _scalar_ewise_add_converted, _udt_eq_without_numba, _element_bytes
  and _element_value.

core/udt.py imports operator, base and scalar names inside functions,
after the built-in early returns, so built-in calls pay nothing extra.
Codecov takes tokenless uploads only from forks. Every upload from this
PR's CI runs was refused with "Token required because branch is
protected", so the PR had no coverage report, and the PRs stacked on it
had no base to compare against. The repository now has the secret.
Tests for the literal-typing paths CI never ran:

- Beside built-in dtypes: a comparison op given as a string with an
  int out of range (ewise_union with "<"), an op string that names no
  BinaryOp (the op reports it), an int beyond INT64 beside a bool
  (OverflowError, as numpy's bools & 2**70), a user's BinaryOp with an
  int out of range, and a complex literal on a backend without complex
  types (vanilla only).
- Beside UDTs: an infinite or NaN Decimal, literals not laid out like
  an element (ragged, too long or too short for a field, not
  broadcastable) or not numbers, which numpy then rejects, a number
  beside a bytes field, a nested record that is not the last field,
  and the conversions that lose a number (an imaginary part, a float32
  overflow, a fraction in a record's array field) or keep a bytes field
  as it is.
- Scalar.isequal with a ragged literal on an array UDT (ValueError).
Tests for the branches of operator/select.py's
_value_select_in_own_type that CI never ran: a complex thunk beside
integers and floats (only valueeq and valuene, which have an answer),
a NaN thunk of another float type, an infinite thunk beside integers,
and an empty thunk Scalar, which GraphBLAS rejects.
Tests for the paths in core/udt.py, Matrix and udt_codegen that CI
never ran:

- store casts between records whose field shapes differ or whose
  bytes fields differ in width (DomainMismatch saying why), a Scalar
  expression assigned to many elements of another UDT type, one
  element extracted from a Matrix into a Scalar of another type, and
  Matrix.power with a semiring whose result is another type;
- without Numba no UDT cast compiles, so every such store raises
  (monkeypatched, so it runs everywhere);
- Matrix.ewise_add with operands of another type than the result's
  (DomainMismatch, and DimensionMismatch first for other shapes), and
  Scalar.ewise_add with a Monoid and with a non-lifted op;
- a fused store with a built-in Scalar operand, and fused stores that
  cannot be made (first, a user's op);
- (x | y) + z of non-BOOL vectors, which reports the infix error.
Tests for operator/udf.py: a UDF on a UDT that returns None, or an
array with no array UDT input, or a tuple beside a UDT that is neither
a record nor an array, raises UdfParseError naming the type; built-in
ops on such a UDT raise their KeyError. The one-line summary of a
Numba error falls back to the first non-empty line, then to a fixed
sentence.
A record with explicit offsets can have a C struct's size with a field
elsewhere, as can such a record nested in one whose own offsets match,
and a nested record with an array field has no C typedef. Each skips
the JIT and still computes with the cfunc.
Behavior-neutral; each guard tests a state its callers never produce:

- literals._check_literal_fits: every caller passes a UDT, never None.
- select._value_select_in_own_type: an int too large for any float
  never reaches the float branch (beside a float type it is typed as
  that type, or float() raises first, as in numpy).
- udt_codegen._c_aligned_version is only called with records.
- udt_codegen._c_expr_unary: both callers pass the field's dtype.
- udt_codegen._make_jit_c_comparison_definition: _udt_c_typedef has
  already returned None for a UDT that is neither array nor record.
- udt_codegen.set_jit_c_on_op and set_jit_c_comparison_on_op: callers
  check _has_jit_set first, since the setter they pass does not exist
  without it.
- numba: _complex_times and _complex_truediv run only inside compiled
  code, as do two UDFs in test_op.py.
- safety: guards no public call reaches: a record leaf lookup_dtype
  refuses (a void field fails to register first), a Numba error in the
  UDF probe (the wrapper has the same operand types), a tuple whose
  length differs from the record _resolve_udt_return_type matched, a
  C expression for an op or dtype no lifted op gives, and the frame
  walk in _warn_from_caller running off the stack.
- only on failure: the udt_op_path fixture's demotion check, and the
  re-raise for a large UDT off Windows.
- test_dtype: the JIT name test is skipped unless _has_jit_set, which
  needs the suitesparse backend, so it always has gb.ss.
- test_ssjit: the two complex JIT tests are skipped without Numba, so
  their own "numba is None" skips never ran.
- test_op: test_compile_codegen_helper compiles plain Python, so its
  Numba skip was needless; it now runs without Numba too, which covers
  _compile_codegen's no-Numba namespace in CI's no-Numba jobs.
Behavior fix: wrong results before, on main too (not a #591
regression). SuiteSparse:GraphBLAS reads the thunk of a value select as
the input's type when the op is typed for another type, and the
workaround from 6917311 converted the thunk only for bool, integer and
float inputs. On an FC32 vector of [1, 2+1j, 3, nan],
select(valueeq, np.complex128(1)) kept nothing (now [0]), valueeq with
np.float64(3) or np.int64(3) kept nothing (now [2]), and valuene with
np.complex128(2+1j) kept everything (now [0, 2, 3]); apply with the same
ops was right. (Main also got a Python 3 wrong there; #591's typing of
a weak int as FC32 already fixed that.)

A complex input now gets the thunk in its own type when that converts
exactly; otherwise (or with a NaN part) the thunk equals no element, so
valueeq keeps nothing and valuene keeps everything. "Exactly" is as for
float inputs: the thunk in the input's type must equal the thunk, part
by part, in Python's int and float comparison, so np.int64(2**53 + 1)
equals no complex64 element, as it equals no float32 one, where apply
compares in complex128 and matches 2**53. The ordered value ops still raise
KeyError on complex values, which have no order. A real input with a
complex thunk keeps its rule: an imaginary part of 0 converts, any
other equals nothing.
Behavior fix. A UDT of bytes, dates or the like holds one value per
element. binary.eq and binary.ne on it raised TypeError ('NoneType'
object is not subscriptable), and so did Vector and Matrix isequal and
==: on main too. A Scalar's == raised it as well, where main compared
the values (a #591 regression, from 6917311's Scalar == through
binary.eq), as did == with a literal (b"ab"), which the literal checks
read as an array UDT. On the vanilla backend a complex literal is such
a UDT, so int8_vec == 1j raised the same TypeError.

eq and ne now compare the two values as numpy does, as a record field
of that type compares (bytes by value, NaT equal to nothing), and only
two elements of the one UDT compare: any other pair raises KeyError,
from _check_udt_pair, with or without Numba. So int8_vec == 1j on
vanilla is a KeyError, and so is a Scalar's == with another type
(another bytes UDT, or a number), where main compared the values (True
or False). A literal beside such a UDT is checked as one value.

A Scalar's == on such a UDT compares in numpy, as without Numba, so it
needs no Numba type for the element: a float16 UDT, which Numba cannot
compile, compares as on main. Vector and Matrix == on float16 still
need Numba and raise its NotImplementedError, as main raised there too.
Behavior fix of a #591 regression: main, which typed UDT ops
otherwise, computed int8 results here. power types the semiring for A,
and GraphBLAS casts a built-in operand of another type to it, but not a
UDT. With a semiring whose result is another type, such as plus_truediv
on an int8 array UDT (float64 results), power(2) worked, while from
n = 3 on the products of float64 squares with A or with each other
raised a bare GrB_DOMAIN_MISMATCH from GrB_mxm.

Each product of UDT operands that are not of the types the semiring
was typed for now takes the semiring typed for its own operands, as any
mxm of them does. power(n) into a float64 array object now equals the
power of a float64 copy of A, as for built-in INT8, for every n.

Unchanged: A.power(n, plus_truediv).new() makes an object of A's int8
type, and an mxm cannot cast a UDT as it stores, so it raises
DomainMismatch by #591's store rule, where main computed int8 results
(built-in INT8 truncates as it stores).
Behavior fix, small: f32_vec.select(valueeq, np.float64(1e300)) gave the
right result but numpy warned "overflow encountered in cast" from
inside python-graphblas (an error under -W error), as the select
workaround from 6917311 converted the thunk to float32. A #591
regression: main had no such conversion (it let SuiteSparse read the
thunk's bytes as float32, which was wrong in another way). numpy's own
f32_array == np.float64(1e300) does not warn.

The conversion now ignores the overflow: the thunk is infinity in
float32, which compares with every element as 1e300 does, so the
selected elements are unchanged.
…keep

Behavior fix. A literal that must become an element of a UDT and does
not fit it by type converts, with a DeprecationWarning, only when its
values convert exactly (#591's MAINT rule). The check skipped every
leaf that holds no number, so whatever was given for a bytes, str, date
or duration leaf counted as exact. Under a user's op, (1.5, 1) for a
record of a bytes field and an int8 field became (b"1.5", 1), and 1/3
became b"0.33", with a warning that said "Its values convert exactly";
np.timedelta64(5, "ms") for a timedelta of seconds became 0 s, and a
structured literal's six bytes became four, under the same warning.

- main: converted silently (1.5 became b"1.5").
- 0e1ad03: a record field converted with that warning; a bare UDT of
  bytes or durations raised "does not fit" (the check failed on its
  layout).
- the commit "Compare UDTs that are neither records nor arrays in eq
  and ne" let a bare UDT convert with the warning too.
- now: such a leaf keeps a value only if it converts back to it, so a
  Python number, a numpy bool, a string cut short or a duration in a
  coarser unit raises ValueError "does not fit". bytes or str that fit
  still convert, as does np.timedelta64(5000, "ms") as 5 s (with the
  warning, since ms to s is not a safe cast).

Only a literal whose type does not fit reaches this check. A numpy
number that numpy casts safely to the leaf's type fits by type: np.uint8(5)
beside a 3-byte bytes UDT is b"5" without a warning, by numpy's rule.
bytes or str alone beside a bare bytes UDT has no type, so numpy still
cuts b"abcdef" to b"abc" there. A record literal holding a numpy
number, text or a date gets no type at all and so no check; the next
commit types it.
Behavior fix, in #591's literal typing (older than its last commits;
main had it too). A tuple or dict literal for a record UDT got no type
at all when a leaf held anything but Python numbers: a numpy number,
text or a date. Then nothing checked it, and numpy converted it into
the record, losing value where a Python number alone raises. In an int8
record, under a user's op, (np.float64(1.5), 1) became (1, 1) while
(1.5, 1) raised "does not fit"; under a lifted op,
int8_rec + (np.float64(1.5), 1) computed with (1, 1), and
int8_rec == (np.float64(1.5), 2) was True. A record of a bytes and an
int8 field holding (b"cd", 1) had (b"cd", 1.5) == it True, and a user's
op stored (b"cd", 1.5) as (b"cd", 1); a record with a date field the
same.

- main and 0e1ad03: all of these as above, silently.
- now every leaf is typed. A numpy number or bool is strong, as alone
  or beside a built-in type: the leaf takes its dtype, promoted with any
  Python numbers in it, and a numpy number may stand for a whole nested
  record. Text or a date in a text or date field takes the field's own
  type when it converts there and back unchanged, so ops still pair the
  field with itself (rec + (b"ab", 1) is (b"abab", 2), and
  (np.timedelta64(5, "m"), 1) adds 300 s to a field of seconds, as on
  main); otherwise it keeps its own type, which does not fit: text at
  its own length, a numpy date or duration in its unit. Other dates
  (datetime, str) take the field's type. So the user's op raises "does
  not fit" for np.float64(1.5) or 1.5 in an int8 field next to text or
  a date, for text longer than its field (it was cut short) and for a
  date or duration in a finer unit that loses value (5 ms in seconds);
  np.int64(2) converts with the DeprecationWarning (exact, but not a
  safe cast) and np.int16(2) fits an int16 field; the lifted op gives a
  float64 field (2.5, 3), and + with 5 ms in a field of seconds raises
  KeyError (main added 0 s); and the comparisons are False.

In a comparison, an int beyond its integer leaf's range now becomes
infinity in that leaf only, not in the whole literal, so a text leaf
beside it still pairs: rec == (b"ab", 300) is False (it raised). A text
date is compared as a date in the exactness check.

A sequence for an array UDT keeps its rule, the array numpy makes of
it.
… Scalar ==

Behavior fix. Numba has a float16 type but no code for it, so every op
compiled on a UDT with a float16 leaf (a field, an array's element, or
a bare float16 UDT) failed with a bare NotImplementedError: float16.

- A record with a float16 field, + (1.0,) or == (1.5,): NotImplementedError
  on main, at 0e1ad03 and before this commit; now KeyError.
- Its Scalar == (1.5,): True on main, NotImplementedError at 0e1ad03
  and before; now True again.
- A bare float16 UDT, Vector == or isequal: TypeError on main and at
  0e1ad03, NotImplementedError before; now KeyError.
- An int8 record + (np.float16(1.5), 1) or == (np.float16(1.5), 2): main
  and 0e1ad03 cut 1.5 to 1 and answered (2, 3) and True, both wrong;
  before, KeyError ("no common type") and NotImplementedError; now
  KeyError.

Every UDT op's _compile_udt (unary, binary, indexunary, select,
indexbinary) now refuses a float16 leaf up front with the KeyError an
op gives for a type it does not take: "binary.eq does not work with
(...): Numba cannot compute on float16, which ... holds." A Scalar's ==
on a UDT with a float16 leaf compares in numpy, as for a UDT that is
neither a record nor an array. A store that would cast to or from a UDT
with a float16 leaf (another record or array UDT, or dup(dtype=...)),
which raised the same NotImplementedError on main, at 0e1ad03 and
before, now raises the DomainMismatch of a store that cannot cast,
saying why: "... Numba cannot compute on float16, which ... holds."
Widening float16 to float32 was not done: the result type would change,
and results stored back as float16 would round.
Behavior fix. A sequence for an array UDT is typed by the rule #591
gives it: Python numbers alone are weak, each taking the element's
type when its kind holds them, and a numpy array, or a sequence holding
a numpy value, is the array numpy makes of it. The same sequence for an
array field of a record was typed element by element, numpy values
strong and Python numbers weak beside the field, and a numpy array was
read as Python numbers. So, with an int8[3] array UDT and a record with
an int8[3] field:

- (np.int8(1), 300, 3): the array UDT gives int64[3] and computes
  [2, 302, 6]; the record raised OverflowError (main and 0e1ad03 too).
- np.array([1, 300, 3]): the array UDT gives int64[3]; the record
  raised OverflowError (0e1ad03), and main wrapped 300 to 44 silently.
- (np.float64(1.5), 2, 3): main and 0e1ad03 cut 1.5 to 1; the commit
  "Type every leaf of a record literal, numpy numbers, text and dates
  too" already gave a float64 field.

A record's array field now takes the array UDT's rule: an array, or a
sequence holding a numpy number or bool, is the array numpy makes of
it, broadcast to the field's shape. So the record gives an int64[3]
field and [2, 302, 6], as the array UDT does. Python numbers alone, and
a numpy number for the whole field, are typed as before.

A comparison keeps a Python int beyond the element's range exact, as it
does everywhere (int8_vec == 300 is False), also inside such an array:
the int is infinity in its place before numpy makes the array. So
rec == ((np.int8(1), 2**70, 3), 4) and int8_array_udt ==
(np.int8(1), 2**70, 3) are False and != True. The array UDT raised
numpy's OverflowError there (older than this commit), and the record
field, False before, would have raised it under the array rule alone.
A user's op with such a literal still raises OverflowError.
Behavior fix. binary.eq and ne compared each pair of record leaves with
Numba, which cannot compare bytes with a number, so such a pair raised
UdfParseError carrying Numba's typing error:

- a record with a bytes and a float64 field, eq with a record of two
  float64 fields: GrB_DOMAIN_MISMATCH on main, UdfParseError at
  0e1ad03 and before this commit;
- the same record == 1 or != (1.0, 1.0), and its Scalar == (1.0, 1.0):
  main converted the number into the bytes field (1 became b"1") and
  compared the converted value; UdfParseError at 0e1ad03 and before;
- a nested record == (b"x", np.float64(1)), whose number stands for a
  nested record holding a bytes field: False on main and at 0e1ad03,
  UdfParseError before this commit.

Text, a date, a duration and a number now compare only with their own
kind; any other pair of leaves raises the KeyError binary.eq gives for
a pair it does not take: "binary.eq does not work with (...): elements
of |S4 and float64 do not compare." Scalar == raises it with or without
Numba. This follows the rules #591 already has: plus finds no common
type for such leaves (a KeyError), and a UDT that is neither a record
nor an array pairs only with its own type. Answering False instead
(a number for a text leaf can never be equal) would turn the record
pair, an error since main, into an answer, and would differ from main
anyway where main compared b"1" with 1 as equal.
… typing

Tests only. Two branches of the record literal typing had no row:

- a date given as text shorter than numpy's own date string
  ("1970-01-01" for a field of seconds), which _converts_back compares
  as a date: with a number that converts exactly, the literal converts
  with the DeprecationWarning, and without the branch it raised "does
  not fit";
- a numpy duration given for a date field, which keeps its own type:
  numpy converts np.timedelta64(5, "s") to a date by value, so without
  the guard it took the date field's type and failed in numpy's own
  conversion; with it, it raises "does not fit".
…array

Behavior fix. A user's op on a UDT of bytes, str or the like that
returns an operand as it is (lambda x, y: x) failed to compile: Numba
types the element as a char sequence, which no DataType is registered
for, so the return type was not found.

- main: UdfParseError "UDT UDF returned an unsupported type [char x 8]".
- 0e1ad03: TypeError 'NoneType' object is not subscriptable (the
  literal beside it was read as an array UDT).
- before this commit: the UdfParseError again.
- now: the return takes the operand's UDT when its Numba type is that
  UDT's, so the op computes; [b"abc"] for x, the literal for y.
… text widths

Behavior fix. Text given beside a UDT of bytes or str (not a record)
had no type, so numpy cut it to the UDT's width: with an S4 element
b"abcd", b"abcdef" became b"abcd".

- v == b"abcdef": TypeError on main and at 0e1ad03; True before this
  commit; now False (!= is True).
- Scalar == and Scalar.isequal with b"abcdef": True on main (isequal
  True at 0e1ad03 too), now False.
- a user's op with b"abcdef": main and the commits before this one cut
  it silently (0e1ad03 raised TypeError); now ValueError "does not fit".

Such text is now typed as a record's text field is: the UDT itself
when it converts into it and back unchanged (b"ab", "ab" into S4, so
bare_s4 == b"ab" works as before), else its own type at its own length,
which does not fit. For eq and ne, that literal is text of the same
kind at another width, so this commit also loosens, there and only
there, the pairing rule of "Compare UDTs that are neither records nor
arrays in eq and ne": two UDTs of text of the same kind (bytes with
bytes, str with str) at different widths now pair under eq and ne and
compare as text, as numpy compares them and as a record's S4 and S2
fields already do. An S4 and an S6 Vector compare element by element
(they raised KeyError before; TypeError on main and at 0e1ad03).
Bytes against str, a number, float16 or any other kind still raise the
pairing KeyError, as does every other op on such a pair.

Stores are unchanged: v[0] = b"abcdef" still stores b"abcd", as every
literal store converts by numpy's cast (int8_vec[0] = 1.5 stores 1).
Behavior fix. numpy before 2.3 converts a NaT timedelta64 scalar into
another unit as the integer NaT is stored as: NaT ms assigned into
timedelta64[s] is 9223372036854774 s, and NaT minutes is 0 s. An array
of durations (astype) and a date convert NaT correctly, and numpy 2.3
and later convert it correctly every way.

On numpy 2.2 (numpy 1.26 converts NaT the same way), with a UDT of
timedelta64[s] or a record with a timedelta64[s] field:

- Scalar.from_value, v[0] = ..., Scalar.value = ..., Vector.from_coo,
  and the Scalar a literal becomes, given NaT ms or NaT minutes: stored
  9223372036854774 s or 0 s, as on main; now NaT.
- A NaT ms literal beside the bare UDT: ValueError "does not fit", since
  it converted to a number and so not exactly; now it converts with the
  DeprecationWarning, as on numpy 2.4.
- (NaT ms, 1) beside the record: typed as timedelta64[ms], so
  binary.plus raised KeyError "no common type"; now it takes the field's
  type, as on numpy 2.4.

The conversion into an element, and the checks of whether a literal
converts exactly, now hand numpy a NaT of no unit in place of each NaT
duration (alone, or in a tuple or list), which every numpy converts to
NaT in any unit. numpy 2.4 gives the same results before and after.
SuiteSparse before 9 has no GrB_NAME setter, so registration names a
UDT by its numpy repr, which must fit in 128 bytes, and warns when it
does not; pytest makes the warning an error.
test_record_layouts_c_cannot_declare_skip_jit registered its record
with an array field with align=True, which gives the dict form of the
repr: 143 bytes. Its float64 fields have the same offsets packed or
aligned, so it is now registered packed (a 71-byte repr); a C struct
still cannot declare it, and the test runs the same lines.

Every other UDT the tests register with a repr over 128 bytes is in a
test that ignores that warning.
"Keep a NaT duration NaT in another unit on numpy before 2.3" handed
numpy a NaT of no unit, np.timedelta64("NaT"), in place of each NaT
duration. numpy 2.5 deprecates that generic unit ("The 'generic' unit
for NumPy timedelta is deprecated"), so on numpy 2.5 every store or
literal of a NaT duration into a duration UDT or field warned with a
DeprecationWarning, which the test suite's filterwarnings makes an
error: test_udt_nat_duration_in_another_unit_stays_nat,
test_udt_sequence_and_array_literals_type_like_numbers and
test_udt_literal_converts_only_when_exact failed.

Each NaT duration is now converted into the unit of the leaf it fills
by converting it as an array (astype), which keeps NaT on every numpy,
before numpy assigns it. The value is walked beside the dtype: a tuple
of a record's length as its fields, a list (or a tuple for any other
dtype) as a sequence of elements, and an array UDT by its element type.
Nothing makes a generic unit.

numpy 2.5.4: DeprecationWarning before, NaT with no warning now. numpy
1.26, 2.2 and 2.4: NaT in the target unit before and after. New rows:
a nested record's duration field, and an array UDT of durations given
a list, where Numba is absent (Numba cannot type an array of durations,
so registering one fails with it).
test_udt_layouts_under_one_name_get_their_own_jit_c_names set
jit_c_control to 'on' and compiled kernels for both layouts, so on a host
whose JIT cannot compile (no C compiler, or a broken one) it failed with
JitError instead of skipping. CI always installs a compiler, so CI never
saw it.

It now asks _enable_jit_for_udt, which probes with a real compile once
per process, and skips the kernel part when the answer is no. The C name
checks and the register_new check need no compiler, so they come first
and run everywhere.
Behavior fix. SuiteSparse exports the iso value of an array UDT with
shape (1, *subarray shape), and normalize_values broadcast it to
shape + values.shape, keeping that axis of length 1. With the
suitesparse backend, for an iso Vector or Matrix of an array UDT such
as INT64[4] (as v[:] = [0, 1, 2, 3] makes):

- Vector.to_dense: TypeError "unsupported operand type(s) for +: 'int'
  and 'tuple'" on main and before this commit (it passed its size as
  an int); now an array of shape (size, 4).
- Matrix.to_dense: shape (nrows, ncols, 1, 4); now (nrows, ncols, 4).
- Matrix.to_csr, to_csc, to_dcsr and to_dcsc: values of shape
  (nvals, 1, 4); now (nvals, 4).
- Matrix.to_dicts, which uses to_dcsr or to_dcsc: each value as
  [[0, 1, 2, 3]]; now [0, 1, 2, 3].

Vector.to_coo, Matrix.to_coo, the vanilla backend and records were
already right. normalize_values now broadcasts to
shape + values.shape[1:], and Vector.to_dense passes (size,).

CI found it through #652's test_udt_mixed_jit_c_name_is_its_own (17
jobs, e.g. job 122633175811): SuiteSparse before 10.5 makes a vector
built from equal values iso, so that test's constant inputs gave an
iso result, and to_dense raised. With 10.5 the vectors are not iso.
…rse 7

Behavior fix. register_anonymous passed a dtype's itemsize to
GrB_Type_new without a check. SuiteSparse 8 and later refuse a size of
0, but SuiteSparse 7 takes it as 1. lookup_dtype auto-registers what it
does not know, so on SuiteSparse 7 a str became the UDT "<U0", which
holds nothing.

- register_anonymous of str, bytes, np.dtype("U"), an empty record or
  a subarray of length 0: InvalidValue with no message on SuiteSparse
  8 and later, a UDT on 7; now ValueError "dtype must not have a size
  of 0" on every version.
- lookup_dtype(str): ValueError "Unknown dtype" on 8 and later, the
  "<U0" UDT on 7; now ValueError "Unknown dtype" on every version.
- Scalar.from_value("x"), Scalar + "x" and v.apply(binary.plus,
  right="x"): on SuiteSparse 7, ValueError "got more than 0 values to
  unpack" from cffi; now the TypeError they raise on 8 and later.

CI found it in wave 2 through #654's test_bad_ewise case (s1, "x"),
which expects the TypeError, in its one job with
python-suitesparse-graphblas 7.4.2 (122632718294).
SuiteSparse:GraphBLAS 10.5.2 fixed the bug _value_select_in_own_type
works around (DrTimothyAldenDavis/GraphBLAS#463): select with a VALUE*
op of another type than the input's now casts both to the op's type.
Its MAINT note said to delete the function once every supported version
has the fix, but that would change two results that
test_select_value_thunk_of_another_type pins: with the function
bypassed on 10.5.2, an INT64 thunk of 2**53 + 1 beside FP32 or FC32
values of 2**53 rounds to 2**53 in the op's type and keeps that element.
The other 61 cases of the test match. The note now says so, and gives
the versions with the bug as 8.0.2 through 10.5.1. Comments only.
@eriknw
eriknw force-pushed the 11-udt-record-leaf-count branch from c045eea to 7efd40b Compare October 29, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant