multi-array, n-dimensional widget that maps arbitrary arrays to arbitrary graphics #951

Activity

1. kushalkolar

   changed the title [-]multi-array, n-dimensional widget that maps groups of arrays to arbitrary graphics[/-] [+]multi-array, n-dimensional widget that maps arbitrary arrays to arbitrary graphics[/+] on Nov 25, 2025
2. kushalkolar

   pinned this issue on Dec 3, 2025
3. kushalkolar commented on Dec 10, 2025

   

   kushalkolar

   on Dec 10, 2025

   Last edited by kushalkolar

   MemberAuthor

   More actions

   Ideas from discussing with @clewis7 :

   NDProcessors

   An NDProcessor for each type of graphical representation, for example: images, line stacks, scatters.

   Need to think about whether a line-stack timeseries representation would be represented by the same NDProcessor class or have separate subclasses for line-stack and time-representing heatmaps.

   The time-series representations can have a "display_window" property that determines the chunk size of the currently displayed slice over a dimension (usually the time dimension). Can have options to align the graphical representation so that the current index is at the start or middle of the line-stack/heatmap's x-axis dimension.

   Scatters can be represented in different ways, need to think about how to implement these NDProcessors and graphical representations:

   • Data where one scatter is shown and a slider highlights a datapoint (or many datapoints maybe?)
   • The scatter itself changes along a dimension, ex: keypoint tracking data where n-points move in [x, y, z] space over time.

   NDWidget orchestrator class

   NDWidget manages a figure. It manages orchestration of NDProcessors and graphics, as well as the sliders and the "reference indices" that these sliders represent.

   The NDWidget can store a "reference index range" for each dimension that it manages, and these index ranges are used to set the sliders. For example, the index range could be a scientifically relevant unit such as seconds (instead of the usual frame/array index). Each NDProcessor instance has a callable that maps sliders in "reference indices" units to their data array's index. This will allow for arbitrary mapping of time (or other units) to an array index, accounting for non-uniform sampling rates or chunked recordings etc.

   NDWidget manages NDProcessors as individual units, so things like window_funcs have to be defined per-NDProcessor instance.

   Using NDWidget

   Ideas of how to populate the widget:

   ndw = NDWidget(..., **kwargs)  # kwargs passed to Figure
   ndw["subplot-name"].add_nd_image(
     data=nd_img_data,
     name="volume-movie",
     window_funcs=...,
     rgb=True,
    **kwargs  # kwargs get passed to ImageGraphic
   )

   I think the **kwargs could get passed to the Graphic that's used to display it, and all the other args go to the processor, can think and decide this later.

   Each NDProcessor could have a class attribute that defines what graphic it should use?

   class NDImageProcessor:
       graphic = ImageGraphic
   
   class NDLineStackProcessor:
       graphic = LineStack
   
   class NDHeatmapProcessor(NDLineStackProcessor):
       graphic = ImageGraphic
   
   ...

   We can have add_<nd_type> for all the default "NDProcessor-Graphic" pairs that we have premade (for images, time series line stacks or heatmaps, scatters, etc.).
   For custom NDProcessors we can maybe do something like:

   processor = CustomNDProcessor(data, ...)
   ndw["subplot-name"].add(processor, name=...)

   Setting NDProcessor or Graphic properties of managed objects in NDWidget

   An NDProcessor-Graphic pair added to a NDWidget subplot always has to be named. So then the way to access it and set things is:

   ndw["subplot-name"]["nd-data-name"].processor.window_funcs = ...
   ndw["subplot-name"]["nd-data-name"].processor.rgb = False
   
   # for line stacks or a heatmap time series
   ndw["subplot-name"]["neural-traces"].processor.display_window = 5
   
   # change the name
   ndw["subplot-name"]["neural-traces"].name = "new_name"

   Accessing the graphic could be like this maybe?

   ndw["subplot-name"]["neural-traces"].graphic.cmap = "jet"

   This may seem tedious but it makes clear separation of responsibilities.
4. kushalkolar

   self-assigned this

   on Dec 12, 2025
5. kushalkolar commented on Dec 25, 2025

   

   kushalkolar

   on Dec 25, 2025

   Last edited by kushalkolar

   MemberAuthor

   More actions

   Need to think a bit about swappable representation of time-series data as lines vs. a heatmap.

   If we have a "linestack", the dims are [n_lines, n_datapoints, 2]. Heatmaps are just images with dims of [rows, col]. We can turn a "linestack" array to a heatmap by just using the y-values, i.e. slice line stack data with: [..., 1]. However one piece of information that is lost is the x-axis positions. We could scale it like this if the values are uniform over the x-axis:

   heatmap = fig[0, 0].add_image(heatmap_data)
   heatmap.world_object.world.scale_x = line_stack_data[:, -1, 0][0] / line_stack_data.shape[1]

   Need to think about the best way to handle this. Do we just rely on the user to know if their data are uniform over x? And we apply the scaling under the hood when viewing as a heatmap? Also need to consider how to scale in x if "reference index" mapping function is used (ex, to map time units such as seconds to integer array indices).

   EDIT: I think the best option is to sample along the x-axis to create evenly spaced columns for displaying heatmaps, can just use linear interpolation. And then scale the x-dim according to the last x-axis value as I showed above.
6. kushalkolar

   closed this as completedon Sep 23, 2026
7. kushalkolar

   unpinned this issue on Sep 23, 2026