feat: Add catmaid spatially indexed skeletons (read-only)

[afonsobspinto]

Closes:

• https://metacell.atlassian.net/browse/NA-546
• https://metacell.atlassian.net/browse/NA-545
• https://metacell.atlassian.net/browse/NA-550
• https://metacell.atlassian.net/browse/NA-560
• https://metacell.atlassian.net/browse/NA-562
• https://metacell.atlassian.net/browse/NA-561

CATMAID Datasource and Spatially Indexed Skeletons

This document explains how the CATMAID datasource is wired up and how spatially
indexed skeletons are loaded, serialized, and rendered.

CATMAID datasource implementation (src/datasource/catmaid/)

Entry points and registration

• register_default.ts registers the provider under the catmaid scheme. The
  URL format is expected to be catmaid://<base_url>/<project_id>.

Credentials flow

• credentials_provider.ts defines CatmaidCredentialsProvider, which fetches
  an anonymous API token from <serverUrl>/accounts/anonymous-api-token.
• This provider is used via fetchOkWithCredentials in the API client to retry
  on 401/403 and request a refreshed token when needed.

API client (api.ts)

CatmaidClient wraps all CATMAID endpoints used by the data source.

Key operations:

• listSkeletons() -> list of skeleton IDs
• getSkeleton(skeletonId) -> full skeleton as nodes from
  skeletons/<id>/compact-detail
• fetchNodes(boundingBox, lod) -> spatially indexed nodes from
  node/list using msgpack; supports multi-LOD data in the response
• getDimensions() and getResolution() -> stack metadata, including voxel
  size and translation
• getGridCellSizes() -> grid cache configurations (from custom metadata object); defaults to the hard-coded
  grid size if none is present

Provider wiring (frontend.ts)

CatmaidDataSourceProvider.get() resolves the source, builds the coordinate
space, and creates the sub-sources:

1. Parse URL

   • Strips catmaid://.
   • Splits base URL and project id.
   • Ensures http prefix.

2. Create CatmaidClient and fetch metadata in parallel:

   • dimensions, resolution, gridCellSizes, skeletonIds.

3. Coordinate space setup

   • Converts resolution (nm per voxel) to meters.
   • Uses stack translation and dimension to establish voxel-space bounds.

4. Create skeleton sources

   • Spatially indexed: CatmaidMultiscaleSpatiallyIndexedSkeletonSource
     provides a multiscale slice-view chunk source for spatially indexed
     skeleton data.
   • Complete: CatmaidSkeletonSource loads full skeletons by ID.

5. Create segment properties for skeleton labels

   • Skeleton IDs are converted to BigInt and stored in a
     SegmentPropertyMap.

6. Subsources returned

   • skeletons-chunked (spatially indexed)
   • skeletons (full skeletons)
   • properties (segment labels)
   • bounds (data bounds)

Multiscale grid mapping

CatmaidMultiscaleSpatiallyIndexedSkeletonSource.getSources():

• Sorts CATMAID cache grid cell sizes from smallest to largest.
• For each grid size, builds a ChunkLayout and SliceViewChunkSpecification.
• Builds a chunkToMultiscaleTransform that scales each grid size relative to
  the smallest grid size.
• Sets useChunkSizeForScaleSelection = true so the backend can bias scale
  selection based on chunk sizes (rather than voxel size alone).

Spatially indexed skeletons (src/skeleton/)

Data model and chunk serialization

The spatially indexed path uses slice-view chunking but renders as a mesh-like
edge list.

Key data types:

• SpatiallyIndexedSkeletonChunk (frontend and backend)
  • vertexPositions (Float32 vec3)
  • vertexAttributes (packed bytes)
  • indices (Uint32 pairs, each edge is two vertex indices)
  • missingConnections (parent edges that cross chunk boundaries)
  • nodeMap (nodeId -> vertexIndex mapping)
• SkeletonChunkData serialization packs vertex positions and attributes into a
  single Uint8Array with vertexAttributeOffsets for GPU upload.

Serialization (skeleton_chunk_serialization.ts):

• Positions are always the first attribute.
• vertexAttributeOffsets describes byte offsets of each attribute.
• missingConnections and nodeMap are serialized alongside.

Chunk download (CATMAID backend)

CatmaidSpatiallyIndexedSkeletonSourceBackend.download():

• Computes a chunk-aligned bounding box from chunkGridPosition and
  chunkDataSize.
• Calls CatmaidClient.fetchNodes(bbox, currentLod).
• Builds:
  • vertexPositions: XYZ for each node.
  • vertexAttributes: segment ID only (stored as float).
  • indices: edge pairs from parent-child relations.
  • nodeMap: maps CATMAID node IDs to vertex indices.

Backend chunk scheduling and LOD

SpatiallyIndexedSkeletonRenderLayerBackend:

• Maintains skeletonLod as a shared watchable value.
• Debounces LOD changes (300 ms) to avoid invalidating the source cache on every
  slider move.
• Uses a multi-scale selection algorithm in recomputeChunkPriorities():
  • Derives pixel size from slice view parameters.
  • Selects which scales to load based on:
    • renderScaleTarget,
    • effectiveVoxelSize,
    • and optionally chunkLayout.size if
      useChunkSizeForScaleSelection is enabled.
  • Requests chunks using forEachVisibleVolumetricChunk and applies a scale
    priority offset (SCALE_PRIORITY_MULTIPLIER).

Rendering pipeline overview

Rendering is split into two paths:

1. Regular skeleton layer (SkeletonLayer)

   • Draws full skeletons (one chunk per skeleton) using the same shader stack.
   • Each chunk renders all visible segments using a single draw call per chunk.

2. Spatially indexed skeleton layer (SpatiallyIndexedSkeletonLayer)

   • Draws skeletons that are chunked spatially across the volume.
   • Supports multi-scale sources and renders each chunk per segment to ensure
     consistent color usage and visibility filtering.

Spatially indexed draw algorithm (per chunk, per segment)

SpatiallyIndexedSkeletonLayer.draw() performs per-segment rendering to ensure
correct color assignment and visibility filtering:

1. Build a global node map across all loaded chunks to resolve inter-chunk
   parent references.

2. For each chunk:

   • Read the segment ID attribute from the packed attribute buffer.
   • Group edge indices by segment ID.
   • For each segment in the chunk:
     • Check visibility (getVisibleSegments).
     • Apply objectAlpha or hiddenObjectAlpha based on visibility.
     • Build a compacted vertex list for just that segment.
     • Remap indices to the compact list.
     • Upload compacted attribute textures.
     • Draw edges and (optionally) nodes with uColor set to the segment color.

3. Per-segment caching

   • The filtered vertex textures are cached per chunk per segment by
     generation, so repeated frames can reuse GPU textures.

4. Inter-chunk edges

   • missingConnections are tracked but not rendered yet.
   • There is a TODO to build combined buffers to render those edges.

Improvements:

1) Per-draw global node map construction

Detail: SpatiallyIndexedSkeletonLayer.draw() rebuilds a global node map
every frame across all loaded chunks.

Risk: This scales with total loaded vertices and can become expensive.

2) Per-segment compaction and buffer swapping

Detail: For each segment in each chunk, the code:

• Filters vertices,
• Remaps indices,
• Builds compact textures,
• Creates a new GL buffer,
• Draws,
• Then restores the original buffers.

Risk: This is CPU-heavy and generates many short-lived GL buffers, which can
cause stalls and GC pressure for large skeleton sets.

3) Inter-chunk edges are not rendered

Detail: missingConnections are tracked, but the renderer does not draw
edges that cross chunk boundaries.

Risk: Skeletons that cross chunk boundaries appear disconnected.

[Copilot]

Pull request overview

This PR introduces support for spatially indexed skeleton rendering in Neuroglancer, specifically adding a CATMAID data source integration for loading skeletons organized in spatial chunks.

Changes:

• Adds spatially indexed skeleton sources that load skeleton data in spatial chunks (similar to volume rendering) with LOD support
• Implements CATMAID data source provider for fetching skeleton data from CATMAID servers
• Adds UI controls for hidden object opacity and skeleton LOD level

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/skeleton/skeleton_chunk_serialization.ts	New utility module for serializing skeleton chunk data for transfer between backend and frontend
src/skeleton/gpu_upload_utils.ts	New utility for uploading vertex attribute data to GPU as 1D textures
src/skeleton/frontend.ts	Extended with new spatially indexed skeleton layer classes and rendering logic
src/skeleton/base.ts	Added RPC IDs for spatially indexed skeleton layers
src/skeleton/backend.ts	Added backend implementation for spatially indexed skeleton chunks with LOD support
src/segmentation_display_state/frontend.ts	Added hiddenObjectAlpha field for controlling opacity of non-visible segments
src/layer/segmentation/layer_controls.ts	Added UI controls for hidden opacity and LOD
src/layer/segmentation/json_keys.ts	Added JSON keys for new configuration options
src/layer/segmentation/index.ts	Integrated spatially indexed skeleton layers into segmentation user layer
src/datasource/index.ts	Extended DataSubsource type to include new skeleton source types
src/datasource/graphene/frontend.ts	Fixed type checking for meshSource RPC access
src/datasource/enabled_frontend_modules.ts	Registered CATMAID frontend module
src/datasource/enabled_backend_modules.ts	Registered CATMAID backend module
src/datasource/catmaid/register_default.ts	CATMAID provider registration
src/datasource/catmaid/register_credentials_provider.ts	CATMAID credentials provider registration
src/datasource/catmaid/frontend.ts	Frontend implementation for CATMAID data source
src/datasource/catmaid/credentials_provider.ts	Credentials provider for CATMAID authentication
src/datasource/catmaid/base.ts	Base parameter classes for CATMAID sources
src/datasource/catmaid/backend.ts	Backend implementation for CATMAID skeleton fetching
src/datasource/catmaid/api.ts	CATMAID API client implementation
package.json	Added package.json exports for CATMAID modules

Comments suppressed due to low confidence (1)

src/datasource/catmaid/frontend.ts:1

• The conditional logic for determining sources is inconsistent. If mesh instanceof CatmaidMultiscaleSpatiallyIndexedSkeletonSource is true, passing allSources[0] (an array) is incorrect as SpatiallyIndexedSkeletonLayer expects a single source or an array of SliceViewSingleResolutionSource. This should pass allSources[0][0].chunkSource in both cases or adjust the constructor signature.

/**

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[afonsobspinto]

https://metacell.atlassian.net/wiki/spaces/NGLA/pages/1199767554/Neuroglancer+Catmaid+Spatially+Indexed+skeletons+diagrams