Skip to content

Commit 7254654

Browse files
committed
docs: add file extension filtering documentation for streaming backends
Document the SUPPORTED_EXTENSIONS pattern and _filter_streamable_files() method for streaming backends. Include examples for: - Declaring supported extensions - Using the filtering method in save_batch() - Adding/removing format support - Creating restricted backends This helps developers understand how to properly filter files in custom streaming backends to avoid crashes with incompatible formats.
1 parent a4326f6 commit 7254654

1 file changed

Lines changed: 74 additions & 0 deletions

File tree

docs/source/guides/custom_backends.rst

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,79 @@ Write comprehensive tests for your backend:
434434
with pytest.raises(FileNotFoundError):
435435
self.backend.load("nonexistent.npy")
436436
437+
File Extension Filtering (Streaming Backends)
438+
---------------------------------------------
439+
440+
Streaming backends (Napari, Fiji) should declare which file types they support
441+
to avoid attempting to stream incompatible files (CSV, JSON, etc.).
442+
443+
**Declare Supported Extensions:**
444+
445+
.. code-block:: python
446+
447+
from polystore import StreamingBackend
448+
449+
class MyStreamingBackend(StreamingBackend):
450+
"""Custom streaming backend with extension filtering."""
451+
452+
_backend_type = 'my_stream'
453+
VIEWER_TYPE = 'myviewer'
454+
SHM_PREFIX = 'myviewer_'
455+
456+
# Declare which extensions this backend can stream
457+
SUPPORTED_EXTENSIONS = {
458+
'.tif', '.tiff', '.png', '.jpg', '.jpeg',
459+
'.roi.zip', # Compound extensions work too
460+
}
461+
462+
def save_batch(self, data_list, file_paths, **kwargs):
463+
"""Stream data, automatically filtering unsupported files."""
464+
# Filter to only supported file types
465+
data_list, file_paths, skipped = self._filter_streamable_files(
466+
data_list, file_paths
467+
)
468+
469+
if not data_list:
470+
return
471+
472+
# Now stream only the supported files...
473+
for data, path in zip(data_list, file_paths):
474+
self._stream_data(data, path, **kwargs)
475+
476+
**How It Works:**
477+
478+
1. **SUPPORTED_EXTENSIONS**: Class attribute defining valid file extensions
479+
2. **_filter_streamable_files()**: Base class method that filters data
480+
3. **Automatic Logging**: Skipped files are logged at INFO level
481+
482+
**Adding/Removing Support:**
483+
484+
To add support for new formats, simply add to the set:
485+
486+
.. code-block:: python
487+
488+
SUPPORTED_EXTENSIONS = {
489+
'.tif', '.tiff',
490+
'.png', '.jpg', '.jpeg',
491+
'.roi.zip',
492+
'.myformat', # Add your format
493+
}
494+
495+
To remove support, remove from the set or override in subclass:
496+
497+
.. code-block:: python
498+
499+
class RestrictedNapariBackend(NapariStreamingBackend):
500+
"""Napari backend that only supports TIFF files."""
501+
SUPPORTED_EXTENSIONS = {'.tif', '.tiff'}
502+
503+
**Why This Pattern:**
504+
505+
- **Prevents Crashes**: CSV/JSON files won't crash the viewer
506+
- **Clear Contracts**: Developers know what each backend supports
507+
- **Extensible**: Easy to add new formats
508+
- **Consistent**: All streaming backends use the same pattern
509+
437510
Best Practices
438511
--------------
439512

@@ -445,6 +518,7 @@ Best Practices
445518
6. **Resource Cleanup**: Implement ``__del__`` or context managers for cleanup
446519
7. **Type Hints**: Add type hints for better IDE support
447520
8. **Logging**: Use logging for debugging and monitoring
521+
9. **Extension Filtering**: For streaming backends, always use ``SUPPORTED_EXTENSIONS``
448522

449523
Publishing Your Backend
450524
-----------------------

0 commit comments

Comments
 (0)