Big renaming and cleanups#45
Merged
shahbaz-humeai merged 11 commits intojpc/modal-shardsfrom Mar 13, 2026
Merged
Conversation
- Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shahbaz-humeai
approved these changes
Mar 11, 2026
Contributor
shahbaz-humeai
left a comment
shahbaz-humeai
left a comment
There was a problem hiding this comment.
Approved, with two comments
shahbaz-humeai
added a commit
that referenced
this pull request
Mar 13, 2026
* Extract audio codec layer from ws_audio.py into audio_codec.py Separates codec concerns (decoder backends, encoder, format utils) from the data model layer (AudioReader, WSAudio) for better reusability and testability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ModalFileReader for Modal Volume range requests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Centralize binary column decoding into ws_decode module Extract duplicated npy/pyd/txt/audio decode logic from WSShard and WSS3Shard into a shared decode_sample() function. Dispatch is now based on column type (binary) rather than column-name heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Added WSModalShard * Big renaming and cleanups (#45) * Move index SQL queries from WSDataset into WSIndex - Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection * Fix env var name in README: WSDS_DATASET_PATH → WSDS_DATASET_SEARCH_PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move is_notebook() to utils, guard _ipython_display_ for terminal use - Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Improve naming consistency across codebase - subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add get_audio() helper in ws_decode, use in WSSample Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add rng parameter to WSDataset for reproducible sampling Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix hume_wsds module path remapping and sql_filter pl.first() usage - Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update module docstring with rich examples and fix doctests - Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Moved the library showcase to README.md * Use shard ref --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai>
shahbaz-humeai
added a commit
that referenced
this pull request
Mar 13, 2026
* WSAudio: disable slots because it breaks code auto-reload * Added WSModalShard (#41) * Extract audio codec layer from ws_audio.py into audio_codec.py Separates codec concerns (decoder backends, encoder, format utils) from the data model layer (AudioReader, WSAudio) for better reusability and testability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ModalFileReader for Modal Volume range requests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Centralize binary column decoding into ws_decode module Extract duplicated npy/pyd/txt/audio decode logic from WSShard and WSS3Shard into a shared decode_sample() function. Dispatch is now based on column type (binary) rather than column-name heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Added WSModalShard * Big renaming and cleanups (#45) * Move index SQL queries from WSDataset into WSIndex - Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection * Fix env var name in README: WSDS_DATASET_PATH → WSDS_DATASET_SEARCH_PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move is_notebook() to utils, guard _ipython_display_ for terminal use - Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Improve naming consistency across codebase - subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add get_audio() helper in ws_decode, use in WSSample Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add rng parameter to WSDataset for reproducible sampling Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix hume_wsds module path remapping and sql_filter pl.first() usage - Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update module docstring with rich examples and fix doctests - Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Moved the library showcase to README.md * Use shard ref --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai>
shahbaz-humeai
added a commit
that referenced
this pull request
Mar 13, 2026
* WSSample: print fields with missing shards last * WSAudio: disable slots because it breaks code auto-reload (#38) * WSAudio: disable slots because it breaks code auto-reload * Added WSModalShard (#41) * Extract audio codec layer from ws_audio.py into audio_codec.py Separates codec concerns (decoder backends, encoder, format utils) from the data model layer (AudioReader, WSAudio) for better reusability and testability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ModalFileReader for Modal Volume range requests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Centralize binary column decoding into ws_decode module Extract duplicated npy/pyd/txt/audio decode logic from WSShard and WSS3Shard into a shared decode_sample() function. Dispatch is now based on column type (binary) rather than column-name heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Added WSModalShard * Big renaming and cleanups (#45) * Move index SQL queries from WSDataset into WSIndex - Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection * Fix env var name in README: WSDS_DATASET_PATH → WSDS_DATASET_SEARCH_PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move is_notebook() to utils, guard _ipython_display_ for terminal use - Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Improve naming consistency across codebase - subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add get_audio() helper in ws_decode, use in WSSample Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add rng parameter to WSDataset for reproducible sampling Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix hume_wsds module path remapping and sql_filter pl.first() usage - Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update module docstring with rich examples and fix doctests - Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Moved the library showcase to README.md * Use shard ref --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai>
shahbaz-humeai
added a commit
that referenced
this pull request
Mar 13, 2026
* WSSample: always validate that keys match across subdirs * WSSample: print fields with missing shards last (#37) * WSSample: print fields with missing shards last * WSAudio: disable slots because it breaks code auto-reload (#38) * WSAudio: disable slots because it breaks code auto-reload * Added WSModalShard (#41) * Extract audio codec layer from ws_audio.py into audio_codec.py Separates codec concerns (decoder backends, encoder, format utils) from the data model layer (AudioReader, WSAudio) for better reusability and testability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ModalFileReader for Modal Volume range requests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Centralize binary column decoding into ws_decode module Extract duplicated npy/pyd/txt/audio decode logic from WSShard and WSS3Shard into a shared decode_sample() function. Dispatch is now based on column type (binary) rather than column-name heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Added WSModalShard * Big renaming and cleanups (#45) * Move index SQL queries from WSDataset into WSIndex - Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection * Fix env var name in README: WSDS_DATASET_PATH → WSDS_DATASET_SEARCH_PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move is_notebook() to utils, guard _ipython_display_ for terminal use - Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Improve naming consistency across codebase - subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add get_audio() helper in ws_decode, use in WSSample Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add rng parameter to WSDataset for reproducible sampling Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix hume_wsds module path remapping and sql_filter pl.first() usage - Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update module docstring with rich examples and fix doctests - Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Moved the library showcase to README.md * Use shard ref --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai>
shahbaz-humeai
added a commit
that referenced
this pull request
Mar 13, 2026
…andling; ws_indexer: ensure we use relative shard paths when possible (#35) * WSDataset: added an ignore_index option * ws_indexer: improved error handling * ws_indexer: ensure we use relative shard paths when possible * WSSample: always validate that keys match across subdirs (#36) * WSSample: always validate that keys match across subdirs * WSSample: print fields with missing shards last (#37) * WSSample: print fields with missing shards last * WSAudio: disable slots because it breaks code auto-reload (#38) * WSAudio: disable slots because it breaks code auto-reload * Added WSModalShard (#41) * Extract audio codec layer from ws_audio.py into audio_codec.py Separates codec concerns (decoder backends, encoder, format utils) from the data model layer (AudioReader, WSAudio) for better reusability and testability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ModalFileReader for Modal Volume range requests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Centralize binary column decoding into ws_decode module Extract duplicated npy/pyd/txt/audio decode logic from WSShard and WSS3Shard into a shared decode_sample() function. Dispatch is now based on column type (binary) rather than column-name heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Added WSModalShard * Big renaming and cleanups (#45) * Move index SQL queries from WSDataset into WSIndex - Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection * Fix env var name in README: WSDS_DATASET_PATH → WSDS_DATASET_SEARCH_PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move is_notebook() to utils, guard _ipython_display_ for terminal use - Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Improve naming consistency across codebase - subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add get_audio() helper in ws_decode, use in WSSample Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add rng parameter to WSDataset for reproducible sampling Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix hume_wsds module path remapping and sql_filter pl.first() usage - Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update module docstring with rich examples and fix doctests - Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Moved the library showcase to README.md * Use shard ref --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai>
shahbaz-humeai
added a commit
that referenced
this pull request
Mar 13, 2026
* WSDataset: scan the dataset folder even if the index contains a field list * WSShardInterface: remove source_dataset from the from_link interface * WSS3Shard: remote audio shards on S3 * pupyarrow: a pure-Python PyArrow implementation with good lazy-loading support * WSDataset: added an ignore_index option; ws_indexer: improved error handling; ws_indexer: ensure we use relative shard paths when possible (#35) * WSDataset: added an ignore_index option * ws_indexer: improved error handling * ws_indexer: ensure we use relative shard paths when possible * WSSample: always validate that keys match across subdirs (#36) * WSSample: always validate that keys match across subdirs * WSSample: print fields with missing shards last (#37) * WSSample: print fields with missing shards last * WSAudio: disable slots because it breaks code auto-reload (#38) * WSAudio: disable slots because it breaks code auto-reload * Added WSModalShard (#41) * Extract audio codec layer from ws_audio.py into audio_codec.py Separates codec concerns (decoder backends, encoder, format utils) from the data model layer (AudioReader, WSAudio) for better reusability and testability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ModalFileReader for Modal Volume range requests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Centralize binary column decoding into ws_decode module Extract duplicated npy/pyd/txt/audio decode logic from WSShard and WSS3Shard into a shared decode_sample() function. Dispatch is now based on column type (binary) rather than column-name heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Added WSModalShard * Big renaming and cleanups (#45) * Move index SQL queries from WSDataset into WSIndex - Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection * Fix env var name in README: WSDS_DATASET_PATH → WSDS_DATASET_SEARCH_PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move is_notebook() to utils, guard _ipython_display_ for terminal use - Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Improve naming consistency across codebase - subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add get_audio() helper in ws_decode, use in WSSample Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add rng parameter to WSDataset for reproducible sampling Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix hume_wsds module path remapping and sql_filter pl.first() usage - Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update module docstring with rich examples and fix doctests - Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Moved the library showcase to README.md * Use shard ref --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai>
shahbaz-humeai
added a commit
that referenced
this pull request
Mar 13, 2026
* Always keep fields as a list of tuples. * WSS3Shard: remote audio shards on S3 (#34) * WSDataset: scan the dataset folder even if the index contains a field list * WSShardInterface: remove source_dataset from the from_link interface * WSS3Shard: remote audio shards on S3 * pupyarrow: a pure-Python PyArrow implementation with good lazy-loading support * WSDataset: added an ignore_index option; ws_indexer: improved error handling; ws_indexer: ensure we use relative shard paths when possible (#35) * WSDataset: added an ignore_index option * ws_indexer: improved error handling * ws_indexer: ensure we use relative shard paths when possible * WSSample: always validate that keys match across subdirs (#36) * WSSample: always validate that keys match across subdirs * WSSample: print fields with missing shards last (#37) * WSSample: print fields with missing shards last * WSAudio: disable slots because it breaks code auto-reload (#38) * WSAudio: disable slots because it breaks code auto-reload * Added WSModalShard (#41) * Extract audio codec layer from ws_audio.py into audio_codec.py Separates codec concerns (decoder backends, encoder, format utils) from the data model layer (AudioReader, WSAudio) for better reusability and testability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ModalFileReader for Modal Volume range requests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Centralize binary column decoding into ws_decode module Extract duplicated npy/pyd/txt/audio decode logic from WSShard and WSS3Shard into a shared decode_sample() function. Dispatch is now based on column type (binary) rather than column-name heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Added WSModalShard * Big renaming and cleanups (#45) * Move index SQL queries from WSDataset into WSIndex - Add old/new index format detection (partition vs dataset_path columns) - Add _partition_col property for unified SQL partition expression - Add lookup_by_index() and lookup_by_key() methods to WSIndex - Add shard_n_samples() and shard_global_offset() via _query_shard() helper - Simplify WSDataset.__getitem__ to delegate to WSIndex lookups - Replace all raw index.query() calls in WSDataset with WSIndex methods - Update ws_tools.py to use new format detection * Fix env var name in README: WSDS_DATASET_PATH → WSDS_DATASET_SEARCH_PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move is_notebook() to utils, guard _ipython_display_ for terminal use - Extract is_notebook() from convplayer.py into utils.py (simplified) - Remove redundant _ipython_display_ from AudioReader and WSAudio (IPython already calls _repr_html_ automatically) - Add is_notebook() guard to WSDataset and WSSample _ipython_display_ so they fall back to print() in terminal IPython sessions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Improve naming consistency across codebase - subdir → column_dir (utils, ws_sample, ws_modal_shard) - shard_name → shard_ref on shard interfaces and WSSample - dataset_path → partition in index and shard code - dataset_dir → dataset_root in ws_indexer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add get_audio() helper in ws_decode, use in WSSample Centralizes audio column lookup logic so it can be reused outside of WSSample (e.g. from plain dicts or other sample types). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add rng parameter to WSDataset for reproducible sampling Allows passing rng=42 (or a Random instance) to get deterministic sample ordering in random_sample() and sql_select(). Also removes unused needs_key variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix hume_wsds module path remapping and sql_filter pl.first() usage - Remap hume_wsds.* loader paths to wsds.* for backward compatibility with old index files that reference the former package name - Use pl.first() instead of exprs[0] in sql_filter for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update module docstring with rich examples and fix doctests - Add comprehensive docstring to __init__.py with working doctests showcasing SQL queries, random access, lazy loading, and audio - Fix ws_dataset.py doctests (AudioReader src type, add shard_subsample=1) - Fix ws_sink.py doctest (remove invalid batch_size param) - Update tests.py to run wsds module doctests and fix imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Moved the library showcase to README.md * Use shard ref --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shahbaz Mogal <shahbaz@hume.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Renaming things to be more consistent, better tests and docstrings, better documentation.