diff --git a/docs/abc.rst b/docs/abc.md similarity index 86% rename from docs/abc.rst rename to docs/abc.md index 16d4e539c..f080766f2 100644 --- a/docs/abc.rst +++ b/docs/abc.md @@ -1,5 +1,6 @@ -Codec API -========= +# Codec API + +```{eval-rst} .. automodule:: numcodecs.abc .. autoclass:: Codec @@ -9,3 +10,4 @@ Codec API .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 000000000..dd9fcdcf7 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,13 @@ +# API reference + +```{toctree} +:maxdepth: 2 + +compression/index +filter/index +other/index +checksum32 +abc +registry +zarr3 +``` diff --git a/docs/api.rst b/docs/api.rst deleted file mode 100644 index 7caf51fd1..000000000 --- a/docs/api.rst +++ /dev/null @@ -1,13 +0,0 @@ -API reference -============= - -.. toctree:: - :maxdepth: 2 - - compression/index - filter/index - other/index - checksum32 - abc - registry - zarr3 \ No newline at end of file diff --git a/docs/checksum32.rst b/docs/checksum32.md similarity index 82% rename from docs/checksum32.rst rename to docs/checksum32.md index fafceca28..b174b77e8 100644 --- a/docs/checksum32.rst +++ b/docs/checksum32.md @@ -1,9 +1,12 @@ -Checksum codecs -=============== +# Checksum codecs + +```{eval-rst} .. automodule:: numcodecs.checksum32 +``` + +## CRC32 -CRC32 ------ +```{eval-rst} .. autoclass:: CRC32 .. autoattribute:: codec_id @@ -11,10 +14,11 @@ CRC32 .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` +## CRC32C -CRC32C ------- +```{eval-rst} .. autoclass:: CRC32C .. autoattribute:: codec_id @@ -22,10 +26,11 @@ CRC32C .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` +## Adler32 -Adler32 -------- +```{eval-rst} .. autoclass:: Adler32 .. autoattribute:: codec_id @@ -33,20 +38,21 @@ Adler32 .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` +## Fletcher32 -Fletcher32 ----------- - +```{eval-rst} .. autoclass:: numcodecs.fletcher32.Fletcher32 .. autoattribute:: codec_id .. automethod:: encode .. automethod:: decode +``` -JenkinsLookup3 --------------- +## JenkinsLookup3 +```{eval-rst} .. autoclass:: JenkinsLookup3 .. autoattribute:: codec_id @@ -54,3 +60,4 @@ JenkinsLookup3 .. autoattribute:: prefix .. automethod:: encode .. automethod:: decode +``` diff --git a/docs/compression/blosc.rst b/docs/compression/blosc.md similarity index 89% rename from docs/compression/blosc.rst rename to docs/compression/blosc.md index dd87b3acc..6103af330 100644 --- a/docs/compression/blosc.rst +++ b/docs/compression/blosc.md @@ -1,5 +1,6 @@ -Blosc -===== +# Blosc + +```{eval-rst} .. automodule:: numcodecs.blosc .. autoclass:: Blosc @@ -14,13 +15,15 @@ Blosc .. automethod:: get_config .. automethod:: from_config .. automethod:: decode_partial +``` -Helper functions ----------------- +## Helper functions +```{eval-rst} .. autofunction:: list_compressors .. autofunction:: get_nthreads .. autofunction:: set_nthreads .. autofunction:: cbuffer_complib .. autofunction:: compress .. autofunction:: decompress +``` diff --git a/docs/compression/bz2.rst b/docs/compression/bz2.md similarity index 88% rename from docs/compression/bz2.rst rename to docs/compression/bz2.md index 81bf81377..20b3e268c 100644 --- a/docs/compression/bz2.rst +++ b/docs/compression/bz2.md @@ -1,5 +1,6 @@ -BZ2 -=== +# BZ2 + +```{eval-rst} .. automodule:: numcodecs.bz2 .. autoclass:: BZ2 @@ -9,3 +10,4 @@ BZ2 .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/compression/gzip.rst b/docs/compression/gzip.md similarity index 88% rename from docs/compression/gzip.rst rename to docs/compression/gzip.md index e261375c3..422b3f026 100644 --- a/docs/compression/gzip.rst +++ b/docs/compression/gzip.md @@ -1,5 +1,6 @@ -GZip -==== +# GZip + +```{eval-rst} .. automodule:: numcodecs.gzip .. autoclass:: GZip @@ -9,3 +10,4 @@ GZip .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/compression/index.md b/docs/compression/index.md new file mode 100644 index 000000000..ffb3ed84b --- /dev/null +++ b/docs/compression/index.md @@ -0,0 +1,15 @@ +# Compression codecs + +```{toctree} +:maxdepth: 2 + +blosc +bz2 +gzip +lzma +lz4 +pcodec +zfpy +zlib +zstd +``` diff --git a/docs/compression/index.rst b/docs/compression/index.rst deleted file mode 100644 index ba8434d25..000000000 --- a/docs/compression/index.rst +++ /dev/null @@ -1,15 +0,0 @@ -Compression codecs ------------------- - -.. toctree:: - :maxdepth: 2 - - blosc - bz2 - gzip - lzma - lz4 - pcodec - zfpy - zlib - zstd diff --git a/docs/compression/lz4.rst b/docs/compression/lz4.md similarity index 80% rename from docs/compression/lz4.rst rename to docs/compression/lz4.md index a12e1213a..5c0d3535d 100644 --- a/docs/compression/lz4.rst +++ b/docs/compression/lz4.md @@ -1,5 +1,6 @@ -LZ4 -=== +# LZ4 + +```{eval-rst} .. automodule:: numcodecs.lz4 .. autoclass:: LZ4 @@ -9,9 +10,11 @@ LZ4 .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` -Helper functions ----------------- +## Helper functions +```{eval-rst} .. autofunction:: compress .. autofunction:: decompress +``` diff --git a/docs/compression/lzma.rst b/docs/compression/lzma.md similarity index 88% rename from docs/compression/lzma.rst rename to docs/compression/lzma.md index 5fc233176..ed06682ca 100644 --- a/docs/compression/lzma.rst +++ b/docs/compression/lzma.md @@ -1,5 +1,6 @@ -LZMA -==== +# LZMA + +```{eval-rst} .. automodule:: numcodecs.lzma .. autoclass:: LZMA @@ -9,3 +10,4 @@ LZMA .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/compression/pcodec.rst b/docs/compression/pcodec.md similarity index 84% rename from docs/compression/pcodec.rst rename to docs/compression/pcodec.md index 34b34fe70..0b5643215 100644 --- a/docs/compression/pcodec.rst +++ b/docs/compression/pcodec.md @@ -1,6 +1,6 @@ -PCodec -====== +# PCodec +```{eval-rst} .. automodule:: numcodecs.pcodec .. autoclass:: PCodec @@ -8,3 +8,4 @@ PCodec .. autoattribute:: codec_id .. automethod:: encode .. automethod:: decode +``` diff --git a/docs/compression/zfpy.rst b/docs/compression/zfpy.md similarity index 88% rename from docs/compression/zfpy.rst rename to docs/compression/zfpy.md index 28bf16382..9e66991ad 100644 --- a/docs/compression/zfpy.rst +++ b/docs/compression/zfpy.md @@ -1,5 +1,6 @@ -ZFPY -==== +# ZFPY + +```{eval-rst} .. automodule:: numcodecs.zfpy .. autoclass:: ZFPY @@ -9,3 +10,4 @@ ZFPY .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/compression/zlib.rst b/docs/compression/zlib.md similarity index 88% rename from docs/compression/zlib.rst rename to docs/compression/zlib.md index 5d1693e62..fa4d404db 100644 --- a/docs/compression/zlib.rst +++ b/docs/compression/zlib.md @@ -1,5 +1,6 @@ -Zlib -==== +# Zlib + +```{eval-rst} .. automodule:: numcodecs.zlib .. autoclass:: Zlib @@ -9,3 +10,4 @@ Zlib .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/compression/zstd.rst b/docs/compression/zstd.md similarity index 56% rename from docs/compression/zstd.rst rename to docs/compression/zstd.md index 51f9628df..6e2ef2d5d 100644 --- a/docs/compression/zstd.rst +++ b/docs/compression/zstd.md @@ -1,5 +1,6 @@ -Zstd -==== +# Zstd + +```{eval-rst} .. automodule:: numcodecs.zstd .. autoclass:: Zstd @@ -7,14 +8,18 @@ Zstd .. autoattribute:: codec_id .. automethod:: encode .. automethod:: decode - .. note:: - If the compressed data does not contain the decompressed size, streaming - decompression will be used. + + .. note:: + If the compressed data does not contain the decompressed size, streaming + decompression will be used. + .. automethod:: get_config .. automethod:: from_config +``` -Helper functions ----------------- +## Helper functions +```{eval-rst} .. autofunction:: compress .. autofunction:: decompress +``` diff --git a/docs/conf.py b/docs/conf.py index 8e6adca61..2b4f95f64 100755 --- a/docs/conf.py +++ b/docs/conf.py @@ -49,6 +49,7 @@ def __getattr__(cls, name): 'sphinx.ext.viewcode', 'numpydoc', 'sphinx_issues', + 'myst_parser', ] numpydoc_show_class_members = False @@ -59,9 +60,10 @@ def __getattr__(cls, name): templates_path = ['_templates'] # The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -# source_suffix = ['.rst', '.md'] -source_suffix = '.rst' +source_suffix = { + '.rst': 'restructuredtext', + '.md': 'markdown', +} # The encoding of source files. # source_encoding = 'utf-8-sig' @@ -129,6 +131,13 @@ def __getattr__(cls, name): todo_include_todos = False +# -- Options for MyST ------------------------------------------------------- + +myst_enable_extensions = [ + "colon_fence", +] + + # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for diff --git a/docs/contributing.rst b/docs/contributing.md similarity index 68% rename from docs/contributing.rst rename to docs/contributing.md index 166fa826c..19a0d8227 100644 --- a/docs/contributing.rst +++ b/docs/contributing.md @@ -1,31 +1,25 @@ -Contributing -============ +# Contributing NumCodecs is a community maintained project. We welcome contributions in the form of bug reports, bug fixes, documentation, enhancement proposals and more. This page provides information on how best to contribute. -Asking for help ---------------- +## Asking for help If you have a question about how to use NumCodecs, please post your question on -StackOverflow using the `"numcodecs" tag `_. -If you don't get a response within a day or two, feel free to raise a `GitHub issue -`_ including a link to your +StackOverflow using the ["numcodecs" tag](https://stackoverflow.com/questions/tagged/numcodecs). +If you don't get a response within a day or two, feel free to raise a [GitHub issue](https://github.com/zarr-developers/numcodecs/issues/new) including a link to your StackOverflow question. We will try to respond to questions as quickly as possible, but please bear in mind that there may be periods where we have limited time to answer questions due to other commitments. -Bug reports ------------ +## Bug reports -If you find a bug, please raise a `GitHub issue -`_. Please include the following items in +If you find a bug, please raise a [GitHub issue](https://github.com/zarr-developers/numcodecs/issues/new). Please include the following items in a bug report: 1. A minimal, self-contained snippet of Python code reproducing the problem. You can - format the code nicely using markdown, e.g.:: - + format the code nicely using markdown, e.g.: ```python >>> import numcodecs @@ -35,25 +29,26 @@ a bug report: 2. Information about the version of NumCodecs, along with versions of dependencies and the Python interpreter, and installation information. The version of NumCodecs can be obtained - from the ``numcodecs.__version__`` property. Please also state how NumCodecs was installed, + from the `numcodecs.__version__` property. Please also state how NumCodecs was installed, e.g., "installed via pip into a virtual environment", or "installed using conda". - Information about other packages installed can be obtained by executing ``pip list`` - (if using pip to install packages) or ``conda list`` (if using conda to install + Information about other packages installed can be obtained by executing `pip list` + (if using pip to install packages) or `conda list` (if using conda to install packages) from the operating system command prompt. The version of the Python - interpreter can be obtained by running a Python interactive session, e.g.:: + interpreter can be obtained by running a Python interactive session, e.g.: + ``` $ python Python 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:06) [Clang 14.0.6 ] on darwin + ``` 3. An explanation of why the current behaviour is wrong/not desired, and what you expect instead. -Enhancement proposals ---------------------- +## Enhancement proposals If you have an idea about a new feature or some other improvement to NumCodecs, please raise a -`GitHub issue `_ first to discuss. +[GitHub issue](https://github.com/zarr-developers/numcodecs/issues/new) first to discuss. We very much welcome ideas and suggestions for how to improve NumCodecs, but please bear in mind that we are likely to be conservative in accepting proposals for new features. The @@ -62,91 +57,106 @@ a core set of functionalities, and available time for development, review and ma of new features is limited. But if you have a great idea, please don't let that stop you posting it on GitHub, just please don't be offended if we respond cautiously. -Contributing code and/or documentation --------------------------------------- +## Contributing code and/or documentation -Forking the repository -~~~~~~~~~~~~~~~~~~~~~~ +### Forking the repository The NumCodecs source code is hosted on GitHub at the following location: -* `https://github.com/zarr-developers/numcodecs `_ +* You will need your own fork to work on the code. Go to the link above and hit -the "Fork" button. Then clone your fork to your local machine:: +the "Fork" button. Then clone your fork to your local machine: - $ git clone --recursive git@github.com:your-user-name/numcodecs.git # with ``ssh`` +``` +$ git clone --recursive git@github.com:your-user-name/numcodecs.git # with ssh +``` -or:: +or: - $ git clone --recursive https://github.com/your-user-name/numcodecs.git # with ``https`` +``` +$ git clone --recursive https://github.com/your-user-name/numcodecs.git # with https +``` -Then ``cd`` into the clone and add the ``upstream`` remote:: +Then `cd` into the clone and add the `upstream` remote: - $ cd numcodecs - $ git remote add upstream https://github.com/zarr-developers/numcodecs.git +``` +$ cd numcodecs +$ git remote add upstream https://github.com/zarr-developers/numcodecs.git +``` -Creating a development environment -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Creating a development environment To work with the NumCodecs source code, it is recommended to set up a Python virtual environment and install all NumCodecs dependencies using the same versions as are used by the core developers and continuous integration services. Assuming you have a Python -3 interpreter already installed matching the ``requires-python`` constraint from -``pyproject.toml``, and you have cloned the NumCodecs source code and your +3 interpreter already installed matching the `requires-python` constraint from +`pyproject.toml`, and you have cloned the NumCodecs source code and your current working directory is the root of the repository, you can do something -like the following:: +like the following: - $ python3 -m venv ~/pyenv/numcodecs-dev - $ source ~/pyenv/numcodecs-dev/bin/activate - $ pip install -e .[docs,test,msgpack,zfpy] +``` +$ python3 -m venv ~/pyenv/numcodecs-dev +$ source ~/pyenv/numcodecs-dev/bin/activate +$ pip install -e .[docs,test,msgpack,zfpy] +``` You may need to initialize the submodule for c-blosc: - $ git submodule update --init --recursive +``` +$ git submodule update --init --recursive +``` -To verify that your development environment is working, you can run the unit tests:: +To verify that your development environment is working, you can run the unit tests: - $ pytest -v +``` +$ pytest -v +``` -Creating a branch -~~~~~~~~~~~~~~~~~ +### Creating a branch Before you do any new work or submit a pull request, please open an issue on GitHub to report the bug or propose the feature you'd like to add. -It's best to create a new, separate branch for each piece of work you want to do. E.g.:: +It's best to create a new, separate branch for each piece of work you want to do. E.g.: - git fetch upstream - git checkout -b shiny-new-feature upstream/main +``` +git fetch upstream +git checkout -b shiny-new-feature upstream/main +``` This changes your working directory to the 'shiny-new-feature' branch. Keep any changes in this branch specific to one bug or feature so it is clear what the branch brings to NumCodecs. To update this branch with latest code from NumCodecs, you can retrieve the changes from -the main branch and perform a rebase:: +the main branch and perform a rebase: - git fetch upstream - git rebase upstream/main +``` +git fetch upstream +git rebase upstream/main +``` This will replay your commits on top of the latest NumCodecs git main. If this leads to merge conflicts, these need to be resolved before submitting a pull request. Alternatively, you can merge the changes in from upstream/main instead of rebasing, -which can be simpler:: +which can be simpler: - git fetch upstream - git merge upstream/main +``` +git fetch upstream +git merge upstream/main +``` Again, any conflicts need to be resolved before submitting a pull request. -Running the test suite -~~~~~~~~~~~~~~~~~~~~~~ +### Running the test suite NumCodecs includes a suite of unit tests, as well as doctests included in function and class -docstrings. The simplest way to run the unit tests is to invoke:: +docstrings. The simplest way to run the unit tests is to invoke: - $ pytest -v +``` +$ pytest -v +``` NumCodecs currently supports Python 6-3.9, so the above command must succeed before code can be accepted into the main code base. @@ -155,21 +165,21 @@ All tests are automatically run via Travis (Linux) and AppVeyor (Windows) contin integration services for every pull request. Tests must pass under both services before code can be accepted. -Code standards -~~~~~~~~~~~~~~ +### Code standards All code must conform to the PEP8 standard. Regarding line length, lines up to 100 characters are allowed, although please try to keep under 90 wherever possible. -Conformance can be checked by running:: +Conformance can be checked by running: - $ pre-commit run ruff +``` +$ pre-commit run ruff +``` -Test coverage -~~~~~~~~~~~~~ +### Test coverage NumCodecs maintains 100% test coverage under the latest Python stable release (currently Python 3.9). Both unit tests and docstring doctests are included when computing -coverage. Running ``pytest -v`` will automatically run the test suite with coverage +coverage. Running `pytest -v` will automatically run the test suite with coverage and produce a coverage report. This should be 100% before code can be accepted into the main code base. @@ -177,35 +187,33 @@ When submitting a pull request, coverage will also be collected across all suppo Python versions via the Codecov service, and will be reported back within the pull request. Codecov coverage must also be 100% before code can be accepted. -Documentation -~~~~~~~~~~~~~ +### Documentation -Docstrings for user-facing classes and functions should follow the `numpydoc -`_ standard, +Docstrings for user-facing classes and functions should follow the [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard) standard, including sections for Parameters and Examples. All examples will be run as doctests under Python 3.9. NumCodecs uses Sphinx for documentation, hosted on readthedocs.org. Documentation is -written in the RestructuredText markup language (.rst files) in the ``docs`` folder. +written in Markdown (.md files) using [MyST](https://myst-parser.readthedocs.io/) in the `docs` folder. The documentation consists both of prose and API documentation. All user-facing classes and functions should be included in the API documentation. Any changes should also be -included in the release notes (``docs/release.rst``). +included in the release notes (`docs/release.md`). -The documentation can be built by running:: +The documentation can be built by running: - $ cd docs - $ make clean; make html +``` +$ cd docs +$ make clean; make html +``` -The resulting built documentation will be available in the ``docs/_build/html`` folder. +The resulting built documentation will be available in the `docs/_build/html` folder. -Development best practices, policies and procedures ---------------------------------------------------- +## Development best practices, policies and procedures The following information is mainly for core developers, but may also be of interest to contributors. -Merging pull requests -~~~~~~~~~~~~~~~~~~~~~ +### Merging pull requests Pull requests should be reviewed and approved by at least one core developer (other than the pull request author) before being merged. @@ -213,17 +221,15 @@ Pull requests should be reviewed and approved by at least one core developer Pull requests should not be merged until all CI checks have passed against code that has had the latest main merged in. -Compatibility and versioning policies -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Compatibility and versioning policies Because NumCodecs is a data encoding/decoding library, there are two types of compatibility to consider: API compatibility and data format compatibility. -API compatibility -""""""""""""""""" +#### API compatibility All functions, classes and methods that are included in the API -documentation (files under ``docs/api/*.rst``) are considered as part of the NumCodecs +documentation (files under `docs/api/*.md`) are considered as part of the NumCodecs **public API**, except if they have been documented as an experimental feature, in which case they are part of the **experimental API**. @@ -247,31 +253,29 @@ backwards-incompatible changes to the experimental API to be included in a major stabilise those features at the same time (i.e., move from experimental to public API), rather than frequently tinkering with the experimental API in minor releases. -Data format compatibility -""""""""""""""""""""""""" +#### Data format compatibility -Each codec class in NumCodecs exposes a ``codec_id`` attribute, which is an identifier for the +Each codec class in NumCodecs exposes a `codec_id` attribute, which is an identifier for the **format of the encoded data** produced by that codec. Thus it is valid for two or more codec -classes to expose the same value for the ``codec_id`` attribute if the format of the encoded data -is identical. The ``codec_id`` is intended to provide a basis for achieving and managing +classes to expose the same value for the `codec_id` attribute if the format of the encoded data +is identical. The `codec_id` is intended to provide a basis for achieving and managing interoperability between versions of the NumCodecs package, as well as between NumCodecs and other software libraries that aim to provide compatible codec implementations. Currently there is no -formal specification of the encoded data format corresponding to each ``codec_id``, so the codec +formal specification of the encoded data format corresponding to each `codec_id`, so the codec classes provided in the NumCodecs package should be taken as the reference implementation for a -given ``codec_id``. +given `codec_id`. -There must be a one-to-one mapping from ``codec_id`` values to encoded data formats, and that -mapping must not change once the first implementation of a ``codec_id`` has been published within a +There must be a one-to-one mapping from `codec_id` values to encoded data formats, and that +mapping must not change once the first implementation of a `codec_id` has been published within a NumCodecs release. If a change is proposed to the encoded data format for a particular type of -codec, then this must be implemented in NumCodecs via a new codec class exposing a new ``codec_id`` +codec, then this must be implemented in NumCodecs via a new codec class exposing a new `codec_id` value. Note that the NumCodecs test suite includes a data fixture and tests to try and ensure that data format compatibility is not accidentally broken. See the -:func:`test_backwards_compatibility` functions in test modules for each codec for examples. +{func}`test_backwards_compatibility` functions in test modules for each codec for examples. -When to make a release -~~~~~~~~~~~~~~~~~~~~~~ +### When to make a release Ideally, any bug fixes that don't change the public API should be released as soon as possible. It is fine for a micro release to contain only a single bug fix. @@ -289,19 +293,22 @@ Major releases obviously need to be given careful consideration, and should be d infrequently as possible, as they will break existing code and/or affect data compatibility in some way. -Release procedure -~~~~~~~~~~~~~~~~~ +### Release procedure -Checkout and update the main branch:: +Checkout and update the main branch: - $ git checkout main - $ git pull +``` +$ git checkout main +$ git pull +``` -Tag the version (where "X.X.X" stands for the version number, e.g., "2.2.0"):: +Tag the version (where "X.X.X" stands for the version number, e.g., "2.2.0"): - $ version=X.X.X - $ git tag -a v$version -m v$version - $ git push --tags +``` +$ version=X.X.X +$ git tag -a v$version -m v$version +$ git push --tags +``` This will trigger a GitHub Action which will build the source distribution as well as wheels for all major platforms. diff --git a/docs/filter/astype.rst b/docs/filter/astype.md similarity index 88% rename from docs/filter/astype.rst rename to docs/filter/astype.md index 624134ec4..31d1e097d 100644 --- a/docs/filter/astype.rst +++ b/docs/filter/astype.md @@ -1,5 +1,6 @@ -AsType -====== +# AsType + +```{eval-rst} .. automodule:: numcodecs.astype .. autoclass:: AsType @@ -9,3 +10,4 @@ AsType .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/filter/bitround.rst b/docs/filter/bitround.md similarity index 87% rename from docs/filter/bitround.rst rename to docs/filter/bitround.md index 3c98b5a7f..e3dbc1df0 100644 --- a/docs/filter/bitround.rst +++ b/docs/filter/bitround.md @@ -1,5 +1,6 @@ -Bitround -======== +# Bitround + +```{eval-rst} .. automodule:: numcodecs.bitround .. autoclass:: BitRound @@ -9,3 +10,4 @@ Bitround .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/filter/categorize.rst b/docs/filter/categorize.md similarity index 86% rename from docs/filter/categorize.rst rename to docs/filter/categorize.md index 1b4de990f..57f20e9b1 100644 --- a/docs/filter/categorize.rst +++ b/docs/filter/categorize.md @@ -1,5 +1,6 @@ -Categorize -========== +# Categorize + +```{eval-rst} .. automodule:: numcodecs.categorize .. autoclass:: Categorize @@ -9,3 +10,4 @@ Categorize .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/filter/delta.rst b/docs/filter/delta.md similarity index 88% rename from docs/filter/delta.rst rename to docs/filter/delta.md index b3145435d..d09a192f5 100644 --- a/docs/filter/delta.rst +++ b/docs/filter/delta.md @@ -1,5 +1,6 @@ -Delta -===== +# Delta + +```{eval-rst} .. automodule:: numcodecs.delta .. autoclass:: Delta @@ -9,3 +10,4 @@ Delta .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/filter/fixedscaleoffset.rst b/docs/filter/fixedscaleoffset.md similarity index 85% rename from docs/filter/fixedscaleoffset.rst rename to docs/filter/fixedscaleoffset.md index 8aeabf184..334a535cc 100644 --- a/docs/filter/fixedscaleoffset.rst +++ b/docs/filter/fixedscaleoffset.md @@ -1,5 +1,6 @@ -FixedScaleOffset -================ +# FixedScaleOffset + +```{eval-rst} .. automodule:: numcodecs.fixedscaleoffset .. autoclass:: FixedScaleOffset @@ -9,3 +10,4 @@ FixedScaleOffset .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/filter/index.md b/docs/filter/index.md new file mode 100644 index 000000000..cd36c71cb --- /dev/null +++ b/docs/filter/index.md @@ -0,0 +1,14 @@ +# Filter codecs + +```{toctree} +:maxdepth: 2 + +delta +fixedscaleoffset +quantize +bitround +packbits +categorize +astype +shuffle +``` diff --git a/docs/filter/index.rst b/docs/filter/index.rst deleted file mode 100644 index c6bfa4db0..000000000 --- a/docs/filter/index.rst +++ /dev/null @@ -1,14 +0,0 @@ -Filter codecs -------------- - -.. toctree:: - :maxdepth: 2 - - delta - fixedscaleoffset - quantize - bitround - packbits - categorize - astype - shuffle diff --git a/docs/filter/packbits.rst b/docs/filter/packbits.md similarity index 87% rename from docs/filter/packbits.rst rename to docs/filter/packbits.md index 32c0532d7..955c9e214 100644 --- a/docs/filter/packbits.rst +++ b/docs/filter/packbits.md @@ -1,5 +1,6 @@ -PackBits -======== +# PackBits + +```{eval-rst} .. automodule:: numcodecs.packbits .. autoclass:: PackBits @@ -9,3 +10,4 @@ PackBits .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/filter/quantize.rst b/docs/filter/quantize.md similarity index 87% rename from docs/filter/quantize.rst rename to docs/filter/quantize.md index f609d2b4d..18e8d0035 100644 --- a/docs/filter/quantize.rst +++ b/docs/filter/quantize.md @@ -1,5 +1,6 @@ -Quantize -======== +# Quantize + +```{eval-rst} .. automodule:: numcodecs.quantize .. autoclass:: Quantize @@ -9,3 +10,4 @@ Quantize .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/filter/shuffle.rst b/docs/filter/shuffle.md similarity index 83% rename from docs/filter/shuffle.rst rename to docs/filter/shuffle.md index 31e28360e..3da217b1e 100644 --- a/docs/filter/shuffle.rst +++ b/docs/filter/shuffle.md @@ -1,5 +1,6 @@ -Shuffle -======= +# Shuffle + +```{eval-rst} .. automodule:: numcodecs.shuffle .. autoclass:: Shuffle @@ -7,3 +8,4 @@ Shuffle .. autoattribute:: codec_id .. automethod:: encode .. automethod:: decode +``` diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 000000000..e6808a86c --- /dev/null +++ b/docs/index.md @@ -0,0 +1,103 @@ +# Numcodecs + +```{eval-rst} +.. automodule:: numcodecs +``` + +## Installation + +Numcodecs depends on NumPy. It is generally best to [install NumPy](https://numpy.org/install/) +first using whatever method is most appropriate for you operating system and Python distribution. + +Install from PyPI: + +``` +$ pip install numcodecs +``` + +Alternatively, install via conda: + +``` +$ conda install -c conda-forge numcodecs +``` + +Numcodecs includes a C extension providing integration with the [Blosc](https://www.blosc.org/) +library. Wheels are available for most platforms. + +Installing a wheel or via conda will install a pre-compiled binary distribution. +However, if you have a newer CPU that supports the AVX2 instruction set (e.g., +Intel Haswell, Broadwell or Skylake) then installing via pip is preferable, +because you can compile the Blosc library from source with optimisations +for AVX2. + +``` +$ pip install -v --no-cache-dir --no-binary numcodecs numcodecs +``` + +Note that if you compile the C extensions on a machine with AVX2 support +you probably then cannot use the same binaries on a machine without AVX2. + +If you specifically want to disable AVX2 or SSE2 when compiling, you can use +the following environment variables: + +``` +$ export DISABLE_NUMCODECS_AVX2=1 +$ export DISABLE_NUMCODECS_SSE2=1 +``` + +To work with Numcodecs source code in development, clone the repository from GitHub +and then install in editable mode using `pip`. + +``` +$ git clone --recursive https://github.com/zarr-developers/numcodecs.git +$ cd numcodecs +$ pip install -e .[test,msgpack,zfpy] +``` + +Note: if you prefer to use the GitHub CLI `gh` you will need to append `-- --recurse-submodules` +to the clone command to everything works properly. + +To verify that Numcodecs has been fully installed (including the Blosc +extension) run the test suite: + +``` +$ pytest -v +``` + +## Contents + +```{toctree} +:maxdepth: 2 + +api +release +contributing +``` + +## Acknowledgments + +The following people have contributed to the development of NumCodecs by contributing code, +documentation, code reviews, comments and/or ideas: + +* {user}`Francesc Alted ` +* {user}`Prakhar Goel ` +* {user}`Jerome Kelleher ` +* {user}`John Kirkham ` +* {user}`Alistair Miles ` +* {user}`Jeff Reback ` +* {user}`Trevor Manz ` +* {user}`Grzegorz Bokota ` +* {user}`Josh Moore ` +* {user}`Martin Durant ` +* {user}`Paul Branson ` + +Numcodecs bundles the [c-blosc](https://github.com/Blosc/c-blosc) library. + +Development of this package is supported by the +[MRC Centre for Genomics and Global Health](https://www.sanger.ac.uk/collaboration/mrc-centre-genomics-and-global-health-cggh/). + +## Indices and tables + +* {ref}`genindex` +* {ref}`modindex` +* {ref}`search` diff --git a/docs/index.rst b/docs/index.rst deleted file mode 100644 index cce02f0cd..000000000 --- a/docs/index.rst +++ /dev/null @@ -1,101 +0,0 @@ -.. numcodecs documentation main file, created by - sphinx-quickstart on Mon May 2 21:40:09 2016. - - -Numcodecs -========= - -.. automodule:: numcodecs - -Installation ------------- - -Numcodecs depends on NumPy. It is generally best to `install NumPy -`_ first using whatever method is most -appropriate for you operating system and Python distribution. - -Install from PyPI:: - - $ pip install numcodecs - -Alternatively, install via conda:: - - $ conda install -c conda-forge numcodecs - -Numcodecs includes a C extension providing integration with the Blosc_ -library. Wheels are available for most platforms. - -Installing a wheel or via conda will install a pre-compiled binary distribution. -However, if you have a newer CPU that supports the AVX2 instruction set (e.g., -Intel Haswell, Broadwell or Skylake) then installing via pip is preferable, -because you can compile the Blosc library from source with optimisations -for AVX2.:: - - $ pip install -v --no-cache-dir --no-binary numcodecs numcodecs - -Note that if you compile the C extensions on a machine with AVX2 support -you probably then cannot use the same binaries on a machine without AVX2. - -If you specifically want to disable AVX2 or SSE2 when compiling, you can use -the following environment variables:: - - $ export DISABLE_NUMCODECS_AVX2=1 - $ export DISABLE_NUMCODECS_SSE2=1 - - -To work with Numcodecs source code in development, clone the repository from GitHub -and then install in editable mode using `pip`.:: - - $ git clone --recursive https://github.com/zarr-developers/numcodecs.git - $ cd numcodecs - $ pip install -e .[test,msgpack,zfpy] - -Note: if you prefer to use the GitHub CLI ``gh`` you will need to append ``-- --recurse-submodules`` -to the clone command to everything works properly. - -To verify that Numcodecs has been fully installed (including the Blosc -extension) run the test suite:: - - $ pytest -v - -Contents --------- - -.. toctree:: - :maxdepth: 2 - - api - release - contributing - -Acknowledgments ---------------- - -The following people have contributed to the development of NumCodecs by contributing code, -documentation, code reviews, comments and/or ideas: - -* :user:`Francesc Alted ` -* :user:`Prakhar Goel ` -* :user:`Jerome Kelleher ` -* :user:`John Kirkham ` -* :user:`Alistair Miles ` -* :user:`Jeff Reback ` -* :user:`Trevor Manz ` -* :user:`Grzegorz Bokota ` -* :user:`Josh Moore ` -* :user:`Martin Durant ` -* :user:`Paul Branson ` - -Numcodecs bundles the `c-blosc `_ library. - -Development of this package is supported by the -`MRC Centre for Genomics and Global Health `_. - -Indices and tables ------------------- - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - -.. _Blosc: https://www.blosc.org/ diff --git a/docs/other/index.md b/docs/other/index.md new file mode 100644 index 000000000..d514d4053 --- /dev/null +++ b/docs/other/index.md @@ -0,0 +1,10 @@ +# Other codecs + +```{toctree} +:maxdepth: 2 + +json +pickles +msgpacks +vlen +``` diff --git a/docs/other/index.rst b/docs/other/index.rst deleted file mode 100644 index 0ebcbdcbb..000000000 --- a/docs/other/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -Other codecs ------------- - - -.. toctree:: - :maxdepth: 2 - - json - pickles - msgpacks - vlen diff --git a/docs/other/json.rst b/docs/other/json.md similarity index 88% rename from docs/other/json.rst rename to docs/other/json.md index 3e540e0af..55ce11163 100644 --- a/docs/other/json.rst +++ b/docs/other/json.md @@ -1,5 +1,6 @@ -JSON -==== +# JSON + +```{eval-rst} .. automodule:: numcodecs.json .. autoclass:: JSON @@ -9,3 +10,4 @@ JSON .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/other/msgpacks.rst b/docs/other/msgpacks.md similarity index 87% rename from docs/other/msgpacks.rst rename to docs/other/msgpacks.md index 47acbf218..5bc3bcf26 100644 --- a/docs/other/msgpacks.rst +++ b/docs/other/msgpacks.md @@ -1,5 +1,6 @@ -MsgPack -======= +# MsgPack + +```{eval-rst} .. automodule:: numcodecs.msgpacks .. autoclass:: MsgPack @@ -9,3 +10,4 @@ MsgPack .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/other/pickles.rst b/docs/other/pickles.md similarity index 88% rename from docs/other/pickles.rst rename to docs/other/pickles.md index f906ad4ea..f32bd64ff 100644 --- a/docs/other/pickles.rst +++ b/docs/other/pickles.md @@ -1,5 +1,6 @@ -Pickle -====== +# Pickle + +```{eval-rst} .. automodule:: numcodecs.pickles .. autoclass:: Pickle @@ -9,3 +10,4 @@ Pickle .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/other/vlen.rst b/docs/other/vlen.md similarity index 78% rename from docs/other/vlen.rst rename to docs/other/vlen.md index aaedb6493..a36c83c72 100644 --- a/docs/other/vlen.rst +++ b/docs/other/vlen.md @@ -1,9 +1,12 @@ -Codecs for variable-length objects -================================== +# Codecs for variable-length objects + +```{eval-rst} .. automodule:: numcodecs.vlen +``` + +## VLenUTF8 -VLenUTF8 --------- +```{eval-rst} .. autoclass:: VLenUTF8 .. autoattribute:: codec_id @@ -11,10 +14,11 @@ VLenUTF8 .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` +## VLenBytes -VLenBytes ---------- +```{eval-rst} .. autoclass:: VLenBytes .. autoattribute:: codec_id @@ -22,10 +26,11 @@ VLenBytes .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` +## VLenArray -VLenArray ---------- +```{eval-rst} .. autoclass:: VLenArray .. autoattribute:: codec_id @@ -33,3 +38,4 @@ VLenArray .. automethod:: decode .. automethod:: get_config .. automethod:: from_config +``` diff --git a/docs/registry.rst b/docs/registry.md similarity index 72% rename from docs/registry.rst rename to docs/registry.md index 0fa3b280d..f279b0bd3 100644 --- a/docs/registry.rst +++ b/docs/registry.md @@ -1,6 +1,8 @@ -Codec registry -============== +# Codec registry + +```{eval-rst} .. automodule:: numcodecs.registry .. autofunction:: get_codec .. autofunction:: register_codec +``` diff --git a/docs/release.md b/docs/release.md new file mode 100644 index 000000000..e2fb936a2 --- /dev/null +++ b/docs/release.md @@ -0,0 +1,1051 @@ +# Release notes + + + +(unreleased)= + +## Unreleased + +### Maintenance + +* Convert documentation from reStructuredText to Markdown using MyST. + By {user}`Max Jones `, :issue:`830` + +* Move tests out of installable package into top-level ``tests/`` directory. + By :user:`Max Jones `, :issue:`829` + +(release_0.16.5)= + +## 0.16.5 + +### Fixes + +* Restore prior optional dependency behavior for `'crc32c'` group. + By {user}`Davis Bennett `, {issue}`802` + +(release_0.16.4)= + +## 0.16.4 + +### Enhancements + +* The Zstandard codec can now decode bytes containing multiple frames. + By {user}`Mark Kittisopikul `, {issue}`757` + +* Add Python 3.14 support to the test matrix. + By {user}`Max Jones `, {issue}`799` + +* Add Python 3.14 wheel builds. + By {user}`Davis Bennett `, {issue}`796` + +### Fixes + +* Fix `PCodec.decode()` to handle numpy arrays by calling `ensure_bytes` on input buffers. + By {user}`Sam Levang `, {issue}`756` + +### Maintenance + +* Replace `crc32c` dependency with `google-crc32c`. + By {user}`Joe Hamman `, {issue}`792` + +* Refactor `numcodecs.zarr3` to re-export classes from `zarr.codecs._numcodecs` + instead of defining them locally. + By {user}`Davis Bennett `, {issue}`780` + +* Add testing against zarr-python main branch. + By {user}`Max Jones `, {issue}`800` + +* Add pcodec and zfpy to 32-bit CI testing. + By {user}`Mark Kittisopikul `, {issue}`786` + +* Improve CRC32C codec test skipping to properly report skipped tests using `pytest.skipif`. + By {user}`Agriya Khetarpal `, {issue}`740` + +* Add nightly wheel upload to scientific-python channel. + By {user}`Ian Hunt-Isaak `, {issue}`787` + +* Fetch tags to version nightly wheels correctly. + By {user}`Ian Hunt-Isaak `, {issue}`790` + +* Bump `pypa/cibuildwheel` from 2.23.2 to 3.1.3. + By {user}`dependabot `, {issue}`771` + +* Bump `actions/checkout` from 4 to 5. + By {user}`dependabot `, {issue}`775` + +* Bump `actions/download-artifact` from 4 to 6. + By {user}`dependabot `, {issue}`794` + +* Bump `prefix-dev/setup-pixi` from 0.8.14 to 0.9.0. + By {user}`dependabot `, {issue}`783` + +* Bump `jirutka/setup-alpine` from 1.2.0 to 1.3.0. + By {user}`dependabot `, {issue}`789` + +* Bump `conda-incubator/setup-miniconda` from 3.1.1 to 3.2.0. + By {user}`dependabot `, {issue}`753` + + +(release_0.16.3)= + +## 0.16.3 + +### Fixes + +* Fix Zstd decompression negative size issue on 32-bit platforms. + By {user}`Mark Kittisopikul `, {issue}`782` + +### Maintenance + +* Test 32-bit support via x86 Alpine Setup Action Linux. + By {user}`Mark Kittisopikul `, {issue}`784` + +* Bump to main branch of c-blosc to fix gcc15 compilation. + By {user}`Thomas Caswell `, {issue}`772` + + +(release_0.16.2)= + +## 0.16.2 + +### Enhancements + +* Add streaming decompression for ZSTD_CONTENTSIZE_UNKNOWN case. + By {user}`Mark Kittisopikul `, {issue}`707` + +### Maintenance + +* Handle zarr 3.1.0 compatibility. + By {user}`Davis Bennett `, {issue}`766` + + +(release_0.16.1)= + +## 0.16.1 + +### Enhancements + +* In `vlen`, define and use `const` `HEADER_LENGTH`. + By {user}`John Kirkham `, {issue}`723` + +* All codecs are now pickleable. + By {user}`Tom Nicholas `, {issue}`745` + +### Fixes + +* Fix `const` discard warnings in `fletcher32`. + By {user}`John Kirkham `, {issue}`728` + +* Update `license*` metadata and include third-party licenses. + By {user}`John Kirkham `, {issue}`729` + +* Add `#ifndef` guard around `PyBytes_RESIZE`. + By {user}`John Kirkham `, {issue}`732` + +* Remove `typesize` from `Blosc.get_config` output. + By {user}`Ilan Gold `, {issue}`739` + + +(release_0.16.0)= + +## 0.16.0 + +### Enhancements + +* Add `typesize` argument to `Blosc` to allow for buffers that are passed to `encode` + use that information. zarr v3 currently has its Blosc codec as bytes-to-bytes but does retain + the size information so using it here allows for massive compression ratio gains. + By {user}`Ilan Gold ` +* Add support for the Linux aarch64 architecture, + By {user}`Agriya Khetarpal `, {issue}`288`. +* Preallocate output buffers and resize directly as needed. + By {user}`John Kirkham `, {issue}`656` +* `Checksum32` is now an abstract base clase, with `checksum()` as an abstract method. + This helps ensure that any subclasses implement the `checksum()` method. + {issue}`711` + +### Removals + +The following `blosc` funcitons are removed, with no replacement. +This is because they were not intended to be public API. + +- `numcodecs.blosc.init` +- `numcodecs.blosc.destroy` +- `numcodecs.blosc.compname_to_compcode` +- `numcodecs.blosc.cbuffer_sizes` +- `numcodecs.blosc.cbuffer_metainfo` + +In addition, `numcodecs.blosc.decompress_partial` is removed as +has always been experimental and there is no equivalent in the official +blsoc Python package. +By {user}`David Stansby `, {issue}`712` + +### Maintenance + +* Replace internal `Buffer` usage with `memoryview`s. + By {user}`John Kirkham `, {issue}`656` +* Bump the minimum macOS deployment target for x86_64 to 10.13. + By {user}`Agriya Khetarpal `, {issue}`288`. + + +## 0.15.1 + +### Improvements + +* Add streaming decompression for ZSTD ({issue}`699`) + By {user}`Mark Kittisopikul `. +* Raise a custom `UnknownCodecError` when trying to retrieve an unavailable codec. + By {user}`Cas Wognum `. + +### Fixes + +* Remove redundant `id` from codec metadata serialization in Zarr3 codecs. + By {user}`Norman Rzepka `, {issue}`685` + +(release_0.15.0)= + +## 0.15.0 + +### Breaking changes + +* All arguments to the `PCodec` constructor except for `level` + are now keyword only, to support the updated API. + By {user}`Sam Levang `, {issue}`623` + +### Deprecations + +The following `blosc` functions are deprecated, with no replacement. +This is because they are not intended to be public API. + +- `numcodecs.blosc.init` +- `numcodecs.blosc.destroy` +- `numcodecs.blosc.compname_to_compcode` +- `numcodecs.blosc.cbuffer_sizes` +- `numcodecs.blosc.cbuffer_metainfo` + +In addition, `numcodecs.blosc.decompress_partial` is deprecated as +has always been experimental and there is no equivalent in the official +blsoc Python package. +By {user}`David Stansby `, {issue}`619` + +### Fixes + +* Fixes issue with `Delta` Zarr 3 codec not working with `astype`. + By {user}`Norman Rzepka `, {issue}`664` +* Cleanup `PCodec` soft dependency. + Previously importing `numcodecs.pcodec` would work if `pcodec` is not installed, + but now it will fail to import. This mirrors the behaviour of other optional dependencies. + By {user}`John Kirkham `, {issue}`647` +* Fixes issues with the upcoming `zarr` 3.0.0 release. + By {user}`Norman Rzepka `, {issue}`675` + +* Removed Version Check: The previous code included a check for the `NumPy` version + and a warning if the version was incompatible with `zfpy`. + This check has been removed because `zfpy` now supports the newer versions of `NumPy`. + By {user}`Meher Gajula `, {issue}`672` + +### Improvements + +* Add support for `pcodec` 0.3. This exposes the new `delta_spec` + and `paging_spec` arguments, but maintains full backwards + compatibility for data written with older package versions. + By {user}`Sam Levang `, {issue}`623` +* If an import error is raised when trying to define a codec that is *not* + an optional dependency, it is no longer silently caught. Instead it will + be propagated to the user, as this indicates an issue with the installed + package. + + Import errors caused by optional dependencies (ZFPY, MsgPack, CRC32C, and PCodec) + are still silently caught. + By {user}`David Stansby `, {issue}`550`. + +(release_0.14.1)= + +## 0.14.1 + +### Fixes + +* Cleanups to the `crc32c` soft dependency. + Whereas in `numcodecs` 0.14.0 a runtime error was raised if + the `CRC32C` codec used without the `crc32c` installed, + the `CRC32C` codec is no longer defined at import time if + `crc32c` is not installed. This has been changed to match + the behaviour of other optional dependencies/codecs. + By {user}`John Kirkham `, {issue}`637` + +### Improvements + +* Add `noexcept` to `_utils` C-equiv functions + By {user}`John Kirkham `, {issue}`641`. + +(release_0.14.0)= + +## 0.14.0 + +### Enhancements + +* Add Crc32c checksum codec. + By {user}`Norman Rzepka `, {issue}`613`. +* Add codec wrappers for Zarr 3. + By {user}`Norman Rzepka `, {issue}`524` +* Added mypy type checking to continuous integration. + By {user}`David Stansby `, {issue}`460`. + +### Fixes + +* Fix in-place mutation of input array in `BitRound`. + By {user}`Sam Levang `, {issue}`608` +* Fix an issue where importing numcodecs would lock the state of `multiprocessing` + and prevent user code to call `multiprocessing.set_start_method("spawn")` + subsequently. + By {user}`Clement Robert ` {issue}`522` + +### Maintenance + +* The minimum supported Python version is now Python 3.11. + By {user}`David Stansby `, {issue}`622` +* The minimum supported numpy version is now 1.24. + By {user}`David Stansby `, {issue}`622` + +(release_0.13.1)= + +## 0.13.1 + +### Breaking changes + +* `Zstd.default_level`, `Zstd.min_level`, and `Zstd.max_level` are now class methods + instead of properties. This means they must now be called like `Zstd.default_level()` + instead of `Zstd.default_level`. This breaking change has been made because Python 3.13 + removes support for class properties. + By {user}`David Stansby `, {issue}`576`. + +### Enhancements + +* Update bundled c-blosc to v1.26.1. This updates Zlib to v1.3.1 + and Zstd to v1.5.6. + By {user}`David Stansby `, {issue}`560`. +* Added support for Python 3.13 {user}`David Stansby `, {issue}`576`. +* Cleaned up the table of contents in the documentation to list codecs by category + {user}`David Stansby `, {issue}`458`. + +### Maintenance + +* Change format() and old string formatting to f-strings. + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`439`. +* Remove pin on Sphinx + By {user}`Elliott Sales de Andrade `, {issue}`552`. + + +(release_0.13.0)= + +## 0.13.0 + +### Enhancements + +* Add checksum flag to zstd and sets the default level to 0. + By {user}`Norman Rzepka `, {issue}`519`. +* Add PCodec + By {user}`Ryan Abernathey `, {issue}`501`. +* Use PyData theme for docs + By {user}`John Kirkham `, {issue}`485`. +* Improve the structure of docs + By {user}`David Stansby `, {issue}`458`. + +### Fix + +* Fix VLenUTF8 encoding for read-only buffers. + By {user}`Isaac Virshup `, {issue}`514`. +* Fix skip of entry points backport tests + By {user}`Elliott Sales de Andrade `, {issue}`487`. +* Fix Upgrade to Zstd 1.5.5 due to potential corruption. + By {user}`Mark Kittisopikul `, {issue}`429` +* Add version constraint(<2.0) for numpy in zfpy. + By {user}`Tom Liang `, {issue}`540`. + +### Maintenance + +* The minimum supported Python version is now Python 3.10. + By {user}`David Stansby `, {issue}`531` +* Add numpy 2 compatibility. + By {user}`David Stansby `, {issue}`535` +* Update c-blosc to 1.21.0 to 1.21.5, zstd from 1.4.8 to 1.5.5, + lz4 from 1.9.3 to 1.9.4, and zlib from 1.2.8 to to 1.2.13 + By {user}`Mark Kittisopikul `, {issue}`500` + + +(release_0.12.1)= + +## 0.12.1 + +### Fix + +* `Codec` is now derived from `abc.ABC` + By {user}`Mads R. B. Kristensen `, {issue}`472`. +* Fix handling of entry points on older Python versions where `importlib_metadata` compatibility is concerned + By {user}`Vyas Ramasubramani `, {issue}`478`. +* Make shuffle pyx functions `noexcept` + By {user}`Martin Durant `, {issue}`477`. + +(release_0.12.0)= + +## 0.12.0 + +### Enhancements + +* Add `fletcher32` checksum codec + By {user}`Martin Durant `, {issue}`410`. +* Add `jenkins_lookup3` checksum codec + By {user}`Mark Kittisopkul `, {issue}`445`. +* Support Python 3.12. + By {user}`John Kirkham `, {issue}`471`. + +### Fix + +* Fixed docs/Makefile error message when sphinx is not present + By {user}`Mark Kittisopikul `, {issue}`451`. +* Add missing RTD requirements + By {user}`John Kirkham `, {issue}`455`. + +### Maintenance + +* Cleanup `import`s in `adhoc/blosc_memleak_check.py` + By {user}`John Kirkham `, {issue}`408`. + +(release_0.11.0)= + +## 0.11.0 + +### Fix + +* Speed up isinstance checks of {class}`numcodecs.ndarray_like.NDArrayLike`, + {class}`numcodecs.ndarray_like.DType` and {class}`numcodecs.ndarray_like.FlagsObj`. + By {user}`Andreas Poehlmann `, {issue}`379`. + +### Maintenance + +* Remove unnecessary None argument to .get(), it is the default value. + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`395`. + +* Apply refurb suggestions. + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`372`. + +* Migrate codespell configuration to pyproject.toml and get rid of setup.cfg. + By {user}`Dimitri Papadopoulos Orfanos ` {issue}`374`. + +* Unvendor cpuinfo. + By {user}`Dimitri Papadopoulos Orfanos ` {issue}`373`. + +* Drop headers. + By {user}`John Kirkham `, {issue}`375`. + +* Remove Python 2 code. + By {user}`Dimitri Papadopoulos Orfanos ` {issue}`368`, + {issue}`387`. + +* Support Python 3.11. + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`369`. + +* Drop Python 3.7. + By {user}`John Kirkham `,, {issue}`405` {issue}`406`. + +* Test with zfpy 1.0.0. + By {user}`John Kirkham `, {issue}`385`. + +* Remove vendored C files and re-generate them on the fly using Cython. + Add a pyproject.toml file to define Cython as a build dependency. + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`369`. + +* Update cpuinfo to 9.0.0. + By {user}`Dimitri Papadopoulos Orfanos ` {issue}`370`. + +* Add tests for all registry classes. + By {user}`Josh Moore `, {issue}`349`. + +* Add workflow to label PRs with "needs release notes". + By {user}`Saransh Chopra ` {issue}`396`. + +* Finish `pyproject.toml` migration. + By {user}`John Kirkham ` {issue}`382`. + +* Drop tox. + By {user}`John Kirkham `, {issue}`384`. + +* Update ReadTheDocs. + By {user}`John Kirkham `, {issue}`383`. + +* Use `conda-incubator/setup-miniconda@v2.2.0` (and use Conda on Linux). + By {user}`John Kirkham ` {issue}`398`. + +* Bring coverage back up to 100%. + By {user}`John Kirkham ` and {user}`Martin Durant `, + {issue}`392` and {issue}`393`. + +* Collect coverage on all OSes & enable Codecov. + By {user}`John Kirkham `, {issue}`386`, {issue}`388`, + {issue}`390`, {issue}`391`. + +(release_0.10.2)= + +## 0.10.2 + +### Fix + +* Add BitRound (0.10.0) to registry. + By {user}`Josh Moore `, {issue}`342`. + +(release_0.10.1)= + +## 0.10.1 + +### Maintenance + +* Add entrypoints to setup.py. + By {user}`Josh Moore `, {issue}`332`. + +* Fix spelling. + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`336`. + +* Drop Python 3.6 from tests + By {user}`Dimitri Papadopoulos Orfanos `, + {issue}`338`, {issue}`339`. + +* Remove trailing spaces and empty lines. + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`341`. + +* Add LGTM.com configuration file + By {user}`Dimitri Papadopoulos Orfanos `, {issue}`337`. + +(release_0.10.0)= + +## 0.10.0 + +### Enhancements + +* Add support of alternative array classes (other than NumPy arrays) + By {user}`Mads R. B. Kristensen `, {issue}`305`. + +* Add ability to find codecs via entrypoint `numcodecs.codecs`. + By {user}`Martin Durant `, {issue}`290`. + +* Add bitround codec + By {user}`Ryan Abernathy ` and {user}`Martin Durant `, {issue}`298`. + +* Introduce a flat option to ensure_contiguous_ndarray to switch off flatten for ZFPY codec + By {user}`Haiying Xu `, {issue}`307`. + +### Bug fixes + +* Fix a flatten array error for ZFPY, ZFPY codec is supported on Python 3.9 + and 3.10 on Linux and MacOS, the docs about ZFPY is also available. + By {user}`Haiying Xu `, {user}`John Kirkham `, + {user}`Ryan Abernathey ` {issue}`303`. + +* Codex: make encode and decode @abstractmethods + By {user}`Mads R. B. Kristensen `, {issue}`306`. + +* Fix expected result test for Shuffle. + By {user}`Elliott Sales de Andrade `, {issue}`282`. + +### Maintenance + +* Multiple code linting fixes. + By {user}`Dimitri Papadopoulos Orfanos `, + {issue}`295`, {issue}`294`, {issue}`293`, and {issue}`292`. + +* Drop Python 3.6 + By {user}`Josh Moore `, {issue}`318`. + +* Fix macOS Python 3.10 + By {user}`John Kirkham `, {issue}`311`. + +* chore: bump cibuildwheel version, use action + By {user}`Henry Schreiner `, {issue}`309`. + +* Specify `language` as `'en'` instead of `None`. + By {user}`John Kirkham `, {issue}`329`. + +* Move `master` to `main`. + By {user}`John Kirkham `, {issue}`322`. + +* Drop `fastparquet` benchmark. + By {user}`John Kirkham `, {issue}`321`. + +* Trim wheel builds. + By {user}`John Kirkham `, {issue}`320`. + +(release_0.9.1)= + +## 0.9.1 + +* Fix inaccurate docstrings for Blosc. + By {user}`James Webber `, {issue}`287`. + +(release_0.9.0)= + +## 0.9.0 + +* c-blosc upgrade 1.18.1 -> 1.21.0. + Warning: this temporarily removes support for snappy compression! + By {user}`kindjacket `, {issue}`283`. + +* Fix an ImportError with Blosc on Android. + By {user}`Daniel Jewell `, {issue}`284`. + +(release_0.8.1)= + +## 0.8.1 + +* Fix an ImportError with Blosc on Android. + By {user}`Daniel Jewell `, {issue}`284`. + +(release_0.8.0)= + +## 0.8.0 + +* The {class}`numcodecs.zfpy.ZFPY` codec is now supported on Python 3.8 if + [zfpy==0.5.5](https://pypi.org/project/zfpy/) is installed. + By {user}`haiying xu `, {issue}`229`. + +* Add support for byte Shuffle filter + By {user}`Paul Branson ` and {user}`Martin Durant ` {issue}`273`. + +* Update Windows + Mac CI to run all tests. + By {user}`Jackson Maxfield Brown `, {issue}`276`. + Help from {user}`Oleg Hoefling `, {issue}`273`. + +* Update cpuinfo to 8.0.0. + By {user}`Florian Jetter `, {issue}`280`. + +* Drop out-of-date manual release docs. + By {user}`John Kirkham `, {issue}`272`. + +* Add support for Python 3.9 and Update GitHub Actions. + +(release_0.7.3)= + +## 0.7.3 + +* Add support for Python 3.9 and Update GitHub Actions. + By {user}`Jackson Maxfield Brown `, {issue}`270`. + +* Remove support for Python 3.5 which is end of life. While the code base might + still be compatible; the source dist and wheel are marked as Python 3.6+ and + pip will not install them. Continuous integration on Python 3.5 has been + disabled. + By {user}`Matthias Bussonnier `, {issue}`266` and {issue}`267`. + +(release_0.7.2)= + +## 0.7.2 + +* Disable avx2 for wheel. + By {user}`Grzegorz Bokota `, {issue}`253`. + +* Add Base64 fixtures. + By {user}`John Kirkham `, {issue}`251`. + +* Update docs regarding wheels. + By {user}`Josh Moore `, {issue}`250`. + + +(release_0.7.1)= + +## 0.7.1 + +* Fix build of wheels. + By {user}`Grzegorz Bokota `, {issue}`244`. + +(release_0.7.0)= + +## 0.7.0 + +* Automatically release to PyPI. + By {user}`Josh Moore `, {issue}`241`. + +* Build wheels on github actions. + By {user}`Grzegorz Bokota `, {issue}`224`. + +* Add Base64 codec. + By {user}`Trevor Manz `, {issue}`176`. + +* Add partial decompression of Blosc compressed arrays. + By {user}`Andrew Fulton `, {issue}`235`. + +* Remove LegacyJSON codec. + By {user}`James Bourbeau `, {issue}`226`. + +* Remove LegacyMsgPack codec. + By {user}`James Bourbeau `, {issue}`218`. + +* Drop support for Python 2. + By {user}`James Bourbeau `, {issue}`220`. + + +(release_0.6.4)= + +## 0.6.4 + +* Update Cython to 0.29.14. + By {user}`John Kirkham `, {issue}`168`, {issue}`177`, {issue}`204`. + +* The bundled c-blosc sources have been upgraded to version 1.17.0. + This fixes compilation with newer versions of gcc. + By {user}`Joe Jevnik `, {issue}`194`. + +* Create `.pep8speaks.yml`. By {user}`Alistair Miles `. + +* Simplify datetime/timedelta check. + By {user}`John Kirkham `, {issue}`170`, {issue}`171`. + +* Update URL metadata for PyPI. + By {user}`Elliott Sales de Andrade `, {issue}`178`. + +* Enable pytest rewriting in test helper functions. + By {user}`Elliott Sales de Andrade `, {issue}`185`. + +* Rewrites the `ensure_text` implementation. + By {user}`John Kirkham `, {issue}`201`, {issue}`205`, {issue}`206`. + +* Add macOS to CI. + By {user}`Alistair Miles `, {issue}`192`. + +* Fix test failures on big-endian systems. + By {user}`Elliott Sales de Andrade `, {issue}`186`. + +* Use unittest.mock on Python 3. + By {user}`Elliott Sales de Andrade `, {issue}`179`. + +* Don't mask compile errors in setup.py. + By {user}`Joe Jevnik `, {issue}`197`. + +* Allow pickles when loading test fixture data. + By {user}`Elliott Sales de Andrade `, {issue}`193`. + +* Update `cpuinfo.py`. + By {user}`John Kirkham `, {issue}`202`. + +* Use `ensure_text` in JSON codecs. + By {user}`John Kirkham `, {issue}`207`. + +* Support Python 3.8. + By {user}`John Kirkham `, {issue}`208`. + + +(release_0.6.3)= + +## 0.6.3 + +* Drop support for 32-bit Windows. + By {user}`Alistair Miles `, {issue}`97`, {issue}`156`. + +* Raise a `TypeError` if an `object` array is passed to `ensure_bytes`. + By {user}`John Kirkham `, {issue}`162`. + +* Update Cython to 0.29.3. + By {user}`John Kirkham `, {issue}`165`. + + +(release_0.6.2)= + +## 0.6.2 + +* Handle (new) buffer protocol conforming types in `Pickle.decode`. + By {user}`John Kirkham `, {issue}`143`, {issue}`150`. + +* Use (new) buffer protocol in `MsgPack` codec `decode()` method. + By {user}`John Kirkham `, {issue}`148`. + +* Use (new) buffer protocol in `JSON` codec `decode()` method. + By {user}`John Kirkham `, {issue}`151`. + +* Avoid copying into data in `GZip`'s `decode()` method on Python 2. + By {user}`John Kirkham `, {issue}`152`. + +* Revert ndarray coercion of encode returned data. + By {user}`John Kirkham `, {issue}`155`. + +* The bundled c-blosc sources have been upgraded to version 1.15.0. By + {user}`Alistair Miles ` and {user}`John Kirkham `, {issue}`142`, {issue}`145`. + +(release_0.6.1)= + +## 0.6.1 + +* Resolved minor issue in backwards-compatibility tests (by {user}`Alistair Miles + `, {issue}`138`, {issue}`139`). + + +(release_0.6.0)= + +## 0.6.0 + +* The encoding format used by the {class}`JSON` and {class}`MsgPack` codecs has been + changed to resolve an issue with correctly encoding and decoding some object arrays. + Now the encoded data includes the original shape of the array, which enables the + correct shape to be restored on decoding. The previous encoding format is still + supported, so that any data encoded using a previous version of numcodecs can still be + read. Thus no changes to user code and applications should be required, other + than upgrading numcodecs. By {user}`Jerome Kelleher `; {issue}`74`, + {issue}`75`. + +* Updated the msgpack dependency (by {user}`Jerome Kelleher `; + {issue}`74`, {issue}`75`). + +* Added support for ppc64le architecture by updating `cpuinfo.py` from upstream (by + {user}`Anand S `; {issue}`82`). + +* Allow {class}`numcodecs.blosc.Blosc` compressor to run on systems where locks are not present (by + {user}`Marcus Kinsella `, {issue}`83`; and {user}`Tom White `, + {issue}`93`). + +* Drop Python 3.4 (by {user}`John Kirkham `; {issue}`89`). + +* Add Python 3.7 (by {user}`John Kirkham `; {issue}`92`). + +* Add codec {class}`numcodecs.gzip.GZip` to replace `gzip` alias for `zlib`, + which was incorrect (by {user}`Jan Funke `; {issue}`87`; and {user}`John Kirkham `, {issue}`134`). + +* Corrects handling of `NaT` in `datetime64` and `timedelta64` in various + compressors (by {user}`John Kirkham `; {issue}`127`, {issue}`131`). + +* Improvements to the compatibility layer used for normalising inputs to encode + and decode methods in most codecs. This removes unnecessary memory copies for + some codecs, and also simplifies the implementation of some codecs, improving + code readability and maintainability. By {user}`John Kirkham ` and + {user}`Alistair Miles `; {issue}`119`, {issue}`121`, {issue}`128`. + +* Return values from encode() and decode() methods are now returned as numpy + arrays for consistency across codecs. By {user}`John Kirkham `, + {issue}`136`. + +* Improvements to handling of errors in the {class}`numcodecs.blosc.Blosc` and + {class}`numcodecs.lz4.LZ4` codecs when the maximum allowed size of an input + buffer is exceeded. By {user}`Jerome Kelleher `, {issue}`80`, + {issue}`81`. + + +(release_0.5.5)= + +## 0.5.5 + +* The bundled c-blosc sources have been upgraded to version 1.14.3 ({issue}`72`). + + +(release_0.5.4)= + +## 0.5.4 + +* The bundled c-blosc sources have been upgraded to version 1.14.0 ({issue}`71`). + + +(release_0.5.3)= + +## 0.5.3 + +* The test suite has been migrated to use pytest instead of nosetests + ({issue}`61`, {issue}`62`). + +* The bundled c-blosc library has been updated to version 1.13.4 ({issue}`63`, + {issue}`64`). + + +(release_0.5.2)= + +## 0.5.2 + +* Add support for encoding None values in VLen... codecs ({issue}`59`). + + +(release_0.5.1)= + +## 0.5.1 + +* Fixed a compatibility issue with the Zlib codec to ensure it can handle + bytearray objects under Python 2.7 ({issue}`57`). +* Restricted the {class}`numcodecs.categorize.Categorize` codec to object + ('O') and unicode ('U') dtypes and disallowed bytes ('S') dtypes because + these do not round-trip through JSON configuration. + + +(release_0.5.0)= + +## 0.5.0 + +* Added new codecs for encoding arrays with variable-length unicode strings + ({class}`numcodecs.vlen.VLenUTF8`), variable-length byte strings + ({class}`numcodecs.vlen.VLenBytes`) and variable-length numerical arrays + ({class}`numcodecs.vlen.VLenArray`) ({issue}`56`). + + +(release_0.4.1)= + +## 0.4.1 + +* Resolved an issue where providing an array with dtype `object` as the destination + when decoding could cause segfaults with some codecs ({issue}`55`). + + +(release_0.4.0)= + +## 0.4.0 + +* Added a new {class}`numcodecs.json.JSON` codec as an alternative for encoding of + object arrays ({issue}`54`). + + +(release_0.3.1)= + +## 0.3.1 + +* Revert the default shuffle argument to SHUFFLE (byte shuffle) for the + {class}`numcodecs.blosc.Blosc` codec for compatibility and consistency with previous + code. + + +(release_0.3.0)= + +## 0.3.0 + +* The {class}`numcodecs.blosc.Blosc` codec has been made robust for usage in both + multithreading and multiprocessing programs, regardless of whether Blosc has been + configured to use multiple threads internally or not ({issue}`41`, {issue}`42`). + +* The {class}`numcodecs.blosc.Blosc` codec now supports an `AUTOSHUFFLE` argument + when encoding (compressing) which activates bit- or byte-shuffle depending on the + itemsize of the incoming buffer ({issue}`37`, {issue}`42`). This is also now the + default. + +* The {class}`numcodecs.blosc.Blosc` codec now raises an exception when an invalid + compressor name is provided under all circumstances ({issue}`40`, {issue}`42`). + +* The bundled version of the c-blosc library has been upgraded to version 1.12.1 + ({issue}`45`, {issue}`42`). + +* An improvement has been made to the system detection capabilities during compilation + of C extensions (by {user}`Prakhar Goel `; {issue}`36`, {issue}`38`). + +* Arrays with datetime64 or timedelta64 can now be passed directly to compressor codecs + ({issue}`39`, {issue}`46`). + + +(release_0.2.1)= + +## 0.2.1 + +The bundled c-blosc library has been upgraded to version 1.11.3 ({issue}`34`, {issue}`35`). + + +(release_0.2.0)= + +## 0.2.0 + +New codecs: + +* The {class}`numcodecs.quantize.Quantize` codec, which provides support for reducing the precision + of floating-point data, has been ported over from Zarr ({issue}`28`, {issue}`31`). + +Other changes: + +* The {class}`numcodecs.zlib.Zlib` codec is now also registered under the alias 'gzip' + ({issue}`29`, {issue}`32`). + +Maintenance work: + +* A data fixture has been added to the test suite to add some protection against changes to codecs + that break backwards-compatibility with data encoded using a previous release of numcodecs + ({issue}`30`, {issue}`33`). + + +(release_0.1.1)= + +## 0.1.1 + +This release includes a small modification to the setup.py script to provide greater control over +how compiler options for different instruction sets are configured ({issue}`24`, +{issue}`27`). + + +(release_0.1.0)= + +## 0.1.0 + +New codecs: + +* Two new compressor codecs {class}`numcodecs.zstd.Zstd` and {class}`numcodecs.lz4.LZ4` + have been added ({issue}`3`, {issue}`22`). These provide direct support for + compression/decompression using [Zstandard](https://github.com/facebook/zstd) and + [LZ4](https://github.com/lz4/lz4) respectively. + +* A new {class}`numcodecs.msgpacks.MsgPack` codec has been added which uses + [msgpack-python](https://github.com/msgpack/msgpack-python) to perform encoding/decoding, + including support for arrays of Python objects + ([Jeff Reback](https://github.com/jreback); {issue}`5`, {issue}`6`, {issue}`8`, + {issue}`21`). + +* A new {class}`numcodecs.pickles.Pickle` codec has been added which uses the Python pickle protocol + to perform encoding/decoding, including support for arrays of Python objects + ([Jeff Reback](https://github.com/jreback); {issue}`5`, {issue}`6`, {issue}`21`). + +* A new {class}`numcodecs.astype.AsType` codec has been added which uses NumPy to perform type + conversion ([John Kirkham](https://github.com/jakirkham); {issue}`7`, {issue}`12`, + {issue}`14`). + +Other new features: + +* The {class}`numcodecs.lzma.LZMA` codec is now supported on Python 2.7 if + [backports.lzma](https://pypi.python.org/pypi/backports.lzma) is installed + ([John Kirkham](https://github.com/jakirkham); {issue}`11`, {issue}`13`). + +* The bundled c-blosc library has been upgraded to version + [1.11.2](https://github.com/Blosc/c-blosc/releases/tag/v1.11.2) ({issue}`10`, + {issue}`18`). + +* An option has been added to the {class}`numcodecs.blosc.Blosc` codec to allow the block size to + be manually configured ({issue}`9`, {issue}`19`). + +* The representation string for the {class}`numcodecs.blosc.Blosc` codec has been tweaked to + help with understanding the shuffle option ({issue}`4`, {issue}`19`). + +* Options have been added to manually control how the C extensions are built regardless of the + architecture of the system on which the build is run. To disable support for AVX2 set the + environment variable "DISABLE_NUMCODECS_AVX2". To disable support for SSE2 set the environment + variable "DISABLE_NUMCODECS_SSE2". To disable C extensions altogether set the environment variable + "DISABLE_NUMCODECS_CEXT" ({issue}`24`, {issue}`26`). + +Maintenance work: + +* CI tests now run under Python 3.6 as well as 2.7, 3.4, 3.5 ({issue}`16`, {issue}`17`). + +* Test coverage is now monitored via + [coveralls](https://coveralls.io/github/alimanfoo/numcodecs?branch=master) + ({issue}`15`, {issue}`20`). + + +(release_0.0.1)= + +## 0.0.1 + +Fixed project description in setup.py. + + +(release_0.0.0)= + +## 0.0.0 + +First release. This version is a port of the `codecs` module from [Zarr](https://zarr.readthedocs.io) 2.1.0. The following changes have been made from +the original Zarr module: + +* Codec classes have been re-organized into separate modules, mostly one per + codec class, for ease of maintenance. +* Two new codec classes have been added based on 32-bit checksums: + {class}`numcodecs.checksum32.CRC32` and {class}`numcodecs.checksum32.Adler32`. +* The Blosc extension has been refactored to remove code duplications related + to handling of buffer compatibility. diff --git a/docs/release.rst b/docs/release.rst deleted file mode 100644 index 093466a6a..000000000 --- a/docs/release.rst +++ /dev/null @@ -1,1124 +0,0 @@ -Release notes -============= - -.. - # Unindent the section between releases in order - # to document your changes. On releases it will be - # re-indented so that it does not show up in the notes. - - .. _unreleased: - - Unreleased - ---------- - - -.. _unreleased: - -Unreleased ----------- - -Maintenance -~~~~~~~~~~~ - -* Move tests out of installable package into top-level ``tests/`` directory. - By :user:`Max Jones `, :issue:`829` - -.. _release_0.16.5: - -0.16.5 ------- - -Fixes -~~~~~ - -* Restore prior optional dependency behavior for `'crc32c'` group. - By :user:`Davis Bennett `, :issue:`802` - -.. _release_0.16.4: - -0.16.4 ------- - -Enhancements -~~~~~~~~~~~~ - -* The Zstandard codec can now decode bytes containing multiple frames. - By :user:`Mark Kittisopikul `, :issue:`757` - -* Add Python 3.14 support to the test matrix. - By :user:`Max Jones `, :issue:`799` - -* Add Python 3.14 wheel builds. - By :user:`Davis Bennett `, :issue:`796` - -Fixes -~~~~~ - -* Fix ``PCodec.decode()`` to handle numpy arrays by calling ``ensure_bytes`` on input buffers. - By :user:`Sam Levang `, :issue:`756` - -Maintenance -~~~~~~~~~~~ - -* Replace ``crc32c`` dependency with ``google-crc32c``. - By :user:`Joe Hamman `, :issue:`792` - -* Refactor ``numcodecs.zarr3`` to re-export classes from ``zarr.codecs._numcodecs`` - instead of defining them locally. - By :user:`Davis Bennett `, :issue:`780` - -* Add testing against zarr-python main branch. - By :user:`Max Jones `, :issue:`800` - -* Add pcodec and zfpy to 32-bit CI testing. - By :user:`Mark Kittisopikul `, :issue:`786` - -* Improve CRC32C codec test skipping to properly report skipped tests using ``pytest.skipif``. - By :user:`Agriya Khetarpal `, :issue:`740` - -* Add nightly wheel upload to scientific-python channel. - By :user:`Ian Hunt-Isaak `, :issue:`787` - -* Fetch tags to version nightly wheels correctly. - By :user:`Ian Hunt-Isaak `, :issue:`790` - -* Bump ``pypa/cibuildwheel`` from 2.23.2 to 3.1.3. - By :user:`dependabot `, :issue:`771` - -* Bump ``actions/checkout`` from 4 to 5. - By :user:`dependabot `, :issue:`775` - -* Bump ``actions/download-artifact`` from 4 to 6. - By :user:`dependabot `, :issue:`794` - -* Bump ``prefix-dev/setup-pixi`` from 0.8.14 to 0.9.0. - By :user:`dependabot `, :issue:`783` - -* Bump ``jirutka/setup-alpine`` from 1.2.0 to 1.3.0. - By :user:`dependabot `, :issue:`789` - -* Bump ``conda-incubator/setup-miniconda`` from 3.1.1 to 3.2.0. - By :user:`dependabot `, :issue:`753` - - -.. _release_0.16.3: - -0.16.3 ------- - -Fixes -~~~~~ - -* Fix Zstd decompression negative size issue on 32-bit platforms. - By :user:`Mark Kittisopikul `, :issue:`782` - -Maintenance -~~~~~~~~~~~ - -* Test 32-bit support via x86 Alpine Setup Action Linux. - By :user:`Mark Kittisopikul `, :issue:`784` - -* Bump to main branch of c-blosc to fix gcc15 compilation. - By :user:`Thomas Caswell `, :issue:`772` - - -.. _release_0.16.2: - -0.16.2 ------- - -Enhancements -~~~~~~~~~~~~ - -* Add streaming decompression for ZSTD_CONTENTSIZE_UNKNOWN case. - By :user:`Mark Kittisopikul `, :issue:`707` - -Maintenance -~~~~~~~~~~~ - -* Handle zarr 3.1.0 compatibility. - By :user:`Davis Bennett `, :issue:`766` - - -.. _release_0.16.1: - -0.16.1 ------- - -Enhancements -~~~~~~~~~~~~ - -* In ``vlen``, define and use ``const`` ``HEADER_LENGTH``. - By :user:`John Kirkham `, :issue:`723` - -* All codecs are now pickleable. - By :user:`Tom Nicholas `, :issue:`745` - -Fixes -~~~~~ - -* Fix ``const`` discard warnings in ``fletcher32``. - By :user:`John Kirkham `, :issue:`728` - -* Update ``license*`` metadata and include third-party licenses. - By :user:`John Kirkham `, :issue:`729` - -* Add ``#ifndef`` guard around ``PyBytes_RESIZE``. - By :user:`John Kirkham `, :issue:`732` - -* Remove ``typesize`` from ``Blosc.get_config`` output. - By :user:`Ilan Gold `, :issue:`739` - - -.. _release_0.16.0: - -0.16.0 ------- - -Enhancements -~~~~~~~~~~~~ - -* Add ``typesize`` argument to ``Blosc`` to allow for buffers that are passed to ``encode`` - use that information. zarr v3 currently has its Blosc codec as bytes-to-bytes but does retain - the size information so using it here allows for massive compression ratio gains. - By :user:`Ilan Gold ` -* Add support for the Linux aarch64 architecture, - By :user:`Agriya Khetarpal `, :issue:`288`. -* Preallocate output buffers and resize directly as needed. - By :user:`John Kirkham `, :issue:`656` -* ``Checksum32`` is now an abstract base clase, with ``checksum()`` as an abstract method. - This helps ensure that any subclasses implement the ``checksum()`` method. - :issue:`711` - -Removals -~~~~~~~~ - -The following ``blosc`` funcitons are removed, with no replacement. -This is because they were not intended to be public API. - -- ``numcodecs.blosc.init`` -- ``numcodecs.blosc.destroy`` -- ``numcodecs.blosc.compname_to_compcode`` -- ``numcodecs.blosc.cbuffer_sizes`` -- ``numcodecs.blosc.cbuffer_metainfo`` - -In addition, ``numcodecs.blosc.decompress_partial`` is removed as -has always been experimental and there is no equivalent in the official -blsoc Python package. -By :user:`David Stansby `, :issue:`712` - -Maintenance -~~~~~~~~~~~ -* Replace internal ``Buffer`` usage with ``memoryview``\ s. - By :user:`John Kirkham `, :issue:`656` -* Bump the minimum macOS deployment target for x86_64 to 10.13. - By :user:`Agriya Khetarpal `, :issue:`288`. - - -0.15.1 ------- - -Improvements -~~~~~~~~~~~~ -* Add streaming decompression for ZSTD (:issue:`699`) - By :user:`Mark Kittisopikul `. -* Raise a custom `UnknownCodecError` when trying to retrieve an unavailable codec. - By :user:`Cas Wognum `. - -Fixes -~~~~~ -* Remove redundant ``id`` from codec metadata serialization in Zarr3 codecs. - By :user:`Norman Rzepka `, :issue:`685` - -.. _release_0.15.0: - -0.15.0 ------- - -Breaking changes -~~~~~~~~~~~~~~~~ -* All arguments to the ``PCodec`` constructor except for ``level`` - are now keyword only, to support the updated API. - By :user:`Sam Levang `, :issue:`623` - -Deprecations -~~~~~~~~~~~~ -The following ``blosc`` functions are deprecated, with no replacement. -This is because they are not intended to be public API. - -- ``numcodecs.blosc.init`` -- ``numcodecs.blosc.destroy`` -- ``numcodecs.blosc.compname_to_compcode`` -- ``numcodecs.blosc.cbuffer_sizes`` -- ``numcodecs.blosc.cbuffer_metainfo`` - -In addition, ``numcodecs.blosc.decompress_partial`` is deprecated as -has always been experimental and there is no equivalent in the official -blsoc Python package. -By :user:`David Stansby `, :issue:`619` - -Fixes -~~~~~ -* Fixes issue with ``Delta`` Zarr 3 codec not working with ``astype``. - By :user:`Norman Rzepka `, :issue:`664` -* Cleanup ``PCodec`` soft dependency. - Previously importing ``numcodecs.pcodec`` would work if ``pcodec`` is not installed, - but now it will fail to import. This mirrors the behaviour of other optional dependencies. - By :user:`John Kirkham `, :issue:`647` -* Fixes issues with the upcoming ``zarr`` 3.0.0 release. - By :user:`Norman Rzepka `, :issue:`675` - -* Removed Version Check: The previous code included a check for the `NumPy` version - and a warning if the version was incompatible with `zfpy`. - This check has been removed because `zfpy` now supports the newer versions of `NumPy`. - By :user:`Meher Gajula `, :issue:`672` - -Improvements -~~~~~~~~~~~~ -* Add support for ``pcodec`` 0.3. This exposes the new ``delta_spec`` - and ``paging_spec`` arguments, but maintains full backwards - compatibility for data written with older package versions. - By :user:`Sam Levang `, :issue:`623` -* If an import error is raised when trying to define a codec that is *not* - an optional dependency, it is no longer silently caught. Instead it will - be propagated to the user, as this indicates an issue with the installed - package. - - Import errors caused by optional dependencies (ZFPY, MsgPack, CRC32C, and PCodec) - are still silently caught. - By :user:`David Stansby `, :issue:`550`. - -.. _release_0.14.1: - -0.14.1 ------- - -Fixes -~~~~~ -* Cleanups to the ``crc32c`` soft dependency. - Whereas in ``numcodecs`` 0.14.0 a runtime error was raised if - the ``CRC32C`` codec used without the ``crc32c`` installed, - the ``CRC32C`` codec is no longer defined at import time if - ``crc32c`` is not installed. This has been changed to match - the behaviour of other optional dependencies/codecs. - By :user:`John Kirkham `, :issue:`637` - -Improvements -~~~~~~~~~~~~ -* Add `noexcept` to `_utils` C-equiv functions - By :user:`John Kirkham `, :issue:`641`. - -.. _release_0.14.0: - -0.14.0 ------- - -Enhancements -~~~~~~~~~~~~ -* Add Crc32c checksum codec. - By :user:`Norman Rzepka `, :issue:`613`. -* Add codec wrappers for Zarr 3. - By :user:`Norman Rzepka `, :issue:`524` -* Added mypy type checking to continuous integration. - By :user:`David Stansby `, :issue:`460`. - -Fixes -~~~~~ -* Fix in-place mutation of input array in `BitRound`. - By :user:`Sam Levang `, :issue:`608` -* Fix an issue where importing numcodecs would lock the state of `multiprocessing` - and prevent user code to call `multiprocessing.set_start_method("spawn")` - subsequently. - By :user:`Clément Robert ` :issue:`522` - -Maintenance -~~~~~~~~~~~ -* The minimum supported Python version is now Python 3.11. - By :user:`David Stansby `, :issue:`622` -* The minimum supported numpy version is now 1.24. - By :user:`David Stansby `, :issue:`622` - -.. _release_0.13.1: - -0.13.1 ------- - -Breaking changes -~~~~~~~~~~~~~~~~ -* `Zstd.default_level`, `Zstd.min_level`, and `Zstd.max_level` are now class methods - instead of properties. This means they must now be called like ``Zstd.default_level()`` - instead of ``Zstd.default_level``. This breaking change has been made because Python 3.13 - removes support for class properties. - By :user:`David Stansby `, :issue:`576`. - -Enhancements -~~~~~~~~~~~~ - -* Update bundled c-blosc to v1.26.1. This updates Zlib to v1.3.1 - and Zstd to v1.5.6. - By :user:`David Stansby `, :issue:`560`. -* Added support for Python 3.13 :user:`David Stansby `, :issue:`576`. -* Cleaned up the table of contents in the documentation to list codecs by category - :user:`David Stansby `, :issue:`458`. - -Maintenance -~~~~~~~~~~~ -* Change format() and old string formatting to f-strings. - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`439`. -* Remove pin on Sphinx - By :user:`Elliott Sales de Andrade `, :issue:`552`. - - -.. _release_0.13.0: - -0.13.0 ------- - -Enhancements -~~~~~~~~~~~~ -* Add checksum flag to zstd and sets the default level to 0. - By :user:`Norman Rzepka `, :issue:`519`. -* Add PCodec - By :user:`Ryan Abernathey `, :issue:`501`. -* Use PyData theme for docs - By :user:`John Kirkham `, :issue:`485`. -* Improve the structure of docs - By :user:`David Stansby `, :issue:`458`. - -Fix -~~~ -* Fix VLenUTF8 encoding for read-only buffers. - By :user:`Isaac Virshup `, :issue:`514`. -* Fix skip of entry points backport tests - By :user:`Elliott Sales de Andrade `, :issue:`487`. -* Fix Upgrade to Zstd 1.5.5 due to potential corruption. - By :user:`Mark Kittisopikul `, :issue:`429` -* Add version constraint(<2.0) for numpy in zfpy. - By :user:`Tom Liang `, :issue:`540`. - -Maintenance -~~~~~~~~~~~ -* The minimum supported Python version is now Python 3.10. - By :user:`David Stansby `, :issue:`531` -* Add numpy 2 compatibility. - By :user:`David Stansby `, :issue:`535` -* Update c-blosc to 1.21.0 to 1.21.5, zstd from 1.4.8 to 1.5.5, - lz4 from 1.9.3 to 1.9.4, and zlib from 1.2.8 to to 1.2.13 - By :user:`Mark Kittisopikul `, :issue:`500` - - -.. _release_0.12.1: - -0.12.1 ------- - -Fix -~~~ - -* ``Codec`` is now derived from ``abc.ABC`` - By :user:`Mads R. B. Kristensen `, :issue:`472`. -* Fix handling of entry points on older Python versions where ``importlib_metadata`` compatibility is concerned - By :user:`Vyas Ramasubramani `, :issue:`478`. -* Make shuffle pyx functions ``noexcept`` - By :user:`Martin Durant `, :issue:`477`. - -.. _release_0.12.0: - -0.12.0 ------- - -Enhancements -~~~~~~~~~~~~ - -* Add ``fletcher32`` checksum codec - By :user:`Martin Durant `, :issue:`410`. -* Add ``jenkins_lookup3`` checksum codec - By :user:`Mark Kittisopkul `, :issue:`445`. -* Support Python 3.12. - By :user:`John Kirkham `, :issue:`471`. - -Fix -~~~ - -* Fixed docs/Makefile error message when sphinx is not present - By :user:`Mark Kittisopikul `, :issue:`451`. -* Add missing RTD requirements - By :user:`John Kirkham `, :issue:`455`. - -Maintenance -~~~~~~~~~~~ - -* Cleanup ``import``\ s in ``adhoc/blosc_memleak_check.py`` - By :user:`John Kirkham `, :issue:`408`. - -.. _release_0.11.0: - -0.11.0 ------- - -Fix -~~~ - -* Speed up isinstance checks of :class:`numcodecs.ndarray_like.NDArrayLike`, - :class:`numcodecs.ndarray_like.DType` and :class:`numcodecs.ndarray_like.FlagsObj`. - By :user:`Andreas Poehlmann `, :issue:`379`. - -Maintenance -~~~~~~~~~~~ - -* Remove unnecessary None argument to .get(), it is the default value. - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`395`. - -* Apply refurb suggestions. - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`372`. - -* Migrate codespell configuration to pyproject.toml and get rid of setup.cfg. - By :user:`Dimitri Papadopoulos Orfanos ` :issue:`374`. - -* Unvendor cpuinfo. - By :user:`Dimitri Papadopoulos Orfanos ` :issue:`373`. - -* Drop headers. - By :user:`John Kirkham `, :issue:`375`. - -* Remove Python 2 code. - By :user:`Dimitri Papadopoulos Orfanos ` :issue:`368`, - :issue:`387`. - -* Support Python 3.11. - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`369`. - -* Drop Python 3.7. - By :user:`John Kirkham `,, :issue:`405` :issue:`406`. - -* Test with zfpy 1.0.0. - By :user:`John Kirkham `, :issue:`385`. - -* Remove vendored C files and re-generate them on the fly using Cython. - Add a pyproject.toml file to define Cython as a build dependency. - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`369`. - -* Update cpuinfo to 9.0.0. - By :user:`Dimitri Papadopoulos Orfanos ` :issue:`370`. - -* Add tests for all registry classes. - By :user:`Josh Moore `, :issue:`349`. - -* Add workflow to label PRs with "needs release notes". - By :user:`Saransh Chopra ` :issue:`396`. - -* Finish ``pyproject.toml`` migration. - By :user:`John Kirkham ` :issue:`382`. - -* Drop tox. - By :user:`John Kirkham `, :issue:`384`. - -* Update ReadTheDocs. - By :user:`John Kirkham `, :issue:`383`. - -* Use `conda-incubator/setup-miniconda@v2.2.0` (and use Conda on Linux). - By :user:`John Kirkham ` :issue:`398`. - -* Bring coverage back up to 100%. - By :user:`John Kirkham ` and :user:`Martin Durant `, - :issue:`392` and :issue:`393`. - -* Collect coverage on all OSes & enable Codecov. - By :user:`John Kirkham `, :issue:`386`, :issue:`388`, - :issue:`390`, :issue:`391`. - -.. _release_0.10.2: - -0.10.2 ------- - -Fix -~~~ - -* Add BitRound (0.10.0) to registry. - By :user:`Josh Moore `, :issue:`342`. - -.. _release_0.10.1: - -0.10.1 ------- - -Maintenance -~~~~~~~~~~~ - -* Add entrypoints to setup.py. - By :user:`Josh Moore `, :issue:`332`. - -* Fix spelling. - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`336`. - -* Drop Python 3.6 from tests - By :user:`Dimitri Papadopoulos Orfanos `, - :issue:`338`, :issue:`339`. - -* Remove trailing spaces and empty lines. - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`341`. - -* Add LGTM.com configuration file - By :user:`Dimitri Papadopoulos Orfanos `, :issue:`337`. - -.. _release_0.10.0: - -0.10.0 ------- - -Enhancements -~~~~~~~~~~~~ - -* Add support of alternative array classes (other than NumPy arrays) - By :user:`Mads R. B. Kristensen `, :issue:`305`. - -* Add ability to find codecs via entrypoint `numcodecs.codecs`. - By :user:`Martin Durant `, :issue:`290`. - -* Add bitround codec - By :user:`Ryan Abernathy ` and :user:`Martin Durant `, :issue:`298`. - -* Introduce a flat option to ensure_contiguous_ndarray to switch off flatten for ZFPY codec - By :user:`Haiying Xu `, :issue:`307`. - -Bug fixes -~~~~~~~~~ - -* Fix a flatten array error for ZFPY, ZFPY codec is supported on Python 3.9 - and 3.10 on Linux and MacOS, the docs about ZFPY is also available. - By :user:`Haiying Xu `, :user:`John Kirkham `, - :user:`Ryan Abernathey ` :issue:`303`. - -* Codex: make encode and decode @abstractmethods - By :user:`Mads R. B. Kristensen `, :issue:`306`. - -* Fix expected result test for Shuffle. - By :user:`Elliott Sales de Andrade `, :issue:`282`. - -Maintenance -~~~~~~~~~~~ - -* Multiple code linting fixes. - By :user:`Dimitri Papadopoulos Orfanos `, - :issue:`295`, :issue:`294`, :issue:`293`, and :issue:`292`. - -* Drop Python 3.6 - By :user:`Josh Moore `, :issue:`318`. - -* Fix macOS Python 3.10 - By :user:`John Kirkham `, :issue:`311`. - -* chore: bump cibuildwheel version, use action - By :user:`Henry Schreiner `, :issue:`309`. - -* Specify `language` as `'en'` instead of `None`. - By :user:`John Kirkham `, :issue:`329`. - -* Move `master` to `main`. - By :user:`John Kirkham `, :issue:`322`. - -* Drop `fastparquet` benchmark. - By :user:`John Kirkham `, :issue:`321`. - -* Trim wheel builds. - By :user:`John Kirkham `, :issue:`320`. - -.. _release_0.9.1: - -0.9.1 ------ - -* Fix inaccurate docstrings for Blosc. - By :user:`James Webber `, :issue:`287`. - -.. _release_0.9.0: - -0.9.0 ------ - -* c-blosc upgrade 1.18.1 -> 1.21.0. - Warning: this temporarily removes support for snappy compression! - By :user:`kindjacket `, :issue:`283`. - -* Fix an ImportError with Blosc on Android. - By :user:`Daniel Jewell `, :issue:`284`. - -.. _release_0.8.1: - -0.8.1 ------ - -* Fix an ImportError with Blosc on Android. - By :user:`Daniel Jewell `, :issue:`284`. - -.. _release_0.8.0: - -0.8.0 ------ - -* The :class:`numcodecs.zfpy.ZFPY` codec is now supported on Python 3.8 if - `zfpy==0.5.5 `_ is installed. - By :user:`haiying xu `, :issue:`229`. - -* Add support for byte Shuffle filter - By :user:`Paul Branson ` and :user:`Martin Durant ` :issue:`273`. - -* Update Windows + Mac CI to run all tests. - By :user:`Jackson Maxfield Brown `, :issue:`276`. - Help from :user:`Oleg Höfling `, :issue:`273`. - -* Update cpuinfo to 8.0.0. - By :user:`Florian Jetter `, :issue:`280`. - -* Drop out-of-date manual release docs. - By :user:`John Kirkham `, :issue:`272`. - -* Add support for Python 3.9 and Update GitHub Actions. - -.. _release_0.7.3: - -0.7.3 ------ - -* Add support for Python 3.9 and Update GitHub Actions. - By :user:`Jackson Maxfield Brown `, :issue:`270`. - -* Remove support for Python 3.5 which is end of life. While the code base might - still be compatible; the source dist and wheel are marked as Python 3.6+ and - pip will not install them. Continuous integration on Python 3.5 has been - disabled. - By :user:`Matthias Bussonnier `, :issue:`266` and :issue:`267`. - -.. _release_0.7.2: - -0.7.2 ------ - -* Disable avx2 for wheel. - By :user:`Grzegorz Bokota `, :issue:`253`. - -* Add Base64 fixtures. - By :user:`John Kirkham `, :issue:`251`. - -* Update docs regarding wheels. - By :user:`Josh Moore `, :issue:`250`. - - -.. _release_0.7.1: - -0.7.1 ------ - -* Fix build of wheels. - By :user:`Grzegorz Bokota `, :issue:`244`. - -.. _release_0.7.0: - -0.7.0 ------ - -* Automatically release to PyPI. - By :user:`Josh Moore `, :issue:`241`. - -* Build wheels on github actions. - By :user:`Grzegorz Bokota `, :issue:`224`. - -* Add Base64 codec. - By :user:`Trevor Manz `, :issue:`176`. - -* Add partial decompression of Blosc compressed arrays. - By :user:`Andrew Fulton `, :issue:`235`. - -* Remove LegacyJSON codec. - By :user:`James Bourbeau `, :issue:`226`. - -* Remove LegacyMsgPack codec. - By :user:`James Bourbeau `, :issue:`218`. - -* Drop support for Python 2. - By :user:`James Bourbeau `, :issue:`220`. - - -.. _release_0.6.4: - -0.6.4 ------ - -* Update Cython to 0.29.14. - By :user:`John Kirkham `, :issue:`168`, :issue:`177`, :issue:`204`. - -* The bundled c-blosc sources have been upgraded to version 1.17.0. - This fixes compilation with newer versions of gcc. - By :user:`Joe Jevnik `, :issue:`194`. - -* Create ``.pep8speaks.yml``. By :user:`Alistair Miles `. - -* Simplify datetime/timedelta check. - By :user:`John Kirkham `, :issue:`170`, :issue:`171`. - -* Update URL metadata for PyPI. - By :user:`Elliott Sales de Andrade `, :issue:`178`. - -* Enable pytest rewriting in test helper functions. - By :user:`Elliott Sales de Andrade `, :issue:`185`. - -* Rewrites the ``ensure_text`` implementation. - By :user:`John Kirkham `, :issue:`201`, :issue:`205`, :issue:`206`. - -* Add macOS to CI. - By :user:`Alistair Miles `, :issue:`192`. - -* Fix test failures on big-endian systems. - By :user:`Elliott Sales de Andrade `, :issue:`186`. - -* Use unittest.mock on Python 3. - By :user:`Elliott Sales de Andrade `, :issue:`179`. - -* Don't mask compile errors in setup.py. - By :user:`Joe Jevnik `, :issue:`197`. - -* Allow pickles when loading test fixture data. - By :user:`Elliott Sales de Andrade `, :issue:`193`. - -* Update ``cpuinfo.py``. - By :user:`John Kirkham `, :issue:`202`. - -* Use ``ensure_text`` in JSON codecs. - By :user:`John Kirkham `, :issue:`207`. - -* Support Python 3.8. - By :user:`John Kirkham `, :issue:`208`. - - -.. _release_0.6.3: - -0.6.3 ------ - -* Drop support for 32-bit Windows. - By :user:`Alistair Miles `, :issue:`97`, :issue:`156`. - -* Raise a ``TypeError`` if an ``object`` array is passed to ``ensure_bytes``. - By :user:`John Kirkham `, :issue:`162`. - -* Update Cython to 0.29.3. - By :user:`John Kirkham `, :issue:`165`. - - -.. _release_0.6.2: - -0.6.2 ------ - -* Handle (new) buffer protocol conforming types in ``Pickle.decode``. - By :user:`John Kirkham `, :issue:`143`, :issue:`150`. - -* Use (new) buffer protocol in ``MsgPack`` codec `decode()` method. - By :user:`John Kirkham `, :issue:`148`. - -* Use (new) buffer protocol in ``JSON`` codec `decode()` method. - By :user:`John Kirkham `, :issue:`151`. - -* Avoid copying into data in ``GZip``'s `decode()` method on Python 2. - By :user:`John Kirkham `, :issue:`152`. - -* Revert ndarray coercion of encode returned data. - By :user:`John Kirkham `, :issue:`155`. - -* The bundled c-blosc sources have been upgraded to version 1.15.0. By - :user:`Alistair Miles ` and :user:`John Kirkham `, :issue:`142`, :issue:`145`. - -.. _release_0.6.1: - -0.6.1 ------ - -* Resolved minor issue in backwards-compatibility tests (by :user:`Alistair Miles - `, :issue:`138`, :issue:`139`). - - -.. _release_0.6.0: - -0.6.0 ------ - -* The encoding format used by the :class:`JSON` and :class:`MsgPack` codecs has been - changed to resolve an issue with correctly encoding and decoding some object arrays. - Now the encoded data includes the original shape of the array, which enables the - correct shape to be restored on decoding. The previous encoding format is still - supported, so that any data encoded using a previous version of numcodecs can still be - read. Thus no changes to user code and applications should be required, other - than upgrading numcodecs. By :user:`Jerome Kelleher `; :issue:`74`, - :issue:`75`. - -* Updated the msgpack dependency (by :user:`Jerome Kelleher `; - :issue:`74`, :issue:`75`). - -* Added support for ppc64le architecture by updating `cpuinfo.py` from upstream (by - :user:`Anand S `; :issue:`82`). - -* Allow :class:`numcodecs.blosc.Blosc` compressor to run on systems where locks are not present (by - :user:`Marcus Kinsella `, :issue:`83`; and :user:`Tom White `, - :issue:`93`). - -* Drop Python 3.4 (by :user:`John Kirkham `; :issue:`89`). - -* Add Python 3.7 (by :user:`John Kirkham `; :issue:`92`). - -* Add codec :class:`numcodecs.gzip.GZip` to replace ``gzip`` alias for ``zlib``, - which was incorrect (by :user:`Jan Funke `; :issue:`87`; and :user:`John Kirkham `, :issue:`134`). - -* Corrects handling of ``NaT`` in ``datetime64`` and ``timedelta64`` in various - compressors (by :user:`John Kirkham `; :issue:`127`, :issue:`131`). - -* Improvements to the compatibility layer used for normalising inputs to encode - and decode methods in most codecs. This removes unnecessary memory copies for - some codecs, and also simplifies the implementation of some codecs, improving - code readability and maintainability. By :user:`John Kirkham ` and - :user:`Alistair Miles `; :issue:`119`, :issue:`121`, :issue:`128`. - -* Return values from encode() and decode() methods are now returned as numpy - arrays for consistency across codecs. By :user:`John Kirkham `, - :issue:`136`. - -* Improvements to handling of errors in the :class:`numcodecs.blosc.Blosc` and - :class:`numcodecs.lz4.LZ4` codecs when the maximum allowed size of an input - buffer is exceeded. By :user:`Jerome Kelleher `, :issue:`80`, - :issue:`81`. - - -.. _release_0.5.5: - -0.5.5 ------ - -* The bundled c-blosc sources have been upgraded to version 1.14.3 (:issue:`72`). - - -.. _release_0.5.4: - -0.5.4 ------ - -* The bundled c-blosc sources have been upgraded to version 1.14.0 (:issue:`71`). - - -.. _release_0.5.3: - -0.5.3 ------ - -* The test suite has been migrated to use pytest instead of nosetests - (:issue:`61`, :issue:`62`). - -* The bundled c-blosc library has been updated to version 1.13.4 (:issue:`63`, - :issue:`64`). - - -.. _release_0.5.2: - -0.5.2 ------ - -* Add support for encoding None values in VLen... codecs (:issue:`59`). - - -.. _release_0.5.1: - -0.5.1 ------ - -* Fixed a compatibility issue with the Zlib codec to ensure it can handle - bytearray objects under Python 2.7 (:issue:`57`). -* Restricted the :class:`numcodecs.categorize.Categorize` codec to object - ('O') and unicode ('U') dtypes and disallowed bytes ('S') dtypes because - these do not round-trip through JSON configuration. - - -.. _release_0.5.0: - -0.5.0 ------ - -* Added new codecs for encoding arrays with variable-length unicode strings - (:class:`numcodecs.vlen.VLenUTF8`), variable-length byte strings - (:class:`numcodecs.vlen.VLenBytes`) and variable-length numerical arrays - ((:class:`numcodecs.vlen.VLenArray`) (:issue:`56`). - - -.. _release_0.4.1: - -0.4.1 ------ - -* Resolved an issue where providing an array with dtype ``object`` as the destination - when decoding could cause segfaults with some codecs (:issue:`55`). - - -.. _release_0.4.0: - -0.4.0 ------ - -* Added a new :class:`numcodecs.json.JSON` codec as an alternative for encoding of - object arrays (:issue:`54`). - - -.. _release_0.3.1: - -0.3.1 ------ - -* Revert the default shuffle argument to SHUFFLE (byte shuffle) for the - :class:`numcodecs.blosc.Blosc` codec for compatibility and consistency with previous - code. - - -.. _release_0.3.0: - -0.3.0 ------ - -* The :class:`numcodecs.blosc.Blosc` codec has been made robust for usage in both - multithreading and multiprocessing programs, regardless of whether Blosc has been - configured to use multiple threads internally or not (:issue:`41`, :issue:`42`). - -* The :class:`numcodecs.blosc.Blosc` codec now supports an ``AUTOSHUFFLE`` argument - when encoding (compressing) which activates bit- or byte-shuffle depending on the - itemsize of the incoming buffer (:issue:`37`, :issue:`42`). This is also now the - default. - -* The :class:`numcodecs.blosc.Blosc` codec now raises an exception when an invalid - compressor name is provided under all circumstances (:issue:`40`, :issue:`42`). - -* The bundled version of the c-blosc library has been upgraded to version 1.12.1 - (:issue:`45`, :issue:`42`). - -* An improvement has been made to the system detection capabilities during compilation - of C extensions (by :user:`Prakhar Goel `; :issue:`36`, :issue:`38`). - -* Arrays with datetime64 or timedelta64 can now be passed directly to compressor codecs - (:issue:`39`, :issue:`46`). - - -.. _release_0.2.1: - -0.2.1 ------ - -The bundled c-blosc library has been upgraded to version 1.11.3 (:issue:`34`, :issue:`35`). - - -.. _release_0.2.0: - -0.2.0 ------ - -New codecs: - -* The :class:`numcodecs.quantize.Quantize` codec, which provides support for reducing the precision - of floating-point data, has been ported over from Zarr (:issue:`28`, :issue:`31`). - -Other changes: - -* The :class:`numcodecs.zlib.Zlib` codec is now also registered under the alias 'gzip' - (:issue:`29`, :issue:`32`). - -Maintenance work: - -* A data fixture has been added to the test suite to add some protection against changes to codecs - that break backwards-compatibility with data encoded using a previous release of numcodecs - (:issue:`30`, :issue:`33`). - - -.. _release_0.1.1: - -0.1.1 ------ - -This release includes a small modification to the setup.py script to provide greater control over -how compiler options for different instruction sets are configured (:issue:`24`, -:issue:`27`). - - -.. _release_0.1.0: - -0.1.0 ------ - -New codecs: - -* Two new compressor codecs :class:`numcodecs.zstd.Zstd` and :class:`numcodecs.lz4.LZ4` - have been added (:issue:`3`, :issue:`22`). These provide direct support for - compression/decompression using `Zstandard `_ and - `LZ4 `_ respectively. - -* A new :class:`numcodecs.msgpacks.MsgPack` codec has been added which uses - `msgpack-python `_ to perform encoding/decoding, - including support for arrays of Python objects - (`Jeff Reback `_; :issue:`5`, :issue:`6`, :issue:`8`, - :issue:`21`). - -* A new :class:`numcodecs.pickles.Pickle` codec has been added which uses the Python pickle protocol - to perform encoding/decoding, including support for arrays of Python objects - (`Jeff Reback `_; :issue:`5`, :issue:`6`, :issue:`21`). - -* A new :class:`numcodecs.astype.AsType` codec has been added which uses NumPy to perform type - conversion (`John Kirkham `_; :issue:`7`, :issue:`12`, - :issue:`14`). - -Other new features: - -* The :class:`numcodecs.lzma.LZMA` codec is now supported on Python 2.7 if - `backports.lzma `_ is installed - (`John Kirkham `_; :issue:`11`, :issue:`13`). - -* The bundled c-blosc library has been upgraded to version - `1.11.2 `_ (:issue:`10`, - :issue:`18`). - -* An option has been added to the :class:`numcodecs.blosc.Blosc` codec to allow the block size to - be manually configured (:issue:`9`, :issue:`19`). - -* The representation string for the :class:`numcodecs.blosc.Blosc` codec has been tweaked to - help with understanding the shuffle option (:issue:`4`, :issue:`19`). - -* Options have been added to manually control how the C extensions are built regardless of the - architecture of the system on which the build is run. To disable support for AVX2 set the - environment variable "DISABLE_NUMCODECS_AVX2". To disable support for SSE2 set the environment - variable "DISABLE_NUMCODECS_SSE2". To disable C extensions altogether set the environment variable - "DISABLE_NUMCODECS_CEXT" (:issue:`24`, :issue:`26`). - -Maintenance work: - -* CI tests now run under Python 3.6 as well as 2.7, 3.4, 3.5 (:issue:`16`, :issue:`17`). - -* Test coverage is now monitored via - `coveralls `_ - (:issue:`15`, :issue:`20`). - - -.. _release_0.0.1: - -0.0.1 ------ - -Fixed project description in setup.py. - - -.. _release_0.0.0: - -0.0.0 ------ - -First release. This version is a port of the ``codecs`` module from `Zarr -`_ 2.1.0. The following changes have been made from -the original Zarr module: - -* Codec classes have been re-organized into separate modules, mostly one per - codec class, for ease of maintenance. -* Two new codec classes have been added based on 32-bit checksums: - :class:`numcodecs.checksum32.CRC32` and :class:`numcodecs.checksum32.Adler32`. -* The Blosc extension has been refactored to remove code duplications related - to handling of buffer compatibility. diff --git a/docs/zarr3.rst b/docs/zarr3.md similarity index 77% rename from docs/zarr3.rst rename to docs/zarr3.md index ea2b15880..67a689092 100644 --- a/docs/zarr3.rst +++ b/docs/zarr3.md @@ -1,12 +1,14 @@ -.. _Zarr 3 codecs: +(Zarr 3 codecs)= -Zarr 3 codecs -============= +# Zarr 3 codecs + +```{eval-rst} .. automodule:: numcodecs.zarr3 +``` +## Compressors (bytes-to-bytes codecs) -Compressors (bytes-to-bytes codecs) ------------------------------------ +```{eval-rst} .. autoclass:: Blosc() .. autoattribute:: codec_name @@ -38,12 +40,13 @@ Compressors (bytes-to-bytes codecs) .. autoclass:: Shuffle() .. autoattribute:: codec_name +``` +### Checksum codecs (bytes-to-bytes codecs) -Checksum codecs (bytes-to-bytes codecs) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Need to be used as ``compressors`` in zarr-python. +Need to be used as `compressors` in zarr-python. +```{eval-rst} .. autoclass:: CRC32() .. autoattribute:: codec_name @@ -63,10 +66,11 @@ Need to be used as ``compressors`` in zarr-python. .. autoclass:: JenkinsLookup3() .. autoattribute:: codec_name +``` +## Filters (array-to-array codecs) -Filters (array-to-array codecs) -------------------------------- +```{eval-rst} .. autoclass:: Delta() .. autoattribute:: codec_name @@ -90,11 +94,11 @@ Filters (array-to-array codecs) .. autoclass:: AsType() .. autoattribute:: codec_name +``` +## Serializers (array-to-bytes codecs) - -Serializers (array-to-bytes codecs) ------------------------------------ +```{eval-rst} .. autoclass:: PCodec() .. autoattribute:: codec_name @@ -102,3 +106,4 @@ Serializers (array-to-bytes codecs) .. autoclass:: ZFPY() .. autoattribute:: codec_name +``` diff --git a/pyproject.toml b/pyproject.toml index 63e2ae5bd..e1e93e8e9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -67,6 +67,7 @@ docs = [ "sphinx-issues", "pydata-sphinx-theme", "numpydoc", + "myst-parser", ] test = [ "coverage",