Skip to content
Snippets Groups Projects
Commit d06f2924 authored by Oriol Tintó's avatar Oriol Tintó
Browse files

Update documentation.

parent 77dd964c
No related branches found
No related tags found
1 merge request!6Get rid of Enums and use a dictionary to define available lossy compressors,...
Compression Specification Format
================================
In order to define compression specifications in a way that makes it readable and easy to use by different applications in differen programming languages, we defined our own Compression Specification Format.
One example looks like this:
We created a Compression Specification Format, which can be easily read and utilized by various applications written in different programming languages.
An example of this format would be:
>>> lossy,zfp,rate,4.0
Its purpose is to represent the specifications in a way that are easy to understand and use.
The currently implemented compressors include `Blosc <https://www.blosc.org/>`_ for lossless compression,
and `ZFP <https://computing.llnl.gov/projects/zfp>`_ and `SZ <https://szcompressor.org/>`_ for lossy compression.
and `ZFP <https://computing.llnl.gov/projects/zfp>`_, `SZ <https://szcompressor.org/>`_ and `SZ3 <https://szcompressor.org/>`_ for lossy compression.
In order to use lossless compression, one can simply write:
>>> lossless
......@@ -30,18 +28,23 @@ One example with a different backend and compression level could be:
For lossy compression, it is mandatory to include the compressor, the mode and the parameter.
At the moment two lossy compressors are supported: **SZ** and **ZFP**.
At the moment the lossy supported compressors are: **ZFP**, **SZ** and **SZ3**.
The two compressors have different methods and different names for them:
-**SZ**:
- **abs**: absolute threshold mode
- **rel**: relative threshold mode
- **pw_rel**: point-wise relative threshold mode
The compressors have different methods and different names for them:
-**ZFP**:
- **accuracy**: absolute threshold mode
- **rate**: number of bits-per-value
- **precision**: keep a certain bits of precision
-**SZ**:
- **abs**: absolute threshold mode
- **rel**: relative threshold mode
- **pw_rel**: point-wise relative threshold mode
-**SZ3**:
- **abs**: absolute threshold mode
- **rel**: relative threshold mode
- **norm2**: using norm2.
- **psnr**: using the peak signal to noise ratio mode.
Few examples of lossy compression specifications:
......@@ -49,6 +52,7 @@ Few examples of lossy compression specifications:
lossy,sz,abs,0.01
lossy,zfp,rate,4.0
lossy,sz3,psnr,40
lossy,sz,rel,1e-3
lossy,zfp,precision,10
lossy,sz,pw_rel,0.05
......
......@@ -7,8 +7,8 @@ Encodings
.. autosummary::
:toctree: Classes
enstools.encoding.api.FilterEncodingForXarray
enstools.encoding.api.FilterEncodingForH5py
enstools.encoding.api.DatasetEncoding
enstools.encoding.api.VariableEncoding
Compressors and Compression modes
......
......@@ -4,7 +4,7 @@ Welcome to Enstools-Encoding's documentation! |docsbadge|
.. attention::
This package is a building block of `enstools-compression <https://github.com/wavestoweather/enstools-compression>`_.
The recomendation is to directly use **enstools-compression**.
The recommendation is to directly use **enstools-compression**.
The only purpose of splitting the package was to provide a way of using compression filters with **xarray** without adding too much dependencies.
It might be the main sound reason to use this package separately.
......
......@@ -5,10 +5,10 @@ Using **lossy compression** with xarray can be as easy as adding a **single line
.. code::
from enstools.encoding.api import FilterEncodingForXarray
from enstools.encoding.api import DatasetEncoding
...
encoding = FilterEncodingForXarray(dataset, "lossy,sz,rel,1.e-4")
encoding = DatasetEncoding(dataset, "lossy,sz,rel,1.e-4")
dataset.to_netcdf(dummy_output_file, encoding=encoding, engine="h5netcdf")
...
......@@ -18,9 +18,9 @@ Save an **xarray** dataset using losslessly compression:
.. code::
from enstools.encoding.api import FilterEncodingForXarray
from enstools.encoding.api import DatasetEncoding
...
encoding = FilterEncodingForXarray(dataset, "lossless")
encoding = DatasetEncoding(dataset, "lossless")
dataset.to_netcdf(dummy_output_file, encoding=encoding, engine="h5netcdf")
......@@ -40,11 +40,11 @@ If we want to directly use **h5py** we can do the following:
.. code::
from enstools.encoding.api import FilterEncodingForH5py
from enstools.encoding.api import VariableEncoding
...
encoding = FilterEncodingForH5py.from_string("lossless")
encoding = VariableEncoding("lossless")
f = h5py.File('test.h5', 'w')
f.create_dataset('lossless_compression_using_blosc',
data=numpy.arange(100),
......@@ -56,11 +56,11 @@ Or without using specification strings:
.. code::
from enstools.encoding.api import FilterEncodingForH5py, Compressors, CompressionModes
from enstools.encoding.api import VariableEncoding
...
encoding = FilterEncodingForH5py(Compressors.ZFP, CompressionModes.RATE, 4.0)
encoding = VariableEncoding(compressor="zfp", mode="rate",parameter="4.0")
f = h5py.File('test.h5', 'w')
f.create_dataset('lossy_compression_with_zfp_rate_4.0',
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment