Motivation

One of the advantages of using HDF5 is that data stored on disk can be compressed, reducing both the space required to store them and the time needed to read those data. This data compression is applied as part of the HDF5 “filter pipeline” that modifies data during I/O operations. HDF5 includes several filter algorithms as standard, and the version of the HDF5 library found in Rhdf5lib is additionally compiled with support for the deflate and szip compression filters which rely on third-party compression libraries. Collectively HDF5 refer to these as the “internal” filters. It is possible to use any combination of these (including none) when writing data using rhdf5. The default filter pipeline is shown in Figure @ref(fig:filter-pipeline).

The default compression pipeline used by rhdf5

This pipeline approach has been designed so that filters can be chained together (as in the diagram above) or easily substituted for alternative filters. This allows tailoring the compression approach to best suit the data or application.

It may be case that for a specific usecase an alternative, third-party, compression algorithm would be the most appropriate to use. Such filters, which are not part of the standard HDF5 distribution, are referred to as “external” filters. In order to allow their use without requiring either the HDF5 library or applications to be built with support for all possible filters HDF5 is able to use dynamically loaded filters. These are compiled independently from the HDF5 library, but are available to an application at run time.

This package currently distributes external HDF5 filters employing bzip2, LZF, VBZ, Zstandard and the Blosc meta-compressor. In total rhdf5filters provides access to 11¹ compression filters than can be applied to HDF5 datasets. The full list of filters currently provided by the package is:

• bzip2
• blosclz
• lz4
• lz4hc
• snappy
• Zstandard
• zlib
• LZF
• VBZ

Usage

rhdf5filters::hdf5_plugin_path()

## [1] "/tmp/RtmpdVy6Ec/Rinst7b769a313b3/rhdf5filters/lib"

## blosc compressed file
blosc_file <- system.file("h5examples/h5ex_d_blosc.h5", 
                          package = "rhdf5filters")

h5dump_out <- system2('h5dump', 
                      args = c('-p', '-d /dset', blosc_file), 
                      stdout = TRUE, stderr = TRUE)
cat(h5dump_out, sep = "\n")

## HDF5 "rhdf5filters/h5examples/h5ex_d_blosc.h5" {
## DATASET "/dset" {
##    DATATYPE  H5T_IEEE_F32LE
##    DATASPACE  SIMPLE { ( 30, 10, 20 ) / ( 30, 10, 20 ) }
##    STORAGE_LAYOUT {
##       CHUNKED ( 10, 10, 20 )
##       SIZE 3347 (7.171:1 COMPRESSION)
##    }
##    FILTERS {
##       USER_DEFINED_FILTER {
##          FILTER_ID 32001
##          COMMENT blosc
##          PARAMS { 2 2 4 8000 4 1 0 }
##       }
##    }
##    FILLVALUE {
##       FILL_TIME H5D_FILL_TIME_IFSET
##       VALUE  H5D_FILL_VALUE_DEFAULT
##    }
##    ALLOCATION_TIME {
##       H5D_ALLOC_TIME_INCR
##    }
##    DATA {h5dump error: unable to print data
## 
##    }
## }
## }

## set environment variable to hdf5filter location
Sys.setenv("HDF5_PLUGIN_PATH" = rhdf5filters::hdf5_plugin_path())
h5dump_out <- system2('h5dump', 
                      args = c('-p', '-d /dset', '-w 50', blosc_file), 
                      stdout = TRUE,  stderr = TRUE)

## find the data entry and print the first few lines
DATA_line <- grep(h5dump_out, pattern = "DATA \\{")
cat( h5dump_out[ (DATA_line):(DATA_line+2) ], sep = "\n" )

##   DATA {
##    (0,0,0): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
##    (0,0,11): 11, 12, 13, 14, 15, 16, 17, 18,

Compiling the compression libraries

In order to compile the compression filters rhdf5filters needs to link them against libraries for the relevant compression tools. These libraries include bzip2, blosc and zstd. By default rhdf5filters will look for system installations of those libraries and link against them. If they are not found then versions that are bundled with the package will be compiled and used instead. It is possible to use system libraries for some filters and the bundled versions for others - rhdf5filters should be able to determine this at compile time.

If you wish to force the use of the bundled libraries, perhaps to ensure linking against the static versions, this can be done by providing the configure argument "--with-bundled-libraries" during package installation e.g.

BiocManager::install('rhdf5filters', 
                     configure.args = "--with-bundled-libraries")