mirror of
https://github.com/Unidata/netcdf-c.git
synced 2024-11-21 03:13:42 +08:00
3ffe7be446
re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
84 lines
1.6 KiB
Plaintext
84 lines
1.6 KiB
Plaintext
netcdf bzip2 {
|
|
dimensions:
|
|
dim0 = 4 ;
|
|
dim1 = 4 ;
|
|
dim2 = 4 ;
|
|
dim3 = 4 ;
|
|
variables:
|
|
float var(dim0, dim1, dim2, dim3) ;
|
|
var:_Storage = "chunked" ;
|
|
var:_ChunkSizes = 4, 4, 4, 4 ;
|
|
var:_Filter = "40000,17" ;
|
|
var:_NoFill = "true" ;
|
|
|
|
// global attributes:
|
|
:_Format = "netCDF-4" ;
|
|
data:
|
|
|
|
var =
|
|
0, 1, 2, 3,
|
|
4, 5, 6, 7,
|
|
8, 9, 10, 11,
|
|
12, 13, 14, 15,
|
|
16, 17, 18, 19,
|
|
20, 21, 22, 23,
|
|
24, 25, 26, 27,
|
|
28, 29, 30, 31,
|
|
32, 33, 34, 35,
|
|
36, 37, 38, 39,
|
|
40, 41, 42, 43,
|
|
44, 45, 46, 47,
|
|
48, 49, 50, 51,
|
|
52, 53, 54, 55,
|
|
56, 57, 58, 59,
|
|
60, 61, 62, 63,
|
|
64, 65, 66, 67,
|
|
68, 69, 70, 71,
|
|
72, 73, 74, 75,
|
|
76, 77, 78, 79,
|
|
80, 81, 82, 83,
|
|
84, 85, 86, 87,
|
|
88, 89, 90, 91,
|
|
92, 93, 94, 95,
|
|
96, 97, 98, 99,
|
|
100, 101, 102, 103,
|
|
104, 105, 106, 107,
|
|
108, 109, 110, 111,
|
|
112, 113, 114, 115,
|
|
116, 117, 118, 119,
|
|
120, 121, 122, 123,
|
|
124, 125, 126, 127,
|
|
128, 129, 130, 131,
|
|
132, 133, 134, 135,
|
|
136, 137, 138, 139,
|
|
140, 141, 142, 143,
|
|
144, 145, 146, 147,
|
|
148, 149, 150, 151,
|
|
152, 153, 154, 155,
|
|
156, 157, 158, 159,
|
|
160, 161, 162, 163,
|
|
164, 165, 166, 167,
|
|
168, 169, 170, 171,
|
|
172, 173, 174, 175,
|
|
176, 177, 178, 179,
|
|
180, 181, 182, 183,
|
|
184, 185, 186, 187,
|
|
188, 189, 190, 191,
|
|
192, 193, 194, 195,
|
|
196, 197, 198, 199,
|
|
200, 201, 202, 203,
|
|
204, 205, 206, 207,
|
|
208, 209, 210, 211,
|
|
212, 213, 214, 215,
|
|
216, 217, 218, 219,
|
|
220, 221, 222, 223,
|
|
224, 225, 226, 227,
|
|
228, 229, 230, 231,
|
|
232, 233, 234, 235,
|
|
236, 237, 238, 239,
|
|
240, 241, 242, 243,
|
|
244, 245, 246, 247,
|
|
248, 249, 250, 251,
|
|
252, 253, 254, 255 ;
|
|
}
|