Commit Graph

114 Commits

Author SHA1 Message Date
Dennis Heimbigner
3ffe7be446 Enhance/Fix filter support
re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214

The primary change is to support so-called "standard filters".
A standard filter is one that is defined by the following
netcdf-c API:
````
int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params);
int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params);
````
So for example, zstandard would be a standard filter by defining
the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*.

In order to define these functions, we need a new dispatch function:
````
int nc_inq_filter_avail(int ncid, unsigned filterid);
````
This function, combined with the existing filter API can be used
to implement arbitrary standard filters using a simple code pattern.
Note that I would have preferred that this function return a list
of all available filters, but HDF5 does not support that functionality.

So this PR implements the dispatch function and implements
the following standard functions:
    + bzip2
    + zstandard
    + blosc
Specific test cases are also provided for HDF5 and NCZarr.
Over time, other specific standard filters will be defined.

## Primary Changes
* Add nc_inq_filter_avail() to netcdf-c API.
* Add standard filter implementations to test use of *nc_inq_filter_avail*.
* Bump the dispatch table version number and add to all the relevant
   dispatch tables (libsrc, libsrcp, etc).
* Create a program to invoke nc_inq_filter_avail so that it is accessible
  to shell scripts.
* Cleanup szip support to properly support szip
  when HDF5 is disabled. This involves detecting
  libsz separately from testing if HDF5 supports szip.
* Integrate shuffle and fletcher32 into the existing
  filter API. This means that, for example, nc_def_var_fletcher32
  is now a wrapper around nc_def_var_filter.
* Extend the Codec defaulting to allow multiple default shared libraries.

## Misc. Changes
* Modify configure.ac/CMakeLists.txt to look for the relevant
  libraries implementing standard filters.
* Modify libnetcdf.settings to list available standard filters
  (including deflate and szip).
* Add CMake test modules to locate libbz2 and libzstd.
* Cleanup the HDF5 memory manager function use in the plugins.
* remove unused file include//ncfilter.h
* remove tests for the HDF5 memory operations e.g. H5allocate_memory.
* Add flag to ncdump to force use of _Filter instead of _Deflate
  or _Shuffle or _Fletcher32. Used for testing.
2022-03-14 12:39:37 -06:00
Greg Sjaardema
cbcee382b0 Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
Orion Poplawski
9fc8ae62a8 Fix tst_h_atts3 for hdf5 1.12 2021-04-10 19:51:58 -06:00
Scot Breitenfeld
efb9b7120b reverted past fix 2021-01-08 11:24:15 -06:00
Scot Breitenfeld
ca3d8c10f4 fixed H5O_info_t incompatiblity with H5Oget_info_by_idx3 2021-01-07 14:30:05 -06:00
Edward Hartnett
9c6b9bb72e fixed tst_h_par_compress.c 2020-08-12 07:28:49 -06:00
Edward Hartnett
b1c0fa07af added tst_h_par_compress to CMake build 2020-07-08 13:26:50 -06:00
Edward Hartnett
73e21b4fd6 took out timing code 2020-07-08 13:21:33 -06:00
Edward Hartnett
c19aa14eaf turned off szip testing while HDF5 issue is resolved 2020-07-08 13:14:58 -06:00
Edward Hartnett
4936edd840 adding other settings of dataset property list to match what is in nc4hdf5.c 2020-07-08 12:56:34 -06:00
Edward Hartnett
d1a13b28cc adding property list for dataset creation to tst_h_par_compress.c 2020-07-08 12:39:35 -06:00
Edward Hartnett
c0b6792515 changed tst_h_par_compress to use H5Dcreate2() just like code in nc4hdf5.c does 2020-07-08 12:36:02 -06:00
Edward Hartnett
1d79a47c6d fixed warning in tst_parallel5.c 2020-07-08 12:24:48 -06:00
Edward Hartnett
90777d6465 added new test tst_h_par_compress.c 2020-07-08 12:17:37 -06:00
Edward Hartnett
993c8259c2 fixed LDFLAFS in plugins Makefile.am, also fixed warning in tst_h_vars.c 2020-06-02 07:34:46 -06:00
Scot Breitenfeld
1a6228f17f fixed missing declaration 2020-04-23 23:32:29 -05:00
Scot Breitenfeld
7b1b06b5ca Merge remote-tracking branch 'upstream/master' 2020-04-23 15:36:14 -05:00
Edward Hartnett
1361deffa3
Merge branch 'master' into ejh_hdf5_1_12_0 2020-03-12 16:03:09 -06:00
Scot Breitenfeld
c5d2e99417 Updated to use H5O_info2_t for HDF5 1.12 and the use of H5Oget_info3 instead of H5Gget_objinfo 2020-03-12 15:50:24 +00:00
Edward Hartnett
2ef818eed0 change to trigger CI 2020-03-09 09:52:08 -06:00
Edward Hartnett
6fa5e677b3 merged in ejh_hdf5_1_12 2020-03-09 07:06:05 -06:00
Edward Hartnett
ddb44b5abb added storage to NC_VAR_INFO_T, removed unneeded test code 2020-03-07 05:08:12 -07:00
Edward Hartnett
ad77f42a90 fixed large file build issue 2020-03-02 16:28:56 -07:00
Dennis Heimbigner
e82d33a9bb fix error in tst_h_vars.c 2020-03-02 11:58:22 -07:00
Edward Hartnett
5a1bbf07e4 resolved conflict in tst_h_vars2.c 2020-02-27 05:25:31 -07:00
Edward Hartnett
58d78a7e4c more HDF5 testing of scalar compact variable 2020-02-26 07:34:56 -07:00
Edward Hartnett
6241a6e7a0 more tests for storage, changed var names to reflect stortage 2020-02-25 15:55:34 -07:00
Edward Hartnett
2ff24bd6fe more tests for compact storage 2020-02-25 13:30:38 -07:00
Edward Hartnett
beb47aeb42 test demonstrating HDF5 use of compact scalar dataset 2020-02-25 07:23:27 -07:00
Edward Hartnett
bb266baef5 fixed comments 2020-02-25 07:10:43 -07:00
Edward Hartnett
10eada9c52 fixed warnings 2020-02-25 07:09:00 -07:00
Edward Hartnett
810f9cdb54 refurbishing old test that was commented out 2020-02-25 07:07:58 -07:00
Edward Hartnett
8952c9b9f1 whitespace cleanup of tst_h_vars2.c 2020-02-25 06:44:40 -07:00
M. Scot Breitenfeld
9f9b125028 Updated H5Oget_info* and H5Oinfo_t to be compatible with HDF5 1.12.0.
Issue: build netcdf with hdf5-1.12.0 beta release fails in h5_test directory #1628
2020-02-20 17:24:29 -06:00
Edward Hartnett
f326b6ad37 added in decl for num_obj for large file builds 2020-02-11 17:21:53 -07:00
Edward Hartnett
03c7e61b20 merged changes from master 2020-02-11 17:20:22 -07:00
Edward Hartnett
e15161ef22 more uncommented test code and whitespace cleanup 2020-02-08 06:59:52 -07:00
Edward Hartnett
26249f4583 added HDF5 parallel test to CMake build 2020-02-08 06:37:43 -07:00
Edward Hartnett
29981888a9 now allow substitution of different mpi launcher in h5_test/run_par_tests.sh 2020-02-08 06:31:19 -07:00
Edward Hartnett
1c3ea221a9 moving run_par_tests.sh to run_par_tests.sh.in 2020-02-08 06:27:32 -07:00
Edward Hartnett
53b1c56ad5 uncommented part of test code 2020-02-08 06:21:37 -07:00
Edward Hartnett
52fb360d8c whitespace and comment cleanup 2020-02-08 06:15:06 -07:00
Edward Hartnett
cd3740e724 finished test for zlib+szip in HDF5 2020-02-07 04:05:46 -07:00
Edward Hartnett
6b2947813f adding test for zlib+szip in HDF5 2020-02-07 03:38:43 -07:00
Edward Hartnett
e793764c18 more HDF5 tests for szip functionality 2020-02-04 05:30:31 -07:00
Greg Sjaardema
56c0d5cf8a Spelling fixes 2019-09-18 08:03:01 -06:00
Dennis Heimbigner
62e2b472b4 Minor config.h changes to support filters in Fortran 2019-04-29 16:36:08 -06:00
Ed Hartnett
cb47eaf895 uncommented some testing 2019-02-02 07:43:56 -07:00
Ed Hartnett
aed2a4c1db got tst_h_rename using H5Lmove 2019-02-02 07:17:57 -07:00
Ed Hartnett
373c0b5665 added tst_h_rename.c to check HDF5 renaming behavior 2019-02-02 06:17:43 -07:00