Commit Graph

928 Commits

Author SHA1 Message Date
Dennis Heimbigner
84c69afca7 Allow redefinition of variable filters
re: Github issue https://github.com/Unidata/netcdf-c/issues/1713

If nc_def_var_filter or nc_def_var_deflate or nc_def_var_szip is
called multiple times with the same filter id, but possibly with
different sets of parameters, then the first invocation is
sticky and later invocations are ignored. The desired behavior
is to have the last invocation be used.

This PR implements that desired behavior, with some special
cases.  If you call nc_def_var_deflate multiple times, then the
last invocation rule applies with respect to deflate. However,
the shuffle filter, if enabled, is always applied just before
applying deflate.

Misc unrelated changes:
1. Make client-side filters be disabled by default
2. Fix the definition of uintptr_t and use in oc2 and libdap4
3. Add some test cases
4. modify filter order tests to use plugin filters rather
   than client-side filters
2020-05-11 09:42:31 -06:00
Ward Fisher
b490c457e9
Merge pull request #1717 from NOAA-GSD/ejh_cache
now properly setting HDF5 file cache for files created/opened sequentially on parallel I/O builds
2020-05-08 15:00:45 -06:00
Edward Hartnett
1fb8d161ce updated RELEASE_NOTES 2020-05-08 11:12:47 -06:00
Edward Hartnett
1267358e9b whitespace cleanup 2020-05-08 11:11:56 -06:00
Edward Hartnett
6aa6eff710 now properly setting HDF5 file cache for files created/opened sequentially on parallel IO builds 2020-05-08 11:00:56 -06:00
Edward Hartnett
e3c9e83ecf adding internal function, plus some documentation 2020-05-08 08:58:42 -06:00
Edward Hartnett
2e2f3e7d29 added demonstration of nc_def_var_deflate() issue 2020-05-07 07:08:52 -06:00
Dennis Heimbigner
f0cd7f8ec1 Support no-op dispatch functions
re: https://github.com/Unidata/netcdf-c/issues/1693

1. Add functions to libdispatch/dnotnc4.c to support
   dispatch table operations that should work for any
   dispatch table, even if they do not do anything.
   Functions such as nc_inq_var_filter.
2. Modify selected dispatch tables to utilize
   the noop functions.
3. Extend nc_test/tst_formats.c to test.

This is an extension of Ed's work to do this for
chunking and deflate and szip. See PRs
https://github.com/Unidata/netcdf-c/pull/1697
and
https://github.com/Unidata/netcdf-c/pull/1692

As a side effect, elide libdispatch/dnotnc3.c since
it is no longer used.
2020-04-15 14:44:58 -06:00
Dennis Heimbigner
6f86660da8 Fix missing forward declarations
re: issue https://github.com/Unidata/netcdf-c/issues/1687

static functions are being used before decl and it causes
errors. Only occurs when BIG_ENDIAN is defined.
Solution is to add the forward declarations.
2020-04-03 20:15:34 -06:00
Ward Fisher
9d482c3f20 Added hard failure to script, modified cmake default. 2020-04-01 16:20:33 -06:00
Ward Fisher
74afd13f44
Merge pull request #1671 from NOAA-GSD/ejh_hdf5_1_12_0
build library with hdf5 1 12 0
2020-03-24 14:33:14 -06:00
Edward Hartnett
edea5e3552 now pass 0 for deflate_level if deflate not in use 2020-03-16 11:01:13 -06:00
Edward Hartnett
6fa5e677b3 merged in ejh_hdf5_1_12 2020-03-09 07:06:05 -06:00
Ward Fisher
34c3b85fc0
Merge pull request #1662 from NOAA-GSD/ejh_scalar_chunking
Return error on attempt to set chunking on scalar var
2020-03-06 14:07:18 -07:00
Ward Fisher
fa0cd583ef
Merge pull request #1659 from NOAA-GSD/ejh_storage
Fix parallel build error in tst_mode.c caused by returning errors for attempts to set filters on scalar vars
2020-03-05 16:45:33 -07:00
Greg Sjaardema
404b0a3ed7
Add missing ERR macro on tests
A few of the tests were missing a call to the ERR macro.
2020-03-04 13:31:47 -07:00
Edward Hartnett
502336c2c7 now return NC_EINVAL on attempt to set chunking on scalar var 2020-03-03 11:57:16 -07:00
Edward Hartnett
c484f8f3d0 fixed tst_mode to not attempt to set filters on scalars 2020-03-03 07:44:37 -07:00
Edward Hartnett
836a8fb55d fixed szip test error in tst_parallel5.c 2020-03-02 16:31:56 -07:00
Dennis Heimbigner
73537603e2 Make scalar X filter return an error instead of ignoring it 2020-03-02 15:10:54 -07:00
Dennis Heimbigner
8a5ae84dc6 fix memory leak 2020-02-27 14:29:28 -07:00
Dennis Heimbigner
b488c272d5 Fix conflicts with master 2020-02-27 14:06:45 -07:00
Edward Hartnett
5a1bbf07e4 resolved conflict in tst_h_vars2.c 2020-02-27 05:25:31 -07:00
Edward Hartnett
6f95f655dd
Merge branch 'master' into ejh_dispatch 2020-02-26 16:12:57 -07:00
Edward Hartnett
418e428a05 fixed problem with scalar compact 2020-02-26 09:13:12 -07:00
Edward Hartnett
b31aedcc8e all tests passing but compact storage for scalars not being properly written in file yet 2020-02-26 08:14:06 -07:00
Edward Hartnett
3241aca93a include storage in rec_print_metadata() output 2020-02-25 16:12:50 -07:00
Edward Hartnett
6241a6e7a0 more tests for storage, changed var names to reflect stortage 2020-02-25 15:55:34 -07:00
Edward Hartnett
2ff24bd6fe more tests for compact storage 2020-02-25 13:30:38 -07:00
Edward Hartnett
8952c9b9f1 whitespace cleanup of tst_h_vars2.c 2020-02-25 06:44:40 -07:00
Edward Hartnett
5b893d8e0a adding more test for compact var 2020-02-25 05:46:09 -07:00
Edward Hartnett
4b163b65e1 added some testing for compact vars 2020-02-24 14:34:27 -07:00
M. Scot Breitenfeld
9f9b125028 Updated H5Oget_info* and H5Oinfo_t to be compatible with HDF5 1.12.0.
Issue: build netcdf with hdf5-1.12.0 beta release fails in h5_test directory #1628
2020-02-20 17:24:29 -06:00
Dennis Heimbigner
44d0dcaad2 Add support for multiple filters per variable.
re: https://github.com/Unidata/netcdf-c/issues/1584

Support has been added for multiple filters per variable.  This
affects a number of components in netcdf. The new APIs are
documented in NUG/filters.md.

The primary changes are:
* A set of new functions are provided (see __include/netcdf_filter.h__).
    - Obtain a list of the filters associated with a variable
    - Obtain the parameters for a specific filter.
* The existing __nc_inq_var_filter__ function now returns info
  about the first defined filter.
* The utilities (ncgen, ncdump, and nccopy) now support
  an extended format for specifying a sequence of filters.
  The general form is __<filter>|<filter>..._.
* The ncdump **_Filter** attribute now dumps a list of all the
  filters associated with a variable using the above new format.
* Filter specifications can now use a filter name instead of number
  for filters known to the netcdf library, which in turn is taken
  from the HDF5 filter registration page.
* New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter
  is returned if an attempt is made to access an unknown filter.
* Internally, the dispatch table has been extended to add a function
  to handle all of the filter functions.
* New, filter-related, tests were added to nc_test4.
* A new plugin was added to the plugins directory to help with testing.

Notes:
1. The shuffle and fletcher32 filters are not part of the multifilter system.

Misc. changes:
1. A debug module was added to libhdf5 to help catch error locations.
2020-02-16 12:59:33 -07:00
Edward Hartnett
07ce91a137
Merge branch 'master' into ejh_h5_par 2020-02-15 09:09:35 -07:00
Edward Hartnett
a8684c730c fixed merge conflict in RELEASE_NOTES 2020-02-15 06:42:49 -07:00
Edward Hartnett
8b9298c58d more checking in 3D test 2020-02-12 08:21:52 -07:00
Edward Hartnett
fe70e818e8 checking 3D file again after close 2020-02-12 08:18:41 -07:00
Edward Hartnett
6e316f44a6 switched to vara functions in 3D test 2020-02-12 08:17:35 -07:00
Edward Hartnett
fe96b6a3e5 checking 2nd var data in 3D test 2020-02-12 08:14:13 -07:00
Edward Hartnett
77ae2cfecc improving 3d test 2020-02-12 08:11:37 -07:00
Edward Hartnett
2addd91215 added another var to 2D test 2020-02-12 08:10:16 -07:00
Edward Hartnett
550b8fa188 better 3d unlimited testing 2020-02-12 08:06:04 -07:00
Edward Hartnett
d922af9820 adding 3d test 2020-02-12 08:04:32 -07:00
Edward Hartnett
93ef9b1bdc adding 3d test 2020-02-12 07:49:04 -07:00
Edward Hartnett
4ea78cc985 testing two unlimited dims 2020-02-12 07:45:59 -07:00
Edward Hartnett
168850c373 testing two unlimited dims 2020-02-12 07:45:29 -07:00
Edward Hartnett
4683a4da3b testing two unlimited dims 2020-02-12 07:44:54 -07:00
Edward Hartnett
2f9856b2e6 testing two unlimited dims 2020-02-12 07:30:00 -07:00
Edward Hartnett
bf87013c6b adding test 2020-02-12 07:27:51 -07:00