Commit Graph

491 Commits

Author SHA1 Message Date
Edward Hartnett
b29f9f34a0 whitespace cleanup 2020-03-08 09:10:07 -06:00
Edward Hartnett
4c7e162f34 less use of contiguous/compact field 2020-03-08 07:31:21 -06:00
Edward Hartnett
053752440b stop setting contiguous field in nc4hdf5.c 2020-03-08 07:18:52 -06:00
Edward Hartnett
04eafff166 stop setting contiguous field in hdf5filter.c 2020-03-08 07:18:11 -06:00
Edward Hartnett
5574317db7 stop setting contiguous/compact fields at file open 2020-03-08 07:17:01 -06:00
Edward Hartnett
61357cfd4d more use of storage field 2020-03-08 07:09:15 -06:00
Edward Hartnett
1761850795 continuing to switch to storage field 2020-03-08 07:05:51 -06:00
Edward Hartnett
b98a37e0b3 using storage field in nc4var.c 2020-03-08 06:38:44 -06:00
Edward Hartnett
119e8e9465 using storage in hdf5filter.c 2020-03-08 06:31:34 -06:00
Edward Hartnett
8dec9f6c99 now setting storage field when setting var storage 2020-03-08 06:29:49 -06:00
Edward Hartnett
d87a073a34 starting to use storage field when opening file 2020-03-08 06:21:08 -06:00
Edward Hartnett
0c419ec582 removed commented-out code 2020-03-06 09:57:33 -07:00
Edward Hartnett
502336c2c7 now return NC_EINVAL on attempt to set chunking on scalar var 2020-03-03 11:57:16 -07:00
Dennis Heimbigner
73537603e2 Make scalar X filter return an error instead of ignoring it 2020-03-02 15:10:54 -07:00
Dennis Heimbigner
420fdf4625 fix memory allocation failure in hdf5var.c 2020-03-02 11:45:41 -07:00
Dennis Heimbigner
7d1ca9ac85 fix references to var->deflate' 2020-03-02 11:12:30 -07:00
Dennis Heimbigner
e66c727c28 Fix Filters x compact 2020-02-29 15:33:27 -07:00
Dennis Heimbigner
f376c23329 Make utilities support NC_COMPACT
re: https://github.com/Unidata/netcdf-c/issues/1642

Modify ncdump, nccopy, and ncgen to support the NC_COMPACT storage option.
Added test cases and added description to the man pages for the utilities.

1. ncdump: For compact storage variable, print special attribute __Storage_ as
````
    <var>: _Storage = "compact";
````

2. ncgen: parse and implement
````
    <var>: _Storage = "compact";
````
in a .cdl file

3. nccopy: Extend the chunk specification (-c flag) to support
   compact using the forms
````
nccopy ... -c <var>:compact
and
nccopy ... -c <var>:contiguous
````

Misc. other changes
1. cleanup the copy_chunking function in ncdump/nccopy.c
2020-02-29 12:06:21 -07:00
Dennis Heimbigner
10d227fc1b fix parallel filter error discovered by Hartnett 2020-02-28 11:36:58 -07:00
Dennis Heimbigner
a3a3e15cb1 fix bad edit 2020-02-27 15:33:39 -07:00
Dennis Heimbigner
afe5a2998c
Merge branch 'master' into multifilter.dmh 2020-02-27 15:02:27 -07:00
Dennis Heimbigner
b488c272d5 Fix conflicts with master 2020-02-27 14:06:45 -07:00
Edward Hartnett
6f95f655dd
Merge branch 'master' into ejh_dispatch 2020-02-26 16:12:57 -07:00
Edward Hartnett
418e428a05 fixed problem with scalar compact 2020-02-26 09:13:12 -07:00
Edward Hartnett
b31aedcc8e all tests passing but compact storage for scalars not being properly written in file yet 2020-02-26 08:14:06 -07:00
Edward Hartnett
6241a6e7a0 more tests for storage, changed var names to reflect stortage 2020-02-25 15:55:34 -07:00
Edward Hartnett
2ff24bd6fe more tests for compact storage 2020-02-25 13:30:38 -07:00
Dennis Heimbigner
44d0dcaad2 Add support for multiple filters per variable.
re: https://github.com/Unidata/netcdf-c/issues/1584

Support has been added for multiple filters per variable.  This
affects a number of components in netcdf. The new APIs are
documented in NUG/filters.md.

The primary changes are:
* A set of new functions are provided (see __include/netcdf_filter.h__).
    - Obtain a list of the filters associated with a variable
    - Obtain the parameters for a specific filter.
* The existing __nc_inq_var_filter__ function now returns info
  about the first defined filter.
* The utilities (ncgen, ncdump, and nccopy) now support
  an extended format for specifying a sequence of filters.
  The general form is __<filter>|<filter>..._.
* The ncdump **_Filter** attribute now dumps a list of all the
  filters associated with a variable using the above new format.
* Filter specifications can now use a filter name instead of number
  for filters known to the netcdf library, which in turn is taken
  from the HDF5 filter registration page.
* New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter
  is returned if an attempt is made to access an unknown filter.
* Internally, the dispatch table has been extended to add a function
  to handle all of the filter functions.
* New, filter-related, tests were added to nc_test4.
* A new plugin was added to the plugins directory to help with testing.

Notes:
1. The shuffle and fletcher32 filters are not part of the multifilter system.

Misc. changes:
1. A debug module was added to libhdf5 to help catch error locations.
2020-02-16 12:59:33 -07:00
Edward Hartnett
a8684c730c fixed merge conflict in RELEASE_NOTES 2020-02-15 06:42:49 -07:00
Edward Hartnett
05a6ff74b2 merged changes from master 2020-02-11 17:19:53 -07:00
Edward Hartnett
15059a18b7 merged changes from master 2020-02-11 17:19:25 -07:00
Edward Hartnett
a0839a2a7a added version to dispatch table 2020-02-09 13:07:58 -07:00
Edward Hartnett
b7ac19a43f only close non-zero typeids 2020-02-09 12:03:21 -07:00
Edward Hartnett
af6b6787bf fix for memory leak due to HDF5 types 2020-02-09 11:47:13 -07:00
Edward Hartnett
8057a552ef move nc_def_var_szip function so it will appear in the documentation 2020-02-07 09:09:01 -07:00
Edward Hartnett
558988bb18 fixed docs, removed unneeded defines in test 2020-02-07 07:54:12 -07:00
Edward Hartnett
c4d3937099 now check number of elements in chunk against pixels_per_block for szip compression 2020-02-07 07:03:40 -07:00
Edward Hartnett
ff7280512e checking for some bad pixels_per_block values for szip 2020-02-07 06:53:52 -07:00
Edward Hartnett
6d2d751e4e disallow zlib if szip already in use 2020-02-07 05:01:06 -07:00
Edward Hartnett
dc4e880c37 disallow szip if zlib already in use 2020-02-07 04:46:15 -07:00
Edward Hartnett
6b2947813f adding test for zlib+szip in HDF5 2020-02-07 03:38:43 -07:00
Edward Hartnett
1817790c6b rely completely on nc_def_var_filter for setting szip 2020-02-05 10:32:16 -07:00
Edward Hartnett
517ef4f257 use nc_def_var_filter in nc_def_var_szip 2020-02-05 10:25:30 -07:00
Edward Hartnett
52d745de68 now remember szip setting in filter fields 2020-02-04 08:40:15 -07:00
Ward Fisher
aadd5a2d81
Merge pull request #1589 from NOAA-GSD/ejh_szip
re-implement the nc_def_var_szip() function, including for parallel I/O
2020-01-22 16:27:33 -07:00
Edward Hartnett
6103f442cb fixed compile error 2020-01-21 08:50:38 -07:00
Edward Hartnett
c839a2d6c5
Merge branch 'master' into ejh_var_cache 2020-01-21 07:48:59 -07:00
Edward Hartnett
e94615a0e5
Merge branch 'master' into ejh_szip 2020-01-16 08:49:12 -07:00
Ward Fisher
8771d0bdf4
Merge pull request #1582 from NOAA-GSD/ejh_parallel_zlib
Allow user to turn on zlib, shuffle, and/or fletcher32 filters with parallel I/O for HDF5-1.10.2+
2020-01-13 16:06:51 -07:00
Dennis Heimbigner
f587654670 Make the dap4 code resistant to various server errors.
Some versions of some servers are returning malformed responses.
Make the library either handle them or gracefully fail.
The three server errors "fixed" here are as follows.
1. The attribute _NCProperties sometimes has a trailing nul character
   in its value. Soln is to elide the nul(s).
2. Sometimes a DAP response has no data part, only a DMR.
   Soln is to detect and return an error code instead of crashing.
3. Sometimes a server returns a redirection, but our current
   openmagic() function was not following the redirect. Soln
   is to follow redirects.
Also because of #2, I am temporarily making --disable-dap-remote-tests
be the default.
2020-01-08 15:18:31 -07:00