Commit Graph

480 Commits

Author SHA1 Message Date
Edward Hartnett
0c419ec582 removed commented-out code 2020-03-06 09:57:33 -07:00
Edward Hartnett
502336c2c7 now return NC_EINVAL on attempt to set chunking on scalar var 2020-03-03 11:57:16 -07:00
Dennis Heimbigner
73537603e2 Make scalar X filter return an error instead of ignoring it 2020-03-02 15:10:54 -07:00
Dennis Heimbigner
420fdf4625 fix memory allocation failure in hdf5var.c 2020-03-02 11:45:41 -07:00
Dennis Heimbigner
7d1ca9ac85 fix references to var->deflate' 2020-03-02 11:12:30 -07:00
Dennis Heimbigner
e66c727c28 Fix Filters x compact 2020-02-29 15:33:27 -07:00
Dennis Heimbigner
f376c23329 Make utilities support NC_COMPACT
re: https://github.com/Unidata/netcdf-c/issues/1642

Modify ncdump, nccopy, and ncgen to support the NC_COMPACT storage option.
Added test cases and added description to the man pages for the utilities.

1. ncdump: For compact storage variable, print special attribute __Storage_ as
````
    <var>: _Storage = "compact";
````

2. ncgen: parse and implement
````
    <var>: _Storage = "compact";
````
in a .cdl file

3. nccopy: Extend the chunk specification (-c flag) to support
   compact using the forms
````
nccopy ... -c <var>:compact
and
nccopy ... -c <var>:contiguous
````

Misc. other changes
1. cleanup the copy_chunking function in ncdump/nccopy.c
2020-02-29 12:06:21 -07:00
Dennis Heimbigner
10d227fc1b fix parallel filter error discovered by Hartnett 2020-02-28 11:36:58 -07:00
Dennis Heimbigner
a3a3e15cb1 fix bad edit 2020-02-27 15:33:39 -07:00
Dennis Heimbigner
afe5a2998c
Merge branch 'master' into multifilter.dmh 2020-02-27 15:02:27 -07:00
Dennis Heimbigner
b488c272d5 Fix conflicts with master 2020-02-27 14:06:45 -07:00
Edward Hartnett
6f95f655dd
Merge branch 'master' into ejh_dispatch 2020-02-26 16:12:57 -07:00
Edward Hartnett
418e428a05 fixed problem with scalar compact 2020-02-26 09:13:12 -07:00
Edward Hartnett
b31aedcc8e all tests passing but compact storage for scalars not being properly written in file yet 2020-02-26 08:14:06 -07:00
Edward Hartnett
6241a6e7a0 more tests for storage, changed var names to reflect stortage 2020-02-25 15:55:34 -07:00
Edward Hartnett
2ff24bd6fe more tests for compact storage 2020-02-25 13:30:38 -07:00
Dennis Heimbigner
44d0dcaad2 Add support for multiple filters per variable.
re: https://github.com/Unidata/netcdf-c/issues/1584

Support has been added for multiple filters per variable.  This
affects a number of components in netcdf. The new APIs are
documented in NUG/filters.md.

The primary changes are:
* A set of new functions are provided (see __include/netcdf_filter.h__).
    - Obtain a list of the filters associated with a variable
    - Obtain the parameters for a specific filter.
* The existing __nc_inq_var_filter__ function now returns info
  about the first defined filter.
* The utilities (ncgen, ncdump, and nccopy) now support
  an extended format for specifying a sequence of filters.
  The general form is __<filter>|<filter>..._.
* The ncdump **_Filter** attribute now dumps a list of all the
  filters associated with a variable using the above new format.
* Filter specifications can now use a filter name instead of number
  for filters known to the netcdf library, which in turn is taken
  from the HDF5 filter registration page.
* New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter
  is returned if an attempt is made to access an unknown filter.
* Internally, the dispatch table has been extended to add a function
  to handle all of the filter functions.
* New, filter-related, tests were added to nc_test4.
* A new plugin was added to the plugins directory to help with testing.

Notes:
1. The shuffle and fletcher32 filters are not part of the multifilter system.

Misc. changes:
1. A debug module was added to libhdf5 to help catch error locations.
2020-02-16 12:59:33 -07:00
Edward Hartnett
a8684c730c fixed merge conflict in RELEASE_NOTES 2020-02-15 06:42:49 -07:00
Edward Hartnett
05a6ff74b2 merged changes from master 2020-02-11 17:19:53 -07:00
Edward Hartnett
15059a18b7 merged changes from master 2020-02-11 17:19:25 -07:00
Edward Hartnett
a0839a2a7a added version to dispatch table 2020-02-09 13:07:58 -07:00
Edward Hartnett
b7ac19a43f only close non-zero typeids 2020-02-09 12:03:21 -07:00
Edward Hartnett
af6b6787bf fix for memory leak due to HDF5 types 2020-02-09 11:47:13 -07:00
Edward Hartnett
8057a552ef move nc_def_var_szip function so it will appear in the documentation 2020-02-07 09:09:01 -07:00
Edward Hartnett
558988bb18 fixed docs, removed unneeded defines in test 2020-02-07 07:54:12 -07:00
Edward Hartnett
c4d3937099 now check number of elements in chunk against pixels_per_block for szip compression 2020-02-07 07:03:40 -07:00
Edward Hartnett
ff7280512e checking for some bad pixels_per_block values for szip 2020-02-07 06:53:52 -07:00
Edward Hartnett
6d2d751e4e disallow zlib if szip already in use 2020-02-07 05:01:06 -07:00
Edward Hartnett
dc4e880c37 disallow szip if zlib already in use 2020-02-07 04:46:15 -07:00
Edward Hartnett
6b2947813f adding test for zlib+szip in HDF5 2020-02-07 03:38:43 -07:00
Edward Hartnett
1817790c6b rely completely on nc_def_var_filter for setting szip 2020-02-05 10:32:16 -07:00
Edward Hartnett
517ef4f257 use nc_def_var_filter in nc_def_var_szip 2020-02-05 10:25:30 -07:00
Edward Hartnett
52d745de68 now remember szip setting in filter fields 2020-02-04 08:40:15 -07:00
Ward Fisher
aadd5a2d81
Merge pull request #1589 from NOAA-GSD/ejh_szip
re-implement the nc_def_var_szip() function, including for parallel I/O
2020-01-22 16:27:33 -07:00
Edward Hartnett
6103f442cb fixed compile error 2020-01-21 08:50:38 -07:00
Edward Hartnett
c839a2d6c5
Merge branch 'master' into ejh_var_cache 2020-01-21 07:48:59 -07:00
Edward Hartnett
e94615a0e5
Merge branch 'master' into ejh_szip 2020-01-16 08:49:12 -07:00
Ward Fisher
8771d0bdf4
Merge pull request #1582 from NOAA-GSD/ejh_parallel_zlib
Allow user to turn on zlib, shuffle, and/or fletcher32 filters with parallel I/O for HDF5-1.10.2+
2020-01-13 16:06:51 -07:00
Dennis Heimbigner
f587654670 Make the dap4 code resistant to various server errors.
Some versions of some servers are returning malformed responses.
Make the library either handle them or gracefully fail.
The three server errors "fixed" here are as follows.
1. The attribute _NCProperties sometimes has a trailing nul character
   in its value. Soln is to elide the nul(s).
2. Sometimes a DAP response has no data part, only a DMR.
   Soln is to detect and return an error code instead of crashing.
3. Sometimes a server returns a redirection, but our current
   openmagic() function was not following the redirect. Soln
   is to follow redirects.
Also because of #2, I am temporarily making --disable-dap-remote-tests
be the default.
2020-01-08 15:18:31 -07:00
Ward Fisher
d9eb078bfd
Merge pull request #1592 from Unidata/travis_typo_fix.wif
Correct a typo in travis.yml
2020-01-07 18:31:53 -07:00
Ward Fisher
438119dd69
Merge pull request #1560 from NOAA-GSD/ejh_cache_docs
increase default cache size for netCDF-4/HDF5 files, also improve cache docs and add benchmarking program
2020-01-07 11:46:30 -07:00
Ward Fisher
72b79ac376 Cleaned up an 'uninitialized variable' issue reported by static analysis. Minor fix, rolling in to this PR rather than spinning up a separate one. 2020-01-07 11:43:50 -07:00
Edward Hartnett
184507be5f now using members in NC_VAR_INFO_T to hold szip info 2020-01-06 08:46:03 -07:00
Edward Hartnett
3e3b83bdbc whitespace cleanup 2020-01-06 08:09:20 -07:00
Edward Hartnett
6af1b0bd91 changed error code in nc_def_var_szip() to NC_EFILTER 2020-01-06 07:51:04 -07:00
Edward Hartnett
e703a7678c first stab at re-adding nc_def_var_szip() 2020-01-03 11:38:45 -07:00
Edward Hartnett
808a0e2be9 merged ejh_parallel_zlib 2020-01-02 14:25:31 -07:00
Ward Fisher
8f2be58d95
Merge pull request #1566 from NetCDF-World-Domination-Council/ejh_unlim_dims
Fix problems with read past end of dataset but within dimension length for vars with multiple unlimited dimensions
2019-12-23 15:08:56 -07:00
Edward Hartnett
680e44f628 changed name of macro 2019-12-20 13:58:01 -07:00
Edward Hartnett
995cfdad96 merged master 2019-12-20 11:16:11 -07:00