Commit Graph

447 Commits

Author SHA1 Message Date
Ward Fisher
aadd5a2d81
Merge pull request #1589 from NOAA-GSD/ejh_szip
re-implement the nc_def_var_szip() function, including for parallel I/O
2020-01-22 16:27:33 -07:00
Edward Hartnett
6103f442cb fixed compile error 2020-01-21 08:50:38 -07:00
Edward Hartnett
c839a2d6c5
Merge branch 'master' into ejh_var_cache 2020-01-21 07:48:59 -07:00
Edward Hartnett
e94615a0e5
Merge branch 'master' into ejh_szip 2020-01-16 08:49:12 -07:00
Ward Fisher
8771d0bdf4
Merge pull request #1582 from NOAA-GSD/ejh_parallel_zlib
Allow user to turn on zlib, shuffle, and/or fletcher32 filters with parallel I/O for HDF5-1.10.2+
2020-01-13 16:06:51 -07:00
Dennis Heimbigner
f587654670 Make the dap4 code resistant to various server errors.
Some versions of some servers are returning malformed responses.
Make the library either handle them or gracefully fail.
The three server errors "fixed" here are as follows.
1. The attribute _NCProperties sometimes has a trailing nul character
   in its value. Soln is to elide the nul(s).
2. Sometimes a DAP response has no data part, only a DMR.
   Soln is to detect and return an error code instead of crashing.
3. Sometimes a server returns a redirection, but our current
   openmagic() function was not following the redirect. Soln
   is to follow redirects.
Also because of #2, I am temporarily making --disable-dap-remote-tests
be the default.
2020-01-08 15:18:31 -07:00
Ward Fisher
d9eb078bfd
Merge pull request #1592 from Unidata/travis_typo_fix.wif
Correct a typo in travis.yml
2020-01-07 18:31:53 -07:00
Ward Fisher
438119dd69
Merge pull request #1560 from NOAA-GSD/ejh_cache_docs
increase default cache size for netCDF-4/HDF5 files, also improve cache docs and add benchmarking program
2020-01-07 11:46:30 -07:00
Ward Fisher
72b79ac376 Cleaned up an 'uninitialized variable' issue reported by static analysis. Minor fix, rolling in to this PR rather than spinning up a separate one. 2020-01-07 11:43:50 -07:00
Edward Hartnett
184507be5f now using members in NC_VAR_INFO_T to hold szip info 2020-01-06 08:46:03 -07:00
Edward Hartnett
3e3b83bdbc whitespace cleanup 2020-01-06 08:09:20 -07:00
Edward Hartnett
6af1b0bd91 changed error code in nc_def_var_szip() to NC_EFILTER 2020-01-06 07:51:04 -07:00
Edward Hartnett
e703a7678c first stab at re-adding nc_def_var_szip() 2020-01-03 11:38:45 -07:00
Edward Hartnett
808a0e2be9 merged ejh_parallel_zlib 2020-01-02 14:25:31 -07:00
Ward Fisher
8f2be58d95
Merge pull request #1566 from NetCDF-World-Domination-Council/ejh_unlim_dims
Fix problems with read past end of dataset but within dimension length for vars with multiple unlimited dimensions
2019-12-23 15:08:56 -07:00
Edward Hartnett
680e44f628 changed name of macro 2019-12-20 13:58:01 -07:00
Edward Hartnett
995cfdad96 merged master 2019-12-20 11:16:11 -07:00
Edward Hartnett
a06df0e4eb fixing for non-parallel builds 2019-12-20 07:52:00 -07:00
Edward Hartnett
accb83a8b5 even more documentation updates 2019-12-20 07:20:02 -07:00
Edward Hartnett
4b7f839666 swtich to collective access when filters are applied 2019-12-20 07:00:12 -07:00
Edward Hartnett
f86c0fb8f9 now check that HDF5 version supports parallel zlib 2019-12-20 05:54:21 -07:00
Edward Hartnett
d534b1298a adding another zlib parallel I/O test 2019-12-20 05:28:20 -07:00
Ward Fisher
6c75e97764
Merge pull request #1570 from NOAA-GSD/ejh_compact
enable compact storage for netcdf-4 vars
2019-12-19 16:47:05 -07:00
Edward Hartnett
3e00967879 allow parallel writes to use zlib 2019-12-19 09:19:23 -07:00
Ward Fisher
29d070c50f
Merge pull request #1564 from NetCDF-World-Domination-Council/ejh_docs_cleanup
fix memory issue that may occur for some HDF5 file opens
2019-12-17 16:18:44 -07:00
Edward Hartnett
bacf017699 better handling of var cache for parallel builds 2019-12-17 06:39:09 -07:00
Edward Hartnett
fd604ddb06 fixed comment 2019-12-16 15:44:14 -07:00
Edward Hartnett
66a2b4c05e more testing for compact vars 2019-12-16 09:37:54 -07:00
Edward Hartnett
06896f432d got compact storage test working 2019-12-04 08:49:37 -07:00
Edward Hartnett
82df2876b6 starting to support compact storage 2019-12-04 07:53:37 -07:00
Edward Hartnett
e52a74520e tests and fix for multiple unlimited dim bug 2019-12-01 15:05:09 -07:00
Edward Hartnett
c5c38148bd moved udata.grps initialization to avoid memory problem on BAIL 2019-12-01 07:37:32 -07:00
Edward Hartnett
5ab7bf7796 now always relax! 2019-11-26 05:36:16 -07:00
Edward Hartnett
2682ffd68d improved docs for cache functions, added libhdf5/hdf5cache.c to Doxyfile.in, added benchmark program for cache settings 2019-11-25 16:33:04 -07:00
Ward Fisher
923d4ccbff
Merge pull request #1530 from NetCDF-World-Domination-Council/ejh_endianness
now testing that endianness can only be set on atomic ints and floats
2019-11-15 15:27:35 -07:00
edwardhartnett
965da1de01 now testing that endianness can only be set on atomic ints and floats 2019-11-15 11:10:10 -07:00
edwardhartnett
8083b3596e fixed problem of unlim dim and var sharing the same name but not being related 2019-11-15 09:18:42 -07:00
Ward Fisher
1a6351dab2
Merge pull request #1521 from ckhroulev/netcdf4-repeated-attribute-modification
Improve the fix for #350 included in #1119
2019-11-14 17:01:09 -07:00
edwardhartnett
d73611de73 now handle two anon dimensions of same size used in same HDF5 var 2019-11-14 06:54:22 -07:00
edwardhartnett
6b9248cef8 adding test 2019-11-14 06:09:45 -07:00
Constantine Khrulev
6abbf8d429 Whitespace changes 2019-11-13 10:07:10 -09:00
Constantine Khrulev
098f2c1056 Modify the condition used to check if an attribute can be re-used
This should make the code a bit cleaner.
2019-11-13 08:38:12 -09:00
Constantine Khrulev
dd181deca9 Improve the fix for #350 included in #1119
1) We have to use H5Tequal() to compare HDF5 type IDs.
2) When checking if we can re-use an NC_CHAR attribute it is enough to
   compare data types (H5Tequal() takes care of the size comparison).
3) This commit adds missing code (reuse_att was set but not used).

Now an attribute in a NetCDF-4 file can be modified as many times as
necessary, as long its type and length remain the same.

Modifications changing either type or length of an attribute require
deleting and re-creating an attribute which increments the attribute
order creation index. Once this index reaches 65535 all attribute
modifications (for a particular group or variable) will fail.

For reference:

Issue 350 title: NetCDF-4 limits the number of times an attribute can
be modified

Pull request 1119 title: Fix checking for HDF5 max dims, no longer
re-create atts if not needed, confirm behavior for HDF5 cyclical
files, allow user to set mpiexec
2019-11-12 21:45:47 -09:00
Dennis Heimbigner
f1506d552e Change (again), and hopefully simplify, the file model inference algorithm.
* For URL paths, the new approach essentially centralizes all information
  in the URL into the "#mode=" fragment key and uses that value
  to determine the dispatcher for (most) URLs.

* The new approach has the following steps:

  1. canonicalize the path if it is a URL.
  2. use the mode= fragment key to determine the dispatcher
  3. if dispatcher still not determined, then use the mode flags
     argument to nc_open/nc_create to determine the dispatcher.
  4. if the path points to something readable, attempt to read the
     magic number at the front, and use that to determine the dispatcher.
     this case may override all previous cases.

* Misc changes.

  1. Update documentation
  2. Moved some unit tests from libdispatch to unit_test directory.
  3. Fixed use of wrong #ifdef macro in test_filter_reg.c
     [I think this may fix an previously reported esupport query].
2019-09-29 12:59:28 -06:00
Greg Sjaardema
56c0d5cf8a Spelling fixes 2019-09-18 08:03:01 -06:00
edwardhartnett
2077729abc removed base_pe functions from dispatch table 2019-08-15 06:51:06 -06:00
edwardhartnett
3c9a25b688 whitespace cleanup 2019-08-03 09:04:58 -06:00
edwardhartnett
7ce322a6f1 now have libhdf5 use nc4_file_list_add() 2019-08-02 09:29:18 -06:00
edwardhartnett
fc1d9baf43 fixed spacing 2019-08-01 18:24:11 -06:00
edwardhartnett
cb3101c59c clean up 2019-08-01 16:13:32 -06:00