Commit Graph

8420 Commits

Author SHA1 Message Date
Ward Fisher
e763e6caf1
Merge pull request #2109 from DennisHeimbigner/ncgenenum.dmh
Fix handling of enum constants nested in compound types.
2021-10-01 17:08:45 -05:00
Ward Fisher
7ec0ac0a08
Merge branch 'main' into mingw-w64-strcasecmp 2021-10-01 17:07:37 -05:00
Ward Fisher
437060b69a
Merge pull request #2088 from edwardhartnett/ejh_quantize_2
Adding nc_def_var_quantize()/nc_inq_var_quantize() - second attempt
2021-10-01 17:06:05 -05:00
Ward Fisher
69ff6fbe78
Merge pull request #2116 from gsjaardema/patch-48
Make sure HDF5_C_LIBRARY and HDF5_HL_LIBRARY set
2021-10-01 17:04:11 -05:00
Greg Sjaardema
6b8a940ab2
Make sure HDF5_C_LIBRARY and HDF5_HL_LIBRARY set
On some versions of the HDF5 find_package call, it sets `HDF5_C_LIBRARIES` and `HDF5_HL_LIBRARIES`, but does not set the `HDF5_C_LIBRARY` or `HDF5_HL_LIBRARY` to anything.  Control then falls out of the if block with these unset and it falls into the default setting at line 792.  This does not include the path, so then when the later `CHECK_LIBRARY_EXISTS` calls are run, they do not have the full path to the library and will not link correctly.  Since the link fails, the code defaults to thinking that none of the symbols are defined.

I don't think this change will have any affect since it only sets the symbols if they are unset.
2021-10-01 13:34:58 -06:00
Dennis Heimbigner
dc2ecc74ac (1) improve INI parser (2) Fix make discheck 2021-09-30 13:45:09 -06:00
Dennis Heimbigner
0f93766599 One additional lgtm fix 2021-09-28 14:37:32 -06:00
Dennis Heimbigner
56f1f595b8 Fix new lgtm alerts 2021-09-28 14:19:07 -06:00
Dennis Heimbigner
e89187d2c2 remove actions debug 2021-09-28 14:09:38 -06:00
Dennis Heimbigner
ca3dfe43b7 Fix FreeBSD fileno problem in the ncgen parsers 2021-09-28 14:03:19 -06:00
Dennis Heimbigner
b970e81e27 Update release notes 2021-09-27 18:44:48 -06:00
Dennis Heimbigner
6b69b9c52c Significantly Improve Amazon S3 Cloud Storage Support
## S3 Related Fixes

* Add comprehensive support for specifying AWS profiles to provide access credentials.
* Parse the files "~/.aws/config" and "~/.aws/credentials to provide credentials for the HDF5 ROS3 driver and to locate default region.
* Add a function to obtain the currently active S3 credentials. The search rules are defined in docs/nczarr.md.
* Provide documentation for the new features.
* Modify the struct NCauth (in include/ncauth.h) to replace specific S3 credentials with a profile name.
* Add a unit test to test the operation of profile and credentials management.
* Add support for URLS of the form "s3://<bucket>/<key>"; this requires obtaining a default region.
* Allows the specification of profile and/or region in a URL of the form "#mode=nczarr,...&aws.region=...&aws.profile=..."

## Misc. Fixes

* Move the ezxml code to libdispatch so that it can be used both by DAP4 and nczarr.
* Modify nclist to provide a deep clone operation.
* Modify ncuri to provide a deep clone operation.
* Modify the .rc file format to allow the specification of a path to be tested when looking for an entry in the .rc file.
* Ensure that the NC_rcload function is called.
* Modify nchttp to support setting request headers.
2021-09-27 18:36:33 -06:00
Even Rouault
0582c2044a
Fix a stack-read-overflow in ncindexlookup()
Fixes an issue with strlen() reading outside the stack allocated buffer
by NC4_HDF5_inq_att, when reading a name whose length is NC_MAX_NAME.

Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=39189 found
on GDAL

==1895951== Conditional jump or move depends on uninitialised value(s)
==1895951==    at 0x483EF58: strlen (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1895951==    by 0x48EF73E: ncindexlookup (ncindex.c:60)
==1895951==    by 0x48E81DF: nc4_find_grp_att (nc4internal.c:587)
==1895951==    by 0x48E5B39: nc4_get_att_ptrs (nc4attr.c:72)
==1895951==    by 0x48F98A0: NC4_HDF5_inq_att (hdf5attr.c:818)
==1895951==    by 0x48847F7: nc_inq_att (dattinq.c:91)
==1895951==    by 0x10D693: pr_att (ncdump.c:767)
==1895951==    by 0x110ADB: do_ncdump_rec (ncdump.c:1887)
==1895951==    by 0x1112F1: do_ncdump (ncdump.c:2038)
==1895951==    by 0x11248B: main (ncdump.c:2478)
==1895951==
==1895951== Use of uninitialised value of size 8
==1895951==    at 0x48A24E4: crc64_little (dcrc64.c:173)
==1895951==    by 0x48A27F4: NC_crc64 (dcrc64.c:229)
==1895951==    by 0x4892D49: NC_hashmapkey (nchashmap.c:159)
==1895951==    by 0x489314B: NC_hashmapget (nchashmap.c:263)
==1895951==    by 0x48EF75F: ncindexlookup (ncindex.c:60)
==1895951==    by 0x48E81DF: nc4_find_grp_att (nc4internal.c:587)
==1895951==    by 0x48E5B39: nc4_get_att_ptrs (nc4attr.c:72)
==1895951==    by 0x48F98A0: NC4_HDF5_inq_att (hdf5attr.c:818)
==1895951==    by 0x48847F7: nc_inq_att (dattinq.c:91)
==1895951==    by 0x10D693: pr_att (ncdump.c:767)
==1895951==    by 0x110ADB: do_ncdump_rec (ncdump.c:1887)
==1895951==    by 0x1112F1: do_ncdump (ncdump.c:2038)
==1895951==
2021-09-24 11:59:48 +02:00
Dennis Heimbigner
72d6894b80 Fix handling of enum constants nested in compound types.
re: Issue https://github.com/Unidata/netcdf-c/issues/2108

Fix ncgen to properly locate a matching enum type when it encounters
a reference to an enum constant. Add test cases.
2021-09-12 18:03:57 -06:00
Edward Hartnett
5200477de1 now nsd of 0 is NC_EINVAL for nc_def_var_quantize() 2021-09-10 06:10:20 -06:00
Edward Hartnett
7806ded5f2 tinker with data algorithm for tst_compress_par.c 2021-09-09 07:53:55 -06:00
Edward Hartnett
e8587b52a8 changed name of tst_gfs_data_1.c to tst_compress_par.c 2021-09-09 07:20:40 -06:00
Edward Hartnett
9cc39fe70f changed makefile to make benchmark bm_file work properly with zlib-ng 2021-09-09 05:58:13 -06:00
Edward Hartnett
db72457db2 changed makefile to allow tst_gfs_data_1 to pick up libz from LD_LIBRARY_PATH first 2021-09-08 09:31:30 -06:00
Ward Fisher
de403a0fe8
Merge pull request #2099 from DennisHeimbigner/killdebugdir.dmh
Remove netcdf-c/debug directory as no longer needed.
2021-09-07 14:45:02 -06:00
Ward Fisher
af41f84b3f
Merge branch 'main' into killdebugdir.dmh 2021-09-07 14:44:51 -06:00
Edward Hartnett
7943172d17 improving benchmark program 2021-09-07 10:47:24 -06:00
Edward Hartnett
0ce463761c
Merge branch 'main' into ejh_quantize_2 2021-09-07 10:44:45 -06:00
Ward Fisher
0a4f4e16ed
Merge pull request #2098 from DennisHeimbigner/fortcache.dmh
Make the fortran cache API always be defined.
2021-09-07 10:30:00 -06:00
Ward Fisher
8eb71290eb
Merge pull request #2101 from DennisHeimbigner/zarrfilters.dmh
Add filter support to NCZarr
2021-09-07 10:28:28 -06:00
Milton Woods
1fa3955226 Remove config.h from ncjson.c and nclist.c 2021-09-06 10:28:56 +10:00
Milton Woods
4fa91d8241 Use strcasecmp definitions from config.h 2021-09-05 17:17:30 +10:00
Milton Woods
3cf75bf088 Avoid duplicate definition of strcasecmp in config.h 2021-09-05 17:16:50 +10:00
Milton Woods
7b3d71b718 Define strcasecmp for Windows in ncconfigure.h 2021-09-05 17:13:23 +10:00
Milton Woods
cb56d5af38 Avoid redefinition of strcasecmp in mingw-w64 2021-09-04 10:33:47 +10:00
Milton Woods
2a66599276 Use canonical path to derive name of dataset 2021-09-04 10:31:25 +10:00
Milton Woods
1c50c4d313 Use relative pathnames in __FILE__ macros on mingw 2021-09-04 10:28:24 +10:00
Milton Woods
918fef9334 hostname lacks arguments on mingw 2021-09-04 10:28:24 +10:00
Milton Woods
2aabae95cc Avoid setting RPATH in mingw 2021-09-04 10:28:24 +10:00
Milton Woods
d03c0056ae valgrind not supported by mingw 2021-09-04 10:28:24 +10:00
Milton Woods
8525818d90 Support ANSI *printf specifiers on mingw-w64 2021-09-04 10:28:24 +10:00
Dennis Heimbigner
05167d1860 Update Release Notes 2021-09-02 17:07:00 -06:00
Dennis Heimbigner
11fe00ea05 Add filter support to NCZarr
Filter support has three goals:

1. Use the existing HDF5 filter implementations,
2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr,
3. Allow filters to be used even when HDF5 is disabled

Detailed usage directions are define in docs/filters.md.

For now, the existing filter API is left in place. So filters
are defined using ''nc_def_var_filter'' using the HDF5 style
where the id and parameters are unsigned integers.

This is a big change since filters affect many parts of the code.

In the following, the terms "compressor" and "filter" and "codec" are generally
used synonomously.

### Filter-Related Changes:
* In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms.
* Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h.
* Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out.
* Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h.
* Add a number of new test to test the new nczarr filters.
* Let ncgen parse _Codecs attribute, although it is ignored.

### Plugin directory changes:
* Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file
* Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip
* Add a Codec defaulter (see docs/filters.md) for the big four filters.
* Make plugins work with windows by properly adding __declspec declaration.

### Misc. Non-Filter Changes
* Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5.
* Improve support for caching
* More fixes for path conversion code
* Fix misc. memory leaks
* Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath.
* Add a number of new test to test the non-filter fixes.
* Update the parsers
* Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-02 17:04:26 -06:00
Edward Hartnett
18aebd9a15 added parallel I/O quantize test 2021-09-02 10:21:44 -06:00
Edward Hartnett
f880a63f73 added parallel I/O quantize test 2021-09-02 10:18:42 -06:00
Edward Hartnett
09defc5c72 more tests for quantize 2021-09-02 08:18:27 -06:00
Tobias Kölling
5e0167b992
Merge branch 'Unidata:master' into open_mem_truncated_file 2021-09-02 14:10:50 +02:00
Dennis Heimbigner
424ba45483 ENOTNC4 -> ENOTBUILT 2021-09-01 17:27:29 -06:00
Dennis Heimbigner
ee25bc0af0 Update RELEASENOTES.md 2021-09-01 15:03:40 -06:00
Dennis Heimbigner
7fb01ad4ed Remove no-longer-needed debug directory 2021-09-01 15:00:51 -06:00
Dennis Heimbigner
b81f8b676a Make the fortran cache API always be defined.
re: Issue https://github.com/Unidata/netcdf-c/issues/2096

The methods nc_set_var_chunk_cache_ints and nc_def_var_chunking_ints
are Fortran entry points for accessing the cache. They are not defined
if netcdf-c is built with --disable-hdf5.

Fix is to create dummy versions that do nothing and return NC_NOERR
when invoked. These dummy versions are defined when USE_HDF5 is false.
2021-09-01 14:10:02 -06:00
Edward Hartnett
e2570c322c refactored quantize code 2021-09-01 04:29:24 -06:00
Edward Hartnett
d2656bae0a code clean up 2021-09-01 04:13:31 -06:00
Edward Hartnett
ae3b083e20 turned off failing quantize test 2021-09-01 02:29:17 -06:00
Edward Hartnett
30448b4adc merged 2021-09-01 02:25:05 -06:00