netcdf-c/ncdump
Dennis Heimbigner f6e25b695e Fix additional S3 support issues
re: https://github.com/Unidata/netcdf-c/issues/2117
re: https://github.com/Unidata/netcdf-c/issues/2119

* Modify libsrc to allow byte-range reading of netcdf-3 files in private S3 buckets; this required using the aws sdk. Also add a test case.
* The aws sdk can sometimes cause problems if the Awd::ShutdownAPI function is not called. So at optional atexit() support to ensure it is called. This is disabled for Windows.
* Add documentation to nczarr.md on how to build and use the aws sdk under windows. Currently it builds, but testing fails.
* Switch testing from stratus to the Unidata bucket on S3.
* Improve support for the s3: url protocol.
* Add a s3 specific utility code file: ds3util.c
* Modify NC_infermodel to attempt to read the magic number of byte-ranged files in S3.

## Misc.

* Move and rename the core S3 SDK wrapper code (libnczarr/zs3sdk.cpp) to libdispatch since it now used in libsrc as well as libnczarr.
* Add calls to nc_finalize in the utilities in case atexit is disabled.
* Add header only json parser to the distribution rather than as a built source.
2021-10-29 20:06:37 -06:00
..
cdl Fix handling of enum constants nested in compound types. 2021-09-12 18:03:57 -06:00
expected Fix handling of enum constants nested in compound types. 2021-09-12 18:03:57 -06:00
bom.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
cdl.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
chunkspec.c Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
chunkspec.h Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
CMakeLists.txt Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
depend moving to trunk subdir 2010-06-03 13:24:43 +00:00
dimmap.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
dimmap.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
dumplib.c Merge branch 'master' into ncdumpvlenbug.dmh 2021-05-27 14:50:05 -06:00
dumplib.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
env re: NCF-317 2014-10-07 12:37:41 -06:00
indent.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
indent.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
inttags4.cdl re: gihub issue https://github.com/Unidata/netcdf-c/issues/380 2017-03-24 18:56:14 -06:00
inttags.cdl ckp 2015-11-19 13:44:55 -07:00
L512.bin re e-support UBS-599337 2017-10-24 16:25:09 -06:00
list.c Extend nccopy -F option syntax. 2019-02-08 18:48:17 -07:00
list.h Extend nccopy -F option syntax. 2019-02-08 18:48:17 -07:00
Makefile.am Merge branch 'main' into ejh_quantize_2 2021-09-07 10:44:45 -06:00
nc4print.c Codify cross-platform file paths 2021-03-04 13:41:31 -07:00
nc4printer.c Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
nccomps.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
nccopy.1 More fixes to the nccopy filter x chunking algorithm 2021-01-31 15:10:39 -07:00
nccopy.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ncdump.1 Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
ncdump.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ncdump.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
nciter.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
nciter.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
ncpathcvt.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
nctime0.c Treat time units as case-insensitive in ncdump 2020-05-14 06:48:03 -06:00
nctime0.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
nctrunc.c Remove netcdf-4 conditional on the dispatch table. 2019-07-20 13:59:40 -06:00
ncvalidator.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ocprint.c Fix Mingw versus XGetopt (again) 2021-05-26 14:27:27 -06:00
printfqn.c Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
ref1.ncml Fix github issue: https://github.com/Unidata/netcdf-c/issues/236 2016-03-17 22:05:46 -06:00
ref_ctest1_nc4.cdl Wiring in more validation 2016-11-18 11:53:53 -07:00
ref_ctest1_nc4c.cdl Fix ncdump clang warnings. Fix invalid encodings for some non-ascii strings in tests. 2015-08-02 16:09:47 -06:00
ref_ctest64.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ctest_small_3.c Add test cases 2019-11-03 12:03:13 -07:00
ref_ctest_small_4.c Add test cases 2019-11-03 12:03:13 -07:00
ref_ctest_special_atts_4.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ctest.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_inttags4.cdl Working on getting cdf5 tests running. 2017-09-15 18:00:28 -06:00
ref_inttags.cdl ckp 2015-11-19 13:44:55 -07:00
ref_keyword1.cdl Allow use of type keywords as identifier in formats that do not support that type. 2020-06-05 17:03:29 -06:00
ref_keyword2.cdl Allow use of type keywords as identifier in formats that do not support that type. 2020-06-05 17:03:29 -06:00
ref_keyword3.cdl Fix the handling of certain alias types on CDL files. 2021-04-13 16:56:43 -06:00
ref_keyword4.cdl Fix the handling of certain alias types on CDL files. 2021-04-13 16:56:43 -06:00
ref_nc_test_netcdf4_4_0.nc Support Dave Allured's ncdump mods for time-valued attributes. Refactor all ncdump time code into nctime.c. Add '-i' option for strict ISO-8601 time notation. Fix backward compatibility test to not use DBL_MAX, which is the same as IEEE infinity when using only 15 significant digits of precision. Document ncdump time functionality in man page. 2011-09-12 21:31:08 +00:00
ref_nc_test_netcdf4.cdl Support Dave Allured's ncdump mods for time-valued attributes. Refactor all ncdump time code into nctime.c. Add '-i' option for strict ISO-8601 time notation. Fix backward compatibility test to not use DBL_MAX, which is the same as IEEE infinity when using only 15 significant digits of precision. Document ncdump time functionality in man page. 2011-09-12 21:31:08 +00:00
ref_nccopy3_subset.nc Wiring new test into autotools. 2017-09-01 11:57:28 -06:00
ref_nccopy_w.cdl Fix nccopy to use NC_PERSIST so that -w actually persists the output. 2019-03-15 12:05:27 -06:00
ref_no_ncproperty.nc Simplify libhdf5/nc5info.c to move to lazy parsing 2019-03-09 20:35:57 -07:00
ref_null_byte_padding_test.nc Wiring in a quick test. 2017-11-20 13:52:06 -07:00
ref_pathcvt.txt Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_provenance_v1.nc There was a request to extend the provenance information 2018-08-25 21:44:41 -06:00
ref_rcmerge1.txt Upgrade the nczarr code to match Zarr V2 2021-04-24 19:48:15 -06:00
ref_rcmerge2.txt Upgrade the nczarr code to match Zarr V2 2021-04-24 19:48:15 -06:00
ref_rcmerge3.txt Upgrade the nczarr code to match Zarr V2 2021-04-24 19:48:15 -06:00
ref_roman_szip_simple.cdl added some ncdump tests for szip when it is present 2020-07-02 13:59:37 -06:00
ref_roman_szip_unlim.cdl added some ncdump tests for szip when it is present 2020-07-02 13:59:37 -06:00
ref_test_360_day_1900.cdl Solve issue https://github.com/Unidata/netcdf-c/issues/359 2017-07-25 11:58:34 -06:00
ref_test_360_day_1900.nc Makefile.am cleanup 2017-11-18 14:20:04 -07:00
ref_test_365_day_1900.cdl Solve issue https://github.com/Unidata/netcdf-c/issues/359 2017-07-25 11:58:34 -06:00
ref_test_365_day_1900.nc Makefile.am cleanup 2017-11-18 14:20:04 -07:00
ref_test_366_day_1900.cdl Solve issue https://github.com/Unidata/netcdf-c/issues/359 2017-07-25 11:58:34 -06:00
ref_test_366_day_1900.nc Makefile.am cleanup 2017-11-18 14:20:04 -07:00
ref_test_corrupt_magic.nc re e-support UBS-599337 2017-10-24 16:25:09 -06:00
ref_times.cdl Test ncdump time unit name case-insensitivity 2020-05-14 06:42:12 -06:00
ref_tst_charfill.cdl add test for fix for Jira NCF-131 2011-12-13 03:08:56 +00:00
ref_tst_comp2.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_comp.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_compounds2.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_compounds2.nc moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_compounds3.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_compounds3.nc moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_compounds4.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_compounds4.nc moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_enum_data.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_fillbug.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_format_att_64.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_format_att.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_group_data_v23.cdl Clarified documentation for nc_inq_grp_ncid(), differentiating it from nc_inq_ncid(). Fixed problem returning values from functions declared void in some libsrc4/ test programs. Added test for bug in ncdump for dimensions with same name in nested groups, and fix for bug. Fixed nccopy bug involving dimensions with smae name in nested groups. Added code for specifying chunking by dimension to nccopy (but not implemented yet). 2010-12-30 18:17:04 +00:00
ref_tst_group_data.cdl Clarified documentation for nc_inq_grp_ncid(), differentiating it from nc_inq_ncid(). Fixed problem returning values from functions declared void in some libsrc4/ test programs. Added test for bug in ncdump for dimensions with same name in nested groups, and fix for bug. Fixed nccopy bug involving dimensions with smae name in nested groups. Added code for specifying chunking by dimension to nccopy (but not implemented yet). 2010-12-30 18:17:04 +00:00
ref_tst_grp_spec0.cdl Add -g option to ncdump and test it 2011-09-21 23:10:03 +00:00
ref_tst_grp_spec.cdl Add -g option to ncdump and test it 2011-09-21 23:10:03 +00:00
ref_tst_irish_rover.nc tests for ncdump issue with irish rover 2018-05-15 08:09:52 -06:00
ref_tst_mud4_chars.cdl Fixed bug NCF-310 (ncdump char vars with multiple unlimited dims). 2014-08-07 14:35:29 -06:00
ref_tst_mud4-bc.cdl Add reference file for ncdump -bc test 2013-11-13 13:45:20 -07:00
ref_tst_mud4.cdl Fixed bug NCF-144 (ncdump of variables with multiple unlimited 2012-11-16 21:37:43 +00:00
ref_tst_nans.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_nc4_utf8_4.cdl Work towards parallel testing via cmake. 2017-11-22 12:10:58 -07:00
ref_tst_ncf213.cdl reverted changes superblock values 2021-01-11 14:43:44 -06:00
ref_tst_nofilters.cdl More fixes to the nccopy filter x chunking algorithm 2021-01-31 15:10:39 -07:00
ref_tst_noncoord.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_opaque_data.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_perdimspecs.cdl Fix nccopy -c dim/x to actually use the dim/x value. 2020-09-01 13:44:24 -06:00
ref_tst_radix.cdl Fix ncgen handling of octal constants (with leading 0). 2019-02-25 20:57:23 -07:00
ref_tst_small.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_solar_1.cdl ckp 2015-11-19 13:44:55 -07:00
ref_tst_solar_2.cdl Fix ncdump bug displaying NC_USHORT attribute values as NC_SHORT in netCDF-4 files. Add test for fix. 2011-06-24 04:51:11 +00:00
ref_tst_special_atts3.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_special_atts.cdl reverted changes superblock values 2021-01-11 14:43:44 -06:00
ref_tst_string_data.cdl Fix NCF-186, ncdump bug not escaping quotes and newlines in string data 2012-08-03 04:40:06 +00:00
ref_tst_unicode.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_utf8_4.cdl re: Jira NCF-320 2015-08-20 15:53:48 -06:00
ref_tst_utf8.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
ref_tst_vlen_data2.cdl premerge 2012-01-29 18:50:02 +00:00
ref_tst_vlen_data.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
rewrite-scalar.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
run_back_comp_tests.sh Primary change: add dap4 support 2017-03-08 17:01:10 -07:00
run_ncgen_nc4_tests.sh Fix more memory leaks in netcdf-c library 2018-11-15 10:00:38 -07:00
run_ncgen_tests.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
run_tests.sh clean up 2019-03-12 09:54:16 -06:00
run_utf8_nc4_tests.sh eliminated dependencies 2017-11-25 05:18:04 -07:00
run_utf8_tests.sh eliminated dependencies 2017-11-25 05:18:04 -07:00
scope_ancestor_only.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
scope_ancestor_subgroup.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
scope_group_only.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
scope_preorder.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
small2.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
small.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
test0.cdl [NCF-267]/SAI-630695 2013-09-23 17:11:59 -06:00
test_corrupt_magic.cdl re e-support UBS-599337 2017-10-24 16:25:09 -06:00
test_keywords.sh Fix the handling of certain alias types on CDL files. 2021-04-13 16:56:43 -06:00
test_radix.sh Fix ncgen handling of octal constants (with leading 0). 2019-02-25 20:57:23 -07:00
test_rcmerge.sh Allow .rc tests to work in parallel by isolation 2021-04-25 22:02:29 -06:00
test_scope.sh Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
test_unicode_directory.sh Codify cross-platform file paths 2021-03-04 13:41:31 -07:00
test_unicode_path.sh Codify cross-platform file paths 2021-03-04 13:41:31 -07:00
testpathcvt.sh Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_64bit.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
tst_bom.sh Primary change: add dap4 support 2017-03-08 17:01:10 -07:00
tst_brecs.cdl Allow double NaNs and Infinities in double to double conversions. Add tests for nccopy deflation, shuffling, and dimension-fixing options. Add man-page documentation for new nccopy topions, with examples. Run tst_nccopy3.sh test even for netCDF-4 builds. 2010-08-31 22:41:00 +00:00
tst_bug321.cdl Add test CDL file and tst_nccopy4.sh test for fix of NCF-321 2015-01-03 17:18:14 -07:00
tst_calendars.cdl Test ncdump time unit name case-insensitivity 2020-05-14 06:42:12 -06:00
tst_calendars.sh Makefile.am cleanup 2017-11-18 14:20:04 -07:00
tst_charfill.cdl add test for fix for Jira NCF-131 2011-12-13 03:08:56 +00:00
tst_charfill.sh Primary change: add dap4 support 2017-03-08 17:01:10 -07:00
tst_chunking.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_comp2.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_comp.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_compress.c Add a comment 2021-08-16 10:06:45 -06:00
tst_create_files.c added some ncdump tests for szip when it is present 2020-07-02 13:59:37 -06:00
tst_ctests.sh Add test cases 2019-11-03 12:03:13 -07:00
tst_dimsizes.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_dimsizes.sh make -j working for classic only in ncdump 2017-11-17 08:22:49 -07:00
tst_enum_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_fileinfo.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_fileinfo.sh Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_fillbug.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_fillbug.sh move C program invokations to scripts to clean up dependencies in build 2017-11-16 13:03:35 -07:00
tst_formatx3.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
tst_formatx4.sh make -j working for classic only in ncdump 2017-11-17 08:22:49 -07:00
tst_group_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_grp_spec.sh Primary change: add dap4 support 2017-03-08 17:01:10 -07:00
tst_h_rdc0.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_h_scalar.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_h_scalar.sh move C program invokations to scripts to clean up dependencies in build 2017-11-16 13:03:35 -07:00
tst_hdf5_offset.sh Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_inmemory_nc3.sh make -j working for classic only in ncdump 2017-11-17 08:22:49 -07:00
tst_inmemory_nc4.sh make -j working for classic only in ncdump 2017-11-17 08:22:49 -07:00
tst_inttags4.sh Working on getting cdf5 tests running. 2017-09-15 18:00:28 -06:00
tst_inttags.sh Primary change: add dap4 support 2017-03-08 17:01:10 -07:00
tst_iter.sh Primary change: add dap4 support 2017-03-08 17:01:10 -07:00
tst_lengths.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
tst_mslp.cdl moving to trunk subdir 2010-06-03 13:24:43 +00:00
tst_mud.sh Primary change: add dap4 support 2017-03-08 17:01:10 -07:00
tst_nans.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_nccopy3_subset.sh Swapped /bin/bash for /bin/sh to test on osx. 2018-08-12 23:01:08 -06:00
tst_nccopy3.sh This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
tst_nccopy4.sh Upgrade the nczarr code to match Zarr V2 2021-04-24 19:48:15 -06:00
tst_nccopy5.sh Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
tst_nccopy_w3.sh Fix nccopy to use NC_PERSIST so that -w actually persists the output. 2019-03-15 12:05:27 -06:00
tst_nccopy_w4.sh Fix nccopy to use NC_PERSIST so that -w actually persists the output. 2019-03-15 12:05:27 -06:00
tst_ncgen4_classic.sh The ncdap_tests were a mess, so I decided to clean them up 2018-03-20 21:31:31 -06:00
tst_ncgen4_cycle.sh Fix handling of enum constants nested in compound types. 2021-09-12 18:03:57 -06:00
tst_ncgen4_diff.sh removed many dependencies in ncdump testing 2017-11-25 07:28:07 -07:00
tst_ncgen4.sh Swapped /bin/bash for /bin/sh to test on osx. 2018-08-12 23:01:08 -06:00
tst_ncgen_shared.sh Fix handling of enum constants nested in compound types. 2021-09-12 18:03:57 -06:00
tst_ncml.cdl Fix github issue: https://github.com/Unidata/netcdf-c/issues/236 2016-03-17 22:05:46 -06:00
tst_netcdf4_4.sh Merge branch '951' of https://github.com/brtnfld/netcdf-c into 4.8.0-wellspring-prs.wif 2021-03-22 11:51:54 -06:00
tst_netcdf4.sh Merge branch '951' of https://github.com/brtnfld/netcdf-c into 4.8.0-wellspring-prs.wif 2021-03-22 11:51:54 -06:00
tst_null_byte_padding.sh Corrected pathing oversight. 2017-11-21 12:17:32 -07:00
tst_opaque_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_output.sh Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_radix.cdl Fix ncgen handling of octal constants (with leading 0). 2019-02-25 20:57:23 -07:00
tst_rcmerge.c Significantly Improve Amazon S3 Cloud Storage Support 2021-09-27 18:36:33 -06:00
tst_special_atts.c Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
tst_string_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_unicode.c Codify cross-platform file paths 2021-03-04 13:41:31 -07:00
tst_utf8.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_vlen_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_vlen_demo.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
utils.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
utils.h Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
vardata.c Correcting a formatting error for scalars when dumping with ncdump -f 2020-04-28 15:49:03 -06:00
vardata.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00