mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-01-30 16:10:44 +08:00
Code Issues Packages Projects Releases Wiki Activity
a5d15a5857
netcdf-c/ncdump
History
Dennis Heimbigner df3636b959 Mitigate S3 test interference + Unlimited Dimensions in NCZarr 
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.

The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.

## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
   - "size" with an integer value representing the (current) size of the dimension.
   - "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.

For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.

Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.

## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
   AWS Transfer Utility mechanism. This is controlled by the
   ```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-26 16:56:48 -06:00
..
cdl Adding a missing file to make dist. 2022-02-01 11:46:01 -07:00
expected Adding a missing reference file. 2022-02-01 11:48:27 -07:00
acpget.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
bom.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
cdl.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
chunkspec.c Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
chunkspec.h Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
CMakeLists.txt "Simplify" XGetopt usage 2023-04-09 13:10:41 -06:00
depend
dimmap.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
dimmap.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
dumplib.c Merge branch 'master' into ncdumpvlenbug.dmh 2021-05-27 14:50:05 -06:00
dumplib.h Fixed Clang -Wstrict-prototypes warnings 2022-03-01 23:21:24 -05:00
env
indent.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
indent.h Fixed Clang -Wstrict-prototypes warnings 2022-03-01 23:21:24 -05:00
inttags4.cdl re: gihub issue https://github.com/Unidata/netcdf-c/issues/380 2017-03-24 18:56:14 -06:00
inttags.cdl
L512.bin re e-support UBS-599337 2017-10-24 16:25:09 -06:00
list.c Extend nccopy -F option syntax. 2019-02-08 18:48:17 -07:00
list.h Extend nccopy -F option syntax. 2019-02-08 18:48:17 -07:00
Makefile.am Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
nc4print.c Codify cross-platform file paths 2021-03-04 13:41:31 -07:00
nc4printer.c Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
nccomps.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
nccopy.1 More fixes to the nccopy filter x chunking algorithm 2021-01-31 15:10:39 -07:00
nccopy.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ncdump.1 Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ncdump.c Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
ncdump.h Cleanup DAP4 testing 2022-11-13 13:15:11 -07:00
ncfilteravail.c Two fixes for the price of one! 2022-04-28 15:51:20 -06:00
nchdf5version.c 1. Fix an additional flaw in fill_value handling where non-atomic default values were not properly being handled. 2022-01-10 15:27:16 -07:00
nciter.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
nciter.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
ncpathcvt.c Initialize potentially uninitialized variable 2022-06-07 14:28:01 -06:00
nctime0.c update against main 2023-01-28 13:45:35 -07:00
nctime0.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
nctrunc.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
ncvalidator.c Merge branch 'Unidata:main' into main 2022-05-13 14:16:15 +02:00
ocprint.c Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
printfqn.c Two fixes for the price of one! 2022-04-28 15:51:20 -06:00
ref1.ncml Docs migration 2021-11-11 10:47:49 -07:00
ref_ctest1_nc4.cdl
ref_ctest1_nc4c.cdl
ref_ctest64.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ctest_small_3.c Add test cases 2019-11-03 12:03:13 -07:00
ref_ctest_small_4.c Add test cases 2019-11-03 12:03:13 -07:00
ref_ctest_special_atts_4.c 1. Fix an additional flaw in fill_value handling where non-atomic default values were not properly being handled. 2022-01-10 22:26:19 -07:00
ref_ctest.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_inttags4.cdl Working on getting cdf5 tests running. 2017-09-15 18:00:28 -06:00
ref_inttags.cdl
ref_keyword1.cdl Allow use of type keywords as identifier in formats that do not support that type. 2020-06-05 17:03:29 -06:00
ref_keyword2.cdl Allow use of type keywords as identifier in formats that do not support that type. 2020-06-05 17:03:29 -06:00
ref_keyword3.cdl Fix the handling of certain alias types on CDL files. 2021-04-13 16:56:43 -06:00
ref_keyword4.cdl Fix the handling of certain alias types on CDL files. 2021-04-13 16:56:43 -06:00
ref_nc_test_netcdf4_4_0.nc
ref_nc_test_netcdf4.cdl
ref_nccopy3_subset.nc Wiring new test into autotools. 2017-09-01 11:57:28 -06:00
ref_nccopy_w.cdl Fix nccopy to use NC_PERSIST so that -w actually persists the output. 2019-03-15 12:05:27 -06:00
ref_no_ncproperty.nc Simplify libhdf5/nc5info.c to move to lazy parsing 2019-03-09 20:35:57 -07:00
ref_null_byte_padding_test.nc Wiring in a quick test. 2017-11-20 13:52:06 -07:00
ref_pathcvt.txt Support MSYS2/Mingw platform 2021-12-23 22:18:56 -07:00
ref_provenance_v1.nc There was a request to extend the provenance information 2018-08-25 21:44:41 -06:00
ref_rcapi.txt Make public a limited API for programmatic access to internal .rc tables 2022-06-17 14:35:12 -06:00
ref_rcmerge1.txt Fix a number of OS specific bugs 2021-11-03 12:49:54 -06:00
ref_rcmerge2.txt Fix a number of OS specific bugs 2021-11-03 12:49:54 -06:00
ref_rcmerge3.txt Fix a number of OS specific bugs 2021-11-03 12:49:54 -06:00
ref_roman_szip_simple.cdl added some ncdump tests for szip when it is present 2020-07-02 13:59:37 -06:00
ref_roman_szip_unlim.cdl added some ncdump tests for szip when it is present 2020-07-02 13:59:37 -06:00
ref_test_360_day_1900.cdl Solve issue https://github.com/Unidata/netcdf-c/issues/359 2017-07-25 11:58:34 -06:00
ref_test_360_day_1900.nc Makefile.am cleanup 2017-11-18 14:20:04 -07:00
ref_test_365_day_1900.cdl Solve issue https://github.com/Unidata/netcdf-c/issues/359 2017-07-25 11:58:34 -06:00
ref_test_365_day_1900.nc Makefile.am cleanup 2017-11-18 14:20:04 -07:00
ref_test_366_day_1900.cdl Solve issue https://github.com/Unidata/netcdf-c/issues/359 2017-07-25 11:58:34 -06:00
ref_test_366_day_1900.nc Makefile.am cleanup 2017-11-18 14:20:04 -07:00
ref_test_corrupt_magic.nc re e-support UBS-599337 2017-10-24 16:25:09 -06:00
ref_times_nc4.cdl Added an ncdump -t calendars test using variable length string 2023-01-08 15:58:20 -07:00
ref_times.cdl Test ncdump time unit name case-insensitivity 2020-05-14 06:42:12 -06:00
ref_tst_charfill.cdl
ref_tst_comp2.cdl
ref_tst_comp.cdl
ref_tst_compounds2.cdl
ref_tst_compounds2.nc
ref_tst_compounds3.cdl
ref_tst_compounds3.nc
ref_tst_compounds4.cdl
ref_tst_compounds4.nc
ref_tst_enum_data.cdl
ref_tst_enum_undef.cdl Provide a default enum const when fill value does not match any enum const. 2022-07-17 14:32:31 -06:00
ref_tst_fillbug.cdl
ref_tst_format_att_64.cdl
ref_tst_format_att.cdl
ref_tst_group_data_v23.cdl
ref_tst_group_data.cdl
ref_tst_grp_spec0.cdl
ref_tst_grp_spec.cdl
ref_tst_irish_rover.nc tests for ncdump issue with irish rover 2018-05-15 08:09:52 -06:00
ref_tst_mud4_chars.cdl
ref_tst_mud4-bc.cdl
ref_tst_mud4.cdl
ref_tst_nans.cdl
ref_tst_nc4_utf8_4.cdl Work towards parallel testing via cmake. 2017-11-22 12:10:58 -07:00
ref_tst_ncf213.cdl Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ref_tst_nofilters.cdl More fixes to the nccopy filter x chunking algorithm 2021-01-31 15:10:39 -07:00
ref_tst_noncoord.cdl
ref_tst_opaque_data.cdl
ref_tst_perdimspecs.cdl Fix nccopy -c dim/x to actually use the dim/x value. 2020-09-01 13:44:24 -06:00
ref_tst_radix.cdl Fix ncgen handling of octal constants (with leading 0). 2019-02-25 20:57:23 -07:00
ref_tst_small.cdl
ref_tst_solar_1.cdl
ref_tst_solar_2.cdl
ref_tst_special_atts3.cdl
ref_tst_special_atts.cdl Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ref_tst_string_data.cdl
ref_tst_unicode.cdl
ref_tst_utf8_4.cdl
ref_tst_utf8.cdl Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
ref_tst_vlen_data2.cdl
ref_tst_vlen_data.cdl
rewrite-scalar.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
run_back_comp_tests.sh Fix race condition in ncdump (and other) tests. 2022-11-08 20:12:38 -07:00
run_cygutf8.sh Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
run_ncgen_nc4_tests.sh Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
run_ncgen_tests.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
run_tests.sh clean up 2019-03-12 09:54:16 -06:00
run_utf8_nc4_tests.sh eliminated dependencies 2017-11-25 05:18:04 -07:00
run_utf8_tests.sh eliminated dependencies 2017-11-25 05:18:04 -07:00
scope_ancestor_only.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
scope_ancestor_subgroup.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
scope_group_only.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
scope_preorder.cdl Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
small2.cdl
small.cdl
test0.cdl
test_corrupt_magic.cdl re e-support UBS-599337 2017-10-24 16:25:09 -06:00
test_keywords.sh Support MSYS2/Mingw platform 2021-12-23 22:18:56 -07:00
test_ncdump.sh Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
test_radix.sh Fix ncgen handling of octal constants (with leading 0). 2019-02-25 20:57:23 -07:00
test_rcmerge.sh Provide a single option to disable all network access and testing. 2023-06-10 14:08:04 -06:00
test_scope.sh Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
test_unicode_directory.sh Codify cross-platform file paths 2021-03-04 13:41:31 -07:00
test_unicode_path.sh Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
testpathcvt.sh Update to latest main 2023-04-04 18:37:20 -06:00
tst_64bit.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
tst_bom.sh Fix race condition in ncdump (and other) tests. 2022-11-08 20:12:38 -07:00
tst_brecs.cdl
tst_bug321.cdl
tst_calendars_nc4.cdl Added an ncdump -t calendars test using variable length string 2023-01-08 15:58:20 -07:00
tst_calendars_nc4.sh Added an ncdump -t calendars test using variable length string 2023-01-08 15:58:20 -07:00
tst_calendars.cdl Test ncdump time unit name case-insensitivity 2020-05-14 06:42:12 -06:00
tst_calendars.sh Makefile.am cleanup 2017-11-18 14:20:04 -07:00
tst_charfill.cdl
tst_charfill.sh
tst_chunking.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
tst_comp2.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_comp.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_compress.c Add a comment 2021-08-16 10:06:45 -06:00
tst_create_files.c added some ncdump tests for szip when it is present 2020-07-02 13:59:37 -06:00
tst_ctests.sh Add test cases 2019-11-03 12:03:13 -07:00
tst_cygutf8.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_dimsizes.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_dimsizes.sh Add complete bitgroom support to NCZarr 2022-01-24 15:22:24 -07:00
tst_enum_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_enum_undef.c Provide a default enum const when fill value does not match any enum const. 2022-07-17 14:32:31 -06:00
tst_fileinfo.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_fileinfo.sh Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_fillbug.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_fillbug.sh move C program invokations to scripts to clean up dependencies in build 2017-11-16 13:03:35 -07:00
tst_formatx3.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
tst_formatx4.sh make -j working for classic only in ncdump 2017-11-17 08:22:49 -07:00
tst_group_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_grp_spec.sh
tst_h_rdc0.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_h_scalar.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_h_scalar.sh move C program invokations to scripts to clean up dependencies in build 2017-11-16 13:03:35 -07:00
tst_hdf5_offset.sh Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_inmemory_nc3.sh make -j working for classic only in ncdump 2017-11-17 08:22:49 -07:00
tst_inmemory_nc4.sh make -j working for classic only in ncdump 2017-11-17 08:22:49 -07:00
tst_inttags4.sh Working on getting cdf5 tests running. 2017-09-15 18:00:28 -06:00
tst_inttags.sh
tst_iter.sh
tst_lengths.sh replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
tst_mslp.cdl
tst_mud.sh Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_nans.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_nccopy3_subset.sh Swapped /bin/bash for /bin/sh to test on osx. 2018-08-12 23:01:08 -06:00
tst_nccopy3.sh Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
tst_nccopy4.sh Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_nccopy5.sh Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_nccopy_w3.sh Fix nccopy to use NC_PERSIST so that -w actually persists the output. 2019-03-15 12:05:27 -06:00
tst_nccopy_w4.sh Fix nccopy to use NC_PERSIST so that -w actually persists the output. 2019-03-15 12:05:27 -06:00
tst_ncgen4_classic.sh The ncdap_tests were a mess, so I decided to clean them up 2018-03-20 21:31:31 -06:00
tst_ncgen4_cycle.sh Fix handling of enum constants nested in compound types. 2021-09-12 18:03:57 -06:00
tst_ncgen4_diff.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
tst_ncgen4.sh Swapped /bin/bash for /bin/sh to test on osx. 2018-08-12 23:01:08 -06:00
tst_ncgen_shared.sh Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
tst_ncml.cdl
tst_netcdf4_4.sh Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_netcdf4.sh Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_null_byte_padding.sh Corrected pathing oversight. 2017-11-21 12:17:32 -07:00
tst_opaque_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_output.sh typo 2022-05-19 22:43:08 -06:00
tst_radix.cdl Fix ncgen handling of octal constants (with leading 0). 2019-02-25 20:57:23 -07:00
tst_rcapi.c Fix errors: 2022-06-20 14:09:05 -06:00
tst_rcmerge.c Make public a limited API for programmatic access to internal .rc tables 2022-06-17 14:35:12 -06:00
tst_special_atts.c Make utilities support NC_COMPACT 2020-02-29 12:06:21 -07:00
tst_string_data.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_unicode.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
tst_utf8.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
tst_vlen_data.c Ensure that netcdf_json.h does not interfere with ncjson. 2022-07-05 22:03:52 -06:00
tst_vlen_demo.c ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00
utils.c update against main 2023-01-28 13:45:35 -07:00
utils.h update against main 2023-01-28 13:45:35 -07:00
vardata.c Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
vardata.h ncdump, other directories updated with copyright stanza. 2018-12-06 15:36:53 -07:00