mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-01-12 15:45:21 +08:00
Code Issues Packages Projects Releases Wiki Activity
cb61d34185
netcdf-c/nc_test4
History
Dennis Heimbigner df3636b959 Mitigate S3 test interference + Unlimited Dimensions in NCZarr 
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.

The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.

## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
   - "size" with an integer value representing the (current) size of the dimension.
   - "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.

For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.

Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.

## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
   AWS Transfer Utility mechanism. This is controlled by the
   ```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-26 16:56:48 -06:00
..
bm_chunking.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
bm_radar.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
build_fixedstring.c Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
bzip2.cdl 1. Keep up to date by merging master 2018-01-16 11:00:09 -07:00
cdm_sea_soundings.c Spelling fixes 2019-09-18 08:03:01 -06:00
clear_cache.sh Swapped /bin/bash for /bin/sh to test on osx. 2018-08-12 23:01:08 -06:00
CMakeLists.txt Merge pull request #2655 from ZedThree/hdf5-transient-types 2023-07-18 16:49:33 -06:00
findplugin.in Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
h5testszip.c Fix for use with szip and also 1.8.21 2021-04-28 15:42:02 -06:00
Makefile.am Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
noop1.cdl Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
noop.cdl Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ref_any.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_bloscx.cdl Fix some bugs in the blosc filter wrapper 2022-07-12 15:19:07 -06:00
ref_bzip2.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_fillonly.cdl Fix error where not converting fill data 2020-09-12 14:49:59 -06:00
ref_filter_order_create.txt Add tests for filter order on read and write cases 2020-12-29 20:12:35 -07:00
ref_filter_order_read.txt Add tests for filter order on read and write cases 2020-12-29 20:12:35 -07:00
ref_filter_repeat.txt Add tests for filter order on read and write cases 2020-12-29 20:12:35 -07:00
ref_filtered.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_filteredvv.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_filterx_hdf5.txt Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_fixedstring.cdl Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
ref_fixedstring.h5 Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
ref_hdf5_compat1.nc
ref_hdf5_compat2.nc
ref_hdf5_compat3.nc
ref_multi.cdl Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
ref_nccopyF.cdl Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
ref_ncgenF.cdl Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
ref_szip.cdl add known szip file against which to test 2017-09-04 18:37:13 -06:00
ref_szip.h5 add known szip file against which to test 2017-09-04 18:37:13 -06:00
ref_tst_compounds.nc
ref_tst_dims.nc
ref_tst_interops4.nc
ref_tst_xplatform2_1.nc
ref_tst_xplatform2_2.nc
ref_unfiltered.cdl Renamed the baseline files for tst_filter.sh 2019-02-09 15:14:13 -07:00
ref_unfilteredvv.cdl Renamed the baseline files for tst_filter.sh 2019-02-09 15:14:13 -07:00
renamegroup.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
run_empty_vlen_test.sh
run_grp_rename.sh Provide byte-range reading of remote datasets 2019-01-01 18:27:36 -07:00
run_par_test.sh.in no longer run tst_parallel_zlib with 1 processor in testing 2022-04-30 07:42:01 -06:00
t_type.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tdset.h5
test_fillonly.sh Fix error where not converting fill data 2020-09-12 14:49:59 -06:00
test_filter_misc.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
test_filter_order.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filter_reg.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filter_repeat.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filter_vlen.c Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
test_filter.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filterx_hdf5.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_szip.c Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
test_wrapper.in 1. Keep up to date by merging master 2018-01-16 11:00:09 -07:00
tst_alignment.c Add support for setting HDF5 alignment property when creating a file 2022-01-29 15:27:52 -07:00
tst_atts1.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_atts2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_atts3.c merge master and fix conflicts 2018-12-12 11:47:54 -07:00
tst_atts_string_rewrite.c more config.h includes for tests 2019-01-02 05:37:54 -07:00
tst_atts.c Add regression tests modifying attributes 2^16 times 2019-11-13 10:08:31 -09:00
tst_bloscfail.sh Fix some bugs in the blosc filter wrapper 2022-07-12 15:19:07 -06:00
tst_broken_files.c Merge branch 'main' into jsonconvention.dmh 2022-05-17 13:24:53 -06:00
tst_bug324.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_bug1442.c tst_bug1442: improve test to retrieve a mix of valid values and values at fillvalue 2019-07-18 03:05:20 +02:00
tst_camrun.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_charvlenbug.c Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
tst_chunks2.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_chunks.c Cleanup the handling of cache parameters. 2023-08-10 16:57:57 -06:00
tst_compounds2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_compounds3.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_compounds.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_converts2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_converts.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_coords2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_coords3.c now using hidden coordinates att to speed file opens 2018-12-20 05:59:31 -07:00
tst_coords.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_dims2.c Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_dims3.c Make public a limited API for programmatic access to internal .rc tables 2022-06-17 14:35:12 -06:00
tst_dims.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_elatefill.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_empty_vlen_unlim.c Separate out the --ansi comment fixes. 2018-12-12 13:23:09 -07:00
tst_endian_fill.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_enums.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files4.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files5.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files6.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files.c now dont return error on second redef call for netcdf/HDF5 files 2020-07-08 11:10:15 -06:00
tst_fill_attr_vanish.c Fix in support of https://github.com/Unidata/netcdf-c/issues/2027 2021-07-06 11:34:30 -06:00
tst_fillbug.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_fillonly.c Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
tst_fills2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_fills.c Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_filter_misc.sh Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_filter_vlen.c Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_filter_vlen.sh Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_filter.sh Fix race condition in ncdump (and other) tests. 2022-11-08 20:12:38 -07:00
tst_filterinstall.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
tst_filterparser.c Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
tst_fixedstring.sh Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
tst_grps2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_grps.c lazy read of some var metadata 2018-12-18 07:48:22 -07:00
tst_h5_endians.c Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_h_converts.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_h_refs.c lazy read of some var metadata 2018-12-18 07:48:22 -07:00
tst_h_scalar.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_h_strbug.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_h_strings3.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_h_transient_types.c Modify PR https://github.com/Unidata/netcdf-c/pull/2655 to ensure transient types have names. 2023-07-22 20:40:53 -06:00
tst_h_vl2.c Merge branch 'virtual_datasets' of https://github.com/d70-t/netcdf-c into gh1983.wif 2021-07-19 09:44:35 -07:00
tst_hdf5_file_compat.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_interops4.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_interops5.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_interops6.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_interops_dims.c another test for two anon dimensions of same size used in same HDF5 var 2019-11-14 07:23:09 -07:00
tst_interops.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_large2.c fixed warnings in tst_large2.c 2019-05-21 15:24:56 -06:00
tst_large3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_large5.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_large.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_misc.sh Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
tst_mode.c fixed tst_mode to not attempt to set filters on scalars 2020-03-03 07:44:37 -07:00
tst_mpi_parallel.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_multifilter.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_nc4perf.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_nc_converts.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_opaques.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_parallel3.c remove some debug statements 2019-02-01 14:39:30 -07:00
tst_parallel4.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_parallel5.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_parallel6.c undoing unintended change 2022-04-30 07:48:03 -06:00
tst_parallel_compress.c fixed problem setting szip on var with unlimited dim and added test 2020-07-02 10:55:34 -06:00
tst_parallel_zlib.c Added error message. 2022-05-02 09:55:08 -06:00
tst_parallel.c more work on tst_parallel5.c 2019-08-14 08:50:09 -06:00
tst_put_vars_two_unlim_dim.c Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_put_vars.c more config.h includes for tests 2019-01-02 05:37:54 -07:00
tst_quantize_par.c added parallel I/O quantize test 2021-09-02 10:21:44 -06:00
tst_quantize.c Update to latest main 2023-04-04 18:37:20 -06:00
tst_rehash.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_rename2.c fixed warnings in tst_rename2 2019-05-21 15:16:35 -06:00
tst_rename3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_rename.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_simplerw_coll_r.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_specific_filters.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
tst_strings2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_strings.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_sync.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_szip.sh Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
tst_types.c merge master and fix conflicts 2018-12-12 11:47:54 -07:00
tst_udf.c update against main 2023-01-28 13:45:35 -07:00
tst_unknown.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
tst_unlim_vars.c Mitigate S3 test interference + Unlimited Dimensions in NCZarr 2023-09-26 16:56:48 -06:00
tst_utf8.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_v2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_varms.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_vars2.c Disable varible rename test 2021-08-12 10:56:39 -06:00
tst_vars3.c Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
tst_vars4.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_vars.c more test development 2021-12-03 06:41:20 -07:00
tst_varsperf.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_virtual_datasets.c For loop initial declarations are only allowed in C99 mode 2022-09-26 13:04:10 -06:00
tst_vl.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_vlenstr.c Improve performance of the nc_reclaim_data and nc_copy_data functions. 2023-05-20 17:11:25 -06:00
tst_xplatform2.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_xplatform.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
unknown.cdl Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00