netcdf-c/nc_test
Dennis Heimbigner 49737888ca Improve S3 Documentation and Support
## Improvements to S3 Documentation
* Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths.
* Modify *nczarr.md* to remove most of the S3 related text.
* Move the S3 text from *nczarr.md* to a new document *cloud.md*.
* Add some S3-related text to the *byterange.md* document.

Hopefully, this will make it easier for users to find the information they want.

## Rebuild NCZarr Testing
In order to avoid problems with running make check in parallel, two changes were made:
1. The *nczarr_test* test system was rebuilt. Now, for each test.
any generated files are kept in a test-specific directory, isolated
from all other test executions.
2. Similarly, since the S3 test bucket is shared, any generated S3 objects
are isolated using a test-specific key path.

## Other S3 Related Changes
* Add code to ensure that files created on S3 are reclaimed at end of testing.
* Used the bash "trap" command to ensure S3 cleanup even if the test fails.
* Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former.
* Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket.
* Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in.
* Merge partial S3 support into dhttp.c.
* Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it.
* Move some definitions from ncrc.h to ncs3sdk.h

## Other Changes
* Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-25 17:15:06 -06:00
..
bad_cdf5_begin.nc
CMakeLists.txt Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
error.c
error.h
f03tst_open_mem.nc
large_files.c
make_test_file.sh
Makefile.am Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
nc_enddef.cdl Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
nc_sync.cdl
nc_test.c Turn verbose output on in nc_test. 2021-06-16 16:34:06 -06:00
nc_test.html
quick_large_files.c Spelling fixes 2019-09-18 08:03:01 -06:00
ref_tst_diskless2.cdl
ref_tst_diskless3_create.cdl
ref_tst_diskless3_open.cdl
ref_tst_http_nc3.cdl Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
ref_tst_http_nc4a.cdl Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
ref_tst_http_nc4b.cdl Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
ref_tst_http_nc4c.cdl Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ref_tst_http_nc4e.cdl Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
ref_tst_http_nc4f.cdl Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
run_cdf5.sh
run_diskless2.sh Two fixes for the price of one! 2022-04-28 15:51:20 -06:00
run_diskless5.sh
run_diskless.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
run_inmemory.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
run_mmap.sh
run_pnetcdf_tests.sh.in Update to default --with-plugin-dir to yes 2022-05-24 20:05:19 -06:00
t_nc_p5.c
t_nc.c
test_byterange.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
test_get.m4 Docs migration 2021-11-11 10:47:49 -07:00
test_put.m4 Fix an accidentally-introduced bug. 2021-07-13 12:07:56 -06:00
test_read.m4 Eliminate a lot of 'result of operation is undefined or garbage' errors reported by static analysis. These errors appeared to be false positives as there was logic that should prevent their occurance, but it's possible that optimization would result in these error states occuring. 2021-07-12 16:40:27 -06:00
test_write.m4 Eliminate a lot of 'result of operation is undefined or garbage' errors reported by static analysis. These errors appeared to be false positives as there was logic that should prevent their occurance, but it's possible that optimization would result in these error states occuring. 2021-07-12 16:40:27 -06:00
testnc3perf.c
tests.h windows: detect Windows using the correct define name 2019-11-07 07:55:47 -05:00
tst_addvar.c
tst_atts3.c fixed comment 2020-08-12 07:59:45 -06:00
tst_big_rvar.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var2.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var6.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_byterange.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_cdf5_begin.c
tst_cdf5format.c
tst_def_var_fill.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_default_format_pnetcdf.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_default_format.c
tst_diskless2.c
tst_diskless3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_diskless4.c
tst_diskless5.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_diskless5.cdl
tst_diskless6.c fixed warnings 2022-04-26 05:31:07 -06:00
tst_diskless.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_err_enddef.c Addendum: This PR has been extended to include 2021-04-02 18:39:50 -06:00
tst_formats.c Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
tst_formatx_pnetcdf.c
tst_global_fillval.c
tst_inmemory.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_inq_type.c
tst_large_cdf5.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_large.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_max_var_dims.c
tst_meta.c
tst_misc.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_names.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_nofill2.c
tst_nofill3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_nofill.c fixing some issuses of pnetcdf tests in nc_test 2022-04-09 13:52:29 -06:00
tst_norm.c
tst_open_cdf5.c
tst_open_mem.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_parallel2.c
tst_pnetcdf.c Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
tst_small.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_utf8_phrases.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_utf8_validate.c
util.c Explicit cast to unsigned char. 2023-02-24 13:37:28 -07:00