netcdf-c/nc_test
Dennis Heimbigner 69e84fe9f1 Fix byterange handling of some URLS
re: Issue

The byterange handling of the following URLS fails.

### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes"
It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD).
I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time.

### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3"
When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case.

## Misc. Related Changes
1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support
2. Added the above URLS as additional test cases

## Misc. Un-Related Changes
1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD
2. Fix some minor warnings

## Open Problems
* Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-02 19:51:02 -07:00
..
bad_cdf5_begin.nc
CMakeLists.txt Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
error.c
error.h
f03tst_open_mem.nc
large_files.c
make_test_file.sh
Makefile.am Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
nc_enddef.cdl Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
nc_sync.cdl
nc_test.c Turn verbose output on in nc_test. 2021-06-16 16:34:06 -06:00
nc_test.html
quick_large_files.c Spelling fixes 2019-09-18 08:03:01 -06:00
ref_tst_diskless2.cdl
ref_tst_diskless3_create.cdl
ref_tst_diskless3_open.cdl
ref_tst_http_nc3.cdl Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
ref_tst_http_nc4a.cdl Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
ref_tst_http_nc4b.cdl Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
ref_tst_http_nc4c.cdl Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ref_tst_http_nc4e.cdl Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
ref_tst_http_nc4f.cdl Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
run_cdf5.sh
run_diskless2.sh Two fixes for the price of one! 2022-04-28 15:51:20 -06:00
run_diskless5.sh
run_diskless.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
run_inmemory.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
run_mmap.sh
run_pnetcdf_tests.sh.in Update to default --with-plugin-dir to yes 2022-05-24 20:05:19 -06:00
t_nc_p5.c
t_nc.c
test_byterange.sh Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
test_get.m4 Docs migration 2021-11-11 10:47:49 -07:00
test_put.m4 Fix an accidentally-introduced bug. 2021-07-13 12:07:56 -06:00
test_read.m4 Eliminate a lot of 'result of operation is undefined or garbage' errors reported by static analysis. These errors appeared to be false positives as there was logic that should prevent their occurance, but it's possible that optimization would result in these error states occuring. 2021-07-12 16:40:27 -06:00
test_write.m4 Eliminate a lot of 'result of operation is undefined or garbage' errors reported by static analysis. These errors appeared to be false positives as there was logic that should prevent their occurance, but it's possible that optimization would result in these error states occuring. 2021-07-12 16:40:27 -06:00
testnc3perf.c
tests.h windows: detect Windows using the correct define name 2019-11-07 07:55:47 -05:00
tst_addvar.c
tst_atts3.c fixed comment 2020-08-12 07:59:45 -06:00
tst_big_rvar.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var2.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var6.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_byterange.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_cdf5_begin.c
tst_cdf5format.c
tst_def_var_fill.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_default_format_pnetcdf.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_default_format.c
tst_diskless2.c
tst_diskless3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_diskless4.c
tst_diskless5.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_diskless5.cdl
tst_diskless6.c fixed warnings 2022-04-26 05:31:07 -06:00
tst_diskless.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_err_enddef.c Addendum: This PR has been extended to include 2021-04-02 18:39:50 -06:00
tst_formats.c Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
tst_formatx_pnetcdf.c
tst_global_fillval.c
tst_inmemory.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_inq_type.c
tst_large_cdf5.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_large.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_max_var_dims.c
tst_meta.c
tst_misc.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_names.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_nofill2.c
tst_nofill3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_nofill.c fixing some issuses of pnetcdf tests in nc_test 2022-04-09 13:52:29 -06:00
tst_norm.c
tst_open_cdf5.c
tst_open_mem.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_parallel2.c
tst_pnetcdf.c Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
tst_small.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_utf8_phrases.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_utf8_validate.c
util.c Explicit cast to unsigned char. 2023-02-24 13:37:28 -07:00