mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2024-12-21 08:39:46 +08:00
Code Issues Packages Projects Releases Wiki Activity
2f103420f6
netcdf-c/nc_test
History
Dennis Heimbigner 69e84fe9f1 Fix byterange handling of some URLS 
re: Issue

The byterange handling of the following URLS fails.

### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes"
It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD).
I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time.

### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3"
When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case.

## Misc. Related Changes
1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support
2. Added the above URLS as additional test cases

## Misc. Un-Related Changes
1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD
2. Fix some minor warnings

## Open Problems
* Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-02 19:51:02 -07:00
..
bad_cdf5_begin.nc
CMakeLists.txt Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
error.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
error.h More copyright stanza updates. 2018-12-06 14:56:42 -07:00
f03tst_open_mem.nc Fix use of 'int' to represent 'hid_t' that caused HDF5 1.10 to fail. 2018-05-30 14:47:37 -06:00
large_files.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
make_test_file.sh Swapped /bin/bash for /bin/sh to test on osx. 2018-08-12 23:01:08 -06:00
Makefile.am Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
nc_enddef.cdl Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
nc_sync.cdl
nc_test.c Turn verbose output on in nc_test. 2021-06-16 16:34:06 -06:00
nc_test.html
quick_large_files.c Spelling fixes 2019-09-18 08:03:01 -06:00
ref_tst_diskless2.cdl
ref_tst_diskless3_create.cdl Fix cmake X mmap 2019-04-19 20:32:26 -06:00
ref_tst_diskless3_open.cdl Fix cmake X mmap 2019-04-19 20:32:26 -06:00
ref_tst_http_nc3.cdl Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
ref_tst_http_nc4a.cdl Use the built-in HDF5 byte-range reader, if available. 2020-09-24 14:33:58 -06:00
ref_tst_http_nc4b.cdl Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
ref_tst_http_nc4c.cdl Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ref_tst_http_nc4e.cdl Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
ref_tst_http_nc4f.cdl Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
run_cdf5.sh
run_diskless2.sh Two fixes for the price of one! 2022-04-28 15:51:20 -06:00
run_diskless5.sh
run_diskless.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
run_inmemory.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
run_mmap.sh Fix cmake X mmap 2019-04-19 20:32:26 -06:00
run_pnetcdf_tests.sh.in Update to default --with-plugin-dir to yes 2022-05-24 20:05:19 -06:00
t_nc_p5.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
t_nc.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
test_byterange.sh Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
test_get.m4 Docs migration 2021-11-11 10:47:49 -07:00
test_put.m4 Fix an accidentally-introduced bug. 2021-07-13 12:07:56 -06:00
test_read.m4 Eliminate a lot of 'result of operation is undefined or garbage' errors reported by static analysis. These errors appeared to be false positives as there was logic that should prevent their occurance, but it's possible that optimization would result in these error states occuring. 2021-07-12 16:40:27 -06:00
test_write.m4 Eliminate a lot of 'result of operation is undefined or garbage' errors reported by static analysis. These errors appeared to be false positives as there was logic that should prevent their occurance, but it's possible that optimization would result in these error states occuring. 2021-07-12 16:40:27 -06:00
testnc3perf.c cleaned up makefile.am, added config.h to some tests 2019-01-02 05:31:15 -07:00
tests.h windows: detect Windows using the correct define name 2019-11-07 07:55:47 -05:00
tst_addvar.c cleaned up makefile.am, added config.h to some tests 2019-01-02 05:31:15 -07:00
tst_atts3.c fixed comment 2020-08-12 07:59:45 -06:00
tst_big_rvar.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var2.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var6.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_big_var.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_byterange.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_cdf5_begin.c cleaned up makefile.am, added config.h to some tests 2019-01-02 05:31:15 -07:00
tst_cdf5format.c Added int i and j declarations 2019-02-12 15:10:39 +00:00
tst_def_var_fill.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_default_format_pnetcdf.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_default_format.c Merge remote-tracking branch 'origin/license_update.wif' into pr-aggregation.wif 2018-12-11 17:08:21 -05:00
tst_diskless2.c Fix cmake X mmap 2019-04-19 20:32:26 -06:00
tst_diskless3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_diskless4.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
tst_diskless5.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_diskless5.cdl
tst_diskless6.c fixed warnings 2022-04-26 05:31:07 -06:00
tst_diskless.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_err_enddef.c Addendum: This PR has been extended to include 2021-04-02 18:39:50 -06:00
tst_formats.c Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
tst_formatx_pnetcdf.c warnings fixes 2019-01-03 07:07:13 -07:00
tst_global_fillval.c Merge remote-tracking branch 'origin/license_update.wif' into pr-aggregation.wif 2018-12-11 17:08:21 -05:00
tst_inmemory.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_inq_type.c merged ejh_test_null_vars_stride this time I mean it! 2019-01-03 06:36:58 -07:00
tst_large_cdf5.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_large.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_max_var_dims.c cleaned up makefile.am, added config.h to some tests 2019-01-02 05:31:15 -07:00
tst_meta.c cleaned up makefile.am, added config.h to some tests 2019-01-02 05:31:15 -07:00
tst_misc.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_names.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_nofill2.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
tst_nofill3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_nofill.c fixing some issuses of pnetcdf tests in nc_test 2022-04-09 13:52:29 -06:00
tst_norm.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
tst_open_cdf5.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
tst_open_mem.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_parallel2.c More copyright stanza updates. 2018-12-06 14:56:42 -07:00
tst_pnetcdf.c Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
tst_small.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_utf8_phrases.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_utf8_validate.c Testing a fix for a failure I observed. 2018-12-06 17:46:30 -07:00
util.c Explicit cast to unsigned char. 2023-02-24 13:37:28 -07:00