mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-01-24 16:04:40 +08:00
Code Issues Packages Projects Releases Wiki Activity
43abd699e1
netcdf-c/nczarr_test
History
Dennis Heimbigner 69e84fe9f1 Fix byterange handling of some URLS 
re: Issue

The byterange handling of the following URLS fails.

### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes"
It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD).
I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time.

### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3"
When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case.

## Misc. Related Changes
1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support
2. Added the above URLS as additional test cases

## Misc. Un-Related Changes
1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD
2. Fix some minor warnings

## Open Problems
* Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-02 19:51:02 -07:00
..
refs This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
bm_chunks3.c Add support for setting HDF5 alignment property when creating a file 2022-01-29 15:27:52 -07:00
bm_timer.h Add missing files so --enable-benchmarks works again. 2021-08-07 21:19:40 -06:00
bm_utils.c Add missing files so --enable-benchmarks works again. 2021-08-07 21:19:40 -06:00
bm_utils.h Add support for setting HDF5 alignment property when creating a file 2022-01-29 15:27:52 -07:00
CMakeLists.txt Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
Makefile.am Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
ncdumpchunks.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ref_any.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_avail1.cdl More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_avail1.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_avail1.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_blosc_zmap.txt Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_byte_fill_value_null.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_byte_fill_value_null.zarr.zip Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_byte.cdl Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
ref_byte.zarr.zip Add test cases 2021-08-10 14:15:24 -06:00
ref_bzip2.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_fillonly.cdl Fix error where not converting fill data 2020-09-12 14:49:59 -06:00
ref_filtered.cdl Add complete bitgroom support to NCZarr 2022-01-24 15:22:24 -07:00
ref_groups_regular.cdl Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
ref_groups.h5 Add test cases 2021-08-10 14:15:24 -06:00
ref_jsonconvention.cdl Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
ref_jsonconvention.zmap Manually bump version in diff-compare to get RC1 out the door, this will need to be automatically excluded from the test at some point otherwise we will see this test fail every time the VERSION string changes. 2022-10-17 14:20:34 -06:00
ref_misc1.cdl Fix some additional edges cases for mapping slices to chunks 2020-12-19 21:17:46 -07:00
ref_misc1.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_misc2.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_multi.cdl Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
ref_nczarr2zarr.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_ndims.cdl Fix some additional edges cases for mapping slices to chunks 2020-12-19 21:17:46 -07:00
ref_ndims.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_newformatpure.cdl Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_nulls_nczarr.baseline Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_nulls_zarr.baseline Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_nulls.cdl Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_oldformat.cdl Move to Version 2 NCZarr Extended Meta-Data 2021-07-17 16:55:30 -06:00
ref_oldformat.zip Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_perdimspecs.cdl Provide a Number of fixes/improvements to NCZarr 2020-11-19 17:01:04 -07:00
ref_power_901_constants_orig.zip Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
ref_power_901_constants.cdl Fix JSON quoted string processing in libnczarr 2021-05-06 16:39:44 -06:00
ref_purezarr_base.cdl Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_purezarr.cdl Add complete bitgroom support to NCZarr 2022-01-24 15:22:24 -07:00
ref_quotes_orig.zip Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
ref_quotes.cdl Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
ref_rem.cdl Fix some additional edges cases for mapping slices to chunks 2020-12-19 21:17:46 -07:00
ref_rem.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_scalar.cdl Fix missing files 2022-09-03 14:57:48 -06:00
ref_skip.cdl Additional Fixes to NCZarr 2020-12-16 20:48:02 -07:00
ref_skip.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_skipw.cdl Fix some additional edges cases for mapping slices to chunks 2020-12-19 21:17:46 -07:00
ref_string_nczarr.baseline Fix missing files 2022-09-03 14:57:48 -06:00
ref_string_zarr.baseline Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_string.cdl Fix some problems with PR https://github.com/Unidata/netcdf-c/pull/2492 2022-09-03 14:21:48 -06:00
ref_t_meta_dim1.cdl This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_t_meta_var1.cdl This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ut_json_build.txt This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ut_json_parse.txt This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ut_map_create.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_map_readdata_nz4.txt This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ut_map_readmeta2.txt Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_readmeta_nz4.txt This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ut_map_readmeta_nzf.txt This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ut_map_readmeta.txt Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_map_search_nz4.txt Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
ref_ut_map_search_nzf.txt Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
ref_ut_map_search.txt Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_writedata.cdl Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_writemeta2.cdl Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_writemeta.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_create.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_data.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_meta.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_search_nz4.txt Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
ref_ut_mapapi_search_nzf.txt Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
ref_ut_mapapi_search_s3.txt This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_ut_mapapi_search.txt Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_proj.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_ut_testmap_create.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_whole.cdl More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_whole.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_xarray.cdl Add complete bitgroom support to NCZarr 2022-01-24 15:22:24 -07:00
ref_zarr_test_data.cdl.gz Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
run_chunkcases.sh Upgrade the nczarr code to match Zarr V2 2021-04-24 19:48:15 -06:00
run_fillonlyz.sh Add zip file support 2021-01-28 20:11:01 -07:00
run_filter.sh merged 2022-05-24 19:59:36 -06:00
run_interop.sh Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
run_it_chunks1.sh Add zip file support 2021-01-28 20:11:01 -07:00
run_jsonconvention.sh Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
run_misc.sh Add test cases 2021-08-10 14:15:24 -06:00
run_nccopyz.sh Fix race condition in ncdump (and other) tests. 2022-11-08 20:12:38 -07:00
run_ncgen4.sh Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
run_nczarr_fill.sh Fix race condition in ncdump (and other) tests. 2022-11-08 20:12:38 -07:00
run_nczfilter.sh Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
run_newformat.sh Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
run_nulls.sh Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
run_perf_chunks1.sh Add zip file support 2021-01-28 20:11:01 -07:00
run_purezarr.sh Turn on the xarray convention for NCZarr format 2022-03-23 19:28:06 -06:00
run_quantize.sh Fix some problems with PR https://github.com/Unidata/netcdf-c/pull/2492 2022-09-03 14:21:48 -06:00
run_s3_cleanup.sh Add zip file support 2021-01-28 20:11:01 -07:00
run_scalar.sh Fix byterange handling of some URLS 2023-03-02 19:51:02 -07:00
run_strings.sh update file permission 2022-09-03 14:21:36 -06:00
run_ut_chunk.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
run_ut_map.sh Add zip file support 2021-01-28 20:11:01 -07:00
run_ut_mapapi.sh Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
run_ut_misc.sh Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
s3util.c Fix a distcheck failure with nczarr_test/run_interop.sh 2023-02-17 13:01:11 -07:00
test_nczarr_utils.h More tweaks. 2022-01-24 15:21:29 -07:00
test_nczarr.sh Support installation of filters into user-specified location 2022-04-29 14:31:55 -06:00
testfilter_misc.c Fix a few memory leaks and erroneous logic in nczarr_test/testfilter_misc 2022-01-09 14:41:35 -05:00
testfilter_multi.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
testfilter_order.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
testfilter_repeat.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
testfilter.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_chunkcases.c Add zip file support 2021-01-28 20:11:01 -07:00
tst_fillonlyz.c Add zip file support 2021-01-28 20:11:01 -07:00
tst_nczfilter.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_utils.c Merge branch 'master' into eliminate_need_for_hdf5-1.6-API 2021-05-28 07:11:13 -06:00
tst_utils.h Add zip file support 2021-01-28 20:11:01 -07:00
tst_zchunks2.c Additional Fixes to NCZarr 2020-12-16 20:48:02 -07:00
tst_zchunks3.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
tst_zchunks.c Remove stray character. 2022-01-24 15:30:01 -07:00
ut_chunking.c Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_includes.h Fix bad chunkpath calculation 2021-07-18 16:20:22 -06:00
ut_json.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
ut_map.c Support MSYS2/Mingw platform 2021-12-23 22:18:56 -07:00
ut_mapapi.c Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_projections.c Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_projtest.h This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ut_test.c Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_test.h Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_util.c Add zip file support 2021-01-28 20:11:01 -07:00
ut_util.h Fix some additional edges cases for mapping slices to chunks 2020-12-19 21:17:46 -07:00
z4dump.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
zhex.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
zisjson.c Allow the read/write of JSON-valued Zarr attributes. 2022-04-06 18:22:59 -06:00
zmapio.c Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
zs3parse.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00