mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-02-05 16:20:10 +08:00
Code Issues Packages Projects Releases Wiki Activity
fb40a72b45
netcdf-c/nczarr_test
History
Dennis Heimbigner fb40a72b45 Improve performance of the nc_reclaim_data and nc_copy_data functions. 
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179

As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.

However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.

This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.

First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.

The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.

The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.

## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-20 17:11:25 -06:00
..
refs
bm_chunks3.c Add support for setting HDF5 alignment property when creating a file 2022-01-29 15:27:52 -07:00
bm_timer.h Add missing files so --enable-benchmarks works again. 2021-08-07 21:19:40 -06:00
bm_utils.c Add missing files so --enable-benchmarks works again. 2021-08-07 21:19:40 -06:00
bm_utils.h Add support for setting HDF5 alignment property when creating a file 2022-01-29 15:27:52 -07:00
CMakeLists.txt ## Addendum [5/9/23] 2023-05-09 21:13:49 -06:00
Makefile.am ## Addendum [5/9/23] 2023-05-09 21:13:49 -06:00
ncdumpchunks.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ref_any.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_avail1.cdl More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_avail1.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_avail1.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_blosc_zmap.txt Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_byte_fill_value_null.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_byte_fill_value_null.zarr.zip Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_byte.cdl Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
ref_byte.zarr.zip Add test cases 2021-08-10 14:15:24 -06:00
ref_bzip2.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_fillonly.cdl
ref_filtered.cdl Add complete bitgroom support to NCZarr 2022-01-24 15:22:24 -07:00
ref_groups_regular.cdl Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
ref_groups.h5 Add test cases 2021-08-10 14:15:24 -06:00
ref_jsonconvention.cdl Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
ref_jsonconvention.zmap Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
ref_misc1.cdl
ref_misc1.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_misc2.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_multi.cdl Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
ref_nczarr2zarr.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_ndims.cdl
ref_ndims.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_newformatpure.cdl Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_notzarr.tar.gz Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
ref_nulls_nczarr.baseline Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_nulls_zarr.baseline Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_nulls.cdl Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_oldformat.cdl Move to Version 2 NCZarr Extended Meta-Data 2021-07-17 16:55:30 -06:00
ref_oldformat.zip Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_perdimspecs.cdl
ref_power_901_constants_orig.zip Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
ref_power_901_constants.cdl Fix JSON quoted string processing in libnczarr 2021-05-06 16:39:44 -06:00
ref_purezarr_base.cdl Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_purezarr.cdl Add complete bitgroom support to NCZarr 2022-01-24 15:22:24 -07:00
ref_quotes_orig.zip Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
ref_quotes.cdl Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
ref_rem.cdl
ref_rem.dmp More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_scalar.cdl Fix missing files 2022-09-03 14:57:48 -06:00
ref_skip.cdl
ref_skip.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_skipw.cdl
ref_string_nczarr.baseline Fix missing files 2022-09-03 14:57:48 -06:00
ref_string_zarr.baseline Fix some addtional errors in NCZarr 2022-09-09 14:25:24 -06:00
ref_string.cdl Fix some problems with PR https://github.com/Unidata/netcdf-c/pull/2492 2022-09-03 14:21:48 -06:00
ref_t_meta_dim1.cdl
ref_t_meta_var1.cdl
ref_ut_json_build.txt
ref_ut_json_parse.txt
ref_ut_map_create.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_map_readdata_nz4.txt
ref_ut_map_readmeta2.txt Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_readmeta_nz4.txt
ref_ut_map_readmeta_nzf.txt
ref_ut_map_readmeta.txt Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_map_search_nz4.txt
ref_ut_map_search_nzf.txt
ref_ut_map_search.txt Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_writedata.cdl Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_writemeta2.cdl Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute 2021-02-24 13:46:11 -07:00
ref_ut_map_writemeta.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_create.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_data.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_meta.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_mapapi_search_nz4.txt
ref_ut_mapapi_search_nzf.txt
ref_ut_mapapi_search_s3.txt
ref_ut_mapapi_search.txt Add zip file support 2021-01-28 20:11:01 -07:00
ref_ut_proj.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_ut_testmap_create.cdl Add zip file support 2021-01-28 20:11:01 -07:00
ref_whole.cdl More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_whole.txt More NCZarr optimizations 2021-01-06 13:35:59 -07:00
ref_xarray.cdl Add complete bitgroom support to NCZarr 2022-01-24 15:22:24 -07:00
ref_zarr_test_data.cdl.gz Add support for Zarr string type to NCZarr 2022-08-27 20:21:13 -06:00
run_chunkcases.sh ## Addendum [5/9/23] 2023-05-09 21:13:49 -06:00
run_external.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_fillonlyz.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_filter.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_interop.sh ## Addendum [5/9/23] 2023-05-09 21:13:49 -06:00
run_it_chunks1.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_jsonconvention.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_misc.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_nccopyz.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_ncgen4.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_nczarr_fill.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_nczfilter.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_newformat.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_notzarr.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_nulls.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_perf_chunks1.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_purezarr.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_quantize.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_s3_cleanup.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_scalar.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_strings.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_ut_chunk.sh
run_ut_map.sh Add zip file support 2021-01-28 20:11:01 -07:00
run_ut_mapapi.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
run_ut_misc.sh
s3util.c Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
test_nczarr_utils.h More tweaks. 2022-01-24 15:21:29 -07:00
test_nczarr.sh ## Addendum [5/9/23] 2023-05-09 21:13:49 -06:00
testfilter_misc.c Fix a few memory leaks and erroneous logic in nczarr_test/testfilter_misc 2022-01-09 14:41:35 -05:00
testfilter_multi.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
testfilter_order.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
testfilter_repeat.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
testfilter.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_chunkcases.c Add zip file support 2021-01-28 20:11:01 -07:00
tst_fillonlyz.c Add zip file support 2021-01-28 20:11:01 -07:00
tst_nczfilter.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_notzarr.c Check at nc_open if file appears to be in NCZarr/Zarr format. 2023-03-13 13:24:14 -06:00
tst_utils.c Merge branch 'master' into eliminate_need_for_hdf5-1.6-API 2021-05-28 07:11:13 -06:00
tst_utils.h Add zip file support 2021-01-28 20:11:01 -07:00
tst_zchunks2.c
tst_zchunks3.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
tst_zchunks.c Remove stray character. 2022-01-24 15:30:01 -07:00
ut_chunking.c Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_includes.h Fix bad chunkpath calculation 2021-07-18 16:20:22 -06:00
ut_json.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
ut_map.c Support MSYS2/Mingw platform 2021-12-23 22:18:56 -07:00
ut_mapapi.c Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
ut_projections.c Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_projtest.h
ut_test.c Check at nc_open if file appears to be in NCZarr/Zarr format. 2023-03-13 13:24:14 -06:00
ut_test.h Fix additional S3 support issues 2021-10-29 20:06:37 -06:00
ut_util.c Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
ut_util.h
z4dump.c
zhex.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
zisjson.c Allow the read/write of JSON-valued Zarr attributes. 2022-04-06 18:22:59 -06:00
zmapio.c Improve performance of the nc_reclaim_data and nc_copy_data functions. 2023-05-20 17:11:25 -06:00
zs3parse.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00