mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-01-30 16:10:44 +08:00
Code Issues Packages Projects Releases Wiki Activity
ce97a67bae
netcdf-c/nc_test4
History
Dennis Heimbigner fb40a72b45 Improve performance of the nc_reclaim_data and nc_copy_data functions. 
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179

As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.

However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.

This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.

First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.

The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.

The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.

## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-20 17:11:25 -06:00
..
bm_chunking.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
bm_radar.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
build_fixedstring.c Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
bzip2.cdl
cdm_sea_soundings.c Spelling fixes 2019-09-18 08:03:01 -06:00
clear_cache.sh
CMakeLists.txt Fix DAP4 remotetest server 2022-11-15 20:29:21 -07:00
findplugin.in TST: Add code for MinGW to findplugin.sh. 2022-10-12 12:55:37 -04:00
h5testszip.c Fix for use with szip and also 1.8.21 2021-04-28 15:42:02 -06:00
Makefile.am Fix DAP4 remotetest server 2022-11-15 20:29:21 -07:00
noop1.cdl Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00
noop.cdl Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
ref_any.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_bloscx.cdl Fix some bugs in the blosc filter wrapper 2022-07-12 15:19:07 -06:00
ref_bzip2.c This PR adds EXPERIMENTAL support for accessing data in the 2020-06-28 18:02:47 -06:00
ref_fillonly.cdl Fix error where not converting fill data 2020-09-12 14:49:59 -06:00
ref_filter_order_create.txt Add tests for filter order on read and write cases 2020-12-29 20:12:35 -07:00
ref_filter_order_read.txt Add tests for filter order on read and write cases 2020-12-29 20:12:35 -07:00
ref_filter_repeat.txt Add tests for filter order on read and write cases 2020-12-29 20:12:35 -07:00
ref_filtered.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_filteredvv.cdl Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_filterx_hdf5.txt Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
ref_fixedstring.cdl Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
ref_fixedstring.h5 Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
ref_hdf5_compat1.nc
ref_hdf5_compat2.nc
ref_hdf5_compat3.nc
ref_multi.cdl Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
ref_nccopyF.cdl Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
ref_ncgenF.cdl Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
ref_szip.cdl
ref_szip.h5
ref_tst_compounds.nc
ref_tst_dims.nc
ref_tst_interops4.nc
ref_tst_xplatform2_1.nc
ref_tst_xplatform2_2.nc
ref_unfiltered.cdl Renamed the baseline files for tst_filter.sh 2019-02-09 15:14:13 -07:00
ref_unfilteredvv.cdl Renamed the baseline files for tst_filter.sh 2019-02-09 15:14:13 -07:00
renamegroup.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
run_empty_vlen_test.sh
run_grp_rename.sh Provide byte-range reading of remote datasets 2019-01-01 18:27:36 -07:00
run_par_test.sh.in no longer run tst_parallel_zlib with 1 processor in testing 2022-04-30 07:42:01 -06:00
t_type.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tdset.h5
test_fillonly.sh Fix error where not converting fill data 2020-09-12 14:49:59 -06:00
test_filter_misc.c Explicitly disallow variable length type compression 2022-02-19 16:47:31 -07:00
test_filter_order.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filter_reg.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filter_repeat.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filter_vlen.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
test_filter.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_filterx_hdf5.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
test_szip.c Add support for multiple filters per variable. 2020-02-16 12:59:33 -07:00
test_wrapper.in
tst_alignment.c Add support for setting HDF5 alignment property when creating a file 2022-01-29 15:27:52 -07:00
tst_atts1.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_atts2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_atts3.c merge master and fix conflicts 2018-12-12 11:47:54 -07:00
tst_atts_string_rewrite.c more config.h includes for tests 2019-01-02 05:37:54 -07:00
tst_atts.c Add regression tests modifying attributes 2^16 times 2019-11-13 10:08:31 -09:00
tst_bloscfail.sh Fix some bugs in the blosc filter wrapper 2022-07-12 15:19:07 -06:00
tst_broken_files.c Merge branch 'main' into jsonconvention.dmh 2022-05-17 13:24:53 -06:00
tst_bug324.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_bug1442.c tst_bug1442: improve test to retrieve a mix of valid values and values at fillvalue 2019-07-18 03:05:20 +02:00
tst_camrun.c Improve UTF8 Support On Windows 2022-02-08 20:53:30 -07:00
tst_charvlenbug.c Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
tst_chunks2.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_chunks.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_compounds2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_compounds3.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_compounds.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_converts2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_converts.c Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"" 2020-08-17 19:15:47 -06:00
tst_coords2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_coords3.c now using hidden coordinates att to speed file opens 2018-12-20 05:59:31 -07:00
tst_coords.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_dims2.c Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_dims3.c Make public a limited API for programmatic access to internal .rc tables 2022-06-17 14:35:12 -06:00
tst_dims.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_elatefill.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_empty_vlen_unlim.c Separate out the --ansi comment fixes. 2018-12-12 13:23:09 -07:00
tst_endian_fill.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_enums.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files4.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files5.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files6.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_files.c now dont return error on second redef call for netcdf/HDF5 files 2020-07-08 11:10:15 -06:00
tst_fill_attr_vanish.c Fix in support of https://github.com/Unidata/netcdf-c/issues/2027 2021-07-06 11:34:30 -06:00
tst_fillbug.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_fillonly.c Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
tst_fills2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_fills.c Fix undefined references when using Visual Studio 2020-05-18 19:36:28 -06:00
tst_filter_avail.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_filter.sh Fix race condition in ncdump (and other) tests. 2022-11-08 20:12:38 -07:00
tst_filterinstall.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
tst_filterparser.c Mostly revert the filter code to reduce its complexity of use. 2020-09-27 12:43:46 -06:00
tst_fixedstring.sh Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
tst_grps2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_grps.c lazy read of some var metadata 2018-12-18 07:48:22 -07:00
tst_h5_endians.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_h_converts.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_h_refs.c lazy read of some var metadata 2018-12-18 07:48:22 -07:00
tst_h_scalar.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_h_strbug.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_h_strings3.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_h_vl2.c Merge branch 'virtual_datasets' of https://github.com/d70-t/netcdf-c into gh1983.wif 2021-07-19 09:44:35 -07:00
tst_hdf5_file_compat.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_interops4.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_interops5.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_interops6.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_interops_dims.c another test for two anon dimensions of same size used in same HDF5 var 2019-11-14 07:23:09 -07:00
tst_interops.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_large2.c fixed warnings in tst_large2.c 2019-05-21 15:24:56 -06:00
tst_large3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_large5.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_large.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_misc.sh Fix various problem around VLEN's 2022-01-08 18:30:00 -07:00
tst_mode.c fixed tst_mode to not attempt to set filters on scalars 2020-03-03 07:44:37 -07:00
tst_mpi_parallel.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_multifilter.c Add filter support to NCZarr 2021-09-02 17:04:26 -06:00
tst_nc4perf.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_nc_converts.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_opaques.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_parallel3.c remove some debug statements 2019-02-01 14:39:30 -07:00
tst_parallel4.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_parallel5.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_parallel6.c undoing unintended change 2022-04-30 07:48:03 -06:00
tst_parallel_compress.c fixed problem setting szip on var with unlimited dim and added test 2020-07-02 10:55:34 -06:00
tst_parallel_zlib.c Added error message. 2022-05-02 09:55:08 -06:00
tst_parallel.c more work on tst_parallel5.c 2019-08-14 08:50:09 -06:00
tst_put_vars_two_unlim_dim.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_put_vars.c more config.h includes for tests 2019-01-02 05:37:54 -07:00
tst_quantize_par.c added parallel I/O quantize test 2021-09-02 10:21:44 -06:00
tst_quantize.c Update to latest main 2023-04-04 18:37:20 -06:00
tst_rehash.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_rename2.c fixed warnings in tst_rename2 2019-05-21 15:16:35 -06:00
tst_rename3.c Spelling fixes 2019-09-18 08:03:01 -06:00
tst_rename.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_simplerw_coll_r.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_specific_filters.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
tst_strings2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_strings.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_sync.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_szip.sh Regularize the scoping of dimensions 2021-05-31 15:51:12 -06:00
tst_types.c merge master and fix conflicts 2018-12-12 11:47:54 -07:00
tst_udf.c update against main 2023-01-28 13:45:35 -07:00
tst_unknown.sh Improve S3 Documentation and Support 2023-04-25 17:15:06 -06:00
tst_unlim_vars.c more checking in 3D test 2020-02-12 08:21:52 -07:00
tst_utf8.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_v2.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_varms.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_vars2.c Disable varible rename test 2021-08-12 10:56:39 -06:00
tst_vars3.c Fix support for reading arrays of HDF5 fixed size strings 2022-07-29 14:47:07 -06:00
tst_vars4.c Enhance/Fix filter support 2022-03-14 12:39:37 -06:00
tst_vars.c more test development 2021-12-03 06:41:20 -07:00
tst_varsperf.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_virtual_datasets.c For loop initial declarations are only allowed in C99 mode 2022-09-26 13:04:10 -06:00
tst_vl.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
tst_vlenstr.c Improve performance of the nc_reclaim_data and nc_copy_data functions. 2023-05-20 17:11:25 -06:00
tst_xplatform2.c Remove need for HDF5-1.6 API being defined 2021-04-28 13:59:24 -06:00
tst_xplatform.c Whew! Updated copyright stanza in nc_test4. 2018-12-06 15:27:32 -07:00
unknown.cdl Improve filter installation process to avoid use of an extra shell script 2022-05-14 16:05:48 -06:00