mirror of
https://github.com/Unidata/netcdf-c.git
synced 2025-01-24 16:04:40 +08:00
1db3d07beb
The current library seems to have some behavior which is N^2 in the number of vars in a file. The `NC4_inq_dim` routine calls down to `nc4_find_dim_len` which iterates through each `var` in the file/group and calls `find_var_dim_max_length` on each var and finds the largest length of the dim on each of those vars. This is done only for unlimited vars. I have a file with 129 dim and 1630 vars. The unlimited dimension is of length 41. In my test program, I am reading data from 4 files which have the same dim and var count and reading every 4th time step (unlimited dimension). If I run a profile, I see that 98.2% of the program time is in the `nc_get_vara_float` call tree and most of that is in `find_var_dim_max_length` (94.8%). There are 66,142 calls to `nc_get_vara_float` resulting in 107,307,290 calls to `find_var_dim_max_length` with twice that number of calls to `malloc/free` and calls to 5 HDF5 routines. All of this, at least in my case, to return the same `41` each time. The proof of concept patch here will check whether the file is read-only (or no_write) and if so, it will cache the value of the dim length the first time it is calculated. With this change, my example run is sped up by a factor of 60. The time for `NC4_inq_dim` and below drops from 97.2% down to 2.7%. I'm not sure whether this is the correct fix, or if there is some behavior that I am overlooking, but my users would definitely like a 10 second run compared to a 10 minute run... This is on current Netcdf master branch. I will try to attach some valgrind/callgrind profiles. |
||
|---|---|---|
| .. | ||
| CMakeLists.txt | ||
| H5FDhttp.c | ||
| H5FDhttp.h | ||
| hdf5attr.c | ||
| hdf5cache.c | ||
| hdf5create.c | ||
| hdf5debug.c | ||
| hdf5debug.h | ||
| hdf5dim.c | ||
| hdf5dispatch.c | ||
| hdf5file.c | ||
| hdf5filter.c | ||
| hdf5grp.c | ||
| hdf5internal.c | ||
| hdf5open.c | ||
| hdf5type.c | ||
| hdf5var.c | ||
| Makefile.am | ||
| nc4hdf.c | ||
| nc4info.c | ||
| nc4mem.c | ||
| nc4memcb.c | ||