mirror of
https://github.com/HDFGroup/hdf5.git
synced 2024-11-27 02:10:55 +08:00
ad3ace3d16
Bug fix
Description:
Raw data I/O on chunked datasets would attempt to allocate data structures
proportional to the number of chunks in the dataset on disk, instead of just
the number of chunks that the I/O operation would interact with, causing
operations on datasets with large #'s of chunks to fail (or become very slow),
even though the actual I/O operation was very modest.
Solution:
This is the "scalability fix" for chunked datasets that I've mentioned
we need to do, althought it's not the complete fix for the issue. Read on
for the details...
Only create data structures for the chunks that the I/O operation will
actually act on, reducing the amount of information allocated in memory,
normally.
I say "normally", because this algorithm has the same problems as the
original algorithm (worse actually, since the data structure for each chunk
is larger now) if _all_ the chunks in a dataset with a lot of chunks are
actually involved in the I/O operation. If that is the case, this code
will fail in a similar way.
To truly fix the problem, we would need to only create data structures for
a particular number of chunks, perform the I/O on just those chunks, then
release the data structures for those chunks and create data structures for
the next set of chunks to access, etc. However, I think this case is pretty
rare right now and we should worry about it after the 1.6.0 release.
Platforms tested:
h5committested
|
||
|---|---|---|
| bin | ||
| c++ | ||
| config | ||
| doc | ||
| examples | ||
| fortran | ||
| hl/tools/gif2h5 | ||
| pablo | ||
| perform | ||
| release_docs | ||
| src | ||
| test | ||
| testpar | ||
| tools | ||
| windows | ||
| .autom4te.cfg | ||
| aclocal.m4 | ||
| configure | ||
| configure.in | ||
| COPYING | ||
| Makefile.dist | ||
| Makefile.in | ||
| MANIFEST | ||
| README.txt | ||
HDF5 version 1.5.54 currently under development
Please refer to the release_docs/INSTALL file for installation instructions.
------------------------------------------------------------------------------
This release is almost fully functional for the entire API defined in the
documentation, see the RELEASE.txt file in this directory for information
specific to this release of the library. The INSTALL file contains
instructions on compiling and installing the library. The INSTALL_parallel
file contains instructions on installing the parallel version of the
library. The INSTALL* files can be found in the release_docs/ directory.
Documentation for this release is in the html directory. Start with the
"index.html" in that directory.
Four mailing lists are currently set up for use with the HDF5
library.
hdf5 - For general discussion of the HDF5 library with
other users.
hdf5dev - For discussion of the HDF5 library development
with developers and other interested parties.
hdf5announce - For announcements of HDF5 related developments,
not a discussion list.
hdf5cvs - For checkin notices of code development on the library,
not a discussion list.
To subscribe to a list, send mail to "majordomo@ncsa.uiuc.edu",
with "subscribe <list>" in the _body_, not the Subject, of the message.
E.g., subscribe hdf5
Messages to be sent to the list should be sent to "<list>@ncsa.uiuc.edu".
Nearly daily code snapshots are now being provided at the following URL:
ftp://hdf.ncsa.uiuc.edu/pub/outgoing/hdf5/snapshots
Please read the readme file in that directory before working with a library
snapshot.
The HDF5 website is located at http://hdf.ncsa.uiuc.edu/HDF5/
Bugs should be reported to hdfhelp@ncsa.uiuc.edu.