mirror of
https://github.com/HDFGroup/hdf5.git
synced 2025-02-17 16:10:24 +08:00
debeaf6e64
Purpose:
A new feature
Description:
While testing h4toh5 utility with real NASA files, we find an example that the data array(one SDS) is so big that it exceeds the physical memory of some machine(>128 MB) and the conversion failed. Before the smart hyperslab operation is out, I am dividing the whole SDS into smaller hyperslabs with each hyperslab propotational to the original SDS array dimensions. For example, a three dimension array with 1000*1000*1000 elements, I can divide them into eight 500*500*500 pieces. I can read and write each piece and remember their starting and ending points. In this way, the memory allocation failure can be avoided; however, it may not be the efficient way.
I've tested this feature using SDS without chunking. It works fine. However, when testing SDS with chunking, it is extremely slow. This happens to be a bug in HDF5 library now. Quincey may fix this later and give me a more efficient way to handle the problem. Currently all my testing files are with UNLIMITED dimensions, so in HDF5 the chunking feature will be required.
SO by default, this feature will not be turned on.
Solution:
see the above
Platforms tested:
linux 2.2.18
|
||
|---|---|---|
| bin | ||
| c++ | ||
| config | ||
| doc | ||
| examples | ||
| fortran | ||
| hl/tools/gif2h5 | ||
| pablo | ||
| perform | ||
| release_docs | ||
| src | ||
| test | ||
| testpar | ||
| tools | ||
| windows | ||
| acconfig.h | ||
| aclocal.m4 | ||
| configure | ||
| configure.in | ||
| COPYING | ||
| Makefile.dist | ||
| Makefile.in | ||
| MANIFEST | ||
| README.txt | ||
HDF5 version 1.5.16 currently under development
Please refer to the INSTALL file for installation instructions.
------------------------------------------------------------------------------
This release is almost fully functional for the entire API defined in the
documentation, see the RELEASE.txt file in this directory for information
specific to this release of the library. The INSTALL file contains
instructions on compiling and installing the library. The INSTALL_parallel
file contains instructions on installing the parallel version of the
library.
Documentation for this release is in the html directory. Start with the
"index.html" in that dictectory.
Four mailing lists are currently set up for use with the HDF5
library.
hdf5 - For general discussion of the HDF5 library with
other users.
hdf5dev - For discussion of the HDF5 library development
with developers and other interested parties.
hdf5announce - For announcements of HDF5 related developments,
not a discussion list.
hdf5cvs - For checkin notices of code development on the library,
not a discussion list.
To subscribe to a list, send mail to "majordomo@ncsa.uiuc.edu",
with "subscribe <list>" in the _body_, not the Subject, of the message.
E.g., subscribe hdf5
Messages to be sent to the list should be sent to "<list>@ncsa.uiuc.edu".
Nearly daily code snapshots are now being provided at the following URL:
ftp://hdf.ncsa.uiuc.edu/pub/outgoing/hdf5/snapshots
Please read the readme file in that directory before working with a library
snapshot.
The HDF5 website is located at http://hdf.ncsa.uiuc.edu/HDF5/
Bugs should be reported to hdfhelp@ncsa.uiuc.edu.