mirror/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2025-04-24 17:51:25 +08:00
Code Issues Packages Projects Releases Wiki Activity

[svn-r467] Restructuring documentation.

Browse Source
This commit is contained in:
Quincey Koziol 1998-07-08 09:54:54 -05:00
parent 7334509589
commit bd1e676c52
71 changed files with 20819 additions and 0 deletions

177
doc/html/Attributes.html Normal file

@ -0,0 +1,177 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Attributes</title>
</head>
<body>
<h1>Attributes</h1>
<h2>1. Introduction</h2>
<p>The appribute API (H5A) is primarily designed to easily allow small
datasets to be attached to primary datasets as metadata information.
Additional goals for the H5A interface include keeping storage
requirement for each attribute to a minimum and easily sharing
attributes among datasets.
<p>Because attributes are intended to be small objects, large datasets
intended as additional information for a primary dataset should be
stored as supplemental datasets in a group with the primary dataset.
Attributes can then be attached to the group containing everything to
indicate a particular type of dataset with supplemental datasets is
located in the group. How small is "small" is not defined by the
library and is up to the user's interpretation.
<p>Attributes are not seperate objects in the file, they are always
contained in the object header of the object they are attached to. The
I/O functions defined below are required to read or write attribute
information, not the H5D I/O routines.
<h2>2. Creating, Opening, Closing and Deleting Attributes</h2>
<p>Attributes are created with the <code>H5Acreate()</code> function,
and existing attributes can be accessed with either the
<code>H5Aopen_name()</code> or <code>H5Aopen_idx()</code> functions. All
three functions return an object ID which should be eventually released
by calling <code>H5Aclose()</code>.
<dl>
<dt><code>hid_t H5Acreate (hid_t <em>loc_id</em>, const char
*<em>name</em>, hid_t <em>type_id</em>, hid_t <em>space_id</em>,
hid_t <em>create_plist_id</em>)</code>
<dd>This function creates an attribute which is attached to the object
specified with <em>loc_id</em>. The name specified with <em>name</em>
for each attribute for an object must be unique for that object. The <em>type_id</em>
and <em>space_id</em> are created with the H5T and H5S interfaces
respectively. Currently only simple dataspaces are allowed for attribute
dataspaces. The <em>create_plist_id</em> property list is currently
unused, but will be used int the future for optional properties of
attributes. The attribute ID returned from this function must be released
with H5Aclose or resource leaks will develop. Attempting to create an
attribute with the same name as an already existing attribute will fail,
leaving the pre-existing attribute in place.
This function returns a attribute ID for success or negative for failure.
<br><br>
<dt><code>hid_t H5Aopen_name (hid_t <em>loc_id</em>, const char
*<em>name</em>)</code>
<dd> This function opens an attribute which is attached to the object
specified with <em>loc_id</em>. The name specified with <em>name</em>
indicates the attribute to access. The attribute ID returned from this
function must be released with H5Aclose or resource leaks will develop.
This function returns a attribute ID for success or negative for failure.
<br><br>
<dt><code>hid_t H5Aopen_idx (hid_t <em>loc_id</em>, unsigned
<em>idx</em>)</code>
<dd>This function opens an attribute which is attached to the object
specified with <em>loc_id</em>. The attribute specified with <em>idx</em>
indicates the <em>idx</em>th attribute to access, starting with '0'. The
attribute ID returned from this function must be released with H5Aclose or
resource leaks will develop.
This function returns a attribute ID for success or negative for failure.
<br><br>
<dt><code>herr_t H5Aclose (hid_t <em>attr_id</em>)</code>
<dd>This function releases an attribute from use. Further use of the
attribute ID will result in undefined behavior.
This function returns non-negative on success, negative on failure.
<br><br>
<dt><code>herr_t H5Adelete (hid_t <em>loc_id</em>,
const char *<em>name</em>)</code>
<dd>This function removes the named attribute from a dataset or group.
This function should not be used when attribute IDs are open on <em>loc_id</em>
as it may cause the internal indexes of the attributes to change and future
writes to the open attributes to produce incorrect results.
Returns non-negative on success, negative on failure.
</dl>
<h2>3. Attribute I/O Functions</h2>
<p>Attributes may only be written as an entire object, no partial I/O
is currently supported.
<dl>
<dt><code>herr_t H5Awrite (hid_t <em>attr_id</em>,
hid_t <em>mem_type_id</em>, void *<em>buf</em>)</code>
<dd>This function writes an attribute, specified with <em>attr_id</em>,
with <em>mem_type_id</em> specifying the datatype in memory. The entire
attribute is written from <em>buf</em> to the file.
This function returns non-negative on success, negative on failure.
<br><br>
<dt><code>herr_t H5Aread (hid_t <em>attr_id</em>,
hid_t <em>mem_type_id</em>, void *<em>buf</em>)</code>
<dd>This function read an attribute, specified with <em>attr_id</em>, with
<em>mem_type_id</em> specifying the datatype in memory. The entire
attribute is read into <em>buf</em> from the file.
This function returns non-negative on success, negative on failure.
</dl>
<h2>4. Attribute Inquiry Functions</h2>
<dl>
<dt><code>int H5Aiterate (hid_t <em>loc_id</em>,
unsigned *<em>attr_number</em>,
H5A_operator <em>operator</em>,
void *<em>operator_data</em>)</code>
<dd> This function interates over the attributes of dataset or group
specified with <em>loc_id</em>. For each attribute of the object, the
<em>operator_data</em> and some additional information (specified below)
are passed to the <em>operator</em> function. The iteration begins with
the <em>*attr_number</em> object in the group and the next attribute to be
processed by the operator is returned in <em>*attr_number</em>.
<P>The iterator returns a negative value if something is wrong, the return
value of the last operator if it was non-zero, or zero if all attributes
were processed.
<P>The prototype for H5A_operator_t is: <br>
<code>typedef herr_t (*H5A_operator_t)(hid_t <em>loc_id</em>,
const char *<em>attr_name</em>, void *<em>operator_data</em>);</code>
<P>The operation receives the ID for the group or dataset being iterated over
(<em>loc_id</em>), the name of the current attribute about the object (<em>attr_name</em>)
and the pointer to the operator data passed in to H5Aiterate
(<em>operator_data</em>). The return values from an operator are:
<ul>
<li>Zero causes the iterator to continue, returning zero when all
attributes have been processed.
<li>Positive causes the iterator to immediately return that positive
value, indicating short-circuit success. The iterator can be
restarted at the next attribute.
<li>Negative causes the iterator to immediately return that value,
indicating failure. The iterator can be restarted at the next
attribute.
</ul>
<br><br>
<dt><code>hid_t H5Aget_space (hid_t <em>attr_id</em>)</code>
<dd>This function retrieves a copy of the dataspace for an attribute.
The dataspace ID returned from this function must be released with H5Sclose
or resource leaks will develop.
This function returns a dataspace ID for success or negative for failure.
<br><br>
<dt><code>hid_t H5Aget_type (hid_t <em>attr_id</em>)</code>
<dd>This function retrieves a copy of the datatype for an attribute.
The datatype ID returned from this function must be released with H5Tclose
or resource leaks will develop.
This function returns a datatype ID for success or negative for failure.
<br><br>
<dt><code>size_t H5Aget_name (hid_t <em>attr_id</em>,
char *<em>buf</em>, size_t <em>buf_size</em>)</code>
<dd>This function retrieves the name of an attribute for an attribute ID.
Up to <em>buf_size</em> characters are stored in <em>buf</em> followed by a
'\0' string terminator. If the name of the attribute is longer than
<em>buf_size</em>-1, the string terminator is stored in the last position
of the buffer to properly terminate the string.
This function returns the length of the attribute's name (which may be
longer than <em>buf_size</em>) on success or negative for failure.
<br><br>
<dt><code>int H5Anum_attrs (hid_t <em>loc_id</em>)</code>
<dd>This function returns the number of attributes attached to a dataset or
group, <em>loc_id</em>.
This function returns non-negative for success or negative for failure.
</dl>
<hr>
<address><a href="mailto:hdfhelp@ncsa.uiuc.edu">HDF Support</a></address>
</body>
</html>

111
doc/html/Big.html Normal file

@ -0,0 +1,111 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Big Datasets on Small Machines</title>
</head>
<body>
<h1>Big Datasets on Small Machines</h1>
<h2>1. Introduction</h2>
<p>The HDF5 library is able to handle files larger than the
maximum file size, and datasets larger than the maximum memory
size. For instance, a machine where <code>sizeof(off_t)</code>
and <code>sizeof(size_t)</code> are both four bytes can handle
datasets and files as large as 18x10^18 bytes. However, most
Unix systems limit the number of concurrently open files, so a
practical file size limit is closer to 512GB or 1TB.
<p>Two "tricks" must be imployed on these small systems in order
to store large datasets. The first trick circumvents the
<code>off_t</code> file size limit and the second circumvents
the <code>size_t</code> main memory limit.
<h2>2. File Size Limits</h2>
<p>Some 32-bit operating systems have special file systems that
can support large (&gt;2GB) files and HDF5 will detect these and
use them automatically. If this is the case, the output from
configure will show:
<p><code><pre>
checking for lseek64... yes
checking for fseek64... yes
</pre></code>
<p>Otherwise one must use an HDF5 file family. Such a family is
created by setting file family properties in a file access
property list and then supplying a file name that includes a
<code>printf</code>-style integer format. For instance:
<p><code><pre>
hid_t plist, file;
plist = H5Pcreate (H5P_FILE_ACCESS);
H5Pset_family (plist, 1<<30, H5P_DEFAULT);
file = H5Fcreate ("big%03d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, plist);
</code></pre>
<p>The second argument (<code>30</code>) to
<code>H5Pset_family()</code> indicates that the family members
are to be 2^30 bytes (1GB) each. In general, family members
cannot be 2GB because writes to byte number 2,147,483,647 will
fail, so the largest safe value for a family member is
2,147,483,647. HDF5 will create family members on demand as the
HDF5 address space increases, but since most Unix systems limit
the number of concurrently open files the effective maximum size
of the HDF5 address space will be limited.
<p>If the effective HDF5 address space is limited then one may be
able to store datasets as external datasets each spanning
multiple files of any length since HDF5 opens external dataset
files one at a time. To arrange storage for a 5TB dataset one
could say:
<p><code><pre>
hid_t plist = H5Pcreate (H5P_DATASET_CREATE);
for (i=0; i&lt;5*1024; i++) {
sprintf (name, "velocity-%04d.raw", i);
H5Pset_external (plist, name, 0, (size_t)1&lt;&lt;30);
}
</code></pre>
<h2>3. Dataset Size Limits</h2>
<p>The second limit which must be overcome is that of
<code>sizeof(size_t)</code>. HDF5 defines a new data type
called <code>hsize_t</code> which is used for sizes of datasets
and is, by default, defined as <code>unsigned long long</code>.
<p>To create a dataset with 8*2^30 4-byte integers for a total of
32GB one first creates the dataspace. We give two examples
here: a 4-dimensional dataset whose dimension sizes are smaller
than the maximum value of a <code>size_t</code>, and a
1-dimensional dataset whose dimension size is too large to fit
in a <code>size_t</code>.
<p><code><pre>
hsize_t size1[4] = {8, 1024, 1024, 1024};
hid_t space1 = H5Screate_simple (4, size1, size1);
hsize_t size2[1] = {8589934592LL};
hid_t space2 = H5Screate_simple (1, size2, size2};
</pre></code>
<p>However, the <code>LL</code> suffix is not portable, so it may
be better to replace the number with
<code>(hsize_t)8*1024*1024*1024</code>.
<p>For compilers that don't support <code>long long</code> large
datasets will not be possible. The library performs too much
arithmetic on <code>hsize_t</code> types to make the use of a
struct feasible.
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Fri Apr 10 13:26:04 EDT 1998 -->
<!-- hhmts start -->
Last modified: Wed May 13 12:36:47 EDT 1998
<!-- hhmts end -->
</body>
</html>

82
doc/html/Caching.html Normal file

@ -0,0 +1,82 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Data Caching</title>
</head>
<body>
<h1>Meta Data Caching</h1>
<p>The HDF5 library caches two types of data: meta data and raw
data. The meta data cache holds file objects like the file
header, symbol table nodes, global heap collections, object
headers and their messages, etc. in a partially decoded
state. The cache has a fixed number of entries which is set with
the file access property list (defaults to 10k) and each entry
can hold a single meta data object. Collisions between objects
are handled by preempting the older object in favor of the new
one.
<h1>Raw Data Chunk Caching</h1>
<p>Raw data chunks are cached because I/O requests at the
application level typically don't map well to chunks at the
storage level. The chunk cache has a maximum size in bytes
set with the file access property list (defaults to 1MB) and
when the limit is reached chunks are preempted based on the
following set of heuristics.
<ul>
<li>Chunks which have not been accessed for a long time
relative to other chunks are penalized.
<li>Chunks which have been accessed frequently in the recent
past are favored.
<li>Chunks which are completely read and not written, completely
written but not read, or completely read and completely
written are penalized according to <em>w0</em>, an
application-defined weight between 0 and 1 inclusive. A weight
of zero does not penalize such chunks while a weight of 1
penalizes those chunks more than all other chunks. The
default is 0.75.
<li>Chunks which are larger than the maximum cache size do not
participate in the cache.
</ul>
<p>One should choose large values for <em>w0</em> if I/O requests
typically do not overlap but smaller values for <em>w0</em> if
the requests do overlap. For instance, reading an entire 2d
array by reading from non-overlapping "windows" in a row-major
order would benefit from a high <em>w0</em> value while reading
a diagonal accross the dataset where each request overlaps the
previous request would benefit from a small <em>w0</em>.
<h1>The API</h1>
<p>The cache parameters for both caches are part of a file access
property list and are set and queried with this pair of
functions:
<dl>
<dt><code>herr_t H5Pset_cache(hid_t <em>plist</em>, unsigned int
<em>mdc_nelmts</em>, size_t <em>rdcc_nbytes</em>, double
<em>w0</em>)</code>
<dt><code>herr_t H5Pget_cache(hid_t <em>plist</em>, unsigned int
*<em>mdc_nelmts</em>, size_t *<em>rdcc_nbytes</em>, double
<em>w0</em>)</code>
<dd>Sets or queries the meta data cache and raw data chunk cache
parameters. The <em>plist</em> is a file access property
list. The number of elements (objects) in the meta data cache
is <em>mdc_nelmts</em>. The total size of the raw data chunk
cache and the preemption policy is <em>rdcc_nbytes</em> and
<em>w0</em>. For <code>H5Pget_cache()</code> any (or all) of
the pointer arguments may be null pointers.
</dl>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Tue May 26 15:20:14 EDT 1998 -->
<!-- hhmts start -->
Last modified: Tue May 26 15:38:27 EDT 1998
<!-- hhmts end -->
</body>
</html>

300
doc/html/CodeReview.html Normal file

@ -0,0 +1,300 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Code Review</title>
</head>
<body>
<center><h1>Code Review 1</h1></center>
<h3>Some background...</h3>
<p>This is one of the functions exported from the
<code>H5B.c</code> file that implements a B-link-tree class
without worrying about concurrency yet (thus the `Note:' in the
function prologue). The <code>H5B.c</code> file provides the
basic machinery for operating on generic B-trees, but it isn't
much use by itself. Various subclasses of the B-tree (like
symbol tables or indirect storage) provide their own interface
and back end to this function. For instance,
<code>H5G_stab_find()</code> takes a symbol table OID and a name
and calls <code>H5B_find()</code> with an appropriate
<code>udata</code> argument that eventually gets passed to the
<code>H5G_stab_find()</code> function.
<p><code><pre>
1 /*-------------------------------------------------------------------------
2 * Function: H5B_find
3 *
4 * Purpose: Locate the specified information in a B-tree and return
5 * that information by filling in fields of the caller-supplied
6 * UDATA pointer depending on the type of leaf node
7 * requested. The UDATA can point to additional data passed
8 * to the key comparison function.
9 *
10 * Note: This function does not follow the left/right sibling
11 * pointers since it assumes that all nodes can be reached
12 * from the parent node.
13 *
14 * Return: Success: SUCCEED if found, values returned through the
15 * UDATA argument.
16 *
17 * Failure: FAIL if not found, UDATA is undefined.
18 *
19 * Programmer: Robb Matzke
20 * matzke@llnl.gov
21 * Jun 23 1997
22 *
23 * Modifications:
24 *
25 *-------------------------------------------------------------------------
26 */
27 herr_t
28 H5B_find (H5F_t *f, const H5B_class_t *type, const haddr_t *addr, void *udata)
29 {
30 H5B_t *bt=NULL;
31 intn idx=-1, lt=0, rt, cmp=1;
32 int ret_value = FAIL;
</pre></code>
<p>All pointer arguments are initialized when defined. I don't
worry much about non-pointers because it's usually obvious when
the value isn't initialized.
<p><code><pre>
33
34 FUNC_ENTER (H5B_find, NULL, FAIL);
35
36 /*
37 * Check arguments.
38 */
39 assert (f);
40 assert (type);
41 assert (type->decode);
42 assert (type->cmp3);
43 assert (type->found);
44 assert (addr && H5F_addr_defined (addr));
</pre></code>
<p>I use <code>assert</code> to check invariant conditions. At
this level of the library, none of these assertions should fail
unless something is majorly wrong. The arguments should have
already been checked by higher layers. It also provides
documentation about what arguments might be optional.
<p><code><pre>
45
46 /*
47 * Perform a binary search to locate the child which contains
48 * the thing for which we're searching.
49 */
50 if (NULL==(bt=H5AC_protect (f, H5AC_BT, addr, type, udata))) {
51 HGOTO_ERROR (H5E_BTREE, H5E_CANTLOAD, FAIL);
52 }
</pre></code>
<p>You'll see this quite often in the low-level stuff and it's
documented in the <code>H5AC.c</code> file. The
<code>H5AC_protect</code> insures that the B-tree node (which
inherits from the H5AC package) whose OID is <code>addr</code>
is locked into memory for the duration of this function (see the
<code>H5AC_unprotect</code> on line 90). Most likely, if this
node has been accessed in the not-to-distant past, it will still
be in memory and the <code>H5AC_protect</code> is almost a
no-op. If cache debugging is compiled in, then the protect also
prevents other parts of the library from accessing the node
while this function is protecting it, so this function can allow
the node to be in an inconsistent state while calling other
parts of the library.
<p>The alternative is to call the slighlty cheaper
<code>H5AC_find</code> and assume that the pointer it returns is
valid only until some other library function is called, but
since we're accessing the pointer throughout this function, I
chose to use the simpler protect scheme. All protected objects
<em>must be unprotected</em> before the file is closed, thus the
use of <code>HGOTO_ERROR</code> instead of
<code>HRETURN_ERROR</code>.
<p><code><pre>
53 rt = bt->nchildren;
54
55 while (lt&lt;rt && cmp) {
56 idx = (lt + rt) / 2;
57 if (H5B_decode_keys (f, bt, idx)&lt;0) {
58 HGOTO_ERROR (H5E_BTREE, H5E_CANTDECODE, FAIL);
59 }
60
61 /* compare */
62 if ((cmp=(type-&gt;cmp3)(f, bt->key[idx].nkey, udata,
63 bt->key[idx+1].nkey))&lt;0) {
64 rt = idx;
65 } else {
66 lt = idx+1;
67 }
68 }
69 if (cmp) {
70 HGOTO_ERROR (H5E_BTREE, H5E_NOTFOUND, FAIL);
71 }
</pre></code>
<p>Code is arranged in paragraphs with a comment starting each
paragraph. The previous paragraph is a standard binary search
algorithm. The <code>(type-&gt;cmp3)()</code> is an indirect
function call into the subclass of the B-tree. All indirect
function calls have the function part in parentheses to document
that it's indirect (quite obvious here, but not so obvious when
the function is a variable).
<p>It's also my standard practice to have side effects in
conditional expressions because I can write code faster and it's
more apparent to me what the condition is testing. But if I
have an assignment in a conditional expr, then I use an extra
set of parens even if they're not required (usually they are, as
in this case) so it's clear that I meant <code>=</code> instead
of <code>==</code>.
<p><code><pre>
72
73 /*
74 * Follow the link to the subtree or to the data node.
75 */
76 assert (idx&gt;=0 && idx<bt->nchildren);
77 if (bt->level > 0) {
78 if ((ret_value = H5B_find (f, type, bt->child+idx, udata))&lt;0) {
79 HGOTO_ERROR (H5E_BTREE, H5E_NOTFOUND, FAIL);
80 }
81 } else {
82 ret_value = (type-&gt;found)(f, bt->child+idx, bt->key[idx].nkey,
83 udata, bt->key[idx+1].nkey);
84 if (ret_value&lt;0) {
85 HGOTO_ERROR (H5E_BTREE, H5E_NOTFOUND, FAIL);
86 }
87 }
</pre></code>
<p>Here I broke the "side effect in conditional" rule, which I
sometimes do if the expression is so long that the
<code>&lt;0</code> gets lost at the end. Another thing to note is
that success/failure is always determined by comparing with zero
instead of <code>SUCCEED</code> or <code>FAIL</code>. I do this
because occassionally one might want to return other meaningful
values (always non-negative) or distinguish between various types of
failure (always negative).
<p><code><pre>
88
89 done:
90 if (bt && H5AC_unprotect (f, H5AC_BT, addr, bt)&lt;0) {
91 HRETURN_ERROR (H5E_BTREE, H5E_PROTECT, FAIL);
92 }
93 FUNC_LEAVE (ret_value);
94 }
</pre></code>
<p>For lack of a better way to handle errors during error cleanup,
I just call the <code>HRETURN_ERROR</code> macro even though it
will make the error stack not quite right. I also use short
circuiting boolean operators instead of nested <code>if</code>
statements since that's standard C practice.
<center><h1>Code Review 2</h1></center>
<p>The following code is an API function from the H5F package...
<p><code><pre>
1 /*--------------------------------------------------------------------------
2 NAME
3 H5Fflush
4
5 PURPOSE
6 Flush all cached data to disk and optionally invalidates all cached
7 data.
8
9 USAGE
10 herr_t H5Fflush(fid, invalidate)
11 hid_t fid; IN: File ID of file to close.
12 hbool_t invalidate; IN: Invalidate all of the cache?
13
14 ERRORS
15 ARGS BADTYPE Not a file atom.
16 ATOM BADATOM Can't get file struct.
17 CACHE CANTFLUSH Flush failed.
18
19 RETURNS
20 SUCCEED/FAIL
21
22 DESCRIPTION
23 This function flushes all cached data to disk and, if INVALIDATE
24 is non-zero, removes cached objects from the cache so they must be
25 re-read from the file on the next access to the object.
26
27 MODIFICATIONS:
28 --------------------------------------------------------------------------*/
</pre></code>
<p>An API prologue is used for each API function instead of my
normal function prologue. I use the prologue from Code Review 1
for non-API functions because it's more suited to C programmers,
it requires less work to keep it synchronized with the code, and
I have better editing tools for it.
<p><code><pre>
29 herr_t
30 H5Fflush (hid_t fid, hbool_t invalidate)
31 {
32 H5F_t *file = NULL;
33
34 FUNC_ENTER (H5Fflush, H5F_init_interface, FAIL);
35 H5ECLEAR;
</pre></code>
<p>API functions are never called internally, therefore I always
clear the error stack before doing anything.
<p><code><pre>
36
37 /* check arguments */
38 if (H5_FILE!=H5Aatom_group (fid)) {
39 HRETURN_ERROR (H5E_ARGS, H5E_BADTYPE, FAIL); /*not a file atom*/
40 }
41 if (NULL==(file=H5Aatom_object (fid))) {
42 HRETURN_ERROR (H5E_ATOM, H5E_BADATOM, FAIL); /*can't get file struct*/
43 }
</pre></code>
<p>If something is wrong with the arguments then we raise an
error. We never <code>assert</code> arguments at this level.
We also convert atoms to pointers since atoms are really just a
pointer-hiding mechanism. Functions that can be called
internally always have pointer arguments instead of atoms
because (1) then they don't have to always convert atoms to
pointers, and (2) the various pointer data types provide more
documentation and type checking than just an <code>hid_t</code>
type.
<p><code><pre>
44
45 /* do work */
46 if (H5F_flush (file, invalidate)&lt;0) {
47 HRETURN_ERROR (H5E_CACHE, H5E_CANTFLUSH, FAIL); /*flush failed*/
48 }
</pre></code>
<p>An internal version of the function does the real work. That
internal version calls <code>assert</code> to check/document
it's arguments and can be called from other library functions.
<p><code><pre>
49
50 FUNC_LEAVE (SUCCEED);
51 }
</pre></code>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Sat Nov 8 17:09:33 EST 1997 -->
<!-- hhmts start -->
Last modified: Mon Nov 10 15:33:33 EST 1997
<!-- hhmts end -->
</body>
</html>

300
doc/html/Coding.html Normal file

@ -0,0 +1,300 @@
<HTML>
<HEAD><TITLE>
HDF5 Naming Scheme
</TITLE> </HEAD>
<BODY bgcolor="#ffffff">
<H1>
<FONT color="#c80028"
<I> <B> <CENTER> HDF5 Naming Scheme for </CENTER> </B> </I> </H1>
</FONT>
<P>
<UL>
<LI> <A HREF = "#01"><I> FILES </I> </A>
<LI> <A HREF = "#02"><I> PACKAGES </I> </A>
<LI> <A HREF = "#03"><I> PUBLIC vs PRIVATE </I> </A>
<LI> <A HREF = "#04"><I> INTEGRAL TYPES </I> </A>
<LI> <A HREF = "#05"><I> OTHER TYPES </I> </A>
<LI> <A HREF = "#06"><I> GLOBAL VARIABLES </I> </A>
<LI> <A HREF = "#07"><I> MACROS, PREPROCESSOR CONSTANTS, AND ENUM MEMEBERs </I> </A>
</UL>
<P>
<center>
Authors: <A HREF = "mailto:koziol@ncsa.uiuc.edu">
<I>Quincey Koziol</I> </A> and
<A HREF = "mailto:matzke@llnl.gov">
<I> Robb Matzke </I> </A>
</center>
<UL>
<FONT color="#c80028"
<LI> <A NAME="01"> <B> <I> FILES </I> </B> </A>
</FONT>
<UL>
<LI> Source files are named according to the package they contain (see
below). All files will begin with `H5' so we can stuff our
object files into someone else's library and not worry about file
name conflicts.
<P>For Example:
<i><b>
<dd> H5.c -- "Generic" library functions
<br>
<dd> H5B.c -- B-link tree functions
</i></b>
<p>
<LI> If a package is in more than one file, then another name is tacked
on. It's all lower case with no underscores or hyphens.
<P>For Example:
<i><b>
<dd> H5F.c -- the file for this package
<br>
<dd> H5Fstdio.c -- stdio functions (just an example)
<br>
<dd> H5Ffcntl.c -- fcntl functions (just an example)
</i></b>
<p>
<LI> Each package file has a header file of API stuff (unless there is
no API component to the package)
<P>For Example:
<i><b>
<dd> H5F.h -- things an application would see. </i> </b>
<P>
and a header file of private stuff
<i><b>
<p>
<dd> H5Fprivate.h -- things an application wouldn't see. The
private header includes the public header.
</i></b>
<p>
and a header for private prototypes
<i><b>
<p>
<dd> H5Fproto.h -- prototypes for internal functions.
</i></b>
<P>
By splitting the prototypes into separate include files we don't
have to recompile everything when just one function prototype
changes.
<LI> The main API header file is `hdf5.h' and it includes each of the
public header files but none of the private header files. Or the
application can include just the public header files it needs.
<LI> There is no main private or prototype header file because it
prevents make from being efficient. Instead, each source file
includes only the private header and prototype files it needs
(first all the private headers, then all the private prototypes).
<LI> Header files should include everything they need and nothing more.
</UL>
<P>
<FONT color="#c80028"
<LI> <A NAME="02"> <B> <I> PACKAGES </I> </B> </A>
</FONT>
<P>
Names exported beyond function scope begin with `H5' followed by zero,
one, or two upper-case letters that describe the class of object.
This prefix is the package name. The implementation of packages
doesn't necessarily have to map 1:1 to the source files.
<P>
<i><b>
<dd> H5 -- library functions
<br>
<dd> H5A -- atoms
<br>
<dd> H5AC -- cache
<br>
<dd> H5B -- B-link trees
<br>
<dd> H5D -- datasets
<br>
<dd> H5E -- error handling
<br>
<dd> H5F -- files
<br>
<dd> H5G -- groups
<br>
<dd> H5M -- meta data
<br>
<dd> H5MM -- core memory management
<br>
<dd> H5MF -- file memory management
<br>
<dd> H5O -- object headers
<br>
<dd> H5P -- Property Lists
<br>
<dd> H5S -- dataspaces
<br>
<dd> H5R -- relationships
<br>
<dd> H5T -- datatype
</i></b>
<p>
Each package implements a single main class of object (e.g., the H5B
package implements B-link trees). The main data type of a package is
the package name followed by `_t'.
<p>
<i><b>
<dd> H5F_t -- HDF5 file type
<br>
<dd> H5B_t -- B-link tree data type
</i></b>
<p>
Not all packages implement a data type (H5, H5MF) and some
packages provide access to a preexisting data type (H5MM, H5S).
<p>
<FONT color="#c80028"
<LI> <A NAME="03"> <B> <I> PUBLIC vs PRIVATE </I> </B> </A>
</FONT>
<p>
If the symbol is for internal use only, then the package name is
followed by an underscore and the rest of the name. Otherwise, the
symbol is part of the API and there is no underscore between the
package name and the rest of the name.
<p>
<i><b>
<dd> H5Fopen -- an API function.
<br>
<dd> H5B_find -- an internal function.
</i></b>
<p>
For functions, this is important because the API functions never pass
pointers around (they use atoms instead for hiding the implementation)
and they perform stringent checks on their arguments. Internal
unctions, on the other hand, check arguments with assert().
<p>
Data types like H5B_t carry no information about whether the type is
public or private since it doesn't matter.
<p>
<FONT color="#c80028"
<LI> <A NAME="04"> <B> <I> INTEGRAL TYPES </I> </B> </A>
</FONT>
<p>
Integral fixed-point type names are an optional `u' followed by `int'
followed by the size in bits (8, 16,
32, or 64). There is no trailing `_t' because these are common
enough and follow their own naming convention.
<p>
<pre><H4>
<dd> hbool_t -- boolean values (BTRUE, BFALSE, BFAIL)
<br>
<dd> int8 -- signed 8-bit integers
<br>
<dd> uint8 -- unsigned 8-bit integers
<br>
<dd> int16 -- signed 16-bit integers
<br>
<dd> uint16 -- unsigned 16-bit integers
<br>
<dd> int32 -- signed 32-bit integers
<br>
<dd> uint32 -- unsigned 32-bit integers
<br>
<dd> int64 -- signed 64-bit integers
<br>
<dd> uint64 -- unsigned 64-bit integers
<br>
<dd> intn -- "native" integers
<br>
<dd> uintn -- "native" unsigned integers
</pre></H4>
<p>
<FONT color="#c80028"
<LI> <A NAME="05"> <B> <I> OTHER TYPES </I> </B> </A>
</FONT>
<p>
Other data types are always followed by `_t'.
<p>
<pre><H4>
<dd> H5B_key_t-- additional data type used by H5B package.
</pre></H4>
<p>
However, if the name is so common that it's used almost everywhere,
then we make an alias for it by removing the package name and leading
underscore and replacing it with an `h' (the main datatype for a
package already has a short enough name, so we don't have aliases for
them).
<P>
<pre><H4>
<dd> typedef H5E_err_t herr_t;
</pre> </H4>
<p>
<FONT color="#c80028"
<LI> <A NAME="06"> <B> <I> GLOBAL VARIABLES </I> </B> </A>
</FONT>
<p>
Global variables include the package name and end with `_g'.
<p>
<pre><H4>
<dd> H5AC_methods_g -- global variable in the H5AC package.
</pre> </H4>
<p>
<FONT color="#c80028"
<LI> <A NAME="07">
<I> <B>
MACROS, PREPROCESSOR CONSTANTS, AND ENUM MEMBERS
</I> </B> </A>
</FONT>
<p>
Same rules as other symbols except the name is all upper case. There
are a few exceptions: <br>
<ul>
<li> Constants and macros defined on a system that is deficient:
<p><pre><H4>
<dd> MIN(x,y), MAX(x,y) and their relatives
</pre></H4>
<li> Platform constants :
<P>
No naming scheme; determined by OS and compiler.<br>
These appear only in one header file anyway.
<p>
<li> Feature test constants (?)<br>
Always start with `HDF5_HAVE_' like HDF5_HAVE_STDARG_H for a
header file, or HDF5_HAVE_DEV_T for a data type, or
HDF5_HAVE_DIV for a function.
</UL>
<p>
</UL>
<p>
<H6>
<center>
This file /hdf3/web/hdf/internal/HDF_standard/HDF5.coding_standard.html is
maintained by Elena Pourmal <A HREF = "mailto:epourmal@ncsa.uiuc.edu">
<I>epourmal@ncsa.uiuc.edu</I> </A>.
</center>
<p>
<center>
Last modified August 5, 1997
</center>
</H6>
</BODY>
<HTML>

409
doc/html/Compression.html Normal file

@ -0,0 +1,409 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Compression</title>
</head>
<body>
<h1>Compression</h1>
<h2>1. Introduction</h2>
<p>HDF5 supports compression of raw data by compression methods
built into the library or defined by an application. A
compression method is associated with a dataset when the dataset
is created and is applied independently to each storage chunk of
the dataset.
The dataset must use the <code>H5D_CHUNKED</code> storage
layout. The library doesn't support compression for contiguous
datasets because of the difficulty of implementing random access
for partial I/O, and compact dataset compression is not
supported because it wouldn't produce significant results.
<h2>2. Supported Compression Methods</h2>
<p>The library identifies compression methods with small
integers, with values less than 16 reserved for use by NCSA and
values between 16 and 255 (inclusive) available for general
use. This range may be extended in the future if it proves to
be too small.
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Method Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td><code>H5Z_NONE</code></td>
<td>The default is to not use compression. Specifying
<code>H5Z_NONE</code> as the compression method results
in better perfomance than writing a function that just
copies data because the library's I/O pipeline
recognizes this method and is able to short circuit
parts of the pipeline.</td>
</tr>
<tr valign=top>
<td><code>H5Z_DEFLATE</code></td>
<td>The <em>deflate</em> method is the algorithm used by
the GNU <code>gzip</code>program. It's a combination of
a Huffman encoding followed by a 1977 Lempel-Ziv (LZ77)
dictionary encoding. The aggressiveness of the
compression can be controlled by passing an integer value
to the compressor with <code>H5Pset_deflate()</code>
(see below). In order for this compression method to be
used, the HDF5 library must be configured and compiled
in the presence of the GNU zlib version 1.1.2 or
later.</td>
</tr>
<tr valign=top>
<td><code>H5Z_RES_<em>N</em></code></td>
<td>These compression methods (where <em>N</em> is in the
range two through 15, inclusive) are reserved by NCSA
for future use.</td>
</tr>
<tr valign=top>
<td>Values of <em>N</em> between 16 and 255, inclusive</td>
<td>These values can be used to represent application-defined
compression methods. We recommend that methods under
testing should be in the high range and when a method is
about to be published it should be given a number near
the low end of the range (or even below 16). Publishing
the compression method and its numeric ID will make a
file sharable.</td>
</tr>
</table>
</center>
<p>Setting the compression for a dataset to a method which was
not compiled into the library and/or not registered by the
application is allowed, but writing to such a dataset will
silently <em>not</em> compress the data. Reading a compressed
dataset for a method which is not available will result in
errors (specifically, <code>H5Dread()</code> will return a
negative value). The errors will be displayed in the
compression statistics if the library was compiled with
debugging turned on for the &quot;z&quot; package. See the
section on diagnostics below for more details.
<h2>3. Application-Defined Methods</h2>
<p>Compression methods 16 through 255 can be defined by an
application. As mentioned above, methods that have not been
released should use high numbers in that range while methods
that have been published will be assigned an official number in
the low region of the range (possibly less than 16). Users
should be aware that using unpublished compression methods
results in unsharable files.
<p>A compression method has two halves: one have handles
compression and the other half handles uncompression. The
halves are implemented as functions
<code><em>method</em>_c</code> and
<code><em>method</em>_u</code> respectively. One should not use
the names <code>compress</code> or <code>uncompress</code> since
they are likely to conflict with other compression libraries
(like the GNU zlib).
<p>Both the <code><em>method</em>_c</code> and
<code><em>method</em>_u</code> functions take the same arguments
and return the same values. They are defined with the type:
<dl>
<dt><code>typedef size_t (*H5Z_func_t)(unsigned int
<em>flags</em>, size_t <em>cd_size</em>, const void
*<em>client_data</em>, size_t <em>src_nbytes</em>, const
void *<em>src</em>, size_t <em>dst_nbytes</em>, void
*<em>dst</em>/*out*/)</code>
<dd>The <em>flags</em> are an 8-bit vector which is stored in
the file and which is defined when the compression method is
defined. The <em>client_data</em> is a pointer to
<em>cd_size</em> bytes of configuration data which is also
stored in the file. The function compresses or uncompresses
<em>src_nbytes</em> from the source buffer <em>src</em> into
at most <em>dst_nbytes</em> of the result buffer <em>dst</em>.
The function returns the number of bytes written to the result
buffer or zero if an error occurs. But if a result buffer
overrun occurs the function should return a value at least as
large as <em>dst_size</em> (the uncompressor will see an
overrun only for corrupt data).
</dl>
<p>The application associates the pair of functions with a name
and a method number by calling <code>H5Zregister()</code>. This
function can also be used to remove a compression method from
the library by supplying null pointers for the functions.
<dl>
<dt><code>herr_t H5Zregister (H5Z_method_t <em>method</em>,
const char *<em>name</em>, H5Z_func_t <em>method_c</em>,
H5Z_func_t <em>method_u</em>)</code>
<dd>The pair of functions to be used for compression
(<em>method_c</em>) and uncompression (<em>method_u</em>) are
associated with a short <em>name</em> used for debugging and a
<em>method</em> number in the range 16 through 255. This
function can be called as often as desired for a particular
compression method with each call replacing the information
stored by the previous call. Sometimes it's convenient to
supply only one half of the compression, for instance in an
application that opens files for read-only. Compression
statistics for the method are accumulated across calls to this
function.
</dl>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Registering an
Application-Defined Compression Method</h4></caption>
<tr>
<td>
<p>Here's a simple-minded &quot;compression&quot; method
that just copies the input value to the output. It's
similar to the <code>H5Z_NONE</code> method but
slower. Compression and uncompression are performed
by the same function.
<p><code><pre>
size_t
bogus (unsigned int flags,
size_t cd_size, const void *client_data,
size_t src_nbytes, const void *src,
size_t dst_nbytes, void *dst/*out*/)
{
memcpy (dst, src, src_nbytes);
return src_nbytes;
}
</pre></code>
<p>The function could be registered as method 250 as
follows:
<p><code><pre>
#define H5Z_BOGUS 250
H5Zregister (H5Z_BOGUS, "bogus", bogus, bogus);
</pre></code>
<p>The function can be unregistered by saying:
<p><code><pre>
H5Zregister (H5Z_BUGUS, "bogus", NULL, NULL);
</pre></code>
<p>Notice that we kept the name &quot;bogus&quot; even
though we unregistered the functions that perform the
compression and uncompression. This makes compression
statistics more understandable when they're printed.
</td>
</tr>
</table>
</center>
<h2>4. Enabling Compression for a Dataset</h2>
<p>If a dataset is to be compressed then the compression
information must be specified when the dataset is created since
once a dataset is created compression parameters cannot be
adjusted. The compression is specified through the dataset
creation property list (see <code>H5Pcreate()</code>).
<dl>
<dt><code>herr_t H5Pset_deflate (hid_t <em>plist</em>, int
<em>level</em>)</code>
<dd>The compression method for dataset creation property list
<em>plist</em> is set to <code>H5Z_DEFLATE</code> and the
aggression level is set to <em>level</em>. The <em>level</em>
must be a value between one and nine, inclusive, where one
indicates no (but fast) compression and nine is aggressive
compression.
<br><br>
<dt><code>int H5Pget_deflate (hid_t <em>plist</em>)</code>
<dd>If dataset creation property list <em>plist</em> is set to
use <code>H5Z_DEFLATE</code> compression then this function
will return the aggression level, an integer between one and
nine inclusive. If <em>plist</em> isn't a valid dataset
creation property list or it isn't set to use the deflate
method then a negative value is returned.
<br><br>
<dt><code>herr_t H5Pset_compression (hid_t <em>plist</em>,
H5Z_method_t <em>method</em>, unsigned int <em>flags</em>,
size_t <em>cd_size</em>, const void *<em>client_data</em>)</code>
<dd>This is a catch-all function for defining compresion methods
and is intended to be called from a wrapper such as
<code>H5Pset_deflate()</code>. The dataset creation property
list <em>plist</em> is adjusted to use the specified
compression method. The <em>flags</em> is an 8-bit vector
which is stored in the file as part of the compression message
and passed to the compress and uncompress functions. The
<em>client_data</em> is a byte array of length
<em>cd_size</em> which is copied to the file and passed to the
compress and uncompress methods.
<br><br>
<dt><code>H5Z_method_t H5Pget_compression (hid_t <em>plist</em>,
unsigned int *<em>flags</em>, size_t *<em>cd_size</em>, void
*<em>client_data</em>)</code>
<dd>This is a catch-all function for querying the compression
method associated with dataset creation property list
<em>plist</em> and is intended to be called from a wrapper
function such as <code>H5Pget_deflate()</code>. The
compression method (or a negative value on error) is returned
by value, and compression flags and client data is returned by
argument. The application should allocate the
<em>client_data</em> and pass its size as the
<em>cd_size</em>. On return, <em>cd_size</em> will contain
the actual size of the client data. If <em>client_data</em>
is not large enough to hold the entire client data then
<em>cd_size</em> bytes are copied into <em>client_data</em>
and <em>cd_size</em> is set to the total size of the client
data, a value larger than the original.
</dl>
<p>It is possible to set the compression to a method which hasn't
been defined with <code>H5Zregister()</code> and which isn't
supported as a predefined method (for instance, setting the
method to <code>H5Z_DEFLATE</code> when the GNU zlib isn't
available). If that happens then data will be written to the
file in its uncompressed form and the compression statistics
will show failures for the compression.
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Statistics for an
Unsupported Compression Method</h4></caption>
<tr>
<td>
<p>If an application attempts to use an unsupported
method then the compression statistics will show large
numbers of compression errors and no data
uncompressed.
<p><code><pre>
H5Z: compression statistics accumulated over life of library:
Method Total Overrun Errors User System Elapsed Bandwidth
------ ----- ------- ------ ---- ------ ------- ---------
deflate-c 160000 0 160000 0.00 0.01 0.01 1.884e+07
deflate-u 0 0 0 0.00 0.00 0.00 NaN
</pre></code>
<p>This example is from a program that tried to use
<code>H5Z_DEFLATE</code> on a system that didn't have
the GNU zlib to write to a dataset and then read the
result. The read and write both succeeded but the
data was not compressed.
</td>
</tr>
</table>
</center>
<h2>5. Compression Diagnostics</h2>
<p>If the library is compiled with debugging turned on for the H5Z
layer (usually as a result of <code>configure --enable-debug=z</code>)
then statistics about data compression are printed when the
application exits normally or the library is closed. The
statistics are written to the standard error stream and include
two lines for each compression method that was used: the first
line shows compression statistics while the second shows
uncompression statistics. The following fields are displayed:
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Method</td>
<td>This is the name of the method as defined with
<code>H5Zregister()</code> with the letters
&quot;-c&quot; or &quot;-u&quot; appended to indicate
compression or uncompression.</td>
</tr>
<tr valign=top>
<td>Total</td>
<td>The total number of bytes compressed or decompressed
including buffer overruns and errors. Bytes of
non-compressed data are counted.</td>
</tr>
<tr valign=top>
<td>Overrun</td>
<td>During compression, if the algorithm causes the result
to be at least as large as the input then a buffer
overrun error occurs. This field shows the total number
of bytes from the Total column which can be attributed to
overruns. Overruns for decompression can only happen if
the data has been corrupted in some way and will result
in failure of <code>H5Dread()</code>.</td>
</tr>
<tr valign=top>
<td>Errors</td>
<td>If an error occurs during compression the data is
stored in it's uncompressed form; and an error during
uncompression causes <code>H5Dread()</code> to return
failure. This field shows the number of bytes of the
Total column which can be attributed to errors.</td>
</tr>
<tr valign=top>
<td>User, System, Elapsed</td>
<td>These are the amount of user time, system time, and
elapsed time in seconds spent by the library to perform
compression. Elapsed time is sensitive to system
load. These times may be zero on operating systems that
don't support the required operations.</td>
</tr>
<tr valign=top>
<td>Bandwidth</td>
<td>This is the compression bandwidth which is the total
number of bytes divided by elapsed time. Since elapsed
time is subject to system load the bandwidth numbers
cannot always be trusted. Furthermore, the bandwidth
includes overrun and error bytes which may significanly
taint the value.</td>
</tr>
</table>
</center>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Compression
Statistics</h4></caption>
<tr>
<td>
<p><code><pre>
H5Z: compression statistics accumulated over life of library:
Method Total Overrun Errors User System Elapsed Bandwidth
------ ----- ------- ------ ---- ------ ------- ---------
deflate-c 160000 200 0 0.62 0.74 1.33 1.204e+05
deflate-u 120000 0 0 0.11 0.00 0.12 9.885e+05
</pre></code>
</td>
</tr>
</table>
</center>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Fri Apr 17 13:39:35 EDT 1998 -->
<!-- hhmts start -->
Last modified: Fri Apr 17 16:15:21 EDT 1998
<!-- hhmts end -->
</body>
</html>

839
doc/html/Datasets.html Normal file

@ -0,0 +1,839 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>The Dataset Interface (H5D)</title>
</head>
<body>
<h1>The Dataset Interface (H5D)</h1>
<h2>1. Introduction</h2>
<p>The purpose of the dataset interface is to provide a mechanism
to describe properties of datasets and to transfer data between
memory and disk. A dataset is composed of a collection of raw
data points and four classes of meta data to describe the data
points. The interface is hopefully designed in such a way as to
allow new features to be added without disrupting current
applications that use the dataset interface.
<p>The four classes of meta data are:
<dl>
<dt>Constant Meta Data
<dd>Meta data that is created when the dataset is created and
exists unchanged for the life of the dataset. For instance,
the data type of stored array elements is defined when the
dataset is created and cannot be subsequently changed.
<dt>Persistent Meta Data
<dd>Meta data that is an integral and permanent part of a
dataset but can change over time. For instance, the size in
any dimension can increase over time if such an increase is
allowed when the dataset was created.
<dt>Memory Meta Data
<dd>Meta data that exists to describe how raw data is organized
in the application's memory space. For instance, the data
type of elements in an application array might not be the same
as the data type of those elements as stored in the HDF5 file.
<dt>Transport Meta Data
<dd>Meta data that is used only during the transfer of raw data
from one location to another. For instance, the number of
processes participating in a collective I/O request or hints
to the library to control caching of raw data.
</dl>
<p>Each of these classes of meta data is handled differently by
the library although the same API might be used to create them.
For instance, the data type exists as constant meta data and as
memory meta data; the same API (the <code>H5T</code> API) is
used to manipulate both pieces of meta data but they're handled
by the dataset API (the <code>H5D</code> API) in different
manners.
<h2>2. Storage Layout Properties</h2>
<p>The dataset API partitions these terms on three orthogonal axes
(layout, compression, and external storage) and uses a
<em>dataset creation property list</em> to hold the various
settings and pass them through the dataset interface. This is
similar to the way HDF5 files are created with a file creation
property list. A dataset creation property list is always
derived from the default dataset creation property list (use
<code>H5Pcreate()</code> to get a copy of the default property
list) by modifying properties with various
<code>H5Pset_<em>property</em>()</code> functions.
<dl>
<dt><code>herr_t H5Pset_layout (hid_t <em>plist_id</em>,
H5D_layout_t <em>layout</em>)</code>
<dd>The storage layout is a piece of constant meta data that
describes what method the library uses to organize the raw
data on disk. The default layout is contiguous storage.
<br><br>
<dl>
<dt><code>H5D_COMPACT</code>
<dd>The raw data is presumably small and can be stored
directly in the object header. Such data is
non-extendible, non-compressible, non-sparse, and cannot
be stored externally. Most of these restrictions are
arbitrary but are enforced because of the small size of
the raw data. Storing data in this format eliminates the
disk seek/read request normally necessary to read raw
data. <b>This layout is not implemented yet.</b>
<br><br>
<dt><code>H5D_CONTIGUOUS</code>
<dd>The raw data is large, non-extendible, non-compressible,
non-sparse, and can be stored externally. This is the
default value for the layout property. The term
<em>large</em> means that it may not be possible to hold
the entire dataset in memory. The non-compressibility is
a side effect of the data being large, contiguous, and
fixed-size at the physical level, which could cause
partial I/O requests to be extremely expensive if
compression were allowed.
<br><br>
<dt><code>H5D_CHUNKED</code>
<dd>The raw data is large and can be extended in any
dimension at any time (provided the data space also allows
the extension). It may be sparse at the chunk level (each
chunk is non-sparse, but there might only be a few chunks)
and each chunk can be compressed and/or stored externally.
A dataset is partitioned into chunks so each chunk is the
same logical size. The chunks are indexed by a B-tree and
are allocated on demand (although it might be useful to be
able to preallocate storage for parts of a chunked array
to reduce contention for the B-tree in a parallel
environment). The chunk size must be defined with
<code>H5Pset_chunk()</code>.
<br><br>
<dt><em>others...</em>
<dd>Other layout types may be defined later without breaking
existing code. However, to be able to correctly read or
modify data stored with one of these new layouts, the
application will need to be linked with a new version of
the library. This happens automatically on systems with
dynamic linking.
</dl>
</dl>
<p>Once the general layout is defined, the user can define
properties of that layout. Currently, the only layout that has
user-settable properties is the <code>H5D_CHUNKED</code> layout,
which needs to know the dimensionality and chunk size.
<dl>
<dt><code>herr_t H5Pset_chunk (hid_t <em>plist_id</em>, int
<em>ndims</em>, hsize_t <em>dim</em>[])</code>
<dd>This function defines the logical size of a chunk for
chunked layout. If the layout property is set to
<code>H5D_CHUNKED</code> and the chunk size is set to
<em>dim</em>. The number of elements in the <em>dim</em> array
is the dimensionality, <em>ndims</em>. One need not call
<code>H5Dset_layout()</code> when using this function since
the chunked layout is implied.
</dl>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Chunked Storage</h4></caption>
<tr>
<td>
<p>This example shows how a two-dimensional dataset
is partitioned into chunks. The library can manage file
memory by moving the chunks around, and each chunk could be
compressed. The chunks are allocated in the file on demand
when data is written to the chunk.
<center>
<img alt="Chunked Storage" src="chunk1.gif">
</center>
<p><code><pre>
size_t hsize[2] = {1000, 1000};
plist = H5Pcreate (H5P_DATASET_CREATE);
H5Pset_chunk (plist, 2, size);
</pre></code>
</td>
</tr>
</table>
</center>
<p>Although it is most efficient if I/O requests are aligned on chunk
boundaries, this is not a constraint. The application can perform I/O
on any set of data points as long as the set can be described by the
data space. The set on which I/O is performed is called the
<em>selection</em>.
<h2>3. Compression Properties</h2>
<p>Some types of storage layout allow data compression which is
defined by the functions described here. <b>Compression is not
implemented yet.</b>
<dl>
<dt><code>herr_t H5Pset_compression (hid_t <em>plist_id</em>,
H5Z_method_t <em>method</em>)</code>
<dt><code>H5Z_method_t H5Pget_compression (hid_t
<em>plist_id</em>)</code>
<dd>These functions set and query the compression method that
is used to compress the raw data of a dataset. The
<em>plist_id</em> is a dataset creation property list. The
possible values for the compression method are:
<br><br>
<dl>
<dt><code>H5Z_NONE</code>
<dd>This is the default and specifies that no compression is
to be performed.
<br><br>
<dt><code>H5Z_DEFLATE</code>
<dd>This specifies that a variation of the Lempel-Ziv 1977
(LZ77) encoding is used, the same encoding used by the
free GNU <code>gzip</code> program.
</dl>
<br><br>
<dt><code>herr_t H5Pset_deflate (hid_t <em>plist_id</em>,
int <em>level</em>)</code>
<dt><code>int H5Pget_deflate (hid_t <em>plist_id</em>)</code>
<dd>These functions set or query the deflate level of
dataset creation property list <em>plist_id</em>. The
<code>H5Pset_deflate()</code> sets the compression method to
<code>H5Z_DEFLATE</code> and sets the compression level to
some integer between one and nine (inclusive). One results in
the fastest compression while nine results in the best
compression ratio. The default value is six if
<code>H5Pset_deflate()</code> isn't called. The
<code>H5Pget_deflate()</code> returns the compression level
for the deflate method, or negative if the method is not the
deflate method.
</dl>
<h2>4. External Storage Properties</h2>
<p>Some storage formats may allow storage of data across a set of
non-HDF5 files. Currently, only the <code>H5D_CONTIGUOUS</code> storage
format allows external storage. A set segments (offsets and sizes) in
one or more files is defined as an external file list, or <em>EFL</em>,
and the contiguous logical addresses of the data storage are mapped onto
these segments.
<dl>
<dt><code>herr_t H5Pset_external (hid_t <em>plist</em>, const
char *<em>name</em>, off_t <em>offset</em>, hsize_t
<em>size</em>)</code>
<dd>This function adds a new segment to the end of the external
file list of the specified dataset creation property list. The
segment begins a byte <em>offset</em> of file <em>name</em> and
continues for <em>size</em> bytes. The space represented by this
segment is adjacent to the space already represented by the external
file list. The last segment in a file list may have the size
<code>H5F_UNLIMITED</em>.
<br><br>
<dt><code>int H5Pget_external_count (hid_t <em>plist</em>)</code>
<dd>Calling this function returns the number of segments in an
external file list. If the dataset creation property list has no
external data then zero is returned.
<br><br>
<dt><code>herr_t H5Pget_external (hid_t <em>plist</em>, int
<em>idx</em>, size_t <em>name_size</em>, char *<em>name</em>, off_t
*<em>offset</em>, hsize_t *<em>size</em>)</code>
<dd>This is the counterpart for the <code>H5Pset_external()</code>
function. Given a dataset creation property list and a zero-based
index into that list, the file name, byte offset, and segment size are
returned through non-null arguments. At most <em>name_size</em>
characters are copied into the <em>name</em> argument which is not
null terminated if the file name is longer than the supplied name
buffer (this is similar to <code>strncpy()</code>).
</dl>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Multiple Segments</h4></caption>
<tr>
<td>
<p>This example shows how a contiguous, one-dimensional dataset
is partitioned into three parts and each of those parts is
stored in a segment of an external file. The top rectangle
represents the logical address space of the dataset
while the bottom rectangle represents an external file.
<center>
<img alt="Multiple Segments" src="extern1.gif">
</center>
<p><code><pre>
plist = H5Pcreate (H5P_DATASET_CREATE);
H5Pset_external (plist, "velocity.data", 3000, 1000);
H5Pset_external (plist, "velocity.data", 0, 2500);
H5Pset_external (plist, "velocity.data", 4500, 1500);
</pre></code>
<p>One should note that the segments are defined in order of the
logical addresses they represent, not their order within the
external file. It would also have been possible to put the
segments in separate files. Care should be taken when setting
up segments in a single file since the library doesn't
automatically check for segments that overlap.
</td>
</tr>
</table>
</center>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Multi-Dimensional</h4></caption>
<tr>
<td>
<p>This example shows how a contiguous, two-dimensional dataset
is partitioned into three parts and each of those parts is
stored in a separate external file. The top rectangle
represents the logical address space of the dataset
while the bottom rectangles represent external files.
<center>
<img alt="Multiple Dimensions" src="extern2.gif">
</center>
<p><code><pre>
plist = H5Pcreate (H5P_DATASET_CREATE);
H5Pset_external (plist, "scan1.data", 0, 24);
H5Pset_external (plist, "scan2.data", 0, 24);
H5Pset_external (plist, "scan3.data", 0, 16);
</pre></code>
<p>The library maps the multi-dimensional array onto a linear
address space like normal, and then maps that address space
into the segments defined in the external file list.
</td>
</tr>
</table>
</center>
<p>The segments of an external file can exist beyond the end of the
file. The library reads that part of a segment as zeros. When writing
to a segment that exists beyond the end of a file, the file is
automatically extended. Using this feature, one can create a segment
(or set of segments) which is larger than the current size of the
dataset, which allows to dataset to be extended at a future time
(provided the data space also allows the extension).
<p>All referenced external data files must exist before performing raw
data I/O on the dataset. This is normally not a problem since those
files are being managed directly by the application, or indirectly
through some other library.
<h2>5. Data Type</h2>
<p>Raw data has a constant data type which describes the data type
of the raw data stored in the file, and a memory data type that
describes the data type stored in application memory. Both data
types are manipulated with the <a
href="Datatypes.html"><code>H5T</code></a> API.
<p>The constant file data type is associated with the dataset when
the dataset is created in a manner described below. Once
assigned, the constant datatype can never be changed.
<p>The memory data type is specified when data is transferred
to/from application memory. In the name of data sharability,
the memory data type must be specified, but can be the same
type identifier as the constant data type.
<p>During dataset I/O operations, the library translates the raw
data from the constant data type to the memory data type or vice
versa. Structured data types include member offsets to allow
reordering of struct members and/or selection of a subset of
members and array data types include index permutation
information to allow things like transpose operations (<b>the
prototype does not support array reordering</b>) Permutations
are relative to some extrinsic descritpion of the dataset.
<h2>6. Data Space</h2>
<p>The dataspace of a dataset defines the number of dimensions
and the size of each dimension and is manipulated with the
<code>H5S</code> API. The <em>simple</em> dataspace consists of
maximum dimension sizes and actual dimension sizes, which are
usually the same. However, maximum dimension sizes can be the
constant <code>H5D_UNLIMITED</code> in which case the actual
dimension size can be incremented with calls to
<code>H5Dextend()</code>. The maximium dimension sizes are
constant meta data while the actual dimension sizes are
persistent meta data. Initial actual dimension sizes are
supplied at the same time as the maximum dimension sizes when
the dataset is created.
<p>The dataspace can also be used to define partial I/O
operations. Since I/O operations have two end-points, the raw
data transfer functions take two data space arguments: one which
describes the application memory data space or subset thereof
and another which describes the file data space or subset
thereof.
<h2>7. Setting Constant or Persistent Properties</h2>
<p>Each dataset has a set of constant and persistent properties
which describe the layout method, pre-compression
transformation, compression method, data type, external storage,
and data space. The constant properties are set as described
above in a dataset creation property list whose identifier is
passed to <code>H5Dcreate()</code>.
<dl>
<dt><code>hid_t H5Dcreate (hid_t <em>file_id</em>, const char
*<em>name</em>, hid_t <em>type_id</em>, hid_t
<em>space_id</em>, hid_t <em>create_plist_id</em>)</code>
<dd>A dataset is created by calling <code>H5Dcreate</code> with
a file identifier, a dataset name, a data type, a data space,
and constant properties. The data type and data space are the
type and space of the dataset as it will exist in the file,
which may be different than in application memory. The
<em>create_plist_id</em> is a <code>H5P_DATASET_CREATE</code>
property list created with <code>H5Pcreate()</code> and
initialized with the various functions described above.
<code>H5Dcreate()</code> returns a dataset handle for success
or negative for failure. The handle should eventually be
closed by calling <code>H5Dclose()</code> to release resources
it uses.
<br><br>
<dt><code>hid_t H5Dopen (hid_t <em>file_id</em>, const char
*<em>name</em>)</code>
<dd>An existing dataset can be opened for access by calling this
function. A dataset handle is returned for success or a
negative value is returned for failure. The handle should
eventually be closed by calling <code>H5Dclose()</code> to
release resources it uses.
<br><br>
<dt><code>herr_t H5Dclose (hid_t <em>dataset_id</em>)</code>
<dd>This function closes a dataset handle and releases all
resources it might have been using. The handle should not be
used in subsequent calls to the library.
<br><br>
<dt><code>herr_t H5Dextend (hid_t <em>dataset_id</em>,
hsize_t <em>dim</em>[])</code>
<dd>This function extends a dataset by increasing the size in
one or more dimensions. Not all datasets can be extended.
</dl>
<h2>8. Querying Constant or Persistent Properties</h2>
<p>Constant or persistent properties can be queried with a set of
three functions. Each function returns an identifier for a copy
of the requested properties. The identifier can be passed to
various functions which modify the underlying object to derive a
new object; the original dataset is completely unchanged. The
return values from these functions should be properly destroyed
when no longer needed.
<dl>
<dt><code>hid_t H5Dget_type (hid_t <em>dataset_id</em>)</code>
<dd>Returns an identifier for a copy of the dataset permanent
data type or negative for failure.
<dt><code>hid_t H5Dget_space (hid_t <em>dataset_id</em>)</code>
<dd>Returns an identifier for a copy of the dataset permanent
data space, which also contains information about the current
size of the dataset if the data set is extendable with
<code>H5Dextend()</code>.
<dt><code>hid_t H5Dget_create_plist (hid_t
<em>dataset_id</em>)</code>
<dd>Returns an identifier for a copy of the dataset creation
property list. The new property list is created by examining
various permanent properties of the dataset. This is mostly a
catch-all for everything but type and space.
</dl>
<h2>9. Setting Memory and Transfer Properties</h2>
<p>A dataset also has memory properties which describe memory
within the application, and transfer properties that control
various aspects of the I/O operations. The memory can have a
data type different than the permanent file data type (different
number types, different struct member offsets, different array
element orderings) and can also be a different size (memory is a
subset of the permanent dataset elements, or vice versa). The
transfer properties might provide caching hints or collective
I/O information. Therefore, each I/O operation must specify
memory and transfer properties.
<p>The memory properties are specified with <em>type_id</em> and
<em>space_id</em> arguments while the transfer properties are
specified with the <em>transfer_id</em> property list for the
<code>H5Dread()</code> and <code>H5Dwrite()</code> functions
(these functions are described below).
<dl>
<dt><code>herr_t H5Pset_buffer (hid_t <em>xfer_plist</em>,
size_t <em>max_buf_size</em>, void *<em>tconv_buf</em>, void
*<em>bkg_buf</em>)</code>
<dt><code>size_t H5Pget_buffer (hid_t <em>xfer_plist</em>, void
**<em>tconv_buf</em>, void **<em>bkg_buf</em>)</code>
<dd>Sets or retrieves the maximum size in bytes of the temporary
buffer used for data type conversion in the I/O pipeline. An
application-defined buffer can also be supplied as the
<em>tconv_buf</em> argument, otherwise a buffer will be
allocated and freed on demand by the library. A second
temporary buffer <em>bkg_buf</em> can also be supplied and
should be the same size as the <em>tconv_buf</em>. The
default values are 1MB for the maximum buffer size, and null
pointers for each buffer indicating that they should be
allocated on demand and freed when no longer needed. The
<code>H5Pget_buffer()</code> function returns the maximum
buffer size or zero on error.
</dl>
<p>If the maximum size of the temporary I/O pipeline buffers is
too small to hold the entire I/O request, then the I/O request
will be fragmented and the transfer operation will be strip
mined. However, certain restrictions apply to the strip
mining. For instance, when performing I/O on a hyperslab of a
simple data space the strip mining is in terms of the slowest
varying dimension. So if a 100x200x300 hyperslab is requested,
the temporary buffer must be large enough to hold a 1x200x300
sub-hyperslab.
<p>To prevent strip mining from happening, the application should
use <code>H5Pset_buffer()</code> to set the size of the
temporary buffer so it's large enough to hold the entire
request.
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example</h4></caption>
<tr>
<td>
<p>This example shows how to define a function that sets
a dataset transfer property list so that strip mining
does not occur. It takes an (optional) dataset transfer
property list, a dataset, a data space that describes
what data points are being transfered, and a data type
for the data points in memory. It returns a (new)
dataset transfer property list with the temporary
buffer size set to an appropriate value. The return
value should be passed as the fifth argument to
<code>H5Dread()</code> or <code>H5Dwrite()</code>.
<p><code><pre>
1 hid_t
2 disable_strip_mining (hid_t xfer_plist, hid_t dataset,
3 hid_t space, hid_t mem_type)
4 {
5 hid_t file_type; /* File data type */
6 size_t type_size; /* Sizeof larger type */
7 size_t size; /* Temp buffer size */
8 hid_t xfer_plist; /* Return value */
9
10 file_type = H5Dget_type (dataset);
11 type_size = MAX(H5Tget_size(file_type), H5Tget_size(mem_type));
12 H5Tclose (file_type);
13 size = H5Sget_npoints(space) * type_size;
14 if (xfer_plist&lt;0) xfer_plist = H5Pcreate (H5P_DATASET_XFER);
15 H5Pset_buffer(xfer_plist, size, NULL, NULL);
16 return xfer_plist;
17 }
</pre></code>
</td>
</tr>
</table>
</center>
<h2>10. Querying Memory or Transfer Properties</h2>
<p>Unlike constant and persistent properties, a dataset cannot be
queried for it's memory or transfer properties. Memory
properties cannot be queried because the application already
stores those properties separate from the buffer that holds the
raw data, and the buffer may hold multiple segments from various
datasets and thus have more than one set of memory properties.
The transfer properties cannot be queried from the dataset
because they're associated with the transfer itself and not with
the dataset (but one can call
<code>H5Pget_<em>property</em>()</code> to query transfer
properties from a tempalate).
<h2>11. Raw Data I/O</h2>
<p>All raw data I/O is accomplished through these functions which
take a dataset handle, a memory data type, a memory data space,
a file data space, transfer properties, and an application
memory buffer. They translate data between the memory data type
and space and the file data type and space. The data spaces can
be used to describe partial I/O operations.
<dl>
<dt><code>herr_t H5Dread (hid_t <em>dataset_id</em>, hid_t
<em>mem_type_id</em>, hid_t <em>mem_space_id</em>, hid_t
<em>file_space_id</em>, hid_t <em>xfer_plist_id</em>,
void *<em>buf</em>/*out*/)</code>
<dd>Reads raw data from the specified dataset into <em>buf</em>
converting from file data type and space to memory data type
and space.
<br><br>
<dt><code>herr_t H5Dwrite (hid_t <em>dataset_id</em>, hid_t
<em>mem_type_id</em>, hid_t <em>mem_space_id</em>, hid_t
<em>file_space_id</em>, hid_t <em>xfer_plist_id</em>,
const void *<em>buf</em>)</code>
<dd>Writes raw data from an application buffer <em>buf</em> to
the specified dataset converting from memory data type and
space to file data type and space.
</dl>
<p>In the name of sharability, the memory datatype must be
supplied. However, it can be the same identifier as was used to
create the dataset or as was returned by
<code>H5Dget_type()</code>; the library will not implicitly
derive memory data types from constant data types.
<p>For complete reads of the dataset one may supply
<code>H5S_ALL</code> as the argument for the file data space.
If <code>H5S_ALL</code> is also supplied as the memory data
space then no data space conversion is performed. This is a
somewhat dangerous situation since the file data space might be
different than what the application expects.
<h2>12. Examples</h2>
<p>The examples in this section illustrate some common dataset
practices.
<p>This example shows how to create a dataset which is stored in
memory as a two-dimensional array of native <code>double</code>
values but is stored in the file in Cray <code>float</code>
format using LZ77 compression. The dataset is written to the
HDF5 file and then read back as a two-dimensional array of
<code>float</code> values.
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example 1</h4></caption>
<tr>
<td>
<p><code><pre>
1 hid_t file, data_space, dataset, properties;
2 double dd[500][600];
3 float ff[500][600];
4 hsize_t dims[2], chunk_size[2];
5
6 /* Describe the size of the array */
7 dims[0] = 500;
8 dims[1] = 600;
9 data_space = H5Screate_simple (2, dims);
10
11
12 /*
13 * Create a new file using with read/write access,
14 * default file creation properties, and default file
15 * access properties.
16 */
17 file = H5Fcreate ("test.h5", H5F_ACC_RDWR, H5P_DEFAULT,
18 H5P_DEFAULT);
19
20 /*
21 * Set the dataset creation plist to specify that
22 * the raw data is to be partitioned into 100x100 element
23 * chunks and that each chunk is to be compressed with
24 * LZ77.
25 */
26 chunk_size[0] = chunk_size[1] = 100;
27 properties = H5Pcreate (H5P_DATASET_CREATE);
28 H5Pset_chunk (properties, 2, chunk_size);
29 H5Pset_compression (properties, H5D_COMPRESS_LZ77);
30
31 /*
32 * Create a new dataset within the file. The data type
33 * and data space describe the data on disk, which may
34 * be different than the format used in the application's
35 * memory.
36 */
37 dataset = H5Dcreate (file, "dataset", H5T_CRAY_FLOAT,
38 data_space, properties);
39
40 /*
41 * Write the array to the file. The data type and data
42 * space describe the format of the data in the `dd'
43 * buffer. The raw data is translated to the format
44 * required on disk defined above. We use default raw
45 * data transfer properties.
46 */
47 H5Dwrite (dataset, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL,
48 H5P_DEFAULT, dd);
49
50 /*
51 * Read the array as floats. This is similar to writing
52 * data except the data flows in the opposite direction.
53 */
54 H5Dread (dataset, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL,
55 H5P_DEFAULT, ff);
56
64 H5Dclose (dataset);
65 H5Sclose (data_space);
66 H5Pclose (properties);
67 H5Fclose (file);
</pre></code>
</td>
</tr>
</table>
</center>
<p>This example uses the file created in Example 1 and reads a
hyperslab of the 500x600 file dataset. The hyperslab size is
100x200 and it is located beginning at element
&lt;200,200&gt;. We read the hyperslab into an 200x400 array in
memory beginning at element &lt;0,0&gt; in memory. Visually,
the transfer looks something like this:
<center>
<img alt="Raw Data Transfer" src="dataset_p1.gif">
</center>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example 2</h4></caption>
<tr>
<td>
<p><code><pre>
1 hid_t file, mem_space, file_space, dataset;
2 double dd[200][400];
3 hssize_t offset[2];
4 hsize size[2];
5
6 /*
7 * Open an existing file and its dataset.
8 */
9 file = H5Fopen ("test.h5", H5F_ACC_RDONLY, H5P_DEFAULT);
10 dataset = H5Dopen (file, "dataset");
11
12 /*
13 * Describe the file data space.
14 */
15 offset[0] = 200; /*offset of hyperslab in file*/
16 offset[1] = 200;
17 size[0] = 100; /*size of hyperslab*/
18 size[1] = 200;
19 file_space = H5Dget_space (dataset);
20 H5Sset_hyperslab (file_space, 2, offset, size);
21
22 /*
23 * Describe the memory data space.
24 */
25 size[0] = 200; /*size of memory array*/
26 size[1] = 400;
27 mem_space = H5Screate_simple (2, size);
28
29 offset[0] = 0; /*offset of hyperslab in memory*/
30 offset[1] = 0;
31 size[0] = 100; /*size of hyperslab*/
32 size[1] = 200;
33 H5Sset_hyperslab (mem_space, 2, offset, size);
34
35 /*
36 * Read the dataset.
37 */
38 H5Dread (dataset, H5T_NATIVE_DOUBLE, mem_space,
39 file_space, H5P_DEFAULT, dd);
40
41 /*
42 * Close/release resources.
43 */
44 H5Dclose (dataset);
45 H5Sclose (mem_space);
46 H5Sclose (file_space);
47 H5Fclose (file);
</pre></code>
</td>
</tr>
</table>
</center>
<p>If the file contains a compound data structure one of whose
members is a floating point value (call it "delta") but the
application is interested in reading an array of floating point
values which are just the "delta" values, then the application
should cast the floating point array as a struct with a single
"delta" member.
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example 3</h4></caption>
<tr>
<td>
<p><code><pre>
1 hid_t file, dataset, type;
2 double delta[200];
3
4 /*
5 * Open an existing file and its dataset.
6 */
7 file = H5Fopen ("test.h5", H5F_ACC_RDONLY, H5P_DEFAULT);
8 dataset = H5Dopen (file, "dataset");
9
10 /*
11 * Describe the memory data type, a struct with a single
12 * "delta" member.
13 */
14 type = H5Tcreate (H5T_COMPOUND, sizeof(double));
15 H5Tinsert (type, "delta", 0, H5T_NATIVE_DOUBLE);
16
17 /*
18 * Read the dataset.
19 */
20 H5Dread (dataset, type, H5S_ALL, H5S_ALL,
21 H5P_DEFAULT, dd);
22
23 /*
24 * Close/release resources.
25 */
26 H5Dclose (dataset);
27 H5Tclose (type);
28 H5Fclose (file);
</pre></code>
</td>
</tr>
</table>
</center>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Tue Dec 2 09:17:09 EST 1997 -->
<!-- hhmts start -->
Last modified: Wed May 13 18:57:47 EDT 1998
<!-- hhmts end -->
</body>
</html>

568
doc/html/Dataspaces.html Normal file

@ -0,0 +1,568 @@
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>The Data Space Interface (H5S)</TITLE>
</HEAD>
<BODY>
<H1>
The Dataspace Interface (H5S)</H1>
<H2>
1. Introduction</H2>
The dataspace interface (H5S) provides a mechanism to describe the positions
of the elements of a dataset and is designed in such a way as to allow
new features to be easily added without disrupting applications that use
the dataspace interface. A dataset (defined with the dataset interface) is
composed of a collection of raw data points of homogeneous type, defined in the
datatype (H5T) interface, organized according to the dataspace with this
interface.
<P>A dataspace describes the locations that dataset elements are located at.
A dataspace is either a regular N-dimensional array of data points,
called a <I>simple</I> dataspace, or a more general collection of data
points organized in another manner, called a <I>complex</I> dataspace.
A <I>scalar</I> dataspace is a special case of the <I>simple</I> data
space and is defined to be a 0-dimensional single data point in size. Currently
only <I>scalar</I> and <I>simple</I> dataspaces are supported with this version
of the H5S interface.
<I>Complex</I> dataspaces will be defined and implemented in a future
version. <I>Complex</I> dataspaces are intended to be used for such structures
which are awkward to express in <I>simple</I> dataspaces, such as irregularly
gridded data or adaptive mesh refinement data. This interface provides
functions to set and query properties of a dataspace.
<P>Operations on a dataspace include defining or extending the extent of
the dataspace, selecting portions of the dataspace for I/O and storing the
dataspaces in the file. The extent of a dataspace is the range of coordinates
over which dataset elements are defined and stored. Dataspace selections are
subsets of the extent (up to the entire extent) which are selected for some
operation.
<P>For example, a 2-dimensional dataspace with an extent of 10 by 10 may have
the following very simple selection:
<br><br><center>
<table border cellpadding=4>
<tr align=center>
<th > </th> <th >0</th> <th >1</th> <th >2</th> <th >3</th> <th >4</th> <th >5</th> <th >6</th> <th >7</th> <th >8</th> <th >9</th>
<tr align=center>
<th>0</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>1</th>
<td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>2</th>
<td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>3</th>
<td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>4</th>
<td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>5</th>
<td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>6</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>7</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>8</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>9</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
</table>
<br><strong>Example 1: Contiguous rectangular selection</strong>
</center>
<br>Or, a more complex selection may be defined:
<br><br><center>
<table border cellpadding=4>
<tr align=center>
<th > </th> <th >0</th> <th >1</th> <th >2</th> <th >3</th> <th >4</th> <th >5</th> <th >6</th> <th >7</th> <th >8</th> <th >9</th>
<tr align=center>
<th>0</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>1</th>
<td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>2</th>
<td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>3</th>
<td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>4</th>
<td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>5</th>
<td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>6</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>7</th>
<td>-</td><td>-</td><td>X</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>8</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>9</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
</table>
<br><strong>Example 2: Non-contiguous selection</strong>
</center>
<P>Selections within dataspaces have an offset within the extent which is used
to locate the selection within the extent of the dataspace. Selection offsets
default to 0 in each dimension, but may be changed to move the selection within
a dataspace. In example 2 above, if the offset was changed to 1,1, the selection
would look like this:
<br><br><center>
<table border cellpadding=4>
<tr align=center>
<th > </th> <th >0</th> <th >1</th> <th >2</th> <th >3</th> <th >4</th> <th >5</th> <th >6</th> <th >7</th> <th >8</th> <th >9</th>
<tr align=center>
<th>0</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>1</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>2</th>
<td>-</td><td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td>
<tr align=center>
<th>3</th>
<td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>4</th>
<td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td>
<tr align=center>
<th>5</th>
<td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>6</th>
<td>-</td><td>-</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td>
<tr align=center>
<th>7</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>8</th>
<td>-</td><td>-</td><td>-</td><td>X</td><td>X</td><td>X</td><td>X</td><td>-</td><td>-</td><td>-</td>
<tr align=center>
<th>9</th>
<td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
</table>
<br><strong>Example 3: Non-contiguous selection with 1,1 offset</strong>
</center>
<P>Selections also have an linearization ordering of the points selected
(defaulting to "C" order, ie. last dimension changing fastest). The
linearization order may be specified for each point or it may be chosen by
the axis of the dataspace. For example, with the default "C" ordering,
example 1's selected points are iterated through in this order: (1,1), (2,1),
(3,1), (1,2), (2,2), etc. With "FORTRAN" ordering, example 1's selected points
would be iterated through in this order: (1,1), (1,2), (1,3), (1,4), (1,5),
(2,1), (2,2), etc.
<P>A dataspace may be stored in the file as a permanent object, to allow many
datasets to use a commonly defined dataspace. Dataspaces with extendable
extents (ie. unlimited dimensions) are not able to be stored as permanent
dataspaces.
<P>Dataspaces may be created using an existing permanent dataspace as a
container to locate the new dataspace within. These dataspaces are complete
dataspaces and may be used to define datasets. A dataspaces with a "parent"
can be queried to determine the parent dataspace and the location within the
parent. These dataspaces must currently be the same number of dimensions as
the parent dataspace.
<H2> 2. General Dataspace Operations</H2>
The functions defined in this section operate on dataspaces as a whole.
New dataspaces can be created from scratch or copied from existing data
spaces. When a dataspace is no longer needed its resources should be released
by calling <TT>H5Sclose()</TT>.
<DL>
<DT>
<TT>hid_t H5Screate(H5S_class_t <I>type</I>)</TT></DT>
<DD>
This function creates a new dataspace of a particular <I>type</I>. The
types currently supported are H5S_SCALAR, H5S_SIMPLE, or H5S_NONE, although
others are planned to be added later. The H5S_NONE dataspace can only hold a
selection, not an extent.
</DD>
<DT>
<TT>hid_t H5Sopen(hid_t <I>location</I>, const char *<I>name</I>)</TT></DT>
<DD>
This function opens a permanent dataspace for use in an application.
The <I>location</I> argument is a file or group ID and <I>name</I> is
an absolute or relative path to the permanent dataspace. The dataspace ID which
is returned is a handle to a permanent dataspace which can't be modified.
</DD>
<DT>
<TT>hid_t H5Scopy (hid_t <I>space</I>)</TT></DT>
<DD>
This function creates a new dataspace which is an exact copy of the
dataspace <I>space</I>.
</DD>
<DT>
<TT>hid_t H5Ssubspace (hid_t <I>space</I>)</TT></DT>
<DD>
This function uses the currently defined selection and offset in <I>space</I>
to create a dataspace which is located within <I>space</I>. The <I>space</I>
dataspace must be a sharable dataspace located in the file, not a dataspace for
a dataset. The relationship of the new dataspace within the existing dataspace
is preserved when the new dataspace is used to create datasets. Currently,
only subspaces which are equivalent to simple dataspaces (ie. rectangular
contiguous areas) are allowed. A subspace is not "simplified" or reduced in
the number of dimensions used if the selection is "flat" in one dimension, they
always have the same number of dimensions as their parent dataspace.
</DD>
<DT>
<TT>herr_t H5Scommit (hid_t <I>location</I>, const char *<I>name</I>, hid_t <I>space</I>)</TT></DT>
<DD>
The dataspaces specified with <I>space</I> is stored in the file specified
by <I>location</I>. <I>Location</I> may be either a file or group handle
and <I>name</I> is an absolute or relative path to the location to store the
dataspace. After this call, the dataspace is permanent and can't be modified.
</DD>
<DT>
<TT>herr_t H5Sclose (hid_t <I>space</I>)</TT></DT>
<DD>
Releases resources associated with a dataspace. Subsequent use of the
dataspace identifier after this call is undefined.
</DD>
<DT>
<TT>H5S_class_t H5Sget_class (hid_t <I>space</I>)</TT></DT>
<DD>
Query a dataspace to determine the current class of a dataspace. The value
which is returned is one of: H5S_SCALAR, H5S_SIMPLE, or H5S_NONE on success or
FAIL on failure.
</DD>
</DL>
<H2> 3. Dataspace Extent Operations</H2>
These functions operate on the extent portion of a dataspace.
<DL>
<DT>
<TT>herr_t H5Sset_extent_simple (hid_t <I>space</I>, int <I>rank</I>, const hsize_t
*<I>current_size</I>, const hsize_t *<I>maximum_size</I>)</TT></DT>
<DD>
Sets or resets the size of an existing dataspace, where <I>rank</I> is
the dimensionality, or number of dimensions, of the dataspace.
<I>current_size</I> is an array of size <I>rank</I> which contains the new size
of each dimension in the dataspace. <I>maximum_size</I> is an array of size
<I>rank</I> which contains the maximum size of each dimension in the dataspace.
Any previous extent is removed from the dataspace, the dataspace type is set to
H5S_SIMPLE and the extent is set as specified.
</DD>
<DT>
<TT>herr_t H5Sset_extent_none (hid_t <I>space</I>)</TT></DT>
<DD>
Removes the extent from a dataspace and sets the type to H5S_NONE.
</DD>
<DT>
<TT>herr_t H5Sextent_copy (hid_t <I>dest_space</I>,
hid_t <I>source_space</I>)</TT></DT>
<DD>
Copies the extent from <I>source_space</I> to <I>dest_space</I>, which may
change the type of the dataspace. Returns non-negative on success, negative on
failure.
</DD>
<DT>
<TT>hsize_t H5Sextent_npoints (hid_t <I>space</I>)</TT></DT>
<DD>
This function determines the number of elements in a dataspace. For example, a
simple 3-dimensional dataspace with dimensions 2, 3 and 4 would have 24
elements.
Returns the number of elements in the dataspace, negative on failure.
</DD>
<DT>
<TT>int H5Sextent_ndims (hid_t <I>space</I>)</TT></DT>
<DD>
This function determines the dimensionality (or rank) of a dataspace.
Returns the number of dimensions in the dataspace, negative on failure.
</DD>
<DT>
<TT>herr_t H5Sextent_dims (hid_t <I>space</I>, hsize_t *<I>dims</I>,
hsize_t *<I>max</I>)</TT></DT>
<DD>
The function retrieves the size of the extent of the dataspace <I>space</I> by
placing the size of each dimension in the array <I>dims</I>. Also retrieves
the size of the maximum extent of the dataspace, placing the results in
<I>max</I>.
Returns non-negative on success, negative on failure.
</DD>
</DL>
<H2> 4. Dataspace Selection Operations</H2>
Selections are maintained separately from extents in dataspaces and operations
on the selection of a dataspace do not affect the extent of the dataspace.
Selections are independent of extent type and the boundaries of selections are
reconciled with the extent at the time of the data transfer. Selection offsets
apply a selection to a location within an extent, allowing the same selection
to be moved within the extent without requiring a new selection to be specified.
Offsets default to 0 when the dataspace is created. Offsets are applied when
an I/O transfer is performed (and checked during calls to H5Sselect_valid).
Selections have an iteration order for the points selected, which can be any
permutation of the dimensions involved (defaulting to 'C' array order) or a
specific order for the selected points, for selections composed of single array
elements with H5Sselect_elements. Selections can also be copied or combined
together in various ways with H5Sselect_op. Further methods of selecting
portions of a dataspace may be added in the future.
<DL>
<DT>
<TT>herr_t H5Sselect_hyperslab (hid_t <I>space</I>, h5s_selopt_t <I>op</I>,
const hssize_t * <I>start</I>, const hsize_t * <I>stride</I>,
const hsize_t * <I>count</I>, const hsize_t * <I>block</I>)</TT></DT>
<DD>
This function selects a hyperslab region to add to the current selected region
for the <I>space</I> dataspace. The <I>start</I>, <I>stride</I>, <I>count</I>
and <I>block</I> arrays must be the same size as the rank of the dataspace.
The selection operator <I>op</I> determines how the new selection is to be
combined with the already existing selection for the dataspace. Currently,
only the H5S_SELECT_SET operator is supported, which replaces the existing
selection with the parameters from this call. Overlapping blocks are not
supported with the H5S_SELECT_SET operator.
<P>The <I>start</I> array determines the starting coordinates of the hyperslab
to select. The <I>stride</I> array chooses array locations from the dataspace
with each value in the <I>stride</I> array determining how many elements to move
in each dimension. Setting a value in the <I>stride</I> array to 1 moves to
each element in that dimension of the dataspace, setting a value of 2 in a
location in the <I>stride</I> array moves to every other element in that
dimension of the dataspace. In other words, the <I>stride</I> determines the
number of elements to move from the <I>start</I> location in each dimension.
Stride values of 0 are not allowed. If the <I>stride</I> parameter is <TT>NULL</TT>,
a contiguous hyperslab is selected (as if each value in the <I>stride</I> array
was set to all 1's). The <I>count</I> array determines how many blocks to
select from the dataspace, in each dimension. The <I>block</I> array determines
the size of the element block selected from the dataspace. If the <I>block</I>
parameter is set to <TT>NULL</TT>, the block size defaults to a single element
in each dimension (as if the <I>block</I> array was set to all 1's).
<P>For example, in a 2-dimensional dataspace, setting <I>start</I> to [1,1],
<I>stride</I> to [4,4], <I>count</I> to [3,7] and <I>block</I> to [2,2] selects
21 2x2 blocks of array elements starting with location (1,1) and selecting
blocks at locations (1,1), (5,1), (9,1), (1,5), (5,5), etc.
<P>Regions selected with this function call default to 'C' order iteration when
I/O is performed.
</DD>
<DT>
<TT>herr_t H5Sselect_elements (hid_t <I>space</I>, h5s_selopt_t <I>op</I>,
const size_t <I>num_elements</I>, const hssize_t *<I>coord</I>[])</TT></DT>
<DD>
This function selects array elements to be included in the selection for the
<I>space</I> dataspace. The number of elements selected must be set with the
<I>num_elements</I>. The <I>coord</I> array is a two-dimensional array of size
&lt;dataspace rank&gt; by &lt;<I>num_elements</I>&gt; in size (ie. a list of
coordinates in the array). The order of the element coordinates in the
<I>coord</I> array also specifies the order that the array elements are
iterated through when I/O is performed. Duplicate coordinate locations are not
checked for.
<P>The selection operator <I>op</I> determines how the new selection is to be
combined with the already existing selection for the dataspace. Currently,
only the H5S_SELECT_SET operator is supported, which replaces the existing
selection with the parameters from this call. When operators other than
H5S_SELECT_SET are used to combine a new selection with an existing selection,
the selection ordering is reset to 'C' array ordering.
</DD>
<DT>
<TT>herr_t H5Sselect_all (hid_t <I>space</I>)</TT></DT>
<DD>
This function selects the special H5S_SELECT_ALL region for the <I>space</I>
dataspace. H5S_SELECT_ALL selects the entire dataspace for any dataspace is is
applied to.
</DD>
<DT>
<TT>herr_t H5Sselect_none (hid_t <I>space</I>)</TT></DT>
<DD>
This function resets the selection region for the <I>space</I>
dataspace not to include any elements.
</DD>
<DT>
<TT>herr_t H5Sselect_op (hid_t <I>space1</I>, h5s_selopt_t <I>op</I>,
hid_t <I>space2</I>)</TT></DT>
<DD>
Uses <I>space2</I> to perform an operation on <I>space1</I>. The valid
operations for <I>op</I> are:
<DL>
<DT>H5S_SELECT_COPY
<DD>Copies the selection from <I>space2</I> into <I>space1</I>, removing any
previously defined selection for <I>space1</I>. The selection order
and offset are also copied to <I>space1</I>
<DT>H5S_SELECT_UNION
<DD>Performs a set union of the selection of the dataspace <I>space2</I>
with the selection from the dataspace <I>space1</I>, with the result
being stored in <I>space1</I>. The selection order for <I>space1</I> is
reset to 'C' order.
<DT>H5S_SELECT_INTERSECT
<DD>Performs an set intersection of the selection from <I>space2</I> with
<I>space1</I>, with the result being stored in <I>space1</I>. The
selection order for <I>space1</I> is reset to 'C' order.
<DT>H5S_SELECT_DIFFERENCE
<DD>Performs a set difference of the selection from <I>space2</I> with
<I>space1</I>, with the result being stored in <I>space1</I>. The
selection order for <I>space1</I> is reset to 'C' order.
</DL>
</DD>
<DT>
<TT>herr_t H5Sselect_order (hid_t <I>space</I>,
hsize_t <I>perm_vector</I>[])</TT></DT>
<DD>
This function selects the order to iterate through the dimensions of a dataspace
when performing I/O on a selection. If a specific order has already been
selected for the selection with H5Sselect_elements, this function will remove
it and use a dimension oriented ordering on the selected elements. The elements
of the <I>perm_vector</I> array must be unique and between 0 and the rank of the
dataspace, minus 1. The order of the elements in <I>perm_vector</I> specify
the order to iterate through the selection for each dimension of the dataspace.
To iterate through a 3-dimensional dataspace selection in 'C' order, specify
the elements of the <I>perm_vector</I> as [0, 1, 2], for FORTRAN order they
would be [2, 1, 0]. Other orderings, such as [1, 2, 0] are also possible, but
may execute slower.
</DD>
<DT>
<TT>hbool_t H5Sselect_valid (hid_t <I>space</I>)</TT></DT>
<DD>
This function verifies that the selection for a dataspace is within the extent
of the dataspace, if the currently set offset for the dataspace is used.
Returns TRUE if the selection is contained within the extent, FALSE if it
is not contained within the extent and FAIL on error conditions (such as if
the selection or extent is not defined).
</DD>
<DT>
<TT>hsize_t H5Sselect_npoints (hid_t <I>space</I>)</TT></DT>
<DD>
This function determines the number of elements in the current selection
of a dataspace.
</DD>
<DT>
<TT>herr_t H5Soffset_simple (hid_t <I>space</I>, const hssize_t *
<I>offset</I>)</TT></DT>
<DD>
Sets the offset of a simple dataspace <I>space</I>. The <I>offset</I> array
must be the same number of elements as the number of dimensions for the
dataspace. If the <I>offset</I> array is set to <TT>NULL</TT>, the offset
for the dataspace is reset to 0.
</DD>
</DL>
<H2> 5. Misc. Dataspace Operations</H2>
<DL>
<DT>
<TT>herr_t H5Slock (hid_t <I>space</I>)</TT></DT>
<DD>
Locks the dataspace so that it cannot be modified or closed. When the library
exits, the dataspace will be unlocked and closed.
</DD>
<DT>
<TT>hid_t H5Screate_simple(int <I>rank</I>, const hsize_t *<I>current_size</I>,
const hsize_t *<I>maximum_size</I>)</TT></DT>
<DD>
This function is a "convenience" wrapper to create a simple dataspace
and set it's extent in one call. It is equivalent to calling H5Screate
and H5Sset_extent_simple() in two steps.
</DD>
<DT>
<TT>int H5Sis_subspace(hid_t <I>space</I>)</TT></DT>
<DD>
This function returns positive if <I>space</I> is located within another
dataspace, zero if it is not, and negative on a failure.
</DD>
<DT>
<TT>char *H5Ssubspace_name(hid_t <I>space</I>)</TT></DT>
<DD>
This function returns the name of the named dataspace that <I>space</I>
is located within. If <I>space</I> is not located within another dataspace,
or an error occurs, NULL is returned. The application is responsible for
freeing the string returned.
</DD>
<DT>
<TT>herr_t H5Ssubspace_location(hid_t <I>space</I>, hsize_t *<I>loc</I>)</TT></DT>
<DD>
If <I>space</I> is located within another dataspace, this function puts
the location of the origin of <I>space</I> in the <I>loc</I> array. The <I>loc</I>
array must be at least as large as the number of dimensions of <I>space</I>.
If <I>space</I> is not located within another dataspace
or an error occurs, a negative value is returned, otherwise a non-negative value
is returned.
</DD>
</DL>
<HR>
<ADDRESS>
<A HREF="mailto:matzke@llnl.gov">Robb Matzke</A></ADDRESS>
<ADDRESS>
<A HREF="mailto:koziol@ncsa.uiuc.edu">Quincey Koziol</A></ADDRESS>
<BR><!-- Created: Thu Dec 4 14:57:32 EST 1997 --><!-- hhmts start -->Last
modified: Thu May 28 15:12:04 EST 1998&nbsp;<!-- hhmts end -->
</BODY>
</HTML>

1370
doc/html/Datatypes.html Normal file

File diff suppressed because it is too large Load Diff

281
doc/html/Errors.html Normal file

@ -0,0 +1,281 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>The Error Handling Interface (H5E)</title>
</head>
<body>
<h1>The Error Handling Interface (H5E)</h1>
<h2>1. Introduction</h2>
<p>When an error occurs deep within the HDF5 library a record is
pushed onto an error stack and that function returns a failure
indication. Its caller detects the failure, pushes another
record onto the stack, and returns a failure indication. This
continues until the application-called API function returns a
failure indication (a negative integer or null pointer). The
next API function which is called (with a few exceptions) resets
the stack.
<p>In normal circumstances, an error causes the stack to be
printed on the standard error stream. The first item, number
"#000" is produced by the API function itself and is usually
sufficient to indicate to the application programmer what went
wrong.
<p>
<center>
<table border align=center width="100%">
<caption align=top><h4>Example: An Error Message</h4></caption>
<tr>
<td>
<p>If an application calls <code>H5Tclose</code> on a
predefined data type then the following message is
printed on the standard error stream. This is a
simple error that has only one component, the API
function; other errors may have many components.
<p><code><pre>
HDF5-DIAG: Error detected in thread 0. Back trace follows.
#000: H5T.c line 462 in H5Tclose(): predefined data type
major(01): Function argument
minor(05): Bad value
</code></pre>
</td>
</tr>
</table>
</center>
<p>The error stack can also be printed and manipulated by these
functions, but if an application wishes make explicit calls to
<code>H5Eprint()</code> then the automatic printing should be
turned off to prevent error messages from being displayed twice
(see <code>H5Eset_auto()</code> below).
<dl>
<dt><code>herr_t H5Eprint (FILE *<em>stream</em>)</code>
<dd>The error stack is printed on the specified stream. Even if
the error stack is empty a one-line message will be printed:
<code>HDF5-DIAG: Error detected in thread 0.</code>
<br><br>
<dt><code>herr_t H5Eclear (void)</code>
<dd>The error stack can be explicitly cleared by calling this
function. The stack is also cleared whenever an API function
is called, with certain exceptions (for instance,
<code>H5Eprint()</code>).
</dl>
<p>Sometimes an application will call a function for the sake of
its return value, fully expecting the function to fail. Under
these conditions, it would be misleading if an error message
were automatically printed. Automatic printing of messages is
controlled by the <code>H5Eset_auto()</code> function:
<dl>
<dt><code>herr_t H5Eset_auto (herr_t(*<em>func</em>)(void*),
void *<em>client_data</em>)</code>
<dd>If <em>func</em> is not a null pointer, then the function to
which it points will be called automatically when an API
function is about to return an indication of failure. The
function is called with a single argument, the
<em>client_data</em> pointer. When the library is first
initialized the auto printing function is set to
<code>H5Eprint()</code> (cast appropriately) and
<em>client_data</em> is the standard error stream pointer,
<code>stderr</code>.
<br><br>
<dt><code>herr_t H5Eget_auto (herr_t(**<em>func</em>)(void*),
void **<em>client_data</em>)</code>
<dd>This function returns the current automatic error traversal
settings through the <em>func</em> and <em>client_data</em>
arguments. Either (or both) arguments may be null pointers in
which case the corresponding information is not returned.
</dl>
<p>
<center>
<table border align=center width="100%">
<caption align=top><h4>Example: Error Control</h4></caption>
<tr>
<td>
<p>An application can temporarily turn off error
messages while "probing" a function.
<p><code><pre>
/* Save old error handler */
herr_t (*old_func)(void*);
void *old_client_data;
H5Eget_auto(&amp;old_func, &amp;old_client_data);
/* Turn off error handling */
H5Eset_auto(NULL, NULL);
/* Probe. Likely to fail, but that's okay */
status = H5Fopen (......);
/* Restore previous error handler */
H5Eset_auto(old_func, old_client_data);
</code></pre>
<p>Or automatic printing can be disabled altogether and
error messages can be explicitly printed.
<p><code><pre>
/* Turn off error handling permanently */
H5Eset_auto (NULL, NULL);
/* If failure, print error message */
if (H5Fopen (....)<0) {
H5Eprint (stderr);
exit (1);
}
</code></pre>
</td>
</tr>
</table>
</center>
<p>The application is allowed to define an automatic error
traversal function other than the default
<code>H5Eprint()</code>. For instance, one could define a
function that prints a simple, one-line error message to the
standard error stream and then exits.
<p>
<center>
<table border align=center width="100%">
<caption align=top><h4>Example: Simple Messages</h4></caption>
<tr>
<td>
<p>The application defines a function to print a simple
error message to the standard error stream.
<p><code><pre>
herr_t
my_hdf5_error_handler (void *unused)
{
fprintf (stderr, "An HDF5 error was detected. Bye.\n");
exit (1);
}
</code></pre>
<p>The function is installed as the error handler by
saying
<p><code><pre>
H5Eset_auto (my_hdf5_error_handler, NULL);
</code></pre>
</td>
</tr>
</table>
</center>
<p>The <code>H5Eprint()</code> function is actually just a wrapper
around the more complex <code>H5Ewalk()</code> function which
traverses an error stack and calls a user-defined function for
each member of the stack.
<dl>
<dt><code>herr_t H5Ewalk (H5E_direction_t <em>direction</em>,
H5E_walk_t <em>func</em>, void *<em>client_data</em>)</code>
<dd>The error stack is traversed and <em>func</em> is called for
each member of the stack. Its arguments are an integer
sequence number beginning at zero (regardless of
<em>direction</em>), a pointer to an error description record,
and the <em>client_data</em> pointer. If <em>direction</em>
is <code>H5E_WALK_UPWARD</em> then traversal begins at the
inner-most function that detected the error and concludes with
the API function. The opposite order is
<code>H5E_WALK_DOWNWARD</code>.
<br><br>
<dt><code>typedef herr_t (*H5E_walk_t)(int <em>n</em>,
H5E_error_t *<em>eptr</em>, void
*<em>client_data</em>)</code>
<dd>An error stack traversal callback function takes three
arguments: <em>n</em> is a sequence number beginning at zero
for each traversal, <em>eptr</em> is a pointer to an error
stack member, and <em>client_data</em> is the same pointer
passed to <code>H5Ewalk()</code>.
<br><br>
<dt><pre><code>typedef struct {
H5E_major_t <em>maj_num</em>;
H5E_minor_t <em>min_num</em>;
const char *<em>func_name</em>;
const char *<em>file_name</em>;
unsigned <em>line</em>;
const char *<em>desc</em>;
} H5E_error_t;</code></pre>
<dd>The <em>maj_num</em> and <em>min_num</em> are major
and minor error numbers, <em>func_name</em> is the name of
the function where the error was detected,
<em>file_name</em> and <em>line</em> locate the error
within the HDF5 library source code, and <em>desc</em>
points to a description of the error.
<br><br>
<dt><code>const char *H5Eget_major (H5E_major_t <em>num</em>)</code>
<dt><code>const char *H5Eget_minor (H5E_minor_t <em>num</em>)</code>
<dd>These functions take a major or minor error number and
return a constant string which describes the error. If
<em>num</em> is out of range than a string like "Invalid major
error number" is returned.
</dl>
<p>
<center>
<table border align=center width="100%">
<caption align=top><h4>Example: H5Ewalk_cb</h4></caption>
<tr>
<td>
<p>This is the implementation of the default error stack
traversal callback.
<p><code><pre>
herr_t
H5Ewalk_cb(int n, H5E_error_t *err_desc, void *client_data)
{
FILE *stream = (FILE *)client_data;
const char *maj_str = NULL;
const char *min_str = NULL;
const int indent = 2;
/* Check arguments */
assert (err_desc);
if (!client_data) client_data = stderr;
/* Get descriptions for the major and minor error numbers */
maj_str = H5Eget_major (err_desc-&gt;maj_num);
min_str = H5Eget_minor (err_desc-&gt;min_num);
/* Print error message */
fprintf (stream, "%*s#%03d: %s line %u in %s(): %s\n",
indent, "", n, err_desc-&gt;file_name, err_desc-&gt;line,
err_desc-&gt;func_name, err_desc-&gt;desc);
fprintf (stream, "%*smajor(%02d): %s\n",
indent*2, "", err_desc-&gt;maj_num, maj_str);
fprintf (stream, "%*sminor(%02d): %s\n",
indent*2, "", err_desc-&gt;min_num, min_str);
return 0;
}
</code></pre>
</td>
</tr>
</table>
</center>
<hr>
<address><a href="mailto:hdf5dev@ncsa.uiuc.edu">Robb Matzke</a></address>
<!-- Created: Fri Feb 27 23:42:52 EST 1998 -->
<!-- hhmts start -->
Last modified: Wed Mar 4 10:06:17 EST 1998
<!-- hhmts end -->
</body>
</html>

278
doc/html/ExternalFiles.html Normal file

@ -0,0 +1,278 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>External Files in HDF5</title>
</head>
<body>
<center><h1>External Files in HDF5</h1></center>
<h3>Overview of Layers</h3>
<p>This table shows some of the layers of HDF5. Each layer calls
functions at the same or lower layers and never functions at
higher layers. An object identifier (OID) takes various forms
at the various layers: at layer 0 an OID is an absolute physical
file address; at layers 1 and 2 it's an absolute virtual file
address. At layers 3 through 6 it's a relative address, and at
layers 7 and above it's an object handle.
<p><center>
<table border cellpadding=4 width="60%">
<tr align=center>
<td>Layer-7</td>
<td>Groups</td>
<td>Datasets</td>
</tr>
<tr align=center>
<td>Layer-6</td>
<td>Indirect Storage</td>
<td>Symbol Tables</td>
</tr>
<tr align=center>
<td>Layer-5</td>
<td>B-trees</td>
<td>Object Hdrs</td>
<td>Heaps</td>
</tr>
<tr align=center>
<td>Layer-4</td>
<td>Caching</td>
</tr>
<tr align=center>
<td>Layer-3</td>
<td>H5F chunk I/O</td>
</tr>
<tr align=center>
<td>Layer-2</td>
<td>H5F low</td>
</tr>
<tr align=center>
<td>Layer-1</td>
<td>File Family</td>
<td>Split Meta/Raw</td>
</tr>
<tr align=center>
<td>Layer-0</td>
<td>Section-2 I/O</td>
<td>Standard I/O</td>
<td>Malloc/Free</td>
</tr>
</table>
</center>
<h3>Single Address Space</h3>
<p>The simplest form of hdf5 file is a single file containing only
hdf5 data. The file begins with the boot block, which is
followed until the end of the file by hdf5 data. The next most
complicated file allows non-hdf5 data (user defined data or
internal wrappers) to appear before the boot block and after the
end of the hdf5 data. The hdf5 data is treated as a single
linear address space in both cases.
<p>The next level of complexity comes when non-hdf5 data is
interspersed with the hdf5 data. We handle that by including
the non-hdf5 interspersed data in the hdf5 address space and
simply not referencing it (eventually we might add those
addresses to a "do-not-disturb" list using the same mechanism as
the hdf5 free list, but it's not absolutely necessary). This is
implemented except for the "do-not-disturb" list.
<p>The most complicated single address space hdf5 file is when we
allow the address space to be split among multiple physical
files. For instance, a >2GB file can be split into smaller
chunks and transfered to a 32 bit machine, then accessed as a
single logical hdf5 file. The library already supports >32 bit
addresses, so at layer 1 we split a 64-bit address into a 32-bit
file number and a 32-bit offset (the 64 and 32 are
arbitrary). The rest of the library still operates with a linear
address space.
<p>Another variation might be a family of two files where all the
meta data is stored in one file and all the raw data is stored
in another file to allow the HDF5 wrapper to be easily replaced
with some other wrapper.
<p>The <code>H5Fcreate</code> and <code>H5Fopen</code> functions
would need to be modified to pass file-type info down to layer 2
so the correct drivers can be called and parameters passed to
the drivers to initialize them.
<h4>Implementation</h4>
<p>I've implemented fixed-size family members. The entire hdf5
file is partitioned into members where each member is the same
size. The family scheme is used if one passes a name to
<code>H5F_open</code> (which is called by <code>H5Fopen()</code>
and <code>H5Fcreate</code>) that contains a
<code>printf(3c)</code>-style integer format specifier.
Currently, the default low-level file driver is used for all
family members (H5F_LOW_DFLT, usually set to be Section 2 I/O or
Section 3 stdio), but we'll probably eventually want to pass
that as a parameter of the file access template, which hasn't
been implemented yet. When creating a family, a default family
member size is used (defined at the top H5Ffamily.c, currently
64MB) but that also should be settable in the file access
template. When opening an existing family, the size of the first
member is used to determine the member size (flushing/closing a
family ensures that the first member is the correct size) but
the other family members don't have to be that large (the local
address space, however, is logically the same size for all
members).
<p>I haven't implemented a split meta/raw family yet but am rather
curious to see how it would perform. I was planning to use the
`.h5' extension for the meta data file and `.raw' for the raw
data file. The high-order bit in the address would determine
whether the address refers to meta data or raw data. If the user
passes a name that ends with `.raw' to <code>H5F_open</code>
then we'll chose the split family and use the default low level
driver for each of the two family members. Eventually we'll
want to pass these kinds of things through the file access
template instead of relying on naming convention.
<h3>External Raw Data</h3>
<p>We also need the ability to point to raw data that isn't in the
HDF5 linear address space. For instance, a dataset might be
striped across several raw data files.
<p>Fortunately, the only two packages that need to be aware of
this are the packages for reading/writing contiguous raw data
and discontiguous raw data. Since contiguous raw data is a
special case, I'll discuss how to implement external raw data in
the discontiguous case.
<p>Discontiguous data is stored as a B-tree whose keys are the
chunk indices and whose leaf nodes point to the raw data by
storing a file address. So what we need is some way to name the
external files, and a way to efficiently store the external file
name for each chunk.
<p>I propose adding to the object header an <em>External File
List</em> message that is a 1-origin array of file names.
Then, in the B-tree, each key has an index into the External
File List (or zero for the HDF5 file) for the file where the
chunk can be found. The external file index is only used at
the leaf nodes to get to the raw data (the entire B-tree is in
the HDF5 file) but because of the way keys are copied among
the B-tree nodes, it's much easier to store the index with
every key.
<h3>Multiple HDF5 Files</h3>
<p>One might also want to combine two or more HDF5 files in a
manner similar to mounting file systems in Unix. That is, the
group structure and meta data from one file appear as though
they exist in the first file. One opens File-A, and then
<em>mounts</em> File-B at some point in File-A, the <em>mount
point</em>, so that traversing into the mount point actually
causes one to enter the root object of File-B. File-A and
File-B are each complete HDF5 files and can be accessed
individually without mounting them.
<p>We need a couple additional pieces of machinery to make this
work. First, an haddr_t type (a file address) doesn't contain
any info about which HDF5 file's address space the address
belongs to. But since haddr_t is an opaque type except at
layers 2 and below, it should be quite easy to add a pointer to
the HDF5 file. This would also remove the H5F_t argument from
most of the low-level functions since it would be part of the
OID.
<p>The other thing we need is a table of mount points and some
functions that understand them. We would add the following
table to each H5F_t struct:
<p><code><pre>
struct H5F_mount_t {
H5F_t *parent; /* Parent HDF5 file if any */
struct {
H5F_t *f; /* File which is mounted */
haddr_t where; /* Address of mount point */
} *mount; /* Array sorted by mount point */
intn nmounts; /* Number of mounted files */
intn alloc; /* Size of mount table */
}
</pre></code>
<p>The <code>H5Fmount</code> function takes the ID of an open
file, the name of a to-be-mounted file, the name of the mount
point, and a file access template (like <code>H5Fopen</code>).
It opens the new file and adds a record to the parent's mount
table. The <code>H5Funmount</code> function takes the parent
file ID and the name of the mount point and closes the file
that's mounted at that point. The <code>H5Fclose</code>
function closes/unmounts files recursively.
<p>The <code>H5G_iname</code> function which translates a name to
a file address (<code>haddr_t</code>) looks at the mount table
at each step in the translation and switches files where
appropriate. All name-to-address translations occur through
this function.
<h3>How Long?</h3>
<p>I'm expecting to be able to implement the two new flavors of
single linear address space in about two days. It took two hours
to implement the malloc/free file driver at level zero and I
don't expect this to be much more work.
<p>I'm expecting three days to implement the external raw data for
discontiguous arrays. Adding the file index to the B-tree is
quite trivial; adding the external file list message shouldn't
be too hard since the object header message class from wich this
message derives is fully implemented; and changing
<code>H5F_istore_read</code> should be trivial. Most of the
time will be spent designing a way to cache Unix file
descriptors efficiently since the total number open files
allowed per process could be much smaller than the total number
of HDF5 files and external raw data files.
<p>I'm expecting four days to implement being able to mount one
HDF5 file on another. I was originally planning a lot more, but
making <code>haddr_t</code> opaque turned out to be much easier
than I planned (I did it last Fri). Most of the work will
probably be removing the redundant H5F_t arguments for lots of
functions.
<h3>Conclusion</h3>
<p>The external raw data could be implemented as a single linear
address space, but doing so would require one to allocate large
enough file addresses throughout the file (>32bits) before the
file was created. It would make mixing an HDF5 file family with
external raw data, or external HDF5 wrapper around an HDF4 file
a more difficult process. So I consider the implementation of
external raw data files as a single HDF5 linear address space a
kludge.
<p>The ability to mount one HDF5 file on another might not be a
very important feature especially since each HDF5 file must be a
complete file by itself. It's not possible to stripe an array
over multiple HDF5 files because the B-tree wouldn't be complete
in any one file, so the only choice is to stripe the array
across multiple raw data files and store the B-tree in the HDF5
file. On the other hand, it might be useful if one file
contains some public data which can be mounted by other files
(e.g., a mesh topology shared among collaborators and mounted by
files that contain other fields defined on the mesh). Of course
the applications can open the two files separately, but it might
be more portable if we support it in the library.
<p>So we're looking at about two weeks to implement all three
versions. I didn't get a chance to do any of them in AIO
although we had long-term plans for the first two with a
possibility of the third. They'll be much easier to implement in
HDF5 than AIO since I've been keeping these in mind from the
start.
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Sat Nov 8 18:08:52 EST 1997 -->
<!-- hhmts start -->
Last modified: Wed Nov 12 15:01:14 EST 1997
<!-- hhmts end -->
</body>
</html>

529
doc/html/Files.html Normal file

@ -0,0 +1,529 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>HDF5 Files</title>
</head>
<body>
<h1>Files</h1>
<h2>1. Introduction</h2>
<p>HDF5 files are composed of a "boot block" describing information
required to portably access files on multiple platforms, followed
by information about the groups in a file and the datasets in the
file. The boot block contains information about the size of offsets
and lengths of objects, the number of entries in symbol tables
(used to store groups) and additional version information for the
file.
<h2>2. File access modes</h2>
<p>The HDF5 library assumes that all files are implicitly opened for read
access at all times. Passing the <code>H5F_ACC_RDWR</code>
parameter to <code>H5Fopen()</code> allows write access to a
file also. <code>H5Fcreate()</code> assumes write access as
well as read access, passing <code>H5F_ACC_TRUNC</code> forces
the truncation of an existing file, otherwise H5Fcreate will
fail to overwrite an existing file.
<h2>3. Creating, Opening, and Closing Files</h2>
<p>Files are created with the <code>H5Fcreate()</code> function,
and existing files can be accessed with <code>H5Fopen()</code>. Both
functions return an object ID which should be eventually released by
calling <code>H5Fclose()</code>.
<dl>
<dt><code>hid_t H5Fcreate (const char *<em>name</em>, uintn
<em>flags</em>, hid_t <em>create_properties</em>, hid_t
<em>access_properties</em>)</code>
<dd>This function creates a new file with the specified name in
the current directory. The file is opened with read and write
permission, and if the <code>H5F_ACC_TRUNC</code> flag is set,
any current file is truncated when the new file is created.
If a file of the same name exists and the
<code>H5F_ACC_TRUNC</code> flag is not set (or the
<code>H5F_ACC_EXCL</code> bit is set), this function will
fail. Passing <code>H5P_DEFAULT</code> for the creation
and/or access property lists uses the library's default
values for those properties. Creating and changing the
values of a property list is documented further below. The
return value is an ID for the open file and it should be
closed by calling <code>H5Fclose()</code> when it's no longer
needed. A negative value is returned for failure.
<br><br>
<dt><code>hid_t H5Fopen (const char *<em>name</em>, uintn
<em>flags</em>, hid_t <em>access_properties</em>)</code>
<dd>This function opens an existing file with read permission
and write permission if the <code>H5F_ACC_RDWR</code> flag is
set. The <em>access_properties</em> is a file access property
list ID or <code>H5P_DEFAULT</code> for the default I/O access
parameters. Creating and changing the parameters for access
templates is documented further below. Files which are opened
more than once return a unique identifier for each
<code>H5Fopen()</code> call and can be accessed through all
file IDs. The return value is an ID for the open file and it
should be closed by calling <code>H5Fclose()</code> when it's
no longer needed. A negative value is returned for failure.
<br><br>
<dt><code>herr_t H5Fclose (hid_t <em>file_id</em>)</code>
<dd>This function releases resources used by a file which was
opened by <code>H5Fcreate()</code> or <code>H5Fopen()</code>. After
closing a file the <em>file_id</em> should not be used again. This
function returns zero for success or a negative value for failure.
</dl>
<h2>4. File Property Lists</h2>
<p>Additional parameters to <code>H5Fcreate()</code> or
<code>H5Fopen()</code> are passed through property list
objects, which are created with the <code>H5Pcreate()</code>
function. These objects allow many parameters of a file's
creation or access to be changed from the default values.
Property lists are used as a portable and extensible method of
modifying multiple parameter values with simple API functions.
There are two kinds of file-related property lists,
namely file creation properties and file access properties.
<h3>4.1. File Creation Properties</h3>
<P>File creation property lists apply to <code>H5Fcreate()</code> only
and are used to control the file meta-data which is maintained
in the boot block of the file. The parameters which can be
modified are:
<dl>
<dt>User-Block Size <dd>The "user-block" is a fixed length block of
data located at the beginning of the file which is ignored by the
HDF5 library and may be used to store any data information found
to be useful to applications. This value may be set to any power
of two equal to 512 or greater (i.e. 512, 1024, 2048, etc). This
parameter is set and queried with the
<code>H5Pset_userblock()</code> and
<code>H5Pget_userblock()</code> calls.
<br><br>
<dt>Offset and Length Sizes
<dd>The number of bytes used to store the offset and length of
objects in the HDF5 file can be controlled with this
parameter. Values of 2, 4 and 8 bytes are currently
supported to allow 16-bit, 32-bit and 64-bit files to
be addressed. These parameters are set and queried
with the <code>H5Pset_sizes()</code> and
<code>H5Pget_sizes()</code> calls.
<br><br>
<dt>Symbol Table Parameters
<dd>The size of symbol table B-trees can be controlled by setting
the 1/2 rank and 1/2 node size parameters of the B-tree. These
parameters are set and queried with the
<code>H5Pset_sym_k()</code> and <code>H5Pget_sym_k()</code> calls.
<br><br>
<dt>Indexed Storage Parameters
<dd>The size of indexed storage B-trees can be controlled by
setting the 1/2 rank and 1/2 node size parameters of the B-tree.
These parameters are set and queried with the
<code>H5Pset_istore_k()</code> and <code>H5Pget_istore_k()</code>
calls.
</dl>
<h3>4.2. File Access Property Lists</h3>
<p>File access property lists apply to <code>H5Fcreate()</code> or
<code>H5Fopen()</code> and are used to control different methods of
performing I/O on files.
<dl>
<dt>Unbuffered I/O
<dd>Local permanent files can be accessed with the functions described
in Section 2 of the Posix manual, namely <code>open()</code>,
<code>lseek()</code>, <code>read()</code>, <code>write()</code>, and
<code>close()</code>. The <code>lseek64()</code> function is used
on operating systems that support it. This driver is enabled and
configured with <code>H5Pset_sec2()</code>, and queried with
<code>H5Pget_sec2()</code>.
<br><br>
<dt>Buffered I/O
<dd>Local permanent files can be accessed with the functions declared
in the <code>stdio.h</code> header file, namely
<code>fopen()</code>, <code>fseek()</code>, <code>fread()</code>,
<code>fwrite()</code>, and <code>fclose()</code>. The
<code>fseek64()</code> function is used on operating systems that
support it. This driver is enabled and configured with
<code>H5Pset_stdio()</code>, and queried with
<code>H5Pget_stdio()</code>.
<br><br>
<dt>Memory I/O
<dd>Local temporary files can be created and accessed directly from
memory without ever creating permanent storage. The library uses
<code>malloc()</code> and <code>free()</code> to create storage
space for the file. The total size of the file must be small enough
to fit in virtual memory. The name supplied to
<code>H5Fcreate()</code> is irrelevant, and <code>H5Fopen()</code>
will always fail.
<br><br>
<dt>Parallel Files using MPI I/O
<dd>This driver allows parallel access to a file through the MPI I/O
library. The parameters which can be modified are the MPI
communicator, the info object, and the access mode.
The communicator and info object are saved and then
passed to <code>MPI_File_open()</code> during file creation or open.
The access_mode controls the kind of parallel access the application
intends. (Note that it is likely that the next API revision will
remove the access_mode parameter and have access control specified
via the raw data transfer property list of <code>H5Dread()</code>
and <code>H5Dwrite()</code>.) These parameters are set and queried
with the <code>H5Pset_mpi()</code> and <code>H5Pget_mpi()</code>
calls.
<br><br>
<dt>Data Alignment
<dd>Sometimes file access is faster if certain things are
aligned on file blocks. This can be controlled by setting
alignment properties of a file access property list with the
<code>H5Pset_alignment()</code> function. Any allocation
request at least as large as some threshold will be aligned on
an address which is a multiple of some number.
</dl> </ul>
<h2>5. Examples of using file templates</h2>
<h3>5.1. Example of using file creation templates</h3>
<p>This following example shows how to create a file with 64-bit object
offsets and lengths:<br>
<pre>
hid_t create_template;
hid_t file_id;
create_template = H5Pcreate(H5P_FILE_CREATE);
H5Pset_sizes(create_template, 8, 8);
file_id = H5Fcreate("test.h5", H5F_ACC_TRUNC,
create_template, H5P_DEFAULT);
.
.
.
H5Fclose(file_id);
</pre>
<h3>5.2. Example of using file creation templates</h3>
<p>This following example shows how to open an existing file for
independent datasets access by MPI parallel I/O:<br>
<pre>
hid_t access_template;
hid_t file_id;
access_template = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_mpi(access_template, MPI_COMM_WORLD, MPI_INFO_NULL);
/* H5Fopen must be called collectively */
file_id = H5Fopen("test.h5", H5F_ACC_RDWR, access_template);
.
.
.
/* H5Fclose must be called collectively */
H5Fclose(file_id);
</pre>
<h2>6. Low-level File Drivers</h2>
<p>HDF5 is able to access its address space through various types of
low-level <em>file drivers</em>. For instance, an address space might
correspond to a single file on a Unix file system, multiple files on a
Unix file system, multiple files on a parallel file system, or a block
of memory within the application. Generally, an HDF5 address space is
referred to as an "HDF5 file" regardless of how the space is organized
at the storage level.
<h3>6.1 Unbuffered Permanent Files</h3>
<p>The <em>sec2</em> driver uses functions from section 2 of the
Posix manual to access files stored on a local file system. These are
the <code>open()</code>, <code>close()</code>, <code>read()</code>,
<code>write()</code>, and <code>lseek()</code> functions. If the
operating system supports <code>lseek64()</code> then it is used instead
of <code>lseek()</code>. The library buffers meta data regardless of
the low-level driver, but using this driver prevents data from being
buffered again by the lowest layers of the HDF5 library.
<dl>
<dt><code>H5F_driver_t H5Pget_driver (hid_t
<em>access_properties</em>)</code>
<dd>This function returns the constant <code>H5F_LOW_SEC2</code> if the
<em>sec2</em> driver is defined as the low-level driver for the
specified access property list.
<br><br>
<dt><code>herr_t H5Pset_sec2 (hid_t <em>access_properties</em>)</code>
<dd>The file access properties are set to use the <em>sec2</em>
driver. Any previously defined driver properties are erased from the
property list. Additional parameters may be added to this function in
the future.
<br><br>
<dt><code>herr_t H5Pget_sec2 (hid_t <em>access_properties</em>)</code>
<dd>If the file access property list is set to the <em>sec2</em> driver
then this function returns zero; otherwise it returns a negative
value. In the future, additional arguments may be added to this
function to match those added to <code>H5Pset_sec2()</code>.
</dl>
<h3>6.2 Buffered Permanent Files</h3>
<p>The <em>stdio</em> driver uses the functions declared in the
<code>stdio.h</code> header file to access permanent files in a local
file system. These are the <code>fopen()</code>, <code>fclose()</code>,
<code>fread()</code>, <code>fwrite()</code>, and <code>fseek()</code>
functions. If the operating system supports <code>fseek64()</code> then
it is used instead of <code>fseek()</code>. Use of this driver
introduces an additional layer of buffering beneath the HDF5 library.
<dl>
<dt><code>H5F_driver_t H5Pget_driver(hid_t
<em>access_properties</em>)</code>
<dd>This function returns the constant <code>H5F_LOW_STDIO</code> if the
<em>stdio</em> driver is defined as the low-level driver for the
specified access property list.
<br><br>
<dt><code>herr_t H5Pset_stdio (hid_t <em>access_properties</em>)</code>
<dd>The file access properties are set to use the <em>stdio</em>
driver. Any previously defined driver properties are erased from the
property list. Additional parameters may be added to this function in
the future.
<br><br>
<dt><code>herr_t H5Pget_stdio (hid_t <em>access_properties</em>)</code>
<dd>If the file access property list is set to the <em>stdio</em> driver
then this function returns zero; otherwise it returns a negative
value. In the future, additional arguments may be added to this
function to match those added to <code>H5Pset_stdio()</code>.
</dl>
<h3>6.3 Buffered Temporary Files</h3>
<p>The <em>core</em> driver uses <code>malloc()</code> and
<code>free()</code> to allocated space for a file in the heap. Reading
and writing to a file of this type results in mem-to-mem copies instead
of disk I/O and as a result is somewhat faster. However, the total file
size must not exceed the amount of available virtual memory, and only
one HDF5 file handle can access the file (because the name of such a
file is insignificant and <code>H5Fopen()</code> always fails).
<dl>
<dt><code>H5F_driver_t H5Pget_driver (hid_t
<em>access_properties</em>)</code>
<dd>This function returns the constant <code>H5F_LOW_CORE</code> if the
<em>core</em> driver is defined as the low-level driver for the
specified access property list.
<br><br>
<dt><code>herr_t H5Pset_core (hid_t <em>access_properties</em>, size_t
<em>block_size</em>)</code>
<dd>The file access properties are set to use the <em>core</em>
driver and any previously defined driver properties are erased from
the property list. Memory for the file will always be allocated in
units of the specified <em>block_size</em>. Additional parameters may
be added to this function in the future.
<br><br>
<dt><code>herr_t H5Pget_core (hid_t <em>access_properties</em>, size_t
*<em>block_size</em>)</code>
<dd>If the file access property list is set to the <em>core</em> driver
then this function returns zero and <em>block_size</em> is set to the
block size used for the file; otherwise it returns a negative
value. In the future, additional arguments may be added to this
function to match those added to <code>H5Pset_core()</code>.
</dl>
<h3>6.4 Parallel Files</h3>
<p>This driver uses MPI I/O to provide parallel access to a file.
<dl>
<dt><code>H5F_driver_t H5Pget_driver (hid_t
<em>access_properties</em>)</code>
<dd>This function returns the constant <code>H5F_LOW_MPI</code> if the
<em>mpi</em> driver is defined as the low-level driver for the
specified access property list.
<br><br>
<dt><code>herr_t H5Pset_mpi (hid_t <em>access_properties</em>, MPI_Comm
<em>comm</em>, MPI_info <em>info</em>)</code>
<dd>The file access properties are set to use the <em>mpi</em>
driver and any previously defined driver properties are erased from
the property list. Additional parameters may be added to this
function in the future.
<br><br>
<dt><code>herr_t H5Pget_mpi (hid_t <em>access_properties</em>, MPI_Comm
*<em>comm</em>, MPI_info *<em>info</em>)</code>
<dd>If the file access property list is set to the <em>mpi</em> driver
then this function returns zero and <em>comm</em>, and <em>info</em>
are set to the values stored in the property
list; otherwise the function returns a negative value. In the future,
additional arguments may be added to this function to match those
added to <code>H5Pset_mpi()</code>.
</dl>
<h3>6.4 File Families</h3>
<p>A single HDF5 address space may be split into multiple files which,
together, form a file family. Each member of the family must be the
same logical size although the size and disk storage reported by
<code>ls</code>(1) may be substantially smaller. The name passed to
<code>H5Fcreate()</code> or <code>H5Fopen()</code> should include a
<code>printf(3c)</code> style integer format specifier which will be
replaced with the family member number (the first family member is
zero).
<p>Any HDF5 file can be split into a family of files by running
the file through <code>split</code>(1) and numbering the output
files. However, because HDF5 is lazy about extending the size
of family members, a valid file cannot generally be created by
concatenation of the family members. Additionally,
<code>split</code> and <code>cat</code> don't attempt to
generate files with holes. The <code>h5repart</code> program
can be used to repartition an HDF5 file or family into another
file or family and preserves holes in the files.
<dl>
<dt><code>h5repart</code> [<code>-v</code>] [<code>-b</code>
<em>block_size</em>[<em>suffix</em>]] [<code>-m</code>
<em>member_size</em>[<em>suffix</em>]] <em>source
destination</em>
<dd>This program repartitions an HDF5 file by copying the source
file or family to the destination file or family preserving
holes in the underlying Unix files. Families are used for the
source and/or destination if the name includes a
<code>printf</code>-style integer format such as "%d". The
<code>-v</code> switch prints input and output file names on
the standard error stream for progress monitoring,
<code>-b</code> sets the I/O block size (the default is 1kB),
and <code>-m</code> sets the output member size if the
destination is a family name (the default is 1GB). The block
and member sizes may be suffixed with the letters
<code>g</code>, <code>m</code>, or <code>k</code> for GB, MB,
or kB respectively.
<br><br>
<dt><code>H5F_driver_t H5Pget_driver (hid_t
<em>access_properties</em>)</code>
<dd>This function returns the constant <code>H5F_LOW_FAMILY</code> if
the <em>family</em> driver is defined as the low-level driver for the
specified access property list.
<br><br>
<dt><code>herr_t H5Pset_family (hid_t <em>access_properties</em>,
hsize_t <em>memb_size</em>, hid_t <em>member_properties</em>)</code>
<dd>The file access properties are set to use the <em>family</em>
driver and any previously defined driver properties are erased
from the property list. Each member of the file family will
use <em>member_properties</em> as its file access property
list. The <em>memb_size</em> argument gives the logical size
in bytes of each family member but the actual size could be
smaller depending on whether the file contains holes. The
member size is only used when creating a new file or
truncating an existing file; otherwise the member size comes
from the size of the first member of the family being
opened. Note: if the size of the <code>off_t</code> type is
four bytes then the maximum family member size is usually
2^31-1 because the byte at offset 2,147,483,647 is generally
inaccessable. Additional parameters may be added to this
function in the future.
<br><br>
<dt><code>herr_t H5Pget_family (hid_t <em>access_properties</em>,
hsize_t *<em>memb_size</em>, hid_t
*<em>member_properties</em>)</code>
<dd>If the file access property list is set to the <em>family</em>
driver then this function returns zero; otherwise the function
returns a negative value. On successful return,
<em>access_properties</em> will point to a copy of the member
access property list which should be closed by calling
<code>H5Pclose()</em> when the application is finished with
it. If <em>memb_size</em> is non-null then it will contain
the logical size in bytes of each family member. In the
future, additional arguments may be added to this function to
match those added to <code>H5Pset_family()</code>.
</dl>
<h3>6.5 Split Meta/Raw Files</h3>
<p>On occasion, it might be useful to separate meta data from raw
data. The <em>split</em> driver does this by creating two files: one for
meta data and another for raw data. The application provides a base
file name to <code>H5Fcreate()</code> or <code>H5Fopen()</code> and this
driver appends a file extension which defaults to ".meta" for the meta
data file and ".raw" for the raw data file. Each file can have its own
file access property list which allows, for instance, a split file with
meta data stored with the <em>core</em> driver and raw data stored with
the <em>sec2</em> driver.
<dl>
<dt><code>H5F_driver_t H5Pget_driver (hid_t
<em>access_properties</em>)</code>
<dd>This function returns the constant <code>H5F_LOW_SPLIT</code> if
the <em>split</em> driver is defined as the low-level driver for the
specified access property list.
<br><br>
<dt><code>herr_t H5Pset_split (hid_t <em>access_properties</em>,
const char *<em>meta_extension</em>, hid_t
<em>meta_properties</em>, const char *<em>raw_extension</em>, hid_t
<em>raw_properties</em>)</code>
<dd>The file access properties are set to use the <em>split</em>
driver and any previously defined driver properties are erased from
the property list. The meta file will have a name which is formed by
adding <em>meta_extension</em> (or ".meta") to the end of the base
name and will be accessed according to the
<em>meta_properties</em>. The raw file will have a name which is
formed by appending <em>raw_extension</em> (or ".raw") to the base
name and will be accessed according to the <em>raw_properties</em>.
Additional parameters may be added to this function in the future.
<br><br>
<dt><code>herr_t H5Pget_split (hid_t <em>access_properties</em>,
size_t <em>meta_ext_size</em>, const char *<em>meta_extension</em>,
hid_t <em>meta_properties</em>, size_t <em>raw_ext_size</em>, const
char *<em>raw_extension</em>, hid_t *<em>raw_properties</em>)</code>
<dd>If the file access property list is set to the <em>split</em>
driver then this function returns zero; otherwise the function
returns a negative value. On successful return,
<em>meta_properties</em> and <em>raw_properties</em> will
point to copies of the meta and raw access property lists
which should be closed by calling <code>H5Pclose()</em> when
the application is finished with them, but if the meta and/or
raw file has no property list then a negative value is
returned for that property list handle. Also, if
<em>meta_extension</em> and/or <em>raw_extension</em> are
non-null pointers, at most <em>meta_ext_size</em> or
<em>raw_ext_size</em> characters of the meta or raw file name
extension will be copied to the specified buffer. If the
actual name is longer than what was requested then the result
will not be null terminated (similar to
<code>strncpy()</code>). In the future, additional arguments
may be added to this function to match those added to
<code>H5Pset_split()</code>.
</dl>
<hr>
<address><a href="mailto:koziol@ncsa.uiuc.edu">Quincey Koziol</a></address>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Tue Jan 27 09:11:27 EST 1998 -->
<!-- hhmts start -->
Last modified: Tue Jun 9 15:03:44 EDT 1998
<!-- hhmts end -->
</body>
</html>

288
doc/html/Groups.html Normal file

@ -0,0 +1,288 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Groups</title>
</head>
<body>
<h1>Groups</h1>
<h2>1. Introduction</h2>
<p>An object in HDF5 consists of an object header at a fixed file
address that contains messages describing various properties of
the object such as its storage location, layout, compression,
etc. and some of these messages point to other data such as the
raw data of a dataset. The address of the object header is also
known as an <em>OID</em> and HDF5 has facilities for translating
names to OIDs.
<p>Every HDF5 object has at least one name and a set of names can
be stored together in a group. Each group implements a name
space where the names are any length and unique with respect to
other names in the group.
<p>Since a group is a type of HDF5 object it has an object header
and a name which exists as a member of some other group. In this
way, groups can be linked together to form a directed graph.
One particular group is called the <em>Root Group</em> and is
the group to which the HDF5 file boot block points. Its name is
"/" by convention. The <em>full name</em> of an object is
created by joining component names with slashes much like Unix.
<p>
<center>
<img alt="Group Graph Example" src="group_p1.gif">
</center>
<p>However, unlike Unix which arranges directories hierarchically,
HDF5 arranges groups in a directed graph. Therefore, there is
no ".." entry in a group since a group can have more than one
parent. There is no "." entry either but the library understands
it internally.
<h2>2. Names</h2>
<p>HDF5 places few restrictions on names: component names may be
any length except zero and may contain any character except
slash ("/") and the null terminator. A full name may be
composed of any number of component names separated by slashes,
with any of the component names being the special name ".". A
name which begins with a slash is an <em>absolute</em> name
which is looked up beginning at the root group of the file while
all other <em>relative</em> names are looked up beginning at the
current working group (described below) or a specified group.
Multiple consecutive slashes in a full name are treated as
single slashes and trailing slashes are not significant. A
special case is the name "/" (or equivalent) which refers to the
root group.
<p>Functions which operate on names generally take a location
identifier which is either a file ID or a group ID and perform
the lookup with respect to that location. Some possibilities
are:
<p>
<center>
<table border cellpadding=4>
<tr>
<th>Location Type</th>
<th>Object Name</th>
<th>Description</th>
</tr>
<tr>
<td>File ID</td>
<td><code>/foo/bar</code></td>
<td>The object <code>bar</code> in group <code>foo</code>
in the root group of the specified file.</td>
</tr>
<tr>
<td>Group ID</td>
<td><code>/foo/bar</code></td>
<td>The object <code>bar</code> in group <code>foo</code>
in the root group of the file containing the specified
group. In other words, the group ID's only purpose is
to supply a file.</td>
</tr>
<tr>
<td>File ID</td>
<td><code>/</code></td>
<td>The root group of the specified file.</td>
</tr>
<tr>
<td>Group ID</td>
<td><code>/</code></td>
<td>The root group of the file containing the specified
group.</td>
</tr>
<tr>
<td>File ID</td>
<td><code>foo/bar</code></td>
<td>The object <code>bar</code> in group <code>foo</code>
in the current working group of the specified file. The
initial current working group is the root group of the
file as described below.</td>
</tr>
<tr>
<td>Group ID</td>
<td><code>foo/bar</code></td>
<td>The object <code>bar</code> in group <code>foo</code>
in the specified group.</td>
</tr>
<tr>
<td>File ID</td>
<td><code>.</code></td>
<td>The current working group of the specified file.</td>
</tr>
<tr>
<td>Group ID</td>
<td><code>.</code></td>
<td>The specified group.</td>
</tr>
<tr>
<td>Other ID</td>
<td><code>.</code></td>
<td>The specified object.</td>
</tr>
</table>
</center>
<h2>3. Creating, Opening, and Closing Groups</h2>
<p>Groups are created with the <code>H5Gcreate()</code> function,
and existing groups can be access with
<code>H5Gopen()</code>. Both functions return an object ID which
should be eventually released by calling
<code>H5Gclose()</code>.
<dl>
<dt><code>hid_t H5Gcreate (hid_t <em>location_id</em>, const char
*<em>name</em>, size_t <em>size_hint</em>)</code>
<dd>This function creates a new group with the specified
name at the specified location which is either a file ID or a
group ID. The name must not already be taken by some other
object and all parent groups must already exist. The
<em>size_hint</em> is a hint for the number of bytes to
reserve to store the names which will be eventually added to
the new group. Passing a value of zero for <em>size_hint</em>
is usually adequate since the library is able to dynamically
resize the name heap, but a correct hint may result in better
performance. The return value is a handle for the open group
and it should be closed by calling <code>H5Gclose()</code>
when it's no longer needed. A negative value is returned for
failure.
<br><br>
<dt><code>hid_t H5Gopen (hid_t <em>location_id</em>, const char
*<em>name</em>)</code>
<dd>This function opens an existing group with the specified
name at the specified location which is either a file ID or a
group ID and returns an object ID. The object ID should be
released by calling <code>H5Gclose()</code> when it is no
longer needed. A negative value is returned for failure.
<br><br>
<dt><code>herr_t H5Gclose (hid_t <em>group_id</em>)</code>
<dd>This function releases resources used by an group which was
opened by <code>H5Gcreate()</code> or
<code>H5Gopen()</code>. After closing a group the
<em>group_id</em> should not be used again. This function
returns zero for success or a negative value for failure.
</dl>
<h2>4. Current Working Group</h2>
<p>Each file handle (<code>hid_t <em>file_id</em></code>) has a
current working group, initially the root group of the file.
Names which do not begin with a slash are relative to the
specified group or to the current working group as described
above. For instance, the name "/Foo/Bar/Baz" is resolved by
first looking up "Foo" in the root group. But the name
"Foo/Bar/Baz" is resolved by first looking up "Foo" in the
current working group.
<dl>
<dt><code>herr_t H5Gset (hid_t <em>location_id</em>, const char
*<em>name</em>)</code>
<dd>The group with the specified name is made the current
working group for the file which contains it. The
<em>location_id</em> can be a file handle or a group handle
and the name is resolved as described above. Each file handle
has it's own current working group and if the
<em>location_id</em> is a group handle then the file handle is
derived from the group handle. This function returns zero for
success or negative for failure.
<br><br>
<dt><code>herr_t H5Gpush (hid_t <em>location_id</em>, const char
*<em>name</em>)</code>
<dd>Each file handle has a stack of groups and the top group on
that stack is the current working group. The stack initially
contains only the root group. This function pushes a new
group onto the stack and returns zero for success or negative
for failure.
<br><br>
<dt><code>herr_t H5Gpop (hid_t <em>location_id</em>)</code>
<dd>This function pops one group off the group stack for the
specified file (if the <em>location_id</em> is a group then
the file is derived from that group), changing the current
working group to the new top-of-stack group. The function
returns zero for success or negative for failure (failure
includes attempting to pop from an empty stack). If the last
item is popped from the stack then the current working group
is set to the root group.
</dl>
<h2>5. Objects with Multiple Names</h2>
<p>An object (including a group) can have more than one
name. Creating the object gives it the first name, and then
functions described here can be used to give it additional
names. The association between a name and the object is called
a <em>link</em> and HDF5 supports two types of links: a <em>hard
link</em> is a direct association between the name and the
object where both exist in a single HDF5 address space, and a
<em>soft link</em> is an indirect association.
<p>
<center>
<img alt="Hard Link Example" src="group_p2.gif">
</center>
<p>
<center>
<img alt="Soft Link Example" src="group_p3.gif">
</center>
<dl>
<dt>Object Creation</dt>
<dd>The creation of an object creates a hard link which is
indistinguishable from other hard links that might be added
later.
<br><br>
<dt><code>herr_t H5Glink (hid_t <em>file_id</em>, H5G_link_t
<em>link_type</em>, const char *<em>current_name</em>,
const char *<em>new_name</em>)</code>
<dd>Creates a new name for an object that has some current name
(possibly one of many names it currently has). If the
<em>link_type</em> is <code>H5G_LINK_HARD</code> then a new
hard link is created. Otherwise if <em>link_type</em> is
<code>H5T_LINK_SOFT</code> a soft link is created which is an
alias for the <em>current_name</em>. When creating a soft
link the object need not exist. This function returns zero
for success or negative for failure. <b>This function is not
part of the prototype API.</b>
<br><br>
<dt><code>herr_t H5Gunlink (hid_t <em>file_id</em>, const char
*<em>name</em>)</code>
<dd>This function removes an association between a name and an
object. Object headers keep track of how many hard links refer
to the object and when the hard link count reaches zero the
object can be removed from the file (but objects which are
open are not removed until all handles to the object are
closed). <b>This function is not part of the prototype
API.</b>
</dl>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Tue Jan 27 09:11:27 EST 1998 -->
<!-- hhmts start -->
Last modified: Tue Mar 24 15:52:14 EST 1998
<!-- hhmts end -->
</body>
</html>

4611
doc/html/H5.api.html Normal file

File diff suppressed because it is too large Load Diff

849
doc/html/H5.api_map.html Normal file

@ -0,0 +1,849 @@
<html><head><title>
HDF5 Legacy API Equivalence
</title></head><body>
<center>
<h1>HDF5: API Mapping to legacy APIs</h1>
</center>
<table border=1 cellpadding=2 cellspacing=0>
<tr>
<th>Functionality</th>
<th>netCDF</th>
<th>SD</th>
<th>AIO</th>
<th>HDF5</th>
<th>Comments</th>
</tr>
<tr align=center>
<td>Open existing file for read/write</td>
<td>ncopen</td>
<td>SDstart</td>
<td>AIO_open</td>
<td>H5Fopen</td>
</tr>
<tr align=center>
<td>Creates new file for read/write.</td>
<td>nccreate</td>
<td><hr></td>
<td><hr></td>
<td>H5Fcreate</td>
<td>SD API handles this with SDopen</td>
</tr>
<tr align=center>
<td>Close file</td>
<td>ncclose</td>
<td>SDend</td>
<td>AIO_close</td>
<td>H5Fclose</td>
</tr>
<tr align=center>
<td>Redefine parameters</td>
<td>ncredef</td>
<td><hr></td>
<td><hr></td>
<td><hr></td>
<td>Unneccessary under SD & HDF5 data-models</td>
</tr>
<tr align=center>
<td>End "define" mode</td>
<td>ncendef</td>
<td><hr></td>
<td><hr></td>
<td><hr></td>
<td>Unneccessary under SD & HDF5 data-models</td>
</tr>
<tr align=center>
<td>Query the number of datasets, dimensions and attributes in a file</td>
<td>ncinquire</td>
<td>SDfileinfo</td>
<td><hr></td>
<td>H5Dget_info<br>H5Rget_num_relations<br>H5Gget_num_contents</td>
<td>HDF5 interface is more granular and flexible</td>
</tr>
<tr align=center>
<td>Update a writeable file with current changes</td>
<td>ncsync</td>
<td><hr></td>
<td>AIO_flush</td>
<td>H5Mflush</td>
<td>HDF5 interface is more flexible because it can be applied to parts of the
file hierarchy instead of the whole file at once. The SD interface does not
have this feature, although most of the lower HDF library supports it.</td>
</tr>
<tr align=center>
<td>Close file access without applying recent changes</td>
<td>ncabort</td>
<td><hr></td>
<td><hr></td>
<td><hr></td>
<td>How useful is this feature?</td>
</tr>
<tr align=center>
<td>Create new dimension</td>
<td>ncdimdef</td>
<td>SDsetdimname</td>
<td><hr></td>
<td>H5Mcreate</td>
<td>SD interface actually creates dimensions with datasets, this just allows
naming them</td>
</tr>
<tr align=center>
<td>Get ID of existing dimension</td>
<td>ncdimid</td>
<td>SDgetdimid</td>
<td><hr></td>
<td>H5Maccess</td>
<td>SD interface looks up dimensions by index and the netCDF interface uses
names, but they are close enough. The HDF5 interface does not current allow
access to particular dimensions, only the dataspace as a whole.</td>
</tr>
<tr align=center>
<td>Get size & name of dimension</td>
<td>ncdiminq</td>
<td>SDdiminfo</td>
<td><hr></td>
<td>H5Mget_name<br>H5Sget_lrank</td>
<td>Only a rough match</td>
</tr>
<tr align=center>
<td>Rename dimension</td>
<td>ncdimrename</td>
<td>SDsetdimname</td>
<td><hr></td>
<td>H5Mset_name</td>
<td></td>
</tr>
<tr align=center>
<td>Create a new dataset</td>
<td>ncvardef</td>
<td>SDcreate</td>
<td>AIO_mkarray</td>
<td>H5Mcreate</td>
<td></td>
</tr>
<tr align=center>
<td>Attach to an existing dataset</td>
<td>ncvarid</td>
<td>SDselect</td>
<td>AIO_arr_load</td>
<td>H5Maccess</td>
<td></td>
</tr>
<tr align=center>
<td>Get basic information about a dataset</td>
<td>ncvarinq</td>
<td>SDgetinfo</td>
<td>AIO_arr_get_btype<br>AIO_arr_get_nelmts<br>AIO_arr_get_nbdims<br>AIO_arr_get_bdims<br>AIO_arr_get_slab</td>
<td>H5Dget_info</td>
<td>All interfaces have different levels of information that they return, some
use of auxilliary functions is required to get equivalent amount of information</td>
</tr>
<tr align=center>
<td>Write a single value to a dataset</td>
<td>ncvarput1</td>
<td>SDwritedata</td>
<td>AIO_write</td>
<td>H5Dwrite</td>
<td>What is this useful for?</td>
</tr>
<tr align=center>
<td>Read a single value from a dataset</td>
<td>ncvarget1</td>
<td>SDreaddata</td>
<td>AIO_read</td>
<td>H5Dread</td>
<td>What is this useful for?</td>
</tr>
<tr align=center>
<td>Write a solid hyperslab of data (i.e. subset) to a dataset</td>
<td>ncvarput</td>
<td>SDwritedata</td>
<td>AIO_write</td>
<td>H5Dwrite</td>
<td></td>
</tr>
<tr align=center>
<td>Read a solid hyperslab of data (i.e. subset) from a dataset</td>
<td>ncvarget</td>
<td>SDreaddata</td>
<td>AIO_read</td>
<td>H5Dread</td>
<td></td>
</tr>
<tr align=center>
<td>Write a general hyperslab of data (i.e. possibly subsampled) to a dataset</td>
<td>ncvarputg</td>
<td>SDwritedata</td>
<td>AIO_write</td>
<td>H5Dwrite</td>
<td></td>
</tr>
<tr align=center>
<td>Read a general hyperslab of data (i.e. possibly subsampled) from a dataset</td>
<td>ncvargetg</td>
<td>SDreaddata</td>
<td>AIO_read</td>
<td>H5Dread</td>
<td></td>
</tr>
<tr align=center>
<td>Rename a dataset variable</td>
<td>ncvarrename</td>
<td><hr></td>
<td><hr></td>
<td>H5Mset_name</td>
<td></td>
</tr>
<tr align=center>
<td>Add an attribute to a dataset</td>
<td>ncattput</td>
<td>SDsetattr</td>
<td><hr></td>
<td>H5Rattach_oid</td>
<td>HDF5 requires creating a seperate object to attach to a dataset, but it also
allows objects to be attributes of any other object, even nested.</td>
</tr>
<tr align=center>
<td>Get attribute information</td>
<td>ncattinq</td>
<td>SDattrinfo</td>
<td><hr></td>
<td>H5Dget_info</td>
<td>HDF5 has no specific function for attributes, they are treated as all other
objects in the file.</td>
</tr>
<tr align=center>
<td>Retrieve attribute for a dataset</td>
<td>ncattget</td>
<td>SDreadattr</td>
<td><hr></td>
<td>H5Dread</td>
<td>HDF5 uses general dataset I/O for attributes.</td>
</tr>
<tr align=center>
<td>Copy attribute from one dataset to another</td>
<td>ncattcopy</td>
<td><hr></td>
<td><hr></td>
<td><hr></td>
<td>What is this used for?</td>
</tr>
<tr align=center>
<td>Get name of attribute</td>
<td>ncattname</td>
<td>SDattrinfo</td>
<td><hr></td>
<td>H5Mget_name</td>
<td></td>
</tr>
<tr align=center>
<td>Rename attribute</td>
<td>ncattrename</td>
<td><hr></td>
<td><hr></td>
<td>H5Mset_name</td>
<td></td>
</tr>
<tr align=center>
<td>Delete attribute</td>
<td>ncattdel</td>
<td><hr></td>
<td><hr></td>
<td>H5Mdelete</td>
<td>This can be faked in current HDF interface with lower-level calls</td>
</tr>
<tr align=center>
<td>Compute # of bytes to store a number-type</td>
<td>nctypelen</td>
<td>DFKNTsize</td>
<td><hr></td>
<td><hr></td>
<td>Hmm, the HDF5 Datatype interface needs this functionality.</td>
</tr>
<tr align=center>
<td>Indicate that fill-values are to be written to dataset</td>
<td>ncsetfill</td>
<td>SDsetfillmode</td>
<td><hr></td>
<td><hr></td>
<td>HDF5 Datatype interface should work on this functionality</td>
</tr>
<tr align=center>
<td>Get information about "record" variables (Those datasets which share the
same unlimited dimension</td>
<td>ncrecinq</td>
<td><hr></td>
<td><hr></td>
<td><hr></td>
<td>This should probably be wrapped in a higher layer interface, if it's
needed for HDF5.</td>
</tr>
<tr align=center>
<td>Get a record from each dataset sharing the unlimited dimension</td>
<td>ncrecget</td>
<td><hr></td>
<td><hr></td>
<td><hr></td>
<td>This is somewhat equivalent to reading a vdata with non-interlaced
fields, only in a dataset oriented way. This should also be wrapped in a
higher layer interface if it's necessary for HDF5.</td>
</tr>
<tr align=center>
<td>Put a record from each dataset sharing the unlimited dimension</td>
<td>ncrecput</td>
<td><hr></td>
<td><hr></td>
<td><hr></td>
<td>This is somewhat equivalent to writing a vdata with non-interlaced
fields, only in a dataset oriented way. This should also be wrapped in a
higher layer interface if it's necessary for HDF5.</td>
</tr>
<tr align=center>
<td>Map a dataset's name to an index to reference it with</td>
<td><hr></td>
<td>SDnametoindex</td>
<td><hr></td>
<td>H5Mfind_name</td>
<td>Equivalent functionality except HDF5 call returns an OID instead of an
index.</td>
</tr>
<tr align=center>
<td>Get the valid range of values for data in a dataset</td>
<td><hr></td>
<td>SDgetrange</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Release access to a dataset</td>
<td><hr></td>
<td>SDendaccess</td>
<td>AIO_arr_destroy</td>
<td>H5Mrelease</td>
<td>Odd that the netCDF API doesn't have this...</td>
</tr>
<tr align=center>
<td>Set the valid range of data in a dataset</td>
<td><hr></td>
<td>SDsetrange</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Set the label, units, format, etc. of the data values in a dataset</td>
<td><hr></td>
<td>SDsetdatastrs</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Get the label, units, format, etc. of the data values in a dataset</td>
<td><hr></td>
<td>SDgetdatastrs</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Set the label, units, format, etc. of the dimensions in a dataset</td>
<td><hr></td>
<td>SDsetdimstrs</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Get the label, units, format, etc. of the dimensions in a dataset</td>
<td><hr></td>
<td>SDgetdimstrs</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Set the scale of the dimensions in a dataset</td>
<td><hr></td>
<td>SDsetdimscale</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Get the scale of the dimensions in a dataset</td>
<td><hr></td>
<td>SDgetdimscale</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Set the calibration parameters of the data values in a dataset</td>
<td><hr></td>
<td>SDsetcal</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Get the calibration parameters of the data values in a dataset</td>
<td><hr></td>
<td>SDgetcal</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented with attributes at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Set the fill value for the data values in a dataset</td>
<td><hr></td>
<td>SDsetfillvalue</td>
<td><hr></td>
<td><hr></td>
<td>HDF5 needs something like this, I'm not certain where to put it.</td>
</tr>
<tr align=center>
<td>Get the fill value for the data values in a dataset</td>
<td><hr></td>
<td>SDgetfillvalue</td>
<td><hr></td>
<td><hr></td>
<td>HDF5 needs something like this, I'm not certain where to put it.</td>
</tr>
<tr align=center>
<td>Move/Set the dataset to be in an 'external' file</td>
<td><hr></td>
<td>SDsetexternalfile</td>
<td><hr></td>
<td>H5Dset_storage</td>
<td>HDF5 has simple functions for this, but needs an API for setting up the
storage flow.</td>
</tr>
<tr align=center>
<td>Move/Set the dataset to be stored using only certain bits from the dataset</td>
<td><hr></td>
<td>SDsetnbitdataset</td>
<td><hr></td>
<td>H5Dset_storage</td>
<td>HDF5 has simple functions for this, but needs an API for setting up the
storage flow.</td>
</tr>
<tr align=center>
<td>Move/Set the dataset to be stored in compressed form</td>
<td><hr></td>
<td>SDsetcompress</td>
<td><hr></td>
<td>H5Dset_storage</td>
<td>HDF5 has simple functions for this, but needs an API for setting up the
storage flow.</td>
</tr>
<tr align=center>
<td>Search for an dataset attribute with particular name</td>
<td><hr></td>
<td>SDfindattr</td>
<td><hr></td>
<td>H5Mfind_name<br>H5Mwild_search</td>
<td>HDF5 can handle wildcard searchs for this feature.</td>
</tr>
<tr align=center>
<td>Map a run-time dataset handle to a persistant disk reference</td>
<td><hr></td>
<td>SDidtoref</td>
<td><hr></td>
<td><hr></td>
<td>I'm not certain this is needed for HDF5.</td>
</tr>
<tr align=center>
<td>Map a persistant disk reference for a dataset to an index in a group</td>
<td><hr></td>
<td>SDreftoindex</td>
<td><hr></td>
<td><hr></td>
<td>I'm not certain this is needed for HDF5.</td>
</tr>
<tr align=center>
<td>Determine if a dataset is a 'record' variable (i.e. it has an unlimited dimension)</td>
<td><hr></td>
<td>SDisrecord</td>
<td><hr></td>
<td><hr></td>
<td>Easily implemented by querying the dimensionality at a higher level for HDF5.</td>
</tr>
<tr align=center>
<td>Determine if a dataset is a 'coordinate' variable (i.e. it is used as a dimension)</td>
<td><hr></td>
<td>SDiscoord</td>
<td><hr></td>
<td><hr></td>
<td>I'm not certain this is needed for HDF5.</td>
</tr>
<tr align=center>
<td>Set the access type (i.e. parallel or serial) for dataset I/O</td>
<td><hr></td>
<td>SDsetaccesstype</td>
<td><hr></td>
<td><hr></td>
<td>HDF5 has functions for reading the information about this, but needs a better
API for setting up the storage flow.</td>
</tr>
<tr align=center>
<td>Set the size of blocks used to store a dataset with unlimited dimensions</td>
<td><hr></td>
<td>SDsetblocksize</td>
<td><hr></td>
<td><hr></td>
<td>HDF5 has functions for reading the information about this, but needs a better
API for setting up the storage flow.</td>
</tr>
<tr align=center>
<td>Sets backward compatibility of dimensions created.</td>
<td><hr></td>
<td>SDsetdimval_comp</td>
<td><hr></td>
<td><hr></td>
<td>Unneccessary in HDF5.</td>
</tr>
<tr align=center>
<td>Checks backward compatibility of dimensions created.</td>
<td><hr></td>
<td>SDisdimval_comp</td>
<td><hr></td>
<td><hr></td>
<td>Unneccessary in HDF5.</td>
</tr>
<tr align=center>
<td>Move/Set the dataset to be stored in chunked form</td>
<td><hr></td>
<td>SDsetchunk</td>
<td><hr></td>
<td>H5Dset_storage</td>
<td>HDF5 has simple functions for this, but needs an API for setting up the
storage flow.</td>
</tr>
<tr align=center>
<td>Get the chunking information for a dataset stored in chunked form</td>
<td><hr></td>
<td>SDgetchunkinfo</td>
<td><hr></td>
<td>H5Dstorage_detail</td>
<td></td>
</tr>
<tr align=center>
<td>Read/Write chunks of a dataset using a chunk index</td>
<td><hr></td>
<td>SDreadchunk<br>SDwritechunk</td>
<td><hr></td>
<td><hr></td>
<td>I'm not certain that HDF5 needs something like this.</td>
</tr>
<tr align=center>
<td>Tune chunk caching parameters for chunked datasets</td>
<td><hr></td>
<td>SDsetchunkcache</td>
<td><hr></td>
<td><hr></td>
<td>HDF5 needs something like this.</td>
</tr>
<tr align=center>
<td>Change some default behavior of the library</td>
<td><hr></td>
<td><hr></td>
<td>AIO_defaults</td>
<td><hr></td>
<td>Something like this would be useful in HDF5, to tune I/O pipelines, etc.</td>
</tr>
<tr align=center>
<td>Flush and close all open files</td>
<td><hr></td>
<td><hr></td>
<td>AIO_exit</td>
<td><hr></td>
<td>Something like this might be useful in HDF5, although it could be
encapsulated with a higher-level function.</td>
</tr>
<tr align=center>
<td>Target an architecture for data-type storage</td>
<td><hr></td>
<td><hr></td>
<td>AIO_target</td>
<td><hr></td>
<td>There are some rough parallels with using the data-type in HDF5 to create
data-type objects which can be used to write out future datasets.</td>
</tr>
<tr align=center>
<td>Map a filename to a file ID</td>
<td><hr></td>
<td><hr></td>
<td>AIO_filename</td>
<td>H5Mget_name</td>
<td></td>
</tr>
<tr align=center>
<td>Get the active directory (where new datasets are created)</td>
<td><hr></td>
<td><hr></td>
<td>AIO_getcwd</td>
<td><hr></td>
<td>HDF5 allows multiple directories (groups) to be attached to, any of which
can have new datasets created within it.</td>
</tr>
<tr align=center>
<td>Change active directory</td>
<td><hr></td>
<td><hr></td>
<td>AIO_chdir</td>
<td><hr></td>
<td>Since HDF5 has a slightly different access method for directories (groups),
this functionality can be wrapped around calls to H5Gget_oid_by_name.</td>
</tr>
<tr align=center>
<td>Create directory</td>
<td><hr></td>
<td><hr></td>
<td>AIO_mkdir</td>
<td>H5Mcreate</td>
<td></td>
</tr>
<tr align=center>
<td>Return detailed information about an object</td>
<td><hr></td>
<td><hr></td>
<td>AIO_stat</td>
<td>H5Dget_info<br>H5Dstorage_detail</td>
<td>Perhaps more information should be provided through another function in
HDF5?</td>
</tr>
<tr align=center>
<td>Get "flag" information</td>
<td><hr></td>
<td><hr></td>
<td>AIO_getflags</td>
<td><hr></td>
<td>Not required in HDF5.</td>
</tr>
<tr align=center>
<td>Set "flag" information</td>
<td><hr></td>
<td><hr></td>
<td>AIO_setflags</td>
<td><hr></td>
<td>Not required in HDF5.</td>
</tr>
<tr align=center>
<td>Get detailed information about all objects in a directory</td>
<td><hr></td>
<td><hr></td>
<td>AIO_ls</td>
<td>H5Gget_content_info_mult<br>H5Dget_info<br>H5Dstorage_detail</td>
<td>Only roughly equivalent functionality in HDF5, perhaps more should be
added?</td>
</tr>
<tr align=center>
<td>Get base type of object</td>
<td><hr></td>
<td><hr></td>
<td>AIO_BASIC</td>
<td>H5Gget_content_info</td>
<td></td>
</tr>
<tr align=center>
<td>Set base type of dataset</td>
<td><hr></td>
<td><hr></td>
<td>AIO_arr_set_btype</td>
<td>H5Mcreate(DATATYPE)</td>
<td></td>
</tr>
<tr align=center>
<td>Set dimensionality of dataset</td>
<td><hr></td>
<td><hr></td>
<td>AIO_arr_set_bdims</td>
<td>H5Mcreate(DATASPACE)</td>
<td></td>
</tr>
<tr align=center>
<td>Set slab of dataset to write</td>
<td><hr></td>
<td><hr></td>
<td>AIO_arr_set_slab</td>
<td><hr></td>
<td>This is similar to the process of creating a dataspace for use when
performing I/O on an HDF5 dataset</td>
</tr>
<tr align=center>
<td>Describe chunking of dataset to write</td>
<td><hr></td>
<td><hr></td>
<td>AIO_arr_set_chunk</td>
<td>H5Dset_storage</td>
<td></td>
</tr>
<tr align=center>
<td>Describe array index permutation of dataset to write</td>
<td><hr></td>
<td><hr></td>
<td>AIO_arr_set_perm</td>
<td>H5Dset_storage</td>
<td></td>
</tr>
<tr align=center>
<td>Create a new dataset with dataspace and datatype information from an
existing dataset.</td>
<td><hr></td>
<td><hr></td>
<td>AIO_arr_copy</td>
<td><hr></td>
<td>This can be mimicked in HDF5 by attaching to the datatype and dataspace of
an existing dataset and using the IDs to create new datasets.</td>
</tr>
<tr align=center>
<td>Create a new directory to group objects within</td>
<td><hr></td>
<td><hr></td>
<td>AIO_mkgroup</td>
<td>H5Mcreate(GROUP)</td>
<td></td>
</tr>
<tr align=center>
<td>Read name of objects in directory</td>
<td><hr></td>
<td><hr></td>
<td>AIO_read_group</td>
<td>H5Gget_content_info_mult</td>
<td></td>
</tr>
<tr align=center>
<td>Add objects to directory</td>
<td><hr></td>
<td><hr></td>
<td>AIO_write_group</td>
<td>H5Ginsert_item_mult</td>
<td></td>
</tr>
<tr align=center>
<td>Combine an architecture and numeric type to derive the format's datatype</td>
<td><hr></td>
<td><hr></td>
<td>AIO_COMBINE</td>
<td><hr></td>
<td>This is a nice feature to add to HDF5.</td>
</tr>
<tr align=center>
<td>Derive an architecture from the format's datatype</td>
<td><hr></td>
<td><hr></td>
<td>AIO_ARCH</td>
<td><hr></td>
<td>This is a nice feature to add to HDF5.</td>
</tr>
<tr align=center>
<td>Derive a numeric type from the format's datatype</td>
<td><hr></td>
<td><hr></td>
<td>AIO_PNT</td>
<td><hr></td>
<td>This is a nice feature to add to HDF5.</td>
</tr>
<tr align=center>
<td>Register error handling function for library to call when errors occur</td>
<td><hr></td>
<td><hr></td>
<td>AIO_error_handler</td>
<td><hr></td>
<td>This should be added to HDF5.</td>
</tr>
</table>

3183
doc/html/H5.format.html Normal file

File diff suppressed because it is too large Load Diff

997
doc/html/H5.intro.html Normal file

@ -0,0 +1,997 @@
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=windows-1252">
<META NAME="Generator" CONTENT="Microsoft Word 97">
<TITLE>H5introH</TITLE>
<META NAME="Template" CONTENT="C:\PROGRAM FILES\MICROSOFT OFFICE\OFFICE\html.dot">
</HEAD>
<BODY LINK="#0000ff" VLINK="#800080">
<B><FONT FACE="Times" SIZE=6><P ALIGN="CENTER">Introduction to HDF5 1.0 Alpha1.0</P>
</B></FONT><FONT FACE="Times"><P>This is a brief introduction to the HDF5 data model and programming model. It is not a full user's guide, but should provide enough information for you to understand how HDF5 is meant to work. Knowledge of the current version of HDF should make it easier to follow the text, but it is not required. For further information on the topics covered here, see the HDF5 documentation at </FONT><A HREF="http://hdf.ncsa.uiuc.edu/nra/BigHDF/"><FONT FACE="Times">http://hdf.ncsa.uiuc.edu/nra/BigHDF/</FONT></A><FONT FACE="Times">. </P>
</FONT><H2>What is the HDF5 prototype?</H2>
<FONT FACE="Times"><P>HDF5 is a new, experimental version of HDF that is designed to address some of the limitations of the current version of HDF (HDF4.1) and to address current and anticipated requirements of modern systems and applications. </P>
<P>This HDF5 prototype is not complete, but it should be sufficient show the basic features of HDF5. We urge you to look at it and give us feedback on what you like or don't like about it, and what features you would like to see added to it.</P>
<B><P>Why HDF5?</B> The development of HDF5 is motivated by a number of limitations in the current HDF format, as well as limitations in the library. Some of these limitations are:</P>
<UL>
</FONT><LI>A single file cannot store more than 20,000 complex objects, and a single file cannot be larger than 2 gigabytes </LI>
<LI>The data models are less consistent than they should be, there are more object types than necessary, and datatypes are too restricted. </LI>
<LI>The library source is old and overly complex, does not support parallel I/O effectively, and is difficult to use in threaded applications.</LI></UL>
<FONT FACE="Times"><P>When complete HDF5 will include the following improvements.</P>
<UL>
</FONT><LI>A new file format designed to address some of the deficiencies of HDF4.1, particularly the need to store larger files and more objects per file. </LI>
<LI>A simpler, more comprehensive data model that includes only two basic structures: a multidimensional array of record structures, and a grouping structure. </LI>
<LI>A simpler, better-engineered library and API, with improved support for parallel i/o, threads, and other requirements imposed by modern systems and applications.</LI></UL>
<H2>Limitations of the current prototype</H2>
<FONT FACE="Times"><P>The prototype release includes most of the <I>basic</I> functionality that is planned for the HDF5 library. However, the library does not implement all of the features detailed in the format and API specifications. Here is a listing of some of the limitations of the current release: </P>
<UL>
</FONT><LI>Attributes for data objects are not supported </LI>
<LI>Data compression is not supported </LI>
<LI>External storage of objects are not supported </LI>
<LI>Some functions for manipulating datasets, dataspaces, and groups have not been implemented </LI>
<FONT FACE="Times"><LI>Some number types, including user-defined number types are not supported. Also number type conversion is limited.</LI></UL>
<P>See the API Specification at </FONT><A HREF="http://hdf.ncsa.uiuc.edu/nra/BigHDF/"><FONT FACE="Times">http://hdf.ncsa.uiuc.edu/nra/BigHDF/</FONT></A><FONT FACE="Times"> for a complete listing of all routines that have been implemented.</P>
</FONT><H2>HDF5 file organization and data model.</H2>
<FONT FACE="Times"><P>HDF5 files are organized in a hierarchical structure, with two primary structures: "groups" and "datasets."</P>
<UL>
</FONT><I><LI>HDF5 group: </I>a grouping structure containing instances of zero or more groups or datasets, together with supporting metadata </LI>
<I><LI>HD5F dataset:</I> a multidimensional array of data elements, together with supporting metadata. </LI></UL>
<FONT FACE="Times"><P>Working with groups and group members is similar in many ways to working with directories and files in UNIX. As with UNIX directories and files, objects in an HDF5 file are often described by giving their full path names. "/" signifies the root group. "/foo" signifies a member of the root group called "foo." "/foo/zoo" signifies a member of the group "foo," which in turn is a member of the root group.</P>
<P>Any HDF5 group or dataset may have an associated <I>attribute list.</I> An HDF5 <I>attribute</I> is a user-defined HDF5 structure that provides extra information about an HDF5 object. Attributes are described in more detail below. <I>(Note: attributes are not supported in the current prototype.)</P>
</I></FONT><H3>HDF5 Groups</H3>
<FONT FACE="Times"><P>An<I> HDF5 group</I> is a structure containing zero or more HDF5 objects. A group has two parts:</P>
<UL>
</FONT><LI>A <I>group header</I>, which contains a group name and a list of group attributes. (Attributes are not yet implemented.) </LI>
<LI>A group symbol table, which is a list of the HDF5 objects that belong to the group.</LI></UL>
<P>&nbsp;</P>
<H3>HDF5 Datasets</H3>
<FONT FACE="Times"><P>A dataset is stored in a file in two parts: a header and a data array. </P>
<P>The header contains information that is needed to interpret the array portion of the dataset, as well as metadata, or pointers to metadata, that describes or annotates the dataset. Header information includes the name of the object, its dimensionality, its number-type, information about how the data itself is stored on disk, and other information used by the library to speed up access to the dataset or maintain the file's integrity.</P>
<P>There are four essential classes of information in any header: <I>name</I>, <I>datatype</I>, <I>dataspace</I>, and <I>storage layout</I>:</P>
<B><P>Name</B><I>.</I> A dataset <I>name</I> is a sequence of alphanumeric ASCII characters.</P>
<B><DFN><P>Datatype</DFN><I>.</B></I> HDF5 allows one to define many different kinds of <B>datatypes</B>. There are two basic categories of data types: "atomic" types and "compound" types. Atomic types are those that are not decomposed at the data type interface level, such as integers and floats. Compound types are made up of atomic types. </P>
<I><P>Atomic datatypes</I> include integers and floating-point numbers. Each atomic type belongs to a particular class and has several properties: size, order, precision, and offset. In this introduction, we consider only a few of these properties.</P>
<P>Atomic datatypes include integer, float, date and time, string, bit field, and opaque. <I>(Note: Only integer and float classes are available in the current implementation.)</P>
</I><P>Properties of integer types include size, order (endian-ness), and signed-ness (signed/unsigned).</P>
<P>Properties of float types include the size and location of the exponent and mantissa, and the location of the sign bit.</P>
<P>The datatypes that are supported in the current implementation are: </P>
<UL>
</FONT><LI>Integer datatypes: 8-bit, 16-bit, 32-bit, and 64-bit integers in both little and big-endian format. </LI>
<LI>Floating-point numbers: IEEE 32-bit and 64-bit floating-point numbers in both little and big-endian format.</LI></UL>
<FONT FACE="Times"><P>A <I>compound datatype</I> is one in which a collection of simple datatypes are represented as a single unit, similar to a "struct" in C. The parts of a compound datatype are called <I>members.</I> The members of a compound datatype may be of any datatype, including another compound datatype. It is possible to read members from a compound type without reading the whole type.</P>
<B><DFN><P>Dataspace.</B> </DFN>A dataset <I>dataspace </I>describes the dimensionality of the dataset. The dimensions of a dataset can be fixed (unchanging), or they may be <I>unlimited</I>, which means that they are extendible (i.e. they can grow larger). </P>
<P>Properties of a dataspace consist of the <I>rank </I>(number of dimensions) of the data array, and the <I>actual sizes of the dimensions</I> of the array, and the <I>maximum sizes of the dimensions </I>of the array. For a fixed-dimension dataset, the actual size is the same as the maximum size of a dimension. When a dimension is unlimited, the maximum size is set to the value </FONT><FONT FACE="Courier">H5S_UNLIMITED</FONT><FONT FACE="Times">. (An example below shows how to create extendible datasets.)</P>
<P>A dataspace can also describe portions of a dataset, making it possible to do partial I/O (hyperslab) operations.</P>
<P>Since I/O operations have two end-points, the raw data transfer functions require two dataspace arguments: one describes the application memory dataspace or subset thereof, and the other describes the file dataspace or subset thereof.</P>
<B><P>Storage layout.</B> The HDF5 format makes it possible to store data in a variety of ways. The default storage layout format is <I>contiguous</I>, meaning that data is stored in the same linear way that it is organized in memory. Two other storage layout formats are currently defined for HDF5: <I>compact, </I>and<I> chunked. </I>In the future, other storage layouts may be added.<I> </P>
<P>Compact</I> storage is used when the amount of data is small and can be stored directly in the object header. <I>(Note: Compact storage is not supported in this prototype.)</I> </P>
<I><P>Chunked</I> storage involves dividing the dataset into equal-sized "chunks" that are stored separately. Chunking has three important benefits. </P>
<OL>
<LI>It makes it possible to achieve good performance when accessing subsets of the datasets, even when the subset to be chosen is orthogonal to the normal storage order of the dataset. </LI>
<LI>It makes it possible to compress large datasets and still achieve good performance when accessing subsets of the dataset. </LI>
<LI>It makes it possible efficiently to extend the dimensions of a dataset in any direction.</LI></OL>
</FONT><H3>HDF5 attribute lists</H3>
<FONT FACE="Times"><P>An <I>attribute list</I> for an dataset or group is a listing of objects in the HDF file that are used as attributes, or metadata for the object. The attribute list is composed of two lists of objects, the first being simple attributes about the object, and the second being pointers to attribute objects. <I>(Note: Attributes are not supported in this prototype.)</P>
</I><P>&nbsp;</P>
</FONT><H2>The HDF5 Applications Programming Interface (API)</H2>
<FONT FACE="Times"><P>The current HDF5 API is implemented only in C. The API provides routines for creating HDF5 files, creating and writing groups, datasets, and their attributes to HDF5 files, and reading groups, datasets and their attributes from HDF5 files.</P>
</FONT><H3>Naming conventions</H3>
<FONT FACE="Times"><P>All C routines on the HDF 5 library begin with a prefix of the form "H5*", where "*" is a single letter indicating the object on which the operation is to be performed:</P>
<UL>
</FONT><B><LI>H5F</B>: <B>F</B>ile-level access routines. <BR>
Example: <FONT FACE="Courier">H5Fopen</FONT>, which opens an HDF5 file. </LI>
<B><LI>H5G</B>: <B>G</B>roup functions, for creating and operating on physical groups of objects. <BR>
Example: <FONT FACE="Courier">H5Gset,</FONT>which sets the working group to the specified group. </LI>
<B><LI>H5T: </B>Data<B>T</B>ype functions, for creating and operating on simple and compound datatypes to be used as the elements in data arrays.<B><BR>
</B>Example: <FONT FACE="Courier">H5Tcopy,</FONT>which creates a copy of an existing data type. </LI>
<B><LI>H5S: </B>DataS<B>P</B>ace functions, which create and manipulate the dataspace in which the elements of a data array are stored.<BR>
Example: <FONT FACE="Courier">H5Sget_ndims</FONT>, which retrieves the number of dimensions of a data array. </LI>
<B><LI>H5D: D</B>ataset functions, which manipulate the data within datasets and determine how the data is to be stored in the file. <BR>
Example: H5D<FONT FACE="Courier">read</FONT>, which reads all or part of a dataset into a buffer in memory. </LI>
<B><LI>H5P</B>: <B>T</B>emplate functions, for manipulating object templates. <BR>
Example: <FONT FACE="Courier">H5Pset_chunk</FONT>, which sets the number of dimensions and the size of a chunk.</LI></UL>
<H3>Include files </H3>
<FONT FACE="Times"><P>There are a number definitions and declarations that should be included with any HDF5 program. These definitions and declarations are contained in several "include" files. The main include file is <I>hdf5.h</I>. This file includes all of the other files that your program is likely to need. <I>Be sure to include hdf5.h in any program that accesses HDF5.</P>
</I></FONT><H3>Predefined simple numeric scalar datatypes</H3>
<FONT FACE="Times"><P>The HDF5 prototype currently supports simple signed and unsigned 8-bit, 16-bit, 32-bit , and 64-bit integers, and floating point numbers. The naming scheme for type definitions uses the following conventions:</P>
<UL>
</FONT><LI>"int" stands for "integer" </LI>
<LI>the prefix "u" stands for "unsigned" </LI>
<LI>the integer suffix indicates the number of bits in the number</LI></UL>
<FONT FACE="Times"><P>For example, "uint16" indicates an unsigned 16-bit integer. Datatypes that are supported in this prototype are:</P>
</FONT><PRE> char
int8
uint8
int16
uint16
int32
uint32
int64
uint64
float32
float64</PRE>
<FONT FACE="Times"><P>These datatypes are defined in the file H5public.h together with keywords used to refer to them. H5public.h is included by the file hdf5.h described earlier. These datatypes should be used whenever you declare a variable to be used with an HDF5 routine. For instance, a 32-bit floating point variable should always be declared using a declaration such as</P>
</FONT><CODE><PRE>float32 x;</PRE>
</CODE><H3>Programming models</H3>
<FONT FACE="Times"><P>In this section we describe how to program some basic operations on files, including how to</P>
<UL>
</FONT><LI>create a file </LI>
<LI>create and initialize a dataset </LI>
<LI>discard objects when they are no longer needed </LI>
<LI>write a dataset to a new file </LI>
<LI>obtain information about a dataset </LI>
<LI>read a portion of a dataset </LI>
<LI>create and write compound datatypes </LI>
<LI>create and write extendible datasets </LI>
<LI>create and populate groups </LI></UL>
<H4>How to create an HDF5 file</H4>
<FONT FACE="Times"><P>This programming model shows how to create a file and also how to close the file.</P>
<OL>
<LI>Create the file using </FONT><FONT FACE="Courier">H5Fcreate.</FONT><FONT FACE="Times"> Obtain a file ID (e.g. </FONT><FONT FACE="Courier">file_id</FONT><FONT FACE="Times">).</LI>
<LI>Close the file with </FONT><FONT FACE="Courier">H5Fclose(file_id)</FONT><FONT FACE="Times">.</LI>
<P>The following code fragment implements the specified model. If there is a possibility that the file already exists, the user must add the flag </FONT><FONT FACE="Courier">H5F_ACC_TRUNC </FONT><FONT FACE="Times">to the access mode to overwrite the previous file's information. </P>
</FONT><CODE><PRE>hid_t file; /* handle */
/*
* Create a new file using H5F_ACC_TRUNC access,
* default file creation properties, and default file
* access properties.
* Then close the file.
*/
file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
status = H5Fclose(file); </PRE>
</CODE><H4>How to create and initialize the essential components of a dataset for writing to a file.</H4>
<FONT FACE="Times"><P>Recall that datatypes and dimensionality (dataspace) are independent objects, which are created separately from any dataset that they might be attached to. Because of this the creation of a dataset requires, at a minimum, separate definitions of datatype, dimensionality, and dataset. Hence, to create a dataset the following steps need to be taken:</P>
<LI VALUE=1>Create and initialize a dataspace for the dataset to be written.</LI>
<LI>Define the datatype for the dataset to be written. </LI>
<LI>Create and initialize the dataset itself.</LI></OL>
</FONT><FONT FACE="Courier New"><P>&nbsp;</P>
</FONT><FONT FACE="Times"><P>The following code illustrates the creation of these three components of a dataset object.</P>
</FONT><CODE><PRE>hid_t dataset, datatype, dataspace; /* declare handles */
/*
* 1. Create dataspace: Describe the size of the array and
* create the data space for fixed size dataset.
*/
dimsf[0] = NX;
dimsf[1] = NY;
dataspace = H5Screate_simple(RANK, dimsf, NULL);
/*
/*
* 2. Define datatype for the data in the file.
* We will store little endian INT32 numbers.
*/
datatype = H5Tcopy(H5T_NATIVE_INT32);
status = H5Tset_order(datatype, H5T_ORDER_LE);
/*
* 3. Create a new dataset within the file using defined
* dataspace and datatype and default dataset creation
* properties.
* NOTE: H5T_NATIVE_INT32 can be used as datatype if conversion
* to little endian is not needed.
*/
dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace, H5P_DEFAULT);</PRE><DIR>
<DIR>
</CODE><H4>How to discard objects when they are no longer needed</H4></DIR>
</DIR>
<FONT FACE="Times"><P>The type, dataspace and dataset objects should be released once they are no longer needed by a program. Since each is an independent object, the must be released ("closed") separately. The following lines of code close the type, dataspace, datasets, and file that were created in the preceding section.</P>
</FONT><CODE><P>H5Tclose(datatype);</P>
<P>H5Dclose(dataset);</P>
<P>H5Sclose(dataspace);</P><DIR>
<DIR>
</CODE><H4>How to write a dataset to a new file</H4></DIR>
</DIR>
<FONT FACE="Times"><P>Having defined the datatype, dataset, and dataspace parameters, you write out the data with a call to </FONT><FONT FACE="Courier">H5Dwrite.</P>
</FONT><CODE><PRE>/*
* Write the data to the dataset using default transfer
* properties.
*/
status = H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);</PRE>
</CODE><FONT FACE="Times"><P>The third and fourth parameters of </FONT><FONT FACE="Courier">H5Dwrite</FONT><FONT FACE="Times"> in the example describe the dataspaces in memory and in the file, respectively. They are set to the value </FONT><FONT FACE="Courier">H5S_ALL</FONT><FONT FACE="Times"> to indicate that an entire dataset is to be written. In a later section we look at how we would access a portion of a dataset.</P>
</FONT><P><A HREF="#CreateExample"><FONT FACE="Times">Example 1</FONT></A><FONT FACE="Times"> contains a program that creates a file and a dataset, and writes the dataset to the file. </P>
<P>Reading is analogous to writing. If, in the previous example, we wish to read an entire dataset, we would use the same basic calls with the same parameters. Of course, the routine </FONT><FONT FACE="Courier">H5Dread</FONT><FONT FACE="Times"> would replace </FONT><FONT FACE="Courier">H5Dwrite.</FONT><FONT FACE="Times"> </P><DIR>
<DIR>
</FONT><H4>Getting information about a dataset</H4></DIR>
</DIR>
<FONT FACE="Times"><P>Although reading is analogous to writing, it is often necessary to query a file to obtain information about a dataset. For instance, we often need to know about the datatype associated with a dataset, as well dataspace information (e.g. rank and dimensions). There are several "get" routines for obtaining this information The following code segment illustrates how we would get this kind of information: </P>
</FONT><CODE><PRE>/*
* Get datatype and dataspace handles and then query
* dataset class, order, size, rank and dimensions.
*/
datatype = H5Dget_type(dataset); /* datatype handle */
class = H5Tget_class(datatype);
if (class == H5T_INTEGER) printf("Data set has INTEGER type \n");
order = H5Tget_order(datatype);
if (order == H5T_ORDER_LE) printf("Little endian order \n");
size = H5Tget_size(datatype);
printf(" Data size is %d \n", size);
dataspace = H5Dget_space(dataset); /* dataspace handle */
rank = H5Sget_ndims(dataspace);
status_n = H5Sget_dims(dataspace, dims_out, NULL);
printf("rank %d, dimensions %d x %d \n", rank, dims_out[0], dims_out[1]);</PRE>
</CODE><FONT FACE="Times"><P>&nbsp;</P><DIR>
<DIR>
</FONT><H4>Reading a portion of a dataset: defining dataspaces</H4></DIR>
</DIR>
<FONT FACE="Times"><P>In the previous discussion, we describe how to access an entire dataset with one write (or read) operation. To read or write a <I>portion</I> of a dataset, we need to provide more contextual information.</P>
<P>Consider the following example. Suppose there is 500x600 dataset in a file, and we wish to read from the dataset a 100x200 hyperslab located beginning at element &lt;200,200&gt;. In addition, suppose we wish to read the hyperslab into an 200x400 array in memory beginning at element &lt;0,0&gt; in memory. Visually, the transfer looks something like this: </P>
</FONT><FONT FACE="Times" SIZE=6><P>&nbsp;<IMG SRC="dataset_p1.gif" WIDTH=521 HEIGHT=420></P>
</FONT><FONT FACE="Times"><P>As the example illustrates, whenever we read part of a dataset from a file we must provide two dataspaces: the dataspace of the object in the file as well as the dataspace of the object in memory into which we read. There are dataspace routines (</FONT><FONT FACE="Courier">H5S...</FONT><FONT FACE="Times">) for doing this. </P>
<P>For example, suppose we want to read a 3x4 hyperslab from a dataset in a file beginning at the element &lt;1,2&gt; in the dataset. In order to do this, we must create a dataspace that describes the overall rank and dimensions of the dataset in the file, as well as the position and size of the hyperslab that we are extracting from that dataset. The following code illustrates how this would be done. </P>
</FONT><CODE><PRE>/*
* Get overall rank and dimensions of dataspace.
*/
dataspace = H5Dget_space(dataset); /* get dataspace handle */
rank = H5Sget_ndims(dataspace);
status_n = H5Sget_dims(dataspace, dims_out, NULL);
/*
* Define hyperslab in the dataset.
*/
offset[0] = 1;
offset[1] = 2;
count[0] = 3;
count[1] = 4;
status = H5Sset_hyperslab(dataspace, offset, count, NULL);</PRE>
</CODE><FONT FACE="Times"><P>This describes the dataspace from which we wish to read. We need to define the dataspace in memory analogously. Suppose, for instance, that we have in memory a 3 dimensional 7x7x3 array into which we wish to read the 3x4 hyperslab described above beginning at the element &lt;3,0,0&gt;. Since the in-memory dataspace has three dimensions, we have to describe the hyperslab as an array with three dimensions, with the last dimension being 1: &lt;3,4,1&gt;.</P>
<P>Notice that now we must describe two things: the dimensions of the in-memory array, and the size and position of the hyperslab that we wish to read in. The following code illustrates how this would be done. </P>
</FONT><CODE><PRE>/*
* Define the memory dataspace.
*/
dimsm[0] = 7;
dimsm[1] = 7;
dimsm[2] = 3;
memspace = H5Screate_simple(RANK_OUT,dimsm,NULL);
/*
* Define memory hyperslab.
*/
offset_out[0] = 3;
offset_out[1] = 0;
offset_out[2] = 0;
count_out[0] = 3;
count_out[1] = 4;
count_out[2] = 1;
status = H5Sset_hyperslab(memspace, offset_out, count_out, NULL);
/*</PRE>
</CODE><P><A HREF="#ReadExample"><FONT FACE="Times">Example 2</FONT></A><FONT FACE="Times"> contains a complete program that performs these operations.</P>
</FONT><H4>Creating compound datatypes</H4>
<P>Properties of compound datatypes.</B>A compound datatype is similar to a struct in C or a common block in Fortran. It is a collection of one or more atomic types or small arrays of such types. To create and use of a compound datatype requires you need to refer to various <I>properties</I> of the data compound datatype:</P>
<UL>
<LI>It is of class <I>compound.</I> </LI>
<LI>It has a fixed total <I>size</I>, in bytes. </LI>
<LI>It consists of zero or more <I>members</I> (defined in any order) with unique names and which occupy non-overlapping regions within the datum. </LI>
<LI>Each member has its own <I>datatype</I>. </LI>
<LI>Each member is referenced by an <I>index number</I> between zero and N-1, where N is the number of members in the compound datatype. </LI>
<LI>Each member has a <I>name</I> which is unique among its siblings in a compound data type. </LI>
<LI>Each member has a fixed <I>byte offset</I>, which is the first byte (smallest byte address) of that member in a compound datatype. </LI>
<LI>Each member can be a small array of up to four dimensions.</LI></UL>
<FONT FACE="Times"><P>Properties of members of a compound data type are defined when the member is added to the compound type and cannot be subsequently modified.</P>
<B><P>Defining compound datatypes.</P> </B>
<P>Compound datatypes must be built out of other datatypes. First, one creates an empty compound data type and specifies its total size. Then members are added to the compound data type in any order.</P>
<I><P>Member names. </I>Each member must have a descriptive name, which is the key used to uniquely identify the member within the compound data type. A member name in an HDF5 data type does not necessarily have to be the same as the name of the corresponding member in the C struct in memory, although this is often the case. Nor does one need to define all members of the C struct in the HDF5 compound data type (or vice versa). </P>
<I><P>Offsets. </I>Usually a C struct will be defined to hold a data point in memory, and the offsets of the members in memory will be the offsets of the struct members from the beginning of an instance of the struct. The library defines two macros to compute the offset of a member within a struct (The only difference between the two is that one uses </FONT><CODE>s.m</CODE><FONT FACE="Times"> as the struct member while the other uses </FONT><CODE>p-&gt;m</CODE><FONT FACE="Times" SIZE=2>)</FONT><FONT FACE="Times">: </P>
</FONT><CODE><P>HOFFSET(s,m)<FONT SIZE=5>. </FONT></CODE><FONT FACE="Times">This macro computes the offset of member </FONT><FONT FACE="Courier"><EM>m</EM> </FONT><FONT FACE="Times">within a struct variable <EM>s</EM>. </P>
</FONT><CODE><P>HPOFFSET(p,m)<FONT SIZE=5>. </FONT></CODE><FONT FACE="Times">This macro computes the offset of member </FONT><FONT FACE="Courier"><EM>m</FONT></EM><FONT FACE="Times"> from a pointer to a struct </FONT><FONT FACE="Courier"><EM>p</FONT></EM><FONT FACE="Times">. </P>
<P>Here is an example in which a compound data type is created to describe complex numbers whose type is defined by the </FONT><CODE>complex_t</CODE><FONT FACE="Times" SIZE=2> </FONT><FONT FACE="Times">struct. </P>
</FONT><CODE><PRE>typedef struct {
double re; /*real part */
double im; /*imaginary part */
} complex_t;
complex_t tmp; /*used only to compute offsets */
hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp);
H5Tinsert (complex_id, "real", HOFFSET(tmp,re),
H5T_NATIVE_DOUBLE);
H5Tinsert (complex_id, "imaginary", HOFFSET(tmp,im),
H5T_NATIVE_DOUBLE);</PRE>
</CODE>
<P> <A HREF="#CompoundExample"><FONT FACE="Times">Example 3</A> shows how to create a compound data type,
write an array that has the compound data type to the file, and read back subsets of the members.</P>
<P>&nbsp;</P>
</FONT><H4>Creating and writing extendible datasets</H4>
<P>An <I>extendible</I> dataset is one whose dimensions can grow. In HDF5, it is possible to define a dataset to have certain initial dimensions, then later to increase the size of any of the initial dimensions. </P>
<P>For example, you can create and store the following 3x3 HDF5 dataset:</P>
<CODE><PRE><P>1 1 1 </P>
<P>1 1 1 </P>
<P>1 1 1 </P>
</PRE>
</CODE>
<FONT FACE="Times"><P>then later to extend this into a 10x3 dataset by adding 7 rows, such as this:</P>
</FONT><CODE><PRE><P>1 1 1 </P>
<P>1 1 1 </P>
<P>1 1 1 </P>
<P>2 2 2</P>
<P>2 2 2</P>
<P>2 2 2</P>
<P>2 2 2</P>
<P>2 2 2</P>
<P>2 2 2</P>
<P>2 2 2</P>
</PRE>
</CODE>
</FONT><FONT FACE="Times"><P>then further extend it to a 10x5 dataset by adding two columns, such as this:</P>
</FONT><CODE><PRE><P>1 1 1 3 3 </P>
<P>1 1 1 3 3 </P>
<P>1 1 1 3 3 </P>
<P>2 2 2 3 3</P>
<P>2 2 2 3 3</P>
<P>2 2 2 3 3</P>
<P>2 2 2 3 3</P>
<P>2 2 2 3 3</P>
<P>2 2 2 3 3</P>
<P>2 2 2 3 3</P>
</PRE>
</CODE>
</FONT><FONT FACE="Times"><P>The current version of HDF 5 requires you to use <I>chunking</I> in order to define extendible datasets. Chunking makes it possible to extend datasets efficiently, without having to reorganize storage excessively. </P>
<P>Three operations are required in order to write an extendible dataset:</P>
<OL>
<LI>Declare the dataspace of the dataset to have <I>unlimited dimensions</I> for all dimensions that might eventually be extended.</LI>
<LI>When creating the dataset, set the storage layout for the dataset to <I>chunked</I>.</LI>
<LI>Extend the size of the dataset.</LI></OL>
<P>For example, suppose we wish to create a dataset similar to the one shown above. We want to start with a 3x3 dataset, then later extend it in both directions. </P>
<B><P>Declaring unlimited dimensions. </B>We could declare the dataspace to have unlimited dimensions with the following code, which uses the predefined constant H5S_UNLIMITED to specify unlimited dimensions.</P>
</FONT><CODE><PRE><P>hsize_t dims[2] = { 3, 3}; /* dataset dimensions at the creation time */ </P>
<P>hsize_t maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED};
</P>
<P>/*</P>
<P>* 1. Create the data space with unlimited dimensions. </P>
<P>*/</P>
<P>dataspace = H5Screate_simple(RANK, dims, maxdims); </P>
</PRE>
</CODE>
<B><P>Enabling chunking. </B>We can then modify the dataset storage layout properties to
enable chunking. We do this using the routine H5Pset_chunk:
<CODE><PRE><P>hid_t cparms; </P>
<P>hsize_t chunk_dims[2] ={2, 5};</P>
<P>/* </P>
<P>* 2. Modify dataset creation properties to enable chunking.</P>
<P>*/</P>
<P>cparms = H5Pcreate (H5P_DATASET_CREATE);</P>
<P>status = H5Pset_chunk( cparms, RANK, chunk_dims);</CODE></PRE></P>
<B>Extending dataset size. </B>Finally, when we want to extend the size of the dataset,
we invoke H5Dextend to extend the size of the dataset. In the following example, we extend the dataset along the first dimension, by seven rows, so that the new dimensions are &lt;10,3&gt;.:
<CODE><PRE><P>/*</P>
<P>* Extend the dataset. Dataset becomes 10 x 3.</P>
<P>*/</P>
<P>dims[0] = dims[0] + 7;</P>
<P>size[0] = dims[0]; </P>
<P>size[1] = dims[1]; </P>
<P>status = H5Dextend (dataset, size);</P>
<P>&nbsp;</P>
</CODE></PRE>
<A HREF="#ExtendibleExample"><FONT FACE="Times">Example 4</A> shows how to create a 3x3 extendible dataset, to extend the dataset to 10x3, then to extend it again to 10x5.</P>
</FONT><H3>Working with groups in a file</H3>
<P>Groups provide a mechanism for organizing datasets in an HDF5 file extendable meaningful ways. The H5G API contains routines for working with groups. </P>
<B>To create a group</B>, use H5Gcreate>. For example, the following code creates two groups that are members of the root group. They are called "/IntData" and "/FloatData." The return value ("dir") is the group ID.
<CODE><PRE>/*<BR>
* Create two groups in a file.
*/
dir = H5Gcreate(file, "/IntData", 0);
status = H5Gclose(dir);
dir = H5Gcreate(file,"/FloatData", 0);
status = H5Gclose(dir);</CODE></PRE></P>
</FONT><PRE>The third parameter in <CODE>H5Gcreate</CODE> optionally specifies how much file space to reserve to store the names that will appear in this group. If a non-positive value is supplied then a default size is chosen.
<CODE>H5Gclose</CODE> closes the group and releases the group ID.<P>
<B><P>Creating an object in a particular group. </B>Except for single-object HDF5 files, every object in an HDF5 file must belong to a group, and hence has a path name. Hence, we put an object in a particular group by giving its path name when we create it. For example, the following code creates a dataset "IntArray" in the group "/IntData":</P>
</FONT><CODE><PRE>/*
* Create dataset in the /IntData group by specifying full path.
*/
dims[0] = 2;
dims[1] = 3;
dataspace = H5Screate_simple(2, dims, NULL);
dataset = H5Dcreate(file, "/IntData/IntArray", H5T_NATIVE_INT, dataspace, H5P_DEFAULT); </PRE>
</CODE><B><P>Changing the current group. </B>The HDF5 Group API supports the idea of a "current," group. This is analogous to the "current working directory" idea in UNIX. You can set the current group in HDF5 with the routine H5Gset. The following code shows how to set a current group, then create a certain dataset ("FloatData") in that group. </P>
</FONT><CODE><PRE>/*
* Set current group to /FloatData.
*/
status = H5Gset (file, "/FloatData");
/*
* Create two datasets
*/
dims[0] = 5;
dims[1] = 10;
dataspace = H5Screate_simple(2, dims, NULL);
dataset = H5Dcreate(file, "FloatArray", H5T_NATIVE_FLOAT, dataspace, H5P_DEFAULT); </PRE>
</CODE>
<A HREF="#GroupExample"><FONT FACE="Times">Example 5</A> shows how to create an HDF5 file with two group, and to place some datasets within those groups.</P>
</FONT><H3>Example code</H3>
<H4><A NAME="CreateExample">Example 1: How to create a homogeneous multi-dimensional dataset</A> and write it to a file.</H4>
<P>This example creates a 2-dimensional HDF 5 dataset of little endian 32-bit integers.</P>
<CODE><PRE><P><A NAME="CheckAndReadExample">/* </P>
<P>* This example writes data to HDF5 file.</P>
<P>* Data conversion is performed during write operation. </P>
<P>*/</P>
<P>#include "hdf5.h"</P>
<P>#define FILE "SDS.h5"</P>
<P>#define DATASETNAME "IntArray" </P>
<P>#define NX 5 /* dataset dimensions */</P>
<P>#define NY 6</P>
<P>#define RANK 2</P>
<P>main ()</P>
<P>{</P>
<P>hid_t file, dataset; /* file and dataset handles */</P>
<P>hid_t datatype, dataspace; /* handles */</P>
<P>hsize_t dimsf[2]; /* dataset dimensions */</P>
<P>herr_t status; </P>
<P>int32 data[NX][NY]; /* data to write */</P>
<P>int i, j;</P>
<P>/* </P>
<P>* Data and output buffer initialization. </P>
<P>*/</P>
<P>for (j = 0; j &lt; NX; j++) {</P>
<P>for (i = 0; i &lt; NY; i++)</P>
<P>data[j][i] = i + j;</P>
<P>} </P>
<P>/* 0 1 2 3 4 5 </P>
<P>1 2 3 4 5 6</P>
<P>2 3 4 5 6 7</P>
<P>3 4 5 6 7 8</P>
<P>4 5 6 7 8 9 */</P>
<P>/*</P>
<P>* Create a new file using H5F_ACC_TRUNC access,</P>
<P>* default file creation properties, and default file</P>
<P>* access properties.</P>
<P>*/</P>
<P>file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);</P>
<P>/*</P>
<P>* Describe the size of the array and create the data space for fixed</P>
<P>* size dataset. </P>
<P>*/</P>
<P>dimsf[0] = NX;</P>
<P>dimsf[1] = NY;</P>
<P>dataspace = H5Screate_simple(RANK, dimsf, NULL); </P>
<P>/* </P>
<P>* Define datatype for the data in the file.</P>
<P>* We will store little endian INT32 numbers.</P>
<P>*/</P>
<P>datatype = H5Tcopy(H5T_NATIVE_INT32);</P>
<P>status = H5Tset_order(datatype, H5T_ORDER_LE);</P>
<P>/*</P>
<P>* Create a new dataset within the file using defined dataspace and</P>
<P>* datatype and default dataset creation properties.</P>
<P>*/</P>
<P>dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace,</P>
<P>H5P_DEFAULT);</P>
<P>/*</P>
<P>* Write the data to the dataset using default transfer properties.</P>
<P>*/</P>
<P>status = H5Dwrite(dataset, H5T_NATIVE_INT32, H5S_ALL, H5S_ALL,</P>
<P>H5P_DEFAULT, data);</P>
<P>/*</P>
<P>* Close/release resources.</P>
<P>*/</P>
<P>H5Sclose(dataspace);</P>
<P>H5Tclose(datatype);</P>
<P>H5Dclose(dataset);</P>
<P>H5Fclose(file);</P>
<P>} </P>
<P>&nbsp;</P>
</CODE></PRE>
<H4><A NAME="ReadExample"> Example 2. How to read a hyperslab from file into memory.</A></H4>
<P>This example reads a hyperslab from a 2-d HDF5 dataset into a 3-d dataset in memory.</P>
<CODE><PRE><P>/* </P>
<P>* This example reads hyperslab from the SDS.h5 file </P>
<P>* created by h5_write.c program into two-dimensional</P>
<P>* plane of the tree-dimensional array. </P>
<P>* Information about dataset in the SDS.h5 file is obtained. </P>
<P>*/</P>
<P>#include "hdf5.h"</P>
<P>#define FILE "SDS.h5"</P>
<P>#define DATASETNAME "IntArray" </P>
<P>#define NX_SUB 3 /* hyperslab dimensions */ </P>
<P>#define NY_SUB 4 </P>
<P>#define NX 7 /* output buffer dimensions */ </P>
<P>#define NY 7 </P>
<P>#define NZ 3 </P>
<P>#define RANK 2</P>
<P>#define RANK_OUT 3</P>
<P>main ()</P>
<P>{</P>
<P>hid_t file, dataset; /* handles */</P>
<P>hid_t datatype, dataspace; </P>
<P>hid_t memspace; </P>
<P>H5T_class_t class; /* data type class */</P>
<P>H5T_order_t order; /* data order */</P>
<P>size_t size; /* size of the data element</P>
<P>stored in file */ </P>
<P>hsize_t dimsm[3]; /* memory space dimensions */</P>
<P>hsize_t dims_out[2]; /* dataset dimensions */ </P>
<P>herr_t status; </P>
<P>int data_out[NX][NY][NZ ]; /* output buffer */</P>
<P>hsize_t count[2]; /* size of the hyperslab in the file */</P>
<P>hssize_t offset[2]; /* hyperslab offset in the file */</P>
<P>hsize_t count_out[3]; /* size of the hyperslab in memory */</P>
<P>hssize_t offset_out[3]; /* hyperslab offset in memory */</P>
<P>int i, j, k, status_n, rank;</P>
<P>for (j = 0; j &lt; NX; j++) {</P>
<P>for (i = 0; i &lt; NY; i++) {</P>
<P>for (k = 0; k &lt; NZ ; k++)</P>
<P>data_out[j][i][k] = 0;</P>
<P>}</P>
<P>} </P>
<P>/*</P>
<P>* Open the file and the dataset.</P>
<P>*/</P>
<P>file = H5Fopen(FILE, H5F_ACC_RDONLY, H5P_DEFAULT);</P>
<P>dataset = H5Dopen(file, DATASETNAME);</P>
<P>/*</P>
<P>* Get datatype and dataspace handles and then query</P>
<P>* dataset class, order, size, rank and dimensions.</P>
<P>*/</P>
<P>datatype = H5Dget_type(dataset); /* datatype handle */ </P>
<P>class = H5Tget_class(datatype);</P>
<P>if (class == H5T_INTEGER) printf("Data set has INTEGER type \n");</P>
<P>order = H5Tget_order(datatype);</P>
<P>if (order == H5T_ORDER_LE) printf("Little endian order \n");</P>
<P>size = H5Tget_size(datatype);</P>
<P>printf(" Data size is %d \n", size);</P>
<P>dataspace = H5Dget_space(dataset); /* dataspace handle */</P>
<P>rank = H5Sget_ndims(dataspace);</P>
<P>status_n = H5Sget_dims(dataspace, dims_out, NULL);</P>
<P>printf("rank %d, dimensions %d x %d \n", rank, dims_out[0], dims_out[1]);</P>
<P>/* </P>
<P>* Define hyperslab in the dataset. </P>
<P>*/</P>
<P>offset[0] = 1;</P>
<P>offset[1] = 2;</P>
<P>count[0] = NX_SUB;</P>
<P>count[1] = NY_SUB;</P>
<P>status = H5Sset_hyperslab(dataspace, offset, count, NULL);</P>
<P>/*</P>
<P>* Define the memory dataspace.</P>
<P>*/</P>
<P>dimsm[0] = NX;</P>
<P>dimsm[1] = NY;</P>
<P>dimsm[2] = NZ ;</P>
<P>memspace = H5Screate_simple(RANK_OUT,dimsm,NULL); </P>
<P>/* </P>
<P>* Define memory hyperslab. </P>
<P>*/</P>
<P>offset_out[0] = 3;</P>
<P>offset_out[1] = 0;</P>
<P>offset_out[2] = 0;</P>
<P>count_out[0] = NX_SUB;</P>
<P>count_out[1] = NY_SUB;</P>
<P>count_out[2] = 1;</P>
<P>status = H5Sset_hyperslab(memspace, offset_out, count_out, NULL);</P>
<P>/*</P>
<P>* Read data from hyperslab in the file into the hyperslab in </P>
<P>* memory and display.</P>
<P>*/</P>
<P>status = H5Dread(dataset, H5T_NATIVE_INT, memspace, dataspace,</P>
<P>H5P_DEFAULT, data_out);</P>
<P>for (j = 0; j &lt; NX; j++) {</P>
<P>for (i = 0; i &lt; NY; i++) printf("%d ", data_out[j][i][0]);</P>
<P>printf("\n");</P>
<P>}</P>
<P>/* 0 0 0 0 0 0 0</P>
<P>0 0 0 0 0 0 0</P>
<P>0 0 0 0 0 0 0</P>
<P>3 4 5 6 0 0 0 </P>
<P>4 5 6 7 0 0 0</P>
<P>5 6 7 8 0 0 0</P>
<P>0 0 0 0 0 0 0 */</P>
<P>/*</P>
<P>* Close/release resources.</P>
<P>*/</P>
<P>H5Tclose(datatype);</P>
<P>H5Dclose(dataset);</P>
<P>H5Sclose(dataspace);</P>
<P>H5Sclose(memspace);</P>
<P>H5Fclose(file);</P>
<P>} </P>
</CODE></PRE>
<P>&nbsp;</P>
<H4><A NAME="CompoundExample">Example 3. Working with compound datatypes.</A></H4>
<P>This example shows how to create a compound data type, write an array which has the compound data type to the file, and read back subsets of fields.</P>
<CODE><PRE><P>/*</P>
<P>* This example shows how to create a compound data type,</P>
<P>* write an array which has the compound data type to the file,</P>
<P>* and read back fields' subsets.</P>
<P>*/</P>
<P>#include "hdf5.h"</P>
<P>#define FILE "SDScompound.h5"</P>
<P>#define DATASETNAME "ArrayOfStructures"</P>
<P>#define LENGTH 10</P>
<P>#define RANK 1</P>
<P>main()</P>
<P>{</P>
<P>/* First structure and dataset*/</P>
<P>typedef struct s1_t {</P>
<P>int a;</P>
<P>float b;</P>
<P>double c; </P>
<P>} s1_t;</P>
<P>s1_t s1[LENGTH];</P>
<P>hid_t s1_tid; /* File datatype handle */</P>
<P>/* Second structure (subset of s1_t) and dataset*/</P>
<P>typedef struct s2_t {</P>
<P>double c;</P>
<P>int a;</P>
<P>} s2_t;</P>
<P>s2_t s2[LENGTH];</P>
<P>hid_t s2_tid; /* Memory datatype handle */</P>
<P>/* Third "structure" ( will be used to read float field of s1) */</P>
<P>hid_t s3_tid; /* Memory datatype handle */</P>
<P>float s3[LENGTH];</P>
<P>int i;</P>
<P>hid_t file, datatype, dataset, space; /* Handles */</P>
<P>herr_t status;</P>
<P>hsize_t dim[] = {LENGTH}; /* Dataspace dimensions */</P>
<P>H5T_class_t class;</P>
<P>size_t size;</P>
<P>/*</P>
<P>* Initialize the data</P>
<P>*/</P>
<P>for (i = 0; i&lt; LENGTH; i++) {</P>
<P>s1[i].a = i;</P>
<P>s1[i].b = i*i;</P>
<P>s1[i].c = 1./(i+1);</P>
<P>}</P>
<P>/*</P>
<P>* Create the data space.</P>
<P>*/</P>
<P>space = H5Screate_simple(RANK, dim, NULL);</P>
<P>/*</P>
<P>* Create the file.</P>
<P>*/</P>
<P>file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);</P>
<P>/*</P>
<P>* Create the memory data type. </P>
<P>*/</P>
<P>s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));</P>
<P>status = H5Tinsert(s1_tid, "a_name", HPOFFSET(s1, a), H5T_NATIVE_INT);</P>
<P>status = H5Tinsert(s1_tid, "c_name", HPOFFSET(s1, c), H5T_NATIVE_DOUBLE);</P>
<P>status = H5Tinsert(s1_tid, "b_name", HPOFFSET(s1, b), H5T_NATIVE_FLOAT);</P>
<P>/* </P>
<P>* Create the dataset.</P>
<P>*/</P>
<P>dataset = H5Dcreate(file, DATASETNAME, s1_tid, space, H5P_DEFAULT);</P>
<P>/*</P>
<P>* Write data to the dataset; </P>
<P>*/</P>
<P>status = H5Dwrite(dataset, s1_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s1);</P>
<P>/*</P>
<P>* Release resources</P>
<P>*/</P>
<P>H5Tclose(s1_tid);</P>
<P>H5Sclose(space);</P>
<P>H5Dclose(dataset);</P>
<P>H5Fclose(file);</P>
<P>/*</P>
<P>* Open the file and the dataset.</P>
<P>*/</P>
<P>file = H5Fopen(FILE, H5F_ACC_RDONLY, H5P_DEFAULT);</P>
<P>dataset = H5Dopen(file, DATASETNAME);</P>
<P>/* </P>
<P>* Create a data type for s2</P>
<P>*/</P>
<P>s2_tid = H5Tcreate(H5T_COMPOUND, sizeof(s2_t));</P>
<P>status = H5Tinsert(s2_tid, "c_name", HPOFFSET(s2, c), H5T_NATIVE_DOUBLE);</P>
<P>status = H5Tinsert(s2_tid, "a_name", HPOFFSET(s2, a), H5T_NATIVE_INT);</P>
<P>/*</P>
<P>* Read two fields c and a from s1 dataset. Fields in the file</P>
<P>* are found by their names "c_name" and "a_name".</P>
<P>*/</P>
<P>status = H5Dread(dataset, s2_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s2);</P>
<P>/*</P>
<P>* Display the fields</P>
<P>*/</P>
<P>printf("\n");</P>
<P>printf("Field c : \n");</P>
<P>for( i = 0; i &lt; LENGTH; i++) printf("%.4f ", s2[i].c);</P>
<P>printf("\n");</P>
<P>printf("\n");</P>
<P>printf("Field a : \n");</P>
<P>for( i = 0; i &lt; LENGTH; i++) printf("%d ", s2[i].a);</P>
<P>printf("\n");</P>
<P>/* </P>
<P>* Create a data type for s3.</P>
<P>*/</P>
<P>s3_tid = H5Tcreate(H5T_COMPOUND, sizeof(float));</P>
<P>status = H5Tinsert(s3_tid, "b_name", 0, H5T_NATIVE_FLOAT);</P>
<P>/*</P>
<P>* Read field b from s1 dataset. Field in the file is found by its name.</P>
<P>*/</P>
<P>status = H5Dread(dataset, s3_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s3);</P>
<P>/*</P>
<P>* Display the field</P>
<P>*/</P>
<P>printf("\n");</P>
<P>printf("Field b : \n");</P>
<P>for( i = 0; i &lt; LENGTH; i++) printf("%.4f ", s3[i]);</P>
<P>printf("\n");</P>
<P>/*</P>
<P>* Release resources</P>
<P>*/</P>
<P>H5Tclose(s2_tid);</P>
<P>H5Tclose(s3_tid);</P>
<P>H5Dclose(dataset);</P>
<P>H5Sclose(space);</P>
<P>H5Fclose(file);</P>
<P>}</P>
</CODE></PRE>
<P>&nbsp;</P>
<H4><A NAME="ExtendibleExample">Example 4. Creating and writing an extendible dataset.</A></H4>
<P>This example shows how to create a 3x3 extendible dataset, to extend the dataset to 10x3, then to extend it again to 10x5.</P>
<CODE><PRE><P>/* </P>
<P>* This example shows how to work with extendible dataset.</P>
<P>* In the current version of the library dataset MUST be</P>
<P>* chunked.</P>
<P>* </P>
<P>*/</P>
<P>#include "hdf5.h"</P>
<P>#define FILE "SDSextendible.h5"</P>
<P>#define DATASETNAME "ExtendibleArray" </P>
<P>#define RANK 2</P>
<P>#define NX 10</P>
<P>#define NY 5 </P>
<P>main ()</P>
<P>{</P>
<P>hid_t file; /* handles */</P>
<P>hid_t datatype, dataspace, dataset; </P>
<P>hid_t filespace; </P>
<P>hid_t cparms; </P>
<P>hsize_t dims[2] = { 3, 3}; /* dataset dimensions</P>
<P>at the creation time */ </P>
<P>hsize_t dims1[2] = { 3, 3}; /* data1 dimensions */ </P>
<P>hsize_t dims2[2] = { 7, 1}; /* data2 dimensions */ </P>
<P>hsize_t dims3[2] = { 2, 2}; /* data3 dimensions */ </P>
<P>hsize_t maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED};</P>
<P>hsize_t chunk_dims[2] ={2, 5};</P>
<P>hsize_t size[2];</P>
<P>hssize_t offset[2];</P>
<P>herr_t status; </P>
<P>int data1[3][3] = { 1, 1, 1, /* data to write */</P>
<P>1, 1, 1,</P>
<P>1, 1, 1 }; </P>
<P>int data2[7] = { 2, 2, 2, 2, 2, 2, 2};</P>
<P>int data3[2][2] = { 3, 3,</P>
<P>3, 3};</P>
<P>/*</P>
<P>* Create the data space with unlimited dimensions. </P>
<P>*/</P>
<P>dataspace = H5Screate_simple(RANK, dims, maxdims); </P>
<P>/*</P>
<P>* Create a new file. If file exists its contents will be overwritten.</P>
<P>*/</P>
<P>file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);</P>
<P>/* </P>
<P>* Modify dataset creation properties, i.e. enable chunking.</P>
<P>*/</P>
<P>cparms = H5Pcreate (H5P_DATASET_CREATE);</P>
<P>status = H5Pset_chunk( cparms, RANK, chunk_dims);</P>
<P>/*</P>
<P>* Create a new dataset within the file using cparms</P>
<P>* creation properties.</P>
<P>*/</P>
<P>dataset = H5Dcreate(file, DATASETNAME, H5T_NATIVE_INT, dataspace,</P>
<P>cparms);</P>
<P>/*</P>
<P>* Extend the dataset. This call assures that dataset is at least 3 x 3.</P>
<P>*/</P>
<P>size[0] = 3; </P>
<P>size[1] = 3; </P>
<P>status = H5Dextend (dataset, size);</P>
<P>/*</P>
<P>* Select a hyperslab.</P>
<P>*/</P>
<P>filespace = H5Dget_space (dataset);</P>
<P>offset[0] = 0;</P>
<P>offset[1] = 0;</P>
<P>status = H5Sset_hyperslab(filespace, offset, dims1, NULL); </P>
<P>/*</P>
<P>* Write the data to the hyperslab.</P>
<P>*/</P>
<P>status = H5Dwrite(dataset, H5T_NATIVE_INT, dataspace, filespace,</P>
<P>H5P_DEFAULT, data1);</P>
<P>/*</P>
<P>* Extend the dataset. Dataset becomes 10 x 3.</P>
<P>*/</P>
<P>dims[0] = dims1[0] + dims2[0];</P>
<P>size[0] = dims[0]; </P>
<P>size[1] = dims[1]; </P>
<P>status = H5Dextend (dataset, size);</P>
<P>/*</P>
<P>* Select a hyperslab.</P>
<P>*/</P>
<P>filespace = H5Dget_space (dataset);</P>
<P>offset[0] = 3;</P>
<P>offset[1] = 0;</P>
<P>status = H5Sset_hyperslab(filespace, offset, dims2, NULL); </P>
<P>/*</P>
<P>* Define memory space</P>
<P>*/</P>
<P>dataspace = H5Screate_simple(RANK, dims2, NULL); </P>
<P>/*</P>
<P>* Write the data to the hyperslab.</P>
<P>*/</P>
<P>status = H5Dwrite(dataset, H5T_NATIVE_INT, dataspace, filespace,</P>
<P>H5P_DEFAULT, data2);</P>
<P>/*</P>
<P>* Extend the dataset. Dataset becomes 10 x 5.</P>
<P>*/</P>
<P>dims[1] = dims1[1] + dims3[1];</P>
<P>size[0] = dims[0]; </P>
<P>size[1] = dims[1]; </P>
<P>status = H5Dextend (dataset, size);</P>
<P>/*</P>
<P>* Select a hyperslab</P>
<P>*/</P>
<P>filespace = H5Dget_space (dataset);</P>
<P>offset[0] = 0;</P>
<P>offset[1] = 3;</P>
<P>status = H5Sset_hyperslab(filespace, offset, dims3, NULL); </P>
<P>/*</P>
<P>* Define memory space.</P>
<P>*/</P>
<P>dataspace = H5Screate_simple(RANK, dims3, NULL); </P>
<P>/*</P>
<P>* Write the data to the hyperslab.</P>
<P>*/</P>
<P>status = H5Dwrite(dataset, H5T_NATIVE_INT, dataspace, filespace,</P>
<P>H5P_DEFAULT, data3);</P>
<P>/*</P>
<P>* Resulting dataset</P>
<P>* </P>
<P>3 3 3 2 2</P>
<P>3 3 3 2 2</P>
<P>3 3 3 0 0</P>
<P>2 0 0 0 0</P>
<P>2 0 0 0 0</P>
<P>2 0 0 0 0</P>
<P>2 0 0 0 0</P>
<P>2 0 0 0 0</P>
<P>2 0 0 0 0</P>
<P>2 0 0 0 0</P>
<P>*/ </P>
<P>/*</P>
<P>* Close/release resources.</P>
<P>*/</P>
<P>H5Dclose(dataset);</P>
<P>H5Sclose(dataspace);</P>
<P>H5Sclose(filespace);</P>
<P>H5Fclose(file);</P>
<P>} </P>
</CODE></PRE>
<P>&nbsp;</P>
<H4><A NAME="GroupExample">Example 5. Creating groups.</A></H4>
<P>This example shows how to create an HDF5 file with two groups, and to place some datasets within those groups.</P>
<CODE><PRE><P>/*</P>
<P>* This example shows how to create groups within the file and </P>
<P>* datasets within the file and groups.</P>
<P>*/ </P>
<P>&nbsp;</P>
<P>#include "hdf5.h"</P>
<P>&nbsp;</P>
<P>#define FILE "DIR.h5"</P>
<P>#define RANK 2</P>
<P>main()</P>
<P>{</P>
<P>hid_t file, dir;</P>
<P>hid_t dataset, dataspace;</P>
<P>herr_t status;</P>
<P>hsize_t dims[2];</P>
<P>hsize_t size[1];</P>
<P>/*</P>
<P>* Create a file.</P>
<P>*/</P>
<P>file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);</P>
<P>/*</P>
<P>* Create two groups in a file.</P>
<P>*/</P>
<P>dir = H5Gcreate(file, "/IntData", 0);</P>
<P>status = H5Gclose(dir);</P>
<P>dir = H5Gcreate(file,"/FloatData", 0);</P>
<P>status = H5Gclose(dir);</P>
<P>/* </P>
<P>* Create dataspace for the character string</P>
<P>*/</P>
<P>size[0] = 80;</P>
<P>dataspace = H5Screate_simple(1, size, NULL);</P>
<P>/*</P>
<P>* Create dataset "String" in the root group. </P>
<P>*/</P>
<P>dataset = H5Dcreate(file, "String", H5T_NATIVE_CHAR, dataspace, H5P_DEFAULT);</P>
<P>H5Dclose(dataset);</P>
<P>/*</P>
<P>* Create dataset "String" in the /IntData group. </P>
<P>*/</P>
<P>dataset = H5Dcreate(file, "/IntData/String", H5T_NATIVE_CHAR, dataspace,</P>
<P>H5P_DEFAULT);</P>
<P>H5Dclose(dataset);</P>
<P>/*</P>
<P>* Create dataset "String" in the /FloatData group. </P>
<P>*/</P>
<P>dataset = H5Dcreate(file, "/FloatData/String", H5T_NATIVE_CHAR, dataspace,</P>
<P>H5P_DEFAULT);</P>
<P>H5Sclose(dataspace);</P>
<P>H5Dclose(dataset);</P>
<P>/*</P>
<P>* Create IntArray dataset in the /IntData group by specifying full path.</P>
<P>*/</P>
<P>dims[0] = 2;</P>
<P>dims[1] = 3;</P>
<P>dataspace = H5Screate_simple(RANK, dims, NULL);</P>
<P>dataset = H5Dcreate(file, "/IntData/IntArray", H5T_NATIVE_INT, dataspace,</P>
<P>H5P_DEFAULT); </P>
<P>H5Sclose(dataspace);</P>
<P>H5Dclose(dataset);</P>
<P>/*</P>
<P>* Set current group to /IntData and attach to the dataset String.</P>
<P>*/</P>
<P>status = H5Gset (file, "/IntData");</P>
<P>dataset = H5Dopen(file, "String");</P>
<P>if (dataset &gt; 0) printf("String dataset in /IntData group is found\n"); </P>
<P>H5Dclose(dataset);</P>
<P>/*</P>
<P>* Set current group to /FloatData.</P>
<P>*/</P>
<P>status = H5Gset (file, "/FloatData");</P>
<P>/* </P>
<P>* Create two datasets FlatArray and DoubleArray.</P>
<P>*/</P>
<P>dims[0] = 5;</P>
<P>dims[1] = 10;</P>
<P>dataspace = H5Screate_simple(RANK, dims, NULL);</P>
<P>dataset = H5Dcreate(file, "FloatArray", H5T_NATIVE_FLOAT, dataspace, H5P_DEFAULT); </P>
<P>H5Sclose(dataspace);</P>
<P>H5Dclose(dataset);</P>
<P>dims[0] = 4;</P>
<P>dims[1] = 6;</P>
<P>dataspace = H5Screate_simple(RANK, dims, NULL);</P>
<P>dataset = H5Dcreate(file, "DoubleArray", H5T_NATIVE_DOUBLE, dataspace,</P>
<P>H5P_DEFAULT); </P>
<P>H5Sclose(dataspace);</P>
<P>H5Dclose(dataset);</P>
<P>/* </P>
<P>* Attach to /FloatData/String dataset.</P>
<P>*/</P>
<P>dataset = H5Dopen(file, "/FloatData/String");</P>
<P>if (dataset &gt; 0) printf("/FloatData/String dataset is found\n"); </P>
<P>H5Dclose(dataset);</P>
<P>H5Fclose(file);</P>
<P>}</P></CODE></PRE></BODY>
</HTML>

123
doc/html/H5.sample_code.html Normal file

@ -0,0 +1,123 @@
<html><head><title>
HDF5 Draft API Example Code
</title></head><body>
<center>
<h1>HDF5: API Example Code</h1>
</center>
<P>Example programs/sections of code below:
<dl COMPACT>
<dt><a href="#Example1">#1</a>
<dd>A simple example showing how to create a file.
<dt><a href="#Example2">#2</a>
<dd>A example showing how to create a homogenous multi-dimensional dataset.
<dt><a href="#Example3">#3</a>
<dd>A example showing how to read a generic dataset.
</dl>
<hr>
<h2><a name="Example1">Simple Example showing how to create a file.</a></h2>
<P>Notes:<br>
This example creates a new HDF5 file and allows write access.
If the file exists already, the H5F_ACC_TRUNC flag would also be necessary to
overwrite the previous file's information.
<P>Code:
<code> <pre>
hid_t file_id;
file_id=<A HREF="H5.apiv2.html#File-Create">H5Fcreate</a>("example1.h5",H5F_ACC_EXCL,H5P_DEFAULT_TEMPLATE,H5P_DEFAULT_TEMPLATE);
<A HREF="H5.apiv2.html#File-Close">H5Fclose</a>(file_id);
</pre> </code>
<hr>
<h2><a name="Example2">Example showing how create a homogenous multi-dimensional dataset.</a></h2>
<P>Notes:<br>
This example creates a 4-dimensional dataset of 32-bit floating-point
numbers, corresponding to the current Scientific Dataset functionality.
<P>Code:
<code> <pre>
1 hid_t file_id; /* new file's ID */
2 hid_t dim_id; /* new dimensionality's ID */
3 int rank=4; /* the number of dimensions */
4 hsize_t dims[4]={6,5,4,3}; /* the size of each dimension */
5 hid_t dataset_id; /* new dataset's ID */
6 float buf[6][5][4][3]; /* storage for the dataset's data */
7 herr_t status; /* function return status */
8
9 file_id = H5Fcreate ("example3.h5", H5F_ACC_TRUNC, H5P_DEFAULT,
10 H5P_DEFAULT);
11 assert (file_id &gt;= 0);
12
13 /* Create & initialize a dimensionality object */
14 dim_id = H5Screate_simple (rank, dims);
15 assert (dim_id &gt;= 0);
16
17 /* Create & initialize the dataset object */
18 dataset_id = H5Dcreate (file_id, "Simple Object", H5T_NATIVE_FLOAT,
19 dim_id, H5P_DEFAULT);
20 assert (dataset_id &gt;= 0);
21
22 &lt;initialize data array&gt;
23
24 /* Write the entire dataset out */
25 status = H5Dwrite (dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL,
26 H5P_DEFAULT, buf);
27 assert (status &gt;= 0);
28
29 /* Release the IDs we've created */
30 H5Sclose (dim_id);
31 H5Dclose (dataset_id);
32 H5Fclose (file_id);
</pre> </code>
<hr>
<h2><a name="Example3">Example showing how read a generic dataset.</a></h2>
<P>Notes:<br>
This example shows how to get the information for and display a generic
dataset.
<P>Code:
<code> <pre>
1 hid_t file_id; /* file's ID */
2 hid_t dataset_id; /* dataset's ID in memory */
3 hid_t space_id; /* dataspace's ID in memory */
4 uintn nelems; /* number of elements in array */
5 double *buf; /* pointer to the dataset's data */
6 herr_t status; /* function return value */
7
8 file_id = H5Fopen ("example6.h5", H5F_ACC_RDONLY, H5P_DEFAULT);
9 assert (file_id &gt;= 0);
10
11 /* Attach to a datatype object */
12 dataset_id = H5Dopen (file_id, "dataset1");
13 assert (dataset_id &gt;= 0);
14
15 /* Get the OID for the dataspace */
16 space_id = H5Dget_space (dataset_id);
17 assert (space_id &gt;= 0);
18
19 /* Allocate space for the data */
20 nelems = H5Sget_npoints (space_id);
21 buf = malloc (nelems * sizeof(double));
22
23 /* Read in the dataset */
24 status = H5Dread (dataset_id, H5T_NATIVE_DOUBLE, H5S_ALL,, H5S_ALL,
25 H5P_DEFAULT, buf);
26 assert (status &gt;= 0);
27
28 /* Release the IDs we've accessed */
29 H5Sclose (space_id);
30 H5Dclose (dataset_id);
31 H5Fclose (file_id);
</pre> </code>

71
doc/html/H5.user.html Normal file

@ -0,0 +1,71 @@
<html>
<head>
<title>
HDF5 Uer's Guide
</title>
</head>
<body>
<center><h1>A User Guide for HDF5</h1></center>
<p>The following documents form a loosely organized user's guide
to the HDF5 library.
<ul>
<li><a href="Files.html">HDF5 Files</a> -
A guide to the H5F interface.
<li><a href="Datasets.html">Datasets</a> -
A guide to the H5D interface.
<li><a href="Datatypes.html">Data types</a> -
A guide to the H5T interface.
<li><a href="Dataspaces.html">Data spaces</a> -
A guide to the H5S interface.
<li><a href="Groups.html">Groups</a> -
A guide to the H5G interface.
<li><a href="Attributes.html">Attributes</a> -
A guide to the H5A interface.
<li><a href="Properties.html">Property lists</a> -
A guide to the H5P interface.
<li><a href="Errors.html">Error handling</a> -
A guide to the H5E interface.
<li><a href="Caching.html">Caching</a> -
A guide for meta and raw data caching.
</ul>
<p>The following documents form a loosely organized developer's guide to
aspects of the HDF5 library. (Some of the following documents
may be rather out of date as they were working papers for design
goals.)
<ul>
<li><a href="Version.html">Version Numbers</a> -
A description of HDF5 version numbers.
<li><a href="IOPipe.html">I/O Pipeline</a> -
A description of the raw data I/O pipeline.
<li><a href="Compression.html">Compression</a> -
A guide to the use of compression for datasets.
<li><a href="ExternalFiles.html">Working with external files</a> -
A guide to the use of multiple files with HDF5.
<li><a href="Big.html">Large Datasets</a> -
A guide to accessing large datasets on small computers.
<li><a href="MemoryManagement.html">Memory management</a> -
A white paper about memory management issues in HDF5.
<li><a href="Coding.html">Coding style</a> -
A white paper about coding style in the HDF5 library code.
<li><a href="compat.html">HDF4/HDF5 compatibility</a> -
A white paper about compatibility issues between HDF4 and HDF5.
<li><a href="study.html">Chunking performance study</a> -
A white paper describing the HDF5 chunking performance.
</ul>
<hr>
<address><a href="mailto:koziol@ncsa.uiuc.edu">Quincey Koziol</a></address>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- hhmts start -->
Last modified: Tue May 26 15:39:47 EDT 1998
<!-- hhmts end -->
</body>
</html>

114
doc/html/IOPipe.html Normal file

@ -0,0 +1,114 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>The Raw Data I/O Pipeline</title>
</head>
<body>
<h1>The Raw Data I/O Pipeline</h1>
<p>The HDF5 raw data pipeline is a complicated beast that handles
all aspects of raw data storage and transfer of that data
between the file and the application. Data can be stored
contiguously (internal or external), in variable size external
segments, or regularly chunked; it can be sparse, extendible,
and/or compressible. Data transfers must be able to convert from
one data space to another, convert from one number type to
another, and perform partial I/O operations. Furthermore,
applications will expect their common usage of the pipeline to
perform well.
<p>To accomplish these goals, the pipeline has been designed in a
modular way so no single subroutine is overly complicated and so
functionality can be inserted easily at the appropriate
locations in the pipeline. A general pipeline was developed and
then certain paths through the pipeline were optimized for
performance.
<p>We describe only the file-to-memory side of the pipeline since
the memory-to-file side is a mirror image. We also assume that a
proper hyperslab of a simple data space is being read from the
file into a proper hyperslab of a simple data space in memory,
and that the data type is a compound type which may require
various number conversions on its members.
<img alt="Figure 1" src="pipe1.gif">
<p>The diagrams should be read from the top down. The Line A
in the figure above shows that <code>H5Dread()</code> copies
data from a hyperslab of a file dataset to a hyperslab of an
application buffer by calling <code>H5D_read()</code>. And
<code>H5D_read()</code> calls, in a loop,
<code>H5S_simp_fgath()</code>, <code>H5T_conv_struct()</code>,
and <code>H5S_simp_mscat()</code>. A temporary buffer, TCONV, is
loaded with data points from the file, then data type conversion
is performed on the temporary buffer, and finally data points
are scattered out to application memory. Thus, data type
conversion is an in-place operation and data space conversion
consists of two steps. An additional temporary buffer, BKG, is
large enough to hold <em>N</em> instances of the destination
data type where <em>N</em> is the same number of data points
that can be held by the TCONV buffer (which is large enough to
hold either source or destination data points).
<p>The application sets an upper limit for the size of the TCONV
buffer and optionally supplies a buffer. If no buffer is
supplied then one will be created by calling
<code>malloc()</code> when the pipeline is executed (when
necessary) and freed when the pipeline exits. The size of the
BKG buffer depends on the size of the TCONV buffer and if the
application supplies a BKG buffer it should be at least as large
as the TCONV buffer. The default size for these buffers is one
megabyte but the buffer might not be used to full capacity if
the buffer size is not an integer multiple of the source or
destination data point size (whichever is larger, but only
destination for the BKG buffer).
<p>Occassionally the destination data points will be partially
initialized and the <code>H5Dread()</code> operation should not
clobber those values. For instance, the destination type might
be a struct with members <code>a</code> and <code>b</code> where
<code>a</code> is already initialized and we're reading
<code>b</code> from the file. An extra line, G, is added to the
pipeline to provide the type conversion functions with the
existing data.
<img alt="Figure 2" src="pipe2.gif">
<p>It will most likely be quite common that no data type
conversion is necessary. In such cases a temporary buffer for
data type conversion is not needed and data space conversion
can happen in a single step. In fact, when the source and
destination data are both contiguous (they aren't in the
picture) the loop degenerates to a single iteration.
<img alt="Figure 3" src="pipe3.gif">
<p>So far we've looked only at internal contiguous storage, but by
replacing Line B in Figures 1 and 2 and Line A in Figure 3 with
Figure 4 the pipeline is able to handle regularly chunked
objects. Line B of Figure 4 is executed once for each chunk
which contains data to be read and the chunk address is found by
looking at a multi-dimensional key in a chunk B-tree which has
one entry per chunk.
<img alt="Figure 4" src="pipe4.gif">
<p>If a single chunk is requested and the destination buffer is
the same size/shape as the chunk, then the CHUNK buffer is
bypassed and the destination buffer is used instead as shown in
Figure 5.
<img alt="Figure 5" src="pipe5.gif">
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Tue Mar 17 11:13:35 EST 1998 -->
<!-- hhmts start -->
Last modified: Wed Mar 18 10:38:30 EST 1998
<!-- hhmts end -->
</body>
</html>

510
doc/html/MemoryManagement.html Normal file

@ -0,0 +1,510 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Memory Management in HDF5</title>
</head>
<body>
<h1>Memory Management in HDF5</h1>
<!-- ---------------------------------------------------------------- -->
<h2>Is a Memory Manager Necessary?</h2>
<p>Some form of memory management may be necessary in HDF5 when
the various deletion operators are implemented so that the
file memory is not permanently orphaned. However, since an
HDF5 file was designed with persistent data in mind, the
importance of a memory manager is questionable.
<p>On the other hand, when certain meta data containers (file glue)
grow, they may need to be relocated in order to keep the
container contiguous.
<blockquote>
<b>Example:</b> An object header consists of up to two
chunks of contiguous memory. The first chunk is a fixed
size at a fixed location when the header link count is
greater than one. Thus, inserting additional items into an
object header may require the second chunk to expand. When
this occurs, the second chunk may need to move to another
location in the file, freeing the file memory which that
chunk originally occupied.
</blockquote>
<p>The relocation of meta data containers could potentially
orphan a significant amount of file memory if the application
has made poor estimates for preallocation sizes.
<!-- ---------------------------------------------------------------- -->
<h2>Levels of Memory Management</h2>
<p>Memory management by the library can be independent of memory
management support by the file format. The file format can
support no memory management, some memory management, or full
memory management. Similarly with the library.
<h3>Support in the Library</h3>
<dl>
<dt><b>No Support: I</b>
<dd>When memory is deallocated it simply becomes unreferenced
(orphaned) in the file. Memory allocation requests are
satisfied by extending the file.
<dd>A separate off-line utility can be used to detect the
unreferenced bytes of a file and "bubble" them up to the end
of the file and then truncate the file.
<dt><b>Some Support: II</b>
<dd>The library could support partial memory management all
the time, or full memory management some of the time.
Orphaning free blocks instead of adding them to a free list
should not affect the file integrity, nor should fulfilling
new requests by extending the file instead of using the free
list.
<dt><b>Full Support: III</b>
<dd>The library supports space-efficient memory management by
always fulfilling allocation requests from the free list when
possible, and by coalescing adjacent free blocks into a
single larger free block.
</dl>
<h3>Support in the File Format</h3>
<dl>
<dt><b>No Support: A</b>
<dd>The file format does not support memory management; any
unreferenced block in the file is assumed to be free. If
the library supports full memory management then it will
have to traverse the entire file to determine which blocks
are unreferenced.
<dt><b>Some Support: B</b>
<dd>Assuming that unreferenced blocks are free can be
dangerous in a situation where the file is not consistent.
For instance, if a directory tree becomes detached from the
main directory hierarchy, then the detached directory and
everything that is referenced only through the detached
directory become unreferenced. File repair utilities will
be unable to determine which unreferenced blocks need to be
linked back into the file hierarchy.
<dd>Therefore, it might be useful to keep an unsorted,
doubly-linked list of free blocks in the file. The library
can add and remove blocks from the list in constant time,
and can generate its own internal free-block data structure
in time proportional to the number of free blocks instead of
the size of the file. Additionally, a library can use a
subset of the free blocks, an alternative which is not
feasible if the file format doesn't support any form of
memory management.
<dt><b>Full Support: C</b>
<dd>The file format can mirror library data structures for
space-efficient memory management. The free blocks are
linked in unsorted, doubly-linked lists with one list per
free block size. The heads of the lists are pointed to by a
B-tree whose nodes are sorted by free block size. At the
same time, all free blocks are the leaf nodes of another
B-tree sorted by starting and ending address. When the
trees are used in combination we can deallocate and allocate
memory in O(log <em>N</em>) time where <em>N</em> is the
number of free blocks.
</dl>
<h3>Combinations of Library and File Format Support</h3>
<p>We now evaluate each combination of library support with file
support:
<dl>
<dt><b>I-A</b>
<dd>If neither the library nor the file support memory
management, then each allocation request will come from the
end of the file and each deallocation request is a no-op
that simply leaves the free block unreferenced.
<ul>
<li>Advantages
<ul>
<li>No file overhead for allocation or deallocation.
<li>No library overhead for allocation or
deallocation.
<li>No file traversal required at time of open.
<li>No data needs to be written back to the file when
it's closed.
<li>Trivial to implement (already implemented).
</ul>
<li>Disadvantages
<ul>
<li>Inefficient use of file space.
<li>A file repair utility must reclaim lost file space.
<li>Difficulties for file repair utilities. (Is an
unreferenced block a free block or orphaned data?)
</ul>
</ul>
<dt><b>II-A</b>
<dd>In order for the library to support memory management, it
will be required to build the internal free block
representation by traversing the entire file looking for
unreferenced blocks.
<ul>
<li>Advantages
<ul>
<li>No file overhead for allocation or deallocation.
<li>Variable amount of library overhead for allocation
and deallocation depending on how much work the
library wants to do.
<li>No data needs to be written back to the file when
it's closed.
<li>Might use file space efficiently.
</ul>
<li>Disadvantages
<ul>
<li>Might use file space inefficiently.
<li>File traversal required at time of open.
<li>A file repair utility must reclaim lost file space.
<li>Difficulties for file repair utilities.
<li>Sharing of the free list between processes falls
outside the HDF5 file format documentation.
</ul>
</ul>
<dt><b>III-A</b>
<dd>In order for the library to support full memory
management, it will be required to build the internal free
block representation by traversing the entire file looking
for unreferenced blocks.
<ul>
<li>Advantages
<ul>
<li>No file overhead for allocation or deallocation.
<li>Efficient use of file space.
<li>No data needs to be written back to the file when
it's closed.
</ul>
<li>Disadvantages
<ul>
<li>Moderate amount of library overhead for allocation
and deallocation.
<li>File traversal required at time of open.
<li>A file repair utility must reclaim lost file space.
<li>Difficulties for file repair utilities.
<li>Sharing of the free list between processes falls
outside the HDF5 file format documentation.
</ul>
</ul>
<dt><b>I-B</b>
<dd>If the library doesn't support memory management but the
file format supports some level of management, then a file
repair utility will have to be run occasionally to reclaim
unreferenced blocks.
<ul>
<li>Advantages
<ul>
<li>No file overhead for allocation or deallocation.
<li>No library overhead for allocation or
deallocation.
<li>No file traversal required at time of open.
<li>No data needs to be written back to the file when
it's closed.
</ul>
<li>Disadvantages
<ul>
<li>A file repair utility must reclaim lost file space.
<li>Difficulties for file repair utilities.
</ul>
</ul>
<dt><b>II-B</b>
<dd>Both the library and the file format support some level
of memory management.
<ul>
<li>Advantages
<ul>
<li>Constant file overhead per allocation or
deallocation.
<li>Variable library overhead per allocation or
deallocation depending on how much work the library
wants to do.
<li>Traversal at file open time is on the order of the
free list size instead of the file size.
<li>The library has the option of reading only part of
the free list.
<li>No data needs to be written at file close time if
it has been amortized into the cost of allocation
and deallocation.
<li>File repair utilties don't have to be run to
reclaim memory.
<li>File repair utilities can detect whether an
unreferenced block is a free block or orphaned data.
<li>Sharing of the free list between processes might
be easier.
<li>Possible efficient use of file space.
</ul>
<li>Disadvantages
<ul>
<li>Possible inefficient use of file space.
</ul>
</ul>
<dt><b>III-B</b>
<dd>The library provides space-efficient memory management but
the file format only supports an unsorted list of free
blocks.
<ul>
<li>Advantages
<ul>
<li>Constant time file overhead per allocation or
deallocation.
<li>No data needs to be written at file close time if
it has been amortized into the cost of allocation
and deallocation.
<li>File repair utilities don't have to be run to
reclaim memory.
<li>File repair utilities can detect whether an
unreferenced block is a free block or orphaned data.
<li>Sharing of the free list between processes might
be easier.
<li>Efficient use of file space.
</ul>
<li>Disadvantages
<ul>
<li>O(log <em>N</em>) library overhead per allocation or
deallocation where <em>N</em> is the total number of
free blocks.
<li>O(<em>N</em>) time to open a file since the entire
free list must be read to construct the in-core
trees used by the library.
<li>Library is more complicated.
</ul>
</ul>
<dt><b>I-C</b>
<dd>This has the same advantages and disadvantages as I-C with
the added disadvantage that the file format is much more
complicated.
<dt><b>II-C</b>
<dd>If the library only provides partial memory management but
the file requires full memory management, then this method
degenerates to the same as II-A with the added disadvantage
that the file format is much more complicated.
<dt><b>III-C</b>
<dd>The library and file format both provide complete data
structures for space-efficient memory management.
<ul>
<li>Advantages
<ul>
<li>Files can be opened in constant time since the
free list is read on demand and amortised into the
allocation and deallocation requests.
<li>No data needs to be written back to the file when
it's closed.
<li>File repair utilities don't have to be run to
reclaim memory.
<li>File repair utilities can detect whether an
unreferenced block is a free block or orphaned data.
<li>Sharing the free list between processes is easy.
<li>Efficient use of file space.
</ul>
<li>Disadvantages
<ul>
<li>O(log <em>N</em>) file allocation and deallocation
cost where <em>N</em> is the total number of free
blocks.
<li>O(log <em>N</em>) library allocation and
deallocation cost.
<li>Much more complicated file format.
<li>More complicated library.
</ul>
</ul>
</dl>
<!-- ---------------------------------------------------------------- -->
<h2>The Algorithm for II-B</h2>
<p>The file contains an unsorted, doubly-linked list of free
blocks. The address of the head of the list appears in the
boot block. Each free block contains the following fields:
<center>
<table border cellpadding=4 width="60%">
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<th colspan=4>Free Block Signature</th>
<tr align=center>
<th colspan=4>Total Free Block Size</th>
<tr align=center>
<th colspan=4>Address of Left Sibling</th>
<tr align=center>
<th colspan=4>Address of Right Sibling</th>
<tr align=center>
<th colspan=4><br><br>Remainder of Free Block<br><br><br></th>
</table>
</center>
<p>The library reads as much of the free list as convenient when
convenient and pushes those entries onto stacks. This can
occur when a file is opened or any time during the life of the
file. There is one stack for each free block size and the
stacks are sorted by size in a balanced tree in memory.
<p>Deallocation involves finding the correct stack or creating
a new one (an O(log <em>K</em>) operation where <em>K</em> is
the number of stacks), pushing the free block info onto the
stack (a constant-time operation), and inserting the free
block into the file free block list (a constant-time operation
which doesn't necessarily involve any I/O since the free blocks
can be cached like other objects). No attempt is made to
coalesce adjacent free blocks into larger blocks.
<p>Allocation involves finding the correct stack (an O(log
<em>K</em>) operation), removing the top item from the stack
(a constant-time operation), and removing the block from the
file free block list (a constant-time operation). If there is
no free block of the requested size or larger, then the file
is extended.
<p>To provide sharability of the free list between processes,
the last step of an allocation will check for the free block
signature and if it doesn't find one will repeat the process.
Alternatively, a process can temporarily remove free blocks
from the file and hold them in it's own private pool.
<p>To summarize...
<dl>
<dt>File opening
<dd>O(<em>N</em>) amortized over the time the file is open,
where <em>N</em> is the number of free blocks. The library
can still function without reading any of the file free
block list.
<dt>Deallocation
<dd>O(log <em>K</em>) where <em>K</em> is the number of unique
sizes of free blocks. File access is constant.
<dt>Allocation
<dd>O(log <em>K</em>). File access is constant.
<dt>File closing
<dd>O(1) even if the library temporarily removes free
blocks from the file to hold them in a private pool since
the pool can still be a linked list on disk.
</dl>
<!-- ---------------------------------------------------------------- -->
<h2>The Algorithm for III-C</h2>
<p>The HDF5 file format supports a general B-tree mechanism
for storing data with keys. If we use a B-tree to represent
all parts of the file that are free and the B-tree is indexed
so that a free file chunk can be found if we know the starting
or ending address, then we can efficiently determine whether a
free chunk begins or ends at the specified address. Call this
the <em>Address B-Tree</em>.
<p>If a second B-tree points to a set of stacks where the
members of a particular stack are all free chunks of the same
size, and the tree is indexed by chunk size, then we can
efficiently find the best-fit chunk size for a memory request.
Call this the <em>Size B-Tree</em>.
<p>All free blocks of a particular size can be linked together
with an unsorted, doubly-linked, circular list and the left
and right sibling addresses can be stored within the free
chunk, allowing us to remove or insert items from the list in
constant time.
<p>Deallocation of a block fo file memory consists of:
<ol type="I">
<li>Add the new free block whose address is <em>ADDR</em> to the
address B-tree.
<ol type="A">
<li>If the address B-tree contains an entry for a free
block that ends at <em>ADDR</em>-1 then remove that
block from the B-tree and from the linked list (if the
block was the first on the list then the size B-tree
must be updated). Adjust the size and address of the
block being freed to include the block just removed from
the free list. The time required to search for and
possibly remove the left block is O(log <em>N</em>)
where <em>N</em> is the number of free blocks.
<li>If the address B-tree contains an entry for the free
block that begins at <em>ADDR</em>+<em>LENGTH</em> then
remove that block from the B-tree and from the linked
list (if the block was the first on the list then the
size B-tree must be updated). Adjust the size of the
block being freed to include the block just removed from
the free list. The time required to search for and
possibly remove the right block is O(log <em>N</em>).
<li>Add the new (adjusted) block to the address B-tree.
The time for this operation is O(log <em>N</em>).
</ol>
<li>Add the new block to the size B-tree and linked list.
<ol type="A">
<li>If the size B-tree has an entry for this particular
size, then add the chunk to the tail of the list. This
is an O(log <em>K</em>) operation where <em>K</em> is
the number of unique free block sizes.
<li>Otherwise make a new entry in the B-tree for chunks of
this size. This is also O(log <em>K</em>).
</ol>
</ol>
<p>Allocation is similar to deallocation.
<p>To summarize...
<dl>
<dt>File opening
<dd>O(1)
<dt>Deallocation
<dd>O(log <em>N</em>) where <em>N</em> is the total number of
free blocks. File access time is O(log <em>N</em>).
<dt>Allocation
<dd>O(log <em>N</em>). File access time is O(log <em>N</em>).
<dt>File closing
<dd>O(1).
</dl>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Thu Jul 24 15:16:40 PDT 1997 -->
<!-- hhmts start -->
Last modified: Thu Jul 31 14:41:01 EST
<!-- hhmts end -->
</body>
</html>

60
doc/html/ObjectHeader.txt Normal file

@ -0,0 +1,60 @@
OBJECT HEADERS
--------------
haddr_t
H5O_new (hdf5_file_t *f, intn nrefs, size_t size_hint)
Creates a new empty object header and returns its address.
The SIZE_HINT is the initial size of the data portion of the
object header and NREFS is the number of symbol table entries
that reference this object header (normally one).
If SIZE_HINT is too small, then at least some default amount
of space is allocated for the object header.
intn /*num remaining links */
H5O_link (hdf5_file_t *f, /*file containing header */
haddr_t addr, /*header file address */
intn adjust) /*link adjustment amount */
size_t
H5O_sizeof (hdf5_file_t *f, /*file containing header */
haddr_t addr, /*header file address */
H5O_class_t *type, /*message type or H5O_ANY */
intn sequence) /*sequence number, usually zero */
Returns the size of a particular instance of a message in an
object header. When an object header has more than one
instance of a particular message type, then SEQUENCE indicates
which instance to return.
void *
H5O_read (hdf5_file_t *f, /*file containing header */
haddr_t addr, /*header file address */
H5G_entry_t *ent, /*optional symbol table entry */
H5O_class_t *type, /*message type or H5O_ANY */
intn sequence, /*sequence number, usually zero */
size_t size, /*size of output message */
void *mesg) /*output buffer */
Reads a message from the object header into memory.
const void *
H5O_peek (hdf5_file_t *f, /*file containing header */
haddr_t addr, /*header file address */
H5G_entry_t *ent, /*optional symbol table entry */
H5O_class_t *type, /*type of message or H5O_ANY */
intn sequence) /*sequence number, usually zero */
haddr_t /*new heap address */
H5O_modify (hdf5_file_t *f, /*file containing header */
haddr_t addr, /*header file address */
H5G_entry_t *ent, /*optional symbol table entry */
hbool_t *ent_modified, /*entry modification flag */
H5O_class_t *type, /*message type */
intn overwrite, /*sequence number or -1 */
void *mesg) /*the message */

81
doc/html/Properties.html Normal file

@ -0,0 +1,81 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Property List Interface (H5P)</title>
</head>
<body>
<h1>Property List Interface (H5P)</h1>
<h2>1. Introduction</h2>
<p>The property list (a.k.a., template) interface provides a
mechanism for default named arguments for a C function
interface. A property list is a collection of name/value pairs
which can be passed to various other HDF5 functions to control
features that are typically unimportant or whose default values
are usually used.
<p>For instance, file creation needs to know various things such
as the size of the user-block at the beginning of the file, or
the size of various file data structures. Wrapping this
information in a property list simplifies the API by reducing
the number of arguments to <code>H5Fcreate()</code>.
<h2>2. General Property List Operations</h2>
<p>Property lists follow the same create/open/close paradigm as
the rest of the library.
<dl>
<dt><code>hid_t H5Pcreate (H5P_class_t <em>class</em>)</code>
<dd>A new property list can be created as an instance of some
property list class. The new property list is initialized
with default values for the specified class. The classes are:
<br><br>
<dl>
<dt><code>H5P_FILE_CREATE</code>
<dd>Properties for file creation. See <a href="Files.html">H5F</a>
for details about the file creation properties.
<dt><code>H5P_FILE_ACCESS</code>
<dd>Properties for file access. See <a href="Files.html">H5F</a>
for details about the file creation properties.
<dt><code>H5P_DATASET_CREATE</code>
<dd>Properties for dataset creation. See
<a href="Datasets.html">H5D</a> for details about dataset
creation properties.
<dt><code>H5P_DATASET_XFER</code>
<dd>Properties for raw data transfer. See
<a href="Datasets.html">H5D</a> for details about raw data
transfer properties.
</dl>
<br><br>
<dt><code>hid_t H5Pcopy (hid_t <em>plist</em>)</code>
<dd>A property list can be copied to create a new property
list. The new property list has the same properties and values
as the original property list.
<br><br>
<dt><code>herr_t H5Pclose (hid_t <em>plist</em>)</code>
<dd>All property lists should be closed when the application is
finished accessing them. This frees resources used by the
property list.
<br><br>
<dt><code>H5P_class_t H5Pget_class (hid_t <em>plist</em>)</code>
<dd>The class of which the property list is a member can be
obtained by calling this function. The property list classes
are defined above for <code>H5Pcreate()</code>.
</dl>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<address><a href="mailto:koziol@ncsa.uiuc.edu">Quincey Koziol</a></address>
<!-- Created: Tue Feb 10 17:06:58 PST 1998 -->
<!-- hhmts start -->
Last modified: Tue Feb 10 17:26:41 PST 1998
<!-- hhmts end -->
</body>
</html>

139
doc/html/Version.html Normal file

@ -0,0 +1,139 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Version Numbers</title>
</head>
<body>
<h1>Version Numbers</h1>
<h2>1. Introduction</h2>
<p>The HDF5 version number is a set of three integer values and
one lower-case letter written as, for example,
<code>hdf5-1.2.0a</code>.
<p>The <code>5</code> is part of the library name and will only
change if the entire file format and library are redesigned
similar in scope to the changes between HDF4 and HDF5.
<p>The <code>1</code> is the <em>major version number</em> and
changes when there is an extensive change to the file format or
library. Such a change will likely require files to be
translated and applications to be modified. This number is not
expected to change frequently.
<p>The <code>2</code> is the <em>minor version number</em> and is
incremented by each public release that presents new features.
Even numbers are reserved for stable public versions of the
library while odd numbers are reserved for developement
versions. See the diagram below for examples.
<p>The <code>0</code> is the <em>release number</em>. For public
versions of the library, the release number is incremented each
time a bug(s) is fixed and the fix is made available to the
public. For development versions, the release number is
incremented automatically each time a CVS commit occurs anywhere
in the source tree.
<p>The <code>a</code> is the <em>patch level</em> and is used only
for public versions. It's incremented only for very minor
changes that don't affect the usability of the library. For
instance, fixing spelling errors, changing warning messages, or
updating documentation.
<h2>2. Abbreviated Versions</h2>
<p>It's often convenient to drop the patch level and release
number when referring to a version of the library, like saying
version 1.2 of HDF5. The release number and patch level can be
any value in this case.
<h2>3. Special Versions</h2>
<p>Version 1.0.0 was released for alpha testing the first week of
March, 1998. The developement version number was incremented to
1.0.1 and remained constant until the the last week of April,
when the release number started to increase and development
versions were made available to people outside the core HDF5
development team.
<p>Version 1.1.0 will be the first official beta release but the
1.1 branch will also serve as a development branch since we're
not concerned about providing bug fixes separate from normal
development for the beta version.
<p>Version 1.2 will be the first official HDF5 version. The
version tree will fork at this point with public bug fixes
provided on the 1.2 branch and development will continue on the
1.3 branch.
<h2>4. Public versus Development</h2>
<p>The motivation for separate public and development versions is
that the public version will receive only bug fixes while the
development version will receive new features.
<p>Eventually, the development version will near completion and a
new development branch will fork while the original one enters a
feature freeze state. When the original development branch is
ready for release the minor version number will be incremented
to an even value.
<p>
<center>
<img alt="Version Example" src="version.gif">
<br><b>Fig 1: Version Example</b>
</center>
<h2>5. Version Support from the Library</h2>
<p>The library provides a set of macros and functions to query and
check version numbers.
<dl>
<dt><code>H5_VERS_MAJOR</code>
<dt><code>H5_VERS_MINOR</code>
<dt><code>H5_VERS_RELEASE</code>
<dt><code>H5_VERS_PATCH</code>
<dd>These preprocessor constants are defined in the public
include file and determine the version of the include files.
<br><br>
<dt><code>herr_t H5version (unsigned *<em>majnum</em>, unsigned
*<em>minnum</em>, unsigned *<em>relnum</em>, unsigned
*<em>patnum</em>)</code>
<dd>This function returns through its arguments the version
numbers for the library to which the application is linked.
<br><br>
<dt><code>void H5check(void)</code>
<dd>This is a macro that verifies that the version number of the
HDF5 include file used to compile the application matches the
version number of the library to which the application is
linked. This check occurs automatically when the first HDF5
file is created or opened and is important because a mismatch
between the include files and the library is likely to result
in corrupted data and/or segmentation faults. If a mismatch
is detected the library issues an error message on the
standard error stream and aborts with a core dump.
<br><br>
<dt><code>herr_t H5vers_check (unsigned <em>majnum</em>,
unsigned <em>minnum</em>, unsigned <em>relnum</em>, unsigned
<em>patnum</em>)</code>
<dd>This function is called by the <code>H5check()</code> macro
with the include file version constants. The function
compares its arguments to the result returned by
<code>H5version()</code> and if a mismatch is detected prints
an error message on the standard error stream and aborts.
</dl>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Wed Apr 22 11:24:40 EDT 1998 -->
<!-- hhmts start -->
Last modified: Wed Apr 22 12:19:53 EDT 1998
<!-- hhmts end -->
</body>
</html>

BIN
doc/html/chunk1.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 5.0 KiB

52
doc/html/chunk1.obj Normal file

@ -0,0 +1,52 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,16,1,9,1,1,0,0,3,0,1,1,'Courier',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',64,64,384,384,5,2,1,29,0,0,0,0,0,'2',[
]).
poly('black',2,[
128,64,128,384],0,2,1,30,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
192,64,192,384],0,2,1,31,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
256,64,256,384],0,2,1,32,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
320,64,320,384],0,2,1,33,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
64,128,384,128],0,2,1,34,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
64,192,384,192],0,2,1,35,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
64,256,384,256],0,2,1,36,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
64,320,384,320],0,2,1,37,0,4,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',128,448,192,512,5,2,1,56,0,0,0,0,0,'2',[
]).
text('black',448,208,'Courier',0,17,2,1,0,1,84,28,61,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Entire array",
"5000 x 5000"]).
text('black',256,464,'Courier',0,17,2,1,0,1,84,28,63,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Single Chunk",
"1000 x 1000"]).
box('black',48,48,512,528,0,1,1,71,0,0,0,0,0,'1',[
]).

271
doc/html/compat.html Normal file

@ -0,0 +1,271 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Backward/Forward Compatability</title>
</head>
<body>
<h1>Backward/Forward Compatability</h1>
<p>The HDF5 development must proceed in such a manner as to
satisfy the following conditions:
<ol type=A>
<li>HDF5 applications can produce data that HDF5
applications can read and write and HDF4 applications can produce
data that HDF4 applications can read and write. The situation
that demands this condition is obvious.</li>
<li>HDF5 applications are able to produce data that HDF4 applications
can read and HDF4 applications can subsequently modify the
file subject to certain constraints depending on the
implementation. This condition is for the temporary
situation where a consumer has neither been relinked with a new
HDF4 API built on top of the HDF5 API nor recompiled with the
HDF5 API.</li>
<li>HDF5 applications can read existing HDF4 files and subsequently
modify the file subject to certain constraints depending on
the implementation. This is condition is for the temporary
situation in which the producer has neither been relinked with a
new HDF4 API built on top of the HDF5 API nor recompiled with
the HDF5 API, or the permanent situation of HDF5 consumers
reading archived HDF4 files.</li>
</ul>
<p>There's at least one invarient: new object features introduced
in the HDF5 file format (like 2-d arrays of structs) might be
impossible to "translate" to a format that an old HDF4
application can understand either because the HDF4 file format
or the HDF4 API has no mechanism to describe the object.
<p>What follows is one possible implementation based on how
Condition B was solved in the AIO/PDB world. It also attempts
to satisfy these goals:
<ol type=1>
<li>The main HDF5 library contains as little extra baggage as
possible by either relying on external programs to take care
of compatability issues or by incorporating the logic of such
programs as optional modules in the HDF5 library. Conditions B
and C are separate programs/modules.</li>
<li>No extra baggage not only means the library proper is small,
but also means it can be implemented (rather than migrated
from HDF4 source) from the ground up with minimal regard for
HDF4 thus keeping the logic straight forward.</li>
<li>Compatability issues are handled behind the scenes when
necessary (and possible) but can be carried out explicitly
during things like data migration.</li>
</ol>
<hr>
<h2>Wrappers</h2>
<p>The proposed implementation uses <i>wrappers</i> to handle
compatability issues. A Format-X file is <i>wrapped</i> in a
Format-Y file by creating a Format-Y skeleton that replicates
the Format-X meta data. The Format-Y skeleton points to the raw
data stored in Format-X without moving the raw data. The
restriction is that raw data storage methods in Format-Y is a
superset of raw data storage methods in Format-X (otherwise the
raw data must be copied to Format-Y). We're assuming that meta
data is small wrt the entire file.
<p>The wrapper can be a separate file that has pointers into the
first file or it can be contained within the first file. If
contained in a single file, the file can appear as a Format-Y
file or simultaneously a Format-Y and Format-X file.
<p>The Format-X meta-data can be thought of as the original
wrapper around raw data and Format-Y is a second wrapper around
the same data. The wrappers are independend of one another;
modifying the meta-data in one wrapper causes the other to
become out of date. Modification of raw data doesn't invalidate
either view as long as the meta data that describes its storage
isn't modifed. For instance, an array element can change values
if storage is already allocated for the element, but if storage
isn't allocated then the meta data describing the storage must
change, invalidating all wrappers but one.
<p>It's perfectly legal to modify the meta data of one wrapper
without modifying the meta data in the other wrapper(s). The
illegal part is accessing the raw data through a wrapper which
is out of date.
<p>If raw data is wrapped by more than one internal wrapper
(<i>internal</i> means that the wrapper is in the same file as
the raw data) then access to that file must assume that
unreferenced parts of that file contain meta data for another
wrapper and cannot be reclaimed as free memory.
<hr>
<h2>Implementation of Condition B</h2>
<p>Since this is a temporary situation which can't be
automatically detected by the HDF5 library, we must rely
on the application to notify the HDF5 library whether or not it
must satisfy Condition B. (Even if we don't rely on the
application, at some point someone is going to remove the
Condition B constraint from the library.) So the module that
handles Condition B is conditionally compiled and then enabled
on a per-file basis.
<p>If the application desires to produce an HDF4 file (determined
by arguments to <code>H5Fopen</code>), and the Condition B
module is compiled into the library, then <code>H5Fclose</code>
calls the module to traverse the HDF5 wrapper and generate an
additional internal or external HDF4 wrapper (wrapper specifics
are described below). If Condition B is implemented as a module
then it can benefit from the metadata already cached by the main
library.
<p>An internal HDF4 wrapper would be used if the HDF5 file is
writable and the user doesn't mind that the HDF5 file is
modified. An external wrapper would be used if the file isn't
writable or if the user wants the data file to be primarily HDF5
but a few applications need an HDF4 view of the data.
<p>Modifying through the HDF5 library an HDF5 file that has
internal HDF4 wrapper should invalidate the HDF4 wrapper (and
optionally regenerate it when <code>H5Fclose</code> is
called). The HDF5 library must understand how wrappers work, but
not necessarily anything about the HDF4 file format.
<p>Modifying through the HDF5 library an HDF5 file that has an
external HDF4 wrapper will cause the HDF4 wrapper to become out
of date (but possibly regenerated during <code>H5Fclose</code>).
<b>Note: Perhaps the next release of the HDF4 library should
insure that the HDF4 wrapper file has a more recent modification
time than the raw data file (the HDF5 file) to which it
points(?)</b>
<p>Modifying through the HDF4 library an HDF5 file that has an
internal or external HDF4 wrapper will cause the HDF5 wrapper to
become out of date. However, there is now way for the old HDF4
library to notify the HDF5 wrapper that it's out of date.
Therefore the HDF5 library must be able to detect when the HDF5
wrapper is out of date and be able to fix it. If the HDF4
wrapper is complete then the easy way is to ignore the original
HDF5 wrapper and generate a new one from the HDF4 wrapper. The
other approach is to compare the HDF4 and HDF5 wrappers and
assume that if they differ HDF4 is the right one, if HDF4 omits
data then it was because HDF4 is a partial wrapper (rather than
assume HDF4 deleted the data), and if HDF4 has new data then
copy the new meta data to the HDF5 wrapper. On the other hand,
perhaps we don't need to allow these situations (modifying an
HDF5 file with the old HDF4 library and then accessing it with
the HDF5 library is either disallowed or causes HDF5 objects
that can't be described by HDF4 to be lost).
<p>To convert an HDF5 file to an HDF4 file on demand, one simply
opens the file with the HDF4 flag and closes it. This is also
how AIO implemented backward compatability with PDB in its file
format.
<hr>
<h2>Implementation of Condition C</h2>
<p>This condition must be satisfied for all time because there
will always be archived HDF4 files. If a pure HDF4 file (that
is, one without HDF5 meta data) is opened with an HDF5 library,
the <code>H5Fopen</code> builds an internal or external HDF5
wrapper and then accesses the raw data through that wrapper. If
the HDF5 library modifies the file then the HDF4 wrapper becomes
out of date. However, since the HDF5 library hasn't been
released, we can at least implement it to disable and/or reclaim
the HDF4 wrapper.
<p>If an external and temporary HDF5 wrapper is desired, the
wrapper is created through the cache like all other HDF5 files.
The data appears on disk only if a particular cached datum is
preempted. Instead of calling <code>H5Fclose</code> on the HDF5
wrapper file we call <code>H5Fabort</code> which immediately
releases all file resources without updating the file, and then
we unlink the file from Unix.
<hr>
<h2>What do wrappers look like?</h2>
<p>External wrappers are quite obvious: they contain only things
from the format specs for the wrapper and nothing from the
format specs of the format which they wrap.
<p>An internal HDF4 wrapper is added to an HDF5 file in such a way
that the file appears to be both an HDF4 file and an HDF5
file. HDF4 requires an HDF4 file header at file offset zero. If
a user block is present then we just move the user block down a
bit (and truncate it) and insert the minimum HDF4 signature.
The HDF4 <code>dd</code> list and any other data it needs are
appended to the end of the file and the HDF5 signature uses the
logical file length field to determine the beginning of the
trailing part of the wrapper.
<p>
<center>
<table border width="60%">
<tr>
<td>HDF4 minimal file header. Its main job is to point to
the <code>dd</code> list at the end of the file.</td>
</tr>
<tr>
<td>User-defined block which is truncated by the size of the
HDF4 file header so that the HDF5 boot block file address
doesn't change.</td>
</tr>
<tr>
<td>The HDF5 boot block and data, unmodified by adding the
HDF4 wrapper.</td>
</tr>
<tr>
<td>The main part of the HDF4 wrapper. The <code>dd</code>
list will have entries for all parts of the file so
hdpack(?) doesn't (re)move anything.</td>
</tr>
</table>
</center>
<p>When such a file is opened by the HDF5 library for
modification it shifts the user block back down to address zero
and fills with zeros, then truncates the file at the end of the
HDF5 data or adds the trailing HDF4 wrapper to the free
list. This prevents HDF4 applications from reading the file with
an out of date wrapper.
<p>If there is no user block then we have a problem. The HDF5
boot block must be moved to make room for the HDF4 file header.
But moving just the boot block causes problems because all file
addresses stored in the file are relative to the boot block
address. The only option is to shift the entire file contents
by 512 bytes to open up a user block (too bad we don't have
hooks into the Unix i-node stuff so we could shift the entire
file contents by the size of a file system page without ever
performing I/O on the file :-)
<p>Is it possible to place an HDF5 wrapper in an HDF4 file? I
don't know enough about the HDF4 format, but I would suspect it
might be possible to open a hole at file address 512 (and
possibly before) by moving some things to the end of the file
to make room for the HDF5 signature. The remainder of the HDF5
wrapper goes at the end of the file and entries are added to the
HDF4 <code>dd</code> list to mark the location(s) of the HDF5
wrapper.
<hr>
<h2>Other Thoughts</h2>
<p>Conversion programs that copy an entire HDF4 file to a separate,
self-contained HDF5 file and vice versa might be useful.
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Fri Oct 3 11:52:31 EST 1997 -->
<!-- hhmts start -->
Last modified: Wed Oct 8 12:34:42 EST 1997
<!-- hhmts end -->
</body>
</html>

BIN
doc/html/dataset_p1.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 3.3 KiB

32
doc/html/dataset_p1.obj Normal file

@ -0,0 +1,32 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,16,1,9,1,1,0,0,1,0,1,1,'Helvetica',0,24,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',128,240,288,432,4,1,1,26,0,0,0,0,0,'1',[
]).
box('black',400,272,464,400,4,1,1,27,0,0,0,0,0,'1',[
]).
box('black',192,304,224,368,6,1,1,28,0,0,0,0,0,'1',[
]).
box('black',400,272,432,336,6,1,1,29,0,0,0,0,0,'1',[
]).
poly('black',2,[
224,304,400,272],1,1,1,32,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
224,368,400,336],1,1,1,33,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',208,208,'Helvetica',0,20,1,1,0,1,77,17,40,0,14,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"File Dataset"]).
text('black',432,208,'Helvetica',0,20,1,1,0,1,106,17,42,0,14,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Memory Dataset"]).
text('black',320,144,'Helvetica',0,24,1,1,0,1,206,29,68,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Raw Data Transfer"]).
box('black',96,128,512,464,0,1,1,70,0,0,0,0,0,'1',[
]).

BIN
doc/html/extern1.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 1.9 KiB

40
doc/html/extern1.obj Normal file

@ -0,0 +1,40 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,16,1,9,1,1,0,0,1,0,1,0,'Courier',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',128,96,192,128,4,1,1,49,0,0,0,0,0,'1',[
]).
box('black',192,96,352,128,12,1,1,50,0,0,0,0,0,'1',[
]).
box('black',352,96,416,128,18,1,1,51,0,0,0,0,0,'1',[
]).
box('black',64,176,224,208,12,1,1,53,0,0,0,0,0,'1',[
]).
box('black',256,176,320,208,4,1,1,54,0,0,0,0,0,'1',[
]).
box('black',352,176,448,208,18,1,1,55,0,0,0,0,0,'1',[
]).
box('black',224,176,256,208,0,1,1,56,0,0,0,0,0,'1',[
]).
box('black',320,176,352,208,0,1,1,57,0,0,0,0,0,'1',[
]).
box('black',448,176,512,208,0,1,1,58,0,0,0,0,0,'1',[
]).
poly('black',2,[
176,128,272,176],1,1,1,59,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
240,128,208,176],1,1,1,60,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
384,128,384,176],1,1,1,61,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
box('black',48,80,528,224,0,1,1,64,0,0,0,0,0,'1',[
]).

BIN
doc/html/extern2.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 4.0 KiB

108
doc/html/extern2.obj Normal file

@ -0,0 +1,108 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,16,1,9,1,1,1,1,0,0,1,1,'Courier',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',48,48,464,432,0,1,1,144,0,0,0,0,0,'1',[
]).
text('black',80,240,'Courier',0,17,1,0,0,1,70,14,146,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"scan1.data"]).
text('black',80,304,'Courier',0,17,1,0,0,1,70,14,148,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"scan2.data"]).
text('black',80,368,'Courier',0,17,1,0,0,1,70,14,150,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"scan3.data"]).
polygon('black',7,[
64,64,64,128,192,128,192,96,320,96,320,64,64,64],20,1,1,0,181,0,0,0,0,0,'1',
"00",[
]).
polygon('black',7,[
64,128,64,160,320,160,320,96,192,96,192,128,64,128],4,1,1,0,182,0,0,0,0,0,'1',
"00",[
]).
box('black',64,160,320,192,26,1,1,183,0,0,0,0,0,'1',[
]).
poly('black',2,[
80,80,304,80],1,1,1,184,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
80,112,176,112],1,1,1,185,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
208,112,304,112],1,1,1,186,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
80,144,304,144],1,1,1,187,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
80,176,304,176],1,1,1,188,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
box('black',64,256,448,288,20,1,1,203,0,0,0,0,0,'1',[
]).
box('black',64,320,448,352,4,1,1,216,0,0,0,0,0,'1',[
]).
box('black',64,384,320,416,26,1,1,225,0,0,0,0,0,'1',[
]).
poly('black',2,[
80,272,304,272],1,1,1,226,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
336,272,432,272],1,1,1,227,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
80,336,176,336],1,1,1,228,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
208,336,432,336],1,1,1,229,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
80,400,304,400],1,1,1,230,0,26,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
192,96,64,96],0,1,1,232,0,26,5,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
192,128,320,128],0,1,1,233,0,26,5,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
256,64,256,192],0,1,1,234,0,26,5,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
192,64,192,192],0,1,1,235,0,26,5,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
128,64,128,192],0,1,1,236,0,26,5,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
320,160,64,160],0,2,1,238,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',4,[
320,96,192,96,192,128,64,128],0,2,1,240,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',6,[
336,64,384,64,384,128,384,128,384,192,336,192],3,1,1,241,1,0,0,0,8,3,0,0,0,'1','8','3',
"78",[
]).
text('black',429,124,'Courier',0,17,2,1,0,1,28,49,250,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,1,0,[
429,124,405,124,454,152,0,1000,-1000,0,-15,2,404,123,455,153],[
"2-d",
"Dataset"]).

BIN
doc/html/group_p1.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 3.6 KiB

85
doc/html/group_p1.obj Normal file

@ -0,0 +1,85 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,8,1,9,1,1,0,2,1,0,1,1,'Times-Roman',0,24,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
text('black',80,168,'Courier',0,17,1,0,0,1,7,14,30,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',80,184,'Courier',0,17,1,0,0,1,7,14,34,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',80,200,'Courier',0,17,1,0,0,1,7,14,36,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',80,216,'Courier',0,17,1,0,0,1,21,14,38,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Foo"]).
text('black',80,232,'Courier',0,17,1,0,0,1,7,14,43,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',80,248,'Courier',0,17,1,0,0,1,7,14,47,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
rcbox('black',64,152,128,280,0,1,1,0,16,49,0,0,0,0,'1',[
]).
text('black',208,152,'Courier',0,17,1,0,0,1,7,14,52,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',80,152,'Courier',0,17,1,0,0,1,7,14,56,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',208,168,'Courier',0,17,1,0,0,1,7,14,58,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',208,184,'Courier',0,17,1,0,0,1,21,14,60,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Bar"]).
text('black',208,200,'Courier',0,17,1,0,0,1,7,14,62,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',208,216,'Courier',0,17,1,0,0,1,7,14,64,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',208,232,'Courier',0,17,1,0,0,1,7,14,68,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',208,248,'Courier',0,17,1,0,0,1,7,14,72,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
rcbox('black',192,152,256,280,0,1,1,0,16,74,0,0,0,0,'1',[
]).
text('black',336,152,'Courier',0,17,1,0,0,1,7,14,75,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',336,168,'Courier',0,17,1,0,0,1,7,14,77,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',336,184,'Courier',0,17,1,0,0,1,7,14,81,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',336,200,'Courier',0,17,1,0,0,1,7,14,88,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',336,216,'Courier',0,17,1,0,0,1,7,14,92,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',336,232,'Courier',0,17,1,0,0,1,7,14,94,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',336,248,'Courier',0,17,1,0,0,1,21,14,96,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Baz"]).
rcbox('black',320,152,384,280,0,1,1,0,16,98,0,0,0,0,'1',[
]).
text('black',224,360,'NewCenturySchlbk-Roman',0,17,2,1,0,1,42,30,99,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Object",
"Header"]).
rcbox('black',192,344,256,408,0,1,1,0,16,101,0,0,0,0,'1',[
]).
poly('black',4,[
112,224,136,216,152,184,192,168],1,1,1,102,2,0,0,0,8,3,0,0,0,'1','8','3',
"",[
]).
poly('black',4,[
232,192,272,184,288,168,320,160],1,1,1,107,2,0,0,0,8,3,0,0,0,'1','8','3',
"",[
]).
poly('black',4,[
368,256,416,272,392,336,256,352],1,1,1,110,2,0,0,0,8,3,0,0,0,'1','8','3',
"",[
]).
text('black',96,128,'Times-Roman',0,17,1,1,0,1,40,15,120,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Group 1"]).
text('black',224,128,'Times-Roman',0,17,1,1,0,1,40,15,126,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Group 2"]).
text('black',352,128,'Times-Roman',0,17,1,1,0,1,40,15,130,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Group 3"]).
text('black',224,320,'Times-Roman',0,17,1,1,0,1,64,15,134,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Some Object"]).
text('black',224,80,'Times-Roman',0,24,1,1,0,1,258,28,138,0,22,6,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"The name \"/Foo/Bar/Baz\""]).
box('black',40,64,448,432,0,1,1,140,0,0,0,0,0,'1',[
]).

BIN
doc/html/group_p2.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 3.4 KiB

57
doc/html/group_p2.obj Normal file

@ -0,0 +1,57 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,8,1,9,1,1,0,2,1,0,1,0,'Courier',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
text('black',144,128,'Courier',0,17,1,0,0,1,7,14,26,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,144,'Courier',0,17,1,0,0,1,7,14,30,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,160,'Courier',0,17,1,0,0,1,21,14,34,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Foo"]).
text('black',144,176,'Courier',0,17,1,0,0,1,7,14,36,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,192,'Courier',0,17,1,0,0,1,7,14,38,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
rcbox('black',128,128,192,256,0,1,1,0,16,40,0,0,0,0,'1',[
]).
text('black',144,320,'Courier',0,17,1,0,0,1,7,14,43,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,336,'Courier',0,17,1,0,0,1,7,14,45,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,352,'Courier',0,17,1,0,0,1,21,14,47,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Bar"]).
text('black',144,368,'Courier',0,17,1,0,0,1,7,14,49,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,384,'Courier',0,17,1,0,0,1,7,14,51,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
rcbox('black',128,320,192,448,0,1,1,0,16,53,0,0,0,0,'1',[
]).
text('black',160,96,'NewCenturySchlbk-Roman',0,17,1,1,0,1,46,15,64,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Group 1"]).
text('black',160,288,'NewCenturySchlbk-Roman',0,17,1,1,0,1,46,15,68,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Group 2"]).
text('black',352,224,'NewCenturySchlbk-Roman',0,17,2,1,0,1,35,30,70,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Some",
"Object"]).
rcbox('black',320,256,384,384,0,1,1,0,16,72,0,0,0,0,'1',[
]).
poly('black',4,[
176,168,224,192,264,240,320,264],1,1,1,73,2,0,0,0,8,3,0,0,0,'1','8','3',
"",[
]).
poly('black',4,[
176,360,232,344,272,288,320,272],1,1,1,74,2,0,0,0,8,3,0,0,0,'1','8','3',
"",[
]).
text('black',264,40,'Helvetica',0,24,1,1,0,1,206,29,93,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Hard Link Example"]).
box('black',88,24,424,496,0,1,1,95,0,0,0,0,0,'1',[
]).
text('black',240,192,'Courier',0,17,1,0,0,1,63,14,129,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"hard link"]).
text('black',248,336,'Courier',0,17,1,0,0,1,63,14,131,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"hard link"]).

BIN
doc/html/group_p3.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 3.3 KiB

59
doc/html/group_p3.obj Normal file

@ -0,0 +1,59 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,8,1,9,1,1,0,2,1,0,1,0,'Courier',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
text('black',144,128,'Courier',0,17,1,0,0,1,7,14,26,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,144,'Courier',0,17,1,0,0,1,7,14,30,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,160,'Courier',0,17,1,0,0,1,21,14,34,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Foo"]).
text('black',144,176,'Courier',0,17,1,0,0,1,7,14,36,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,192,'Courier',0,17,1,0,0,1,7,14,38,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
rcbox('black',128,128,192,256,0,1,1,0,16,40,0,0,0,0,'1',[
]).
text('black',144,320,'Courier',0,17,1,0,0,1,7,14,43,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,336,'Courier',0,17,1,0,0,1,7,14,45,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,352,'Courier',0,17,1,0,0,1,21,14,47,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Bar"]).
text('black',144,368,'Courier',0,17,1,0,0,1,7,14,49,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
text('black',144,384,'Courier',0,17,1,0,0,1,7,14,51,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"."]).
rcbox('black',128,320,192,448,0,1,1,0,16,53,0,0,0,0,'1',[
]).
text('black',160,96,'NewCenturySchlbk-Roman',0,17,1,1,0,1,46,15,64,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Group 1"]).
text('black',160,288,'NewCenturySchlbk-Roman',0,17,1,1,0,1,46,15,68,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Group 2"]).
text('black',352,96,'NewCenturySchlbk-Roman',0,17,2,1,0,1,35,30,70,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Some",
"Object"]).
rcbox('black',320,128,384,256,0,1,1,0,16,72,0,0,0,0,'1',[
]).
text('black',264,40,'Helvetica',0,24,1,1,0,1,197,29,93,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Soft Link Example"]).
box('black',88,24,424,496,0,1,1,95,0,0,0,0,0,'1',[
]).
text('black',320,352,'Courier',0,17,1,0,0,1,35,14,105,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"\"Foo\""]).
poly('black',4,[
176,168,232,160,264,144,320,136],1,1,1,111,2,0,0,0,8,3,0,0,0,'1','8','3',
"",[
]).
poly('black',2,[
176,360,312,360],1,1,1,116,2,0,0,0,8,3,0,0,0,'1','8','3',
"",[
]).
text('black',240,160,'Courier',0,17,1,0,0,1,63,14,119,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"hard link"]).
text('black',216,368,'Courier',0,17,1,0,0,1,63,14,121,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"soft link"]).

347
doc/html/h5s.examples Normal file

@ -0,0 +1,347 @@
Example 1: Create a simple fixed size 3-D dataspace in memory and on disk and
copy the entire dataset to disk.
{
hid_t file; /* File ID */
hid_t dataset; /* Dataset ID */
hid_t mem_space, file_space; /* Dataspaces for memory and the file */
uint8 *buf; /* Buffer for data */
hsize_t curr_dims[3]={3,4,5}; /* Dimensions of the dataset */
/* Create file */
file = H5Fcreate("example1.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Create dataspace for dataset in the file */
/* Selection for dataspace defaults to entire space */
file_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the dataset's dataspace */
H5Sset_extent_simple(file_space,3,curr_dims,curr_dims);
/* Create the dataspace for the dataset in memory */
/* Selection for dataspace defaults to entire space */
mem_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the memory dataspace */
H5Sset_extent_simple(mem_space,3,curr_dims,curr_dims);
/* Create the dataset on disk */
dataset=H5Dcreate(file,"Dataset",H5T_NATIVE_UINT8,file_space,H5P_DEFAULT);
/* Write the dataset to the file */
H5Dwrite(dataset,H5T_NATIVE_UINT8,mem_space,file_space,H5P_DEFAULT,buf);
/* Close dataspaces */
H5Sclose(mem_space);
H5Sclose(file_space);
/* Close dataset & file */
H5Dclose(dataset);
H5Fclose(file);
}
Example 2: Create a simple fixed size 3-D dataspace in memory and on disk and
copy a hyperslab to disk. The hyperslab blocks are packed and
contiguous in memory, but are scattered when written to the dataset
on disk.
{
hid_t file; /* File ID */
hid_t dataset; /* Dataset ID */
hid_t mem_space, file_space; /* Dataspaces for memory and the file */
uint8 *buf; /* Buffer for data */
hssize_t start[3]={3,4,5}; /* Start of hyperslab */
hsize_t stride[3]={1,2,2}; /* Stride for hyperslab */
hsize_t count[3]={3,3,3}; /* Hyperslab block count in each dimension */
hsize_t block[3]={2,2,2}; /* Hyperslab block size in each dimension */
hsize_t curr_dims[3]={13,14,15}; /* Dimensions of the dataset */
/* Create file */
file = H5Fcreate("example2.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Create dataspace for dataset in the file */
/* Selection for dataspace defaults to entire space */
file_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the dataset's dataspace */
H5Sset_extent_simple(file_space,3,curr_dims,curr_dims);
/* Set the hyperslab selection for a file dataspace */
H5Sselect_hyperslab(file_space,H5S_SELECT_SET,start,stride,count,block);
/* Create the dataspace for the dataset in memory */
/* Selection for dataspace defaults to entire space */
mem_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the memory dataspace */
/* Compute the memory dimensions based on the hyperslab blocks to write */
for(i=0; i<3; i++)
curr_dims[i]=count[i]*block[i];
H5Sset_extent_simple(mem_space,3,curr_dims,curr_dims);
/* Create the dataset on disk */
dataset=H5Dcreate(file,"Dataset",H5T_NATIVE_UINT8,file_space,H5P_DEFAULT);
/* Write the hyperslab to the file */
H5Dwrite(dataset,H5T_NATIVE_UINT8,mem_space,file_space,H5P_DEFAULT,buf);
/* Close dataspaces */
H5Sclose(mem_space);
H5Sclose(file_space);
/* Close dataset & file */
H5Dclose(dataset);
H5Fclose(file);
}
Example 3: Create a simple fixed size 3-D dataspace in memory and on disk and
copy a specific selection of points (with a particular order) to
disk. The memory and file dataspaces are different sizes, but the number
of points selected are the same.
{
hid_t file; /* File ID */
hid_t dataset; /* Dataset ID */
hid_t mem_space, file_space; /* Dataspaces for memory and the file */
uint8 *buf; /* Buffer for data */
hsize_t elements[5][3]; /* Dataspace elements selected */
hsize_t curr_dims[3]={13,14,15}; /* Dimensions of the dataset */
/* Create file */
file = H5Fcreate("example3.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Create dataspace for dataset in the file */
/* Selection for dataspace defaults to entire space */
file_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the dataset's dataspace */
H5Sset_extent_simple(file_space,3,curr_dims,curr_dims);
/* Set the elements for the selection in the file dataspace */
elements[0]={0,2,4}; /* Yes, I know this won't compile.. :-) */
elements[1]={3,4,1};
elements[2]={9,8,3};
elements[3]={7,2,0};
elements[4]={6,5,8};
H5Sselect_elements(file_space,H5S_SELECT_SET,5,elements);
/* Create the dataspace for the dataset in memory */
/* Selection for dataspace defaults to entire space */
mem_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the memory dataspace */
curr_dims={23,15,18}; /* This won't compile either :-) */
H5Sset_extent_simple(mem_space,3,curr_dims,curr_dims);
/* Set the elements for the selection in the file dataspace */
elements[0]={9,2,1};
elements[1]={13,1,12};
elements[2]={4,1,7};
elements[3]={0,12,0};
elements[4]={20,10,17};
H5Sselect_elements(mem_space,H5S_SELECT_SET,5,elements);
/* Create the dataset on disk */
dataset=H5Dcreate(file,"Dataset",H5T_NATIVE_UINT8,file_space,H5P_DEFAULT);
/* Write the hyperslab to the file */
H5Dwrite(dataset,H5T_NATIVE_UINT8,mem_space,file_space,H5P_DEFAULT,buf);
/* Close dataspaces */
H5Sclose(mem_space);
H5Sclose(file_space);
/* Close dataset & file */
H5Dclose(dataset);
H5Fclose(file);
}
Example 4: Create a simple fixed size 3-D dataspace in memory and on disk and
build up selection hyperslab selections to copy from memory to disk. The
selection is the same for both dataspaces, but a different offset is used,
to illustrate the selection offsets.
{
hid_t file; /* File ID */
hid_t dataset; /* Dataset ID */
hid_t mem_space, file_space; /* Dataspaces for memory and the file */
uint8 *buf; /* Buffer for data */
hssize_t start[3]; /* Start of hyperslab */
hsize_t stride[3]; /* Stride for hyperslab */
hsize_t count[3]; /* Hyperslab block count in each dimension */
hsize_t block[3]; /* Hyperslab block size in each dimension */
hssize_t offset[3]; /* Selection offset */
hsize_t curr_dims[3]={13,14,15}; /* Dimensions of the dataset */
/* Create file */
file = H5Fcreate("example4.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Create dataspace for dataset in the file */
/* Selection for dataspace defaults to entire space */
file_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the dataset's dataspace */
H5Sset_extent_simple(file_space,3,curr_dims,curr_dims);
/* Build up the selection with a series of hyperslab selections */
start={0,2,4}; /* Again, this won't compile.. :-) */
stride={1,1,1};
count={6,5,8};
block={1,1,1};
/* Set the first selection, union the rest in */
H5Sselect_hyperslab(file_space,H5S_SELECT_SET,start,stride,count,block);
/* initialize the second hyperslab */
start={10,9,1}; /* Again, this won't compile.. :-) */
stride={1,1,1};
count={2,3,10};
block={1,1,1};
/* Union the second hyperslab into the file dataspace's selection */
H5Sselect_hyperslab(file_space,H5S_SELECT_UNION,start,stride,count,block);
/* initialize the third hyperslab */
start={3,10,5}; /* Again, this won't compile.. :-) */
stride={1,1,1};
count={8,2,6};
block={1,1,1};
/* Union the final hyperslab into the file dataspace's selection */
H5Sselect_hyperslab(file_space,H5S_SELECT_UNION,start,stride,count,block);
/* Create the dataspace for the dataset in memory */
/* Selection for dataspace defaults to entire space */
mem_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the memory dataspace */
curr_dims={23,15,18}; /* This won't compile either :-) */
H5Sset_extent_simple(mem_space,3,curr_dims,curr_dims);
/* Copy the selection from the file dataspace */
H5Sselect_op(mem_space,H5S_SELECT_COPY,file_space);
/* Adjust the offset of the selection in the memory dataspace */
offset={1,1,1};
H5Soffset_simple(mem_space,offset);
/* Create the dataset on disk */
dataset=H5Dcreate(file,"Dataset",H5T_NATIVE_UINT8,file_space,H5P_DEFAULT);
/* Write the hyperslab to the file */
H5Dwrite(dataset,H5T_NATIVE_UINT8,mem_space,file_space,H5P_DEFAULT,buf);
/* Close dataspaces */
H5Sclose(mem_space);
H5Sclose(file_space);
/* Close dataset & file */
H5Dclose(dataset);
H5Fclose(file);
}
Example 5: Same as example 1 (create a simple fixed size 3-D dataspace in memory and on disk and
copy the entire dataset to disk), except that the selection order is changed
for the memory dataspace, to change between FORTRAN and C array ordering.
{
hid_t file; /* File ID */
hid_t dataset; /* Dataset ID */
hid_t mem_space, file_space; /* Dataspaces for memory and the file */
uint8 *buf; /* Buffer for data */
hsize_t order[3]; /* Dimension ordering for selection */
hsize_t curr_dims[3]={3,4,5}; /* Dimensions of the dataset */
/* Create file */
file = H5Fcreate("example5.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Create dataspace for dataset in the file */
/* Selection for dataspace defaults to entire space and C array order */
file_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the dataset's dataspace */
H5Sset_extent_simple(file_space,3,curr_dims,curr_dims);
/* Create the dataspace for the dataset in memory */
/* Selection for dataspace defaults to entire space and C array order */
mem_space=H5Screate(H5S_SIMPLE);
/* Set the extent & type of the memory dataspace */
H5Sset_extent_simple(mem_space,3,curr_dims,curr_dims);
/* Change selection ordering to FORTRAN order for memory dataspace */
order={0,1,2};
H5Sselect_order(mem_space,order);
/* Create the dataset on disk */
dataset=H5Dcreate(file,"Dataset",H5T_NATIVE_UINT8,file_space,H5P_DEFAULT);
/* Write the dataset to the file */
H5Dwrite(dataset,H5T_NATIVE_UINT8,mem_space,file_space,H5P_DEFAULT,buf);
/* Close dataspaces */
H5Sclose(mem_space);
H5Sclose(file_space);
/* Close dataset & file */
H5Dclose(dataset);
H5Fclose(file);
}
Example 6: Create a stored dataspace on disk and use the H5Ssubspace function
create a dataspace located within that space.
{
hid_t file; /* File ID */
hid_t space1, space2; /* Dataspace IDs */
hssize_t start[3]; /* Start of hyperslab */
hsize_t count[3]; /* Hyperslab block count in each dimension */
hsize_t curr_dims[3]={13,14,15};/* Dimensions of the dataset */
/* Create file */
file = H5Fcreate("example6.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Create dataspace #1 */
space1=H5Screate(H5S_SIMPLE);
/* Set the extent & type of dataspace #1 */
H5Sset_extent_simple(space1,3,curr_dims,curr_dims);
/* Store dataspace #1 on disk */
H5Scommit(file,"/Dataspaces/Dataspace #1",space1);
/* Select a contiguous hyperslab in dataspace #1 to create dataspace #2 with */
start={0,2,4};
count={6,5,8};
/*
* Use stride and block set to NULL to get contiguous, single element sized
* hyperslab. The stride and block parameters could also be set to all
* 1's, but this is simpler and easier.
*/
H5Sselect_hyperslab(space1,H5S_SELECT_SET,start,NULL,count,NULL);
/* Create dataspace #2 as a dataspace located within dataspace #1 */
space2=H5Ssubspace(space1);
/* Store dataspace #2 on disk also */
H5Scommit(file,"/Dataspaces/Dataspace #2",space2);
/*
* space1 & space2 can be used to create datasets, etc. Any datasets
* created with space2 can have their dataspace queried to find the parent
* dataspace and the location within the parent dataspace
*/
/* Close dataspaces */
H5Sclose(space1);
H5Sclose(space2);
/* Close file */
H5Fclose(file);
}

72
doc/html/heap.txt Normal file

@ -0,0 +1,72 @@
HEAP MANAGEMENT IN HDF5
------------------------
Heap functions are in the H5H package.
off_t
H5H_new (hdf5_file_t *f, size_t size_hint, size_t realloc_hint);
Creates a new heap in the specified file which can efficiently
store at least SIZE_HINT bytes. The heap can store more than
that, but doing so may cause the heap to become less efficient
(for instance, a heap implemented as a B-tree might become
discontigous). The REALLOC_HINT is the minimum number of bytes
by which the heap will grow when it must be resized. The hints
may be zero in which case reasonable (but probably not
optimal) values will be chosen.
The return value is the address of the new heap relative to
the beginning of the file boot block.
off_t
H5H_insert (hdf5_file_t *f, off_t addr, size_t size, const void *buf);
Copies SIZE bytes of data from BUF into the heap whose address
is ADDR in file F. BUF must be the _entire_ heap object. The
return value is the byte offset of the new data in the heap.
void *
H5H_read (hdf5_file_t *f, off_t addr, off_t offset, size_t size, void *buf);
Copies SIZE bytes of data from the heap whose address is ADDR
in file F into BUF and then returns the address of BUF. If
BUF is the null pointer then a new buffer will be malloc'd by
this function and its address is returned.
Returns buffer address or null.
const void *
H5H_peek (hdf5_file_t *f, off_t addr, off_t offset)
A more efficient version of H5H_read that returns a pointer
directly into the cache; the data is not copied from the cache
to a buffer. The pointer is valid until the next call to an
H5AC function directly or indirectly.
Returns a pointer or null. Do not free the pointer.
void *
H5H_write (hdf5_file_t *f, off_t addr, off_t offset, size_t size,
const void *buf);
Modifies (part of) an object in the heap at address ADDR of
file F by copying SIZE bytes from the beginning of BUF to the
file. OFFSET is the address withing the heap where the output
is to occur.
This function can fail if the combination of OFFSET and SIZE
would write over a boundary between two heap objects.
herr_t
H5H_remove (hdf5_file_t *f, off_t addr, off_t offset, size_t size);
Removes an object or part of an object which begins at byte
OFFSET within a heap whose address is ADDR in file F. SIZE
bytes are returned to the free list. Removing the middle of
an object has the side effect that one object is now split
into two objects.
Returns success or failure.

40
doc/html/index.html Normal file

@ -0,0 +1,40 @@
<html>
<head>
<title>
HDF5 - The Next Generation of the HDF library & tools
</title>
</head>
<body>
<center><h1>HDF5 - A New Generation of HDF</h1></center>
<UL>
<LI><A HREF="H5.intro.html">An Introduction to HDF5</A> - A brief overview of
the design goals behind the HDF5 library & file format
</LI>
<LI><A HREF="H5.api.html">HDF5 Reference Manual</A> - A complete reference
manual for the HDF5 API
</LI>
<LI><A HREF="H5.user.html">HDF5 User Guide</A> - A collection of chapters on
each of the main HDF5 APIs as well as other supporting information
</LI>
<LI><A HREF="H5.format.html">HDF5 Format Specification</A> - The complete
specification of the HDF5 file format
</LI>
<LI><A HREF="ph5design.html">Parallel HDF5 Design</A> - The design
specification of the parallel HDF5 API and an example program
</LI>
<LI><A HREF="ph5implement.txt">Parallel HDF5 Implementation</A> - The
current status of implmentation of the parallel HDF5 API
</LI>
</UL>
<hr>
<address><a href="mailto:koziol@ncsa.uiuc.edu">Quincey Koziol</a></address>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- hhmts start -->
Last modified: Feb 16, 1998
<!-- hhmts end -->
</body>
</html>

66
doc/html/move.html Normal file

@ -0,0 +1,66 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>How to Relocate a File Data Structure</title>
</head>
<body>
<h1>How to Relocate a File Data Structure</h1>
<p>Since file data structures can be cached in memory by the H5AC
package it becomes problematic to move such a data structure in
the file. One cannot just copy a portion of the file from one
location to another because:
<ol>
<li>the file might not contain the latest information, and</li>
<li>the H5AC package might not realize that the object's
address has changed and attempt to write the object to disk
at the old address.</li>
</ol>
<p>Here's a correct method to move data from one location to
another. The example code assumes that one is moving a B-link
tree node from <code>old_addr</code> to <code>new_addr</code>.
<ol>
<li>Make sure the disk is up-to-date with respect to the
cache. There is no need to remove the item from the cache,
hence the final argument to <code>H5AC_flush</code> is
<code>FALSE</code>.
<br><br>
<code>
H5AC_flush (f, H5AC_BT, old_addr, FALSE);<br>
</code>
<br>
</li>
<li>Read the data from the old address and write it to the new
address.
<br><br>
<code>
H5F_block_read (f, old_addr, size, buf);<br>
H5F_block_write (f, new_addr, size, buf);<br>
</code>
<br>
</li>
<li>Notify the cache that the address of the object changed.
<br><br>
<code>
H5AC_rename (f, H5AC_BT, old_addr, new_addr);<br>
</code>
<br>
</li>
</ol>
<hr>
<address><a href="mailto:robb@maya.nuance.com">Robb Matzke</a></address>
<!-- Created: Mon Jul 14 15:09:06 EST 1997 -->
<!-- hhmts start -->
Last modified: Mon Jul 14 15:38:29 EST
<!-- hhmts end -->
</body>
</html>

77
doc/html/ph5design.html Normal file

@ -0,0 +1,77 @@
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=windows-1252">
<META NAME="Generator" CONTENT="Microsoft Word 97">
<TITLE>new</TITLE>
<META NAME="Template" CONTENT="C:\PROGRAM FILES\MICROSOFT OFFICE\OFFICE\html.dot">
</HEAD>
<BODY LINK="#0000ff" VLINK="#800080">
<B><FONT SIZE=6><P ALIGN="CENTER">Parallel HDF5 Design</P>
</B></FONT><P ALIGN="CENTER">&nbsp;</P>
<H1>1. Design Overview</H1>
<P>In this section, I first describe the function requirements of the Parallel HDF5 (PHDF5) software and the assumed system requirements. Section 2 describes the programming model of the PHDF5 interface. Section 3 shows an example PHDF5 program. </P>
<H2>1.1. Function requirements</H2>
<UL>
<LI>An API to support parallel file access for HDF5 files in a message passing environment. </LI>
<LI>Fast parallel I/O to large datasets through standard parallel I/O interface.</LI>
<LI>Processes are required to do collective API calls only when structural changes are needed for the HDF5 file. </LI>
<LI>Each process may do independent I/O requests to different datasets in the same or different HDF5 files. </LI>
<LI>Supports collective I/O requests for datasets (to be included in next version). </LI>
<LI>Minimize diviation from HDF5 interface.</LI>
</UL>
<H2>1.2. System requirements</H2>
<UL>
<LI>C language interface is the initial requirement. Fortran77 interface will be added later. </LI>
<LI>Use Message Passing Interface (MPI) for interprocess communication. </LI>
<LI>Use MPI-IO calls for parallel file accesses. </LI>
<LI>Initial platforms—IBM SP2, Intel TFLOPS and SGI Origin 2000. </LI></UL>
<H1>2. Programming Model</H1>
<P>HDF5 uses optional access template object to control the file access
mechanism. The general model in accessing an HDF5 file in parallel
contains the following steps: </P>
<UL>
<LI>Setup access template</LI>
<LI>File open </LI>
<LI>Dataset open </LI>
<LI>Dataset data access (zero or more) </LI>
<LI>Dataset close </LI>
<LI>File close </LI></UL>
<H2>2.1. Setup access template</H2>
<P>Each processes of the MPI communicator creates an access template and sets
it up with MPI parallel access information (communicator, info object,
access-mode). </P>
<H2>2.1. File open</H2>
<P>All processes of the MPI communicator open an HDF5 file by a collective call
(H5FCreate or H5Fopen) with the access template. </P>
<H2>2.2. Dataset open</H2>
<P>All processes of the MPI communicator open a dataset by a collective call (H5Dcreate or H5Dopen).&nbsp; This version supports only collective dataset open.&nbsp; Future version may support datasets open by a subset of the processes that have opened the file. </P>
<H2>2.3. Dataset access</H2>
<H3>2.3.1. Independent dataset access</H3>
<P>Each process may do independent and arbitrary number of data I/O access by independent calls (H5Dread or H5Dwrite) to the dataset with the transfer template set for independent access.&nbsp; (The default transfer mode is independent transfer).&nbsp; If the dataset is an unlimited dimension one and if the H5Dwrite is writing data beyond the current dimension size of the dataset, all processes that have opened the dataset must make a collective call (H5Dallocate) to allocate more space for the dataset BEFORE the independent H5Dwrite call. </P>
<H3>2.3.2. Collective dataset access</H3>
<P>All processes that have opened the dataset may do collective data I/O access by collective calls (H5Dread or H5Dwrite) to the dataset with the transfer template set for collective access.&nbsp; Pre-allocation (H5Dallocate) is not needed for unlimited dimension datasets since the H5Dallocate call, if needed, is done internally by the collective data access call. </P>
<H3>2.3.3. Dataset attributes access</H3>
<P>Changes to attributes can only occur at the <I>"main process" </I>(process 0).&nbsp; Read only access to attributes can occur independent in each process that has opened the dataset.&nbsp; (API to be defined later.) <BR>
&nbsp; </P>
<H2>2.4. Dataset close</H2>
<P>All processes that have opened the dataset must close the dataset by a collective call (H5Dclose). </P>
<H2>2.5. File close</H2>
<P>All processes that have opened the file must close the file by a collective call (H5Fclose). <BR>
&nbsp; </P>
<H1>3. Parallel HDF5 Example</H1>
<PRE>
<CODE>
</CODE><A HREF="ph5example.c">Example code</A>
</PRE>
<P><HR></P>
<P>Send comments to <BR>
<A HREF="mailto:hdfparallel@ncsa.uiuc.edu">hdfparallel@ncsa.uiuc.edu</A> </P>
<H6>Last Modified: Feb 16, 1998</H6></BODY>
</HTML>

1003
doc/html/ph5example.c Normal file

File diff suppressed because it is too large Load Diff

27
doc/html/ph5implement.txt Normal file

@ -0,0 +1,27 @@
Release information for parallel HDF5
-------------------------------------
+) Current release supports independent access to fixed dimension datasets
only.
+) The comm and info arguments of H5Pset_mpi are not used. All parallel
I/O are done via MPI_COMM_WORLD. Access_mode for H5Pset_mpi can be
H5ACC_INDEPENDENT only.
+) This release of parallel HDF5 has been tested on IBM SP2 and SGI
Origin 2000 systems. It uses the ROMIO version of MPIO interface
for parallel I/O supports.
+) Useful URL's.
Parallel HDF webpage: "http://hdf.ncsa.uiuc.edu/Parallel_HDF/"
ROMIO webpage: "http://www.mcs.anl.gov/home/thakur/romio/"
+) Some to-do items for future releases
support for Intel Teraflop platform.
support for unlimited dimension datasets.
support for file access via a communicator besides MPI_COMM_WORLD.
support for collective access to datasets.
support for independent create/open of datasets.
----
Last updated: Feb 16, 1998.

BIN
doc/html/pipe1.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 9.9 KiB

136
doc/html/pipe1.obj Normal file

@ -0,0 +1,136 @@
%TGIF 3.0-p5
state(1,33,100,0,0,0,8,1,9,1,1,0,0,0,0,1,1,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[
]).
box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[
]).
poly('black',2,[
128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',5,[
160,160,144,224,160,272,176,224,160,160],1,2,1,25,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
848,160,832,224,848,272,864,224,848,160],1,2,1,34,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
box('black',464,192,496,256,26,1,1,39,0,0,0,0,0,'1',[
]).
poly('black',2,[
160,224,464,224],1,2,1,40,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
496,224,848,224],1,2,1,41,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',5,[
192,224,176,288,192,336,208,288,192,224],1,2,1,42,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
432,224,416,288,432,336,448,288,432,224],1,2,1,43,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
192,288,432,288],1,2,1,44,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',464,352,496,416,26,1,1,45,0,0,0,0,0,'1',[
]).
poly('black',5,[
528,224,512,288,528,336,544,288,528,224],1,2,1,46,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
816,224,800,288,816,336,832,288,816,224],1,2,1,47,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
528,288,816,288],1,2,1,48,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',5,[
464,256,456,304,464,328,488,304,488,256],1,2,1,62,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
480,352,488,304],2,2,1,85,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',912,64,976,256,0,1,1,87,0,0,0,0,0,'1',[
]).
box('black',928,96,960,224,26,1,1,88,0,0,0,0,0,'1',[
]).
text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"File"]).
text('black',944,48,'Helvetica',0,17,1,1,0,1,64,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Application"]).
text('black',480,144,'Helvetica',0,17,1,1,0,1,65,15,99,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5D_read()"]).
text('black',480,128,'Helvetica',0,17,1,1,0,1,58,15,108,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5Dread()"]).
text('black',304,208,'Helvetica',0,17,1,1,0,1,86,15,115,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_arr_read()"]).
text('black',304,192,'Helvetica',0,17,1,1,0,1,99,15,119,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5S_simp_fgath()"]).
text('black',296,288,'Helvetica',0,17,1,1,0,1,101,15,125,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_block_read()"]).
text('black',296,304,'Helvetica',0,17,1,1,0,1,90,15,132,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_low_read()"]).
text('black',296,320,'Helvetica',0,17,1,1,0,1,98,15,136,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_sec2_read()"]).
text('black',296,336,'Helvetica',0,17,1,1,0,1,33,15,140,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"read()"]).
text('black',664,208,'Helvetica',0,17,1,1,0,1,106,15,146,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_stride_copy()"]).
text('black',664,176,'Helvetica',0,17,1,1,0,1,104,15,150,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5S_simp_mscat()"]).
text('black',664,272,'Helvetica',0,17,1,1,0,1,54,15,154,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"memcpy()"]).
text('black',384,392,'Helvetica',0,17,1,1,0,1,105,15,170,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5T_conv_struct()"]).
poly('black',4,[
392,384,400,352,440,368,456,336],1,1,1,172,1,0,0,0,8,3,0,0,0,'1','8','3',
"6",[
]).
text('black',480,176,'Helvetica',0,17,1,1,0,1,44,15,176,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"TCONV"]).
text('black',480,416,'Helvetica',0,17,1,1,0,1,25,15,182,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"BKG"]).
box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[
]).
poly('black',5,[
72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"== Loop"]).
poly('black',3,[
48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',480,40,'Helvetica',0,24,1,1,0,1,380,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Fig 1: Internal Contiguous Storage"]).
text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"A"]).
text('black',160,208,'Helvetica',0,17,1,1,0,1,8,15,207,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"B"]).
text('black',192,272,'Helvetica',0,17,1,1,0,1,9,15,211,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"C"]).
text('black',504,208,'Helvetica',0,17,1,1,0,1,8,15,215,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"E"]).
text('black',528,272,'Helvetica',0,17,1,1,0,1,8,15,223,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"F"]).
text('black',464,304,'Helvetica',0,17,1,1,0,1,9,15,231,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"D"]).
text('black',664,192,'Helvetica',0,17,1,1,0,1,107,15,324,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_hyper_copy()"]).

BIN
doc/html/pipe2.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 11 KiB

168
doc/html/pipe2.obj Normal file

@ -0,0 +1,168 @@
%TGIF 3.0-p5
state(1,33,100,0,0,0,8,1,9,1,1,1,1,0,0,1,1,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[
]).
box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[
]).
poly('black',2,[
128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',5,[
160,160,144,224,160,272,176,224,160,160],1,2,1,25,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
848,160,832,224,848,272,864,224,848,160],1,2,1,34,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
box('black',464,192,496,256,26,1,1,39,0,0,0,0,0,'1',[
]).
poly('black',2,[
160,224,464,224],1,2,1,40,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
496,224,848,224],1,2,1,41,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',5,[
192,224,176,288,192,336,208,288,192,224],1,2,1,42,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
432,224,416,288,432,336,448,288,432,224],1,2,1,43,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
192,288,432,288],1,2,1,44,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',464,352,496,416,26,1,1,45,0,0,0,0,0,'1',[
]).
poly('black',5,[
528,224,512,288,528,336,544,288,528,224],1,2,1,46,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
816,224,800,288,816,336,832,288,816,224],1,2,1,47,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
528,288,816,288],1,2,1,48,0,26,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',5,[
848,240,848,352,832,384,800,384,496,384],1,2,1,55,1,0,0,0,10,4,0,0,0,'2','10','4',
"70",[
]).
poly('black',5,[
528,384,512,448,528,496,544,448,528,384],1,2,1,57,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
800,384,784,448,800,496,816,448,800,384],1,2,1,58,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
800,448,528,448],1,2,1,61,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',5,[
464,256,456,304,464,328,488,304,488,256],1,2,1,62,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
480,352,488,304],0,2,1,85,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',912,64,976,256,0,1,1,87,0,0,0,0,0,'1',[
]).
box('black',928,96,960,224,26,1,1,88,0,0,0,0,0,'1',[
]).
text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"File"]).
text('black',944,48,'Helvetica',0,17,1,1,0,1,64,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Application"]).
text('black',480,144,'Helvetica',0,17,1,1,0,1,65,15,99,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5D_read()"]).
text('black',480,128,'Helvetica',0,17,1,1,0,1,58,15,108,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5Dread()"]).
text('black',304,208,'Helvetica',0,17,1,1,0,1,86,15,115,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_arr_read()"]).
text('black',304,192,'Helvetica',0,17,1,1,0,1,99,15,119,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5S_simp_fgath()"]).
text('black',296,288,'Helvetica',0,17,1,1,0,1,101,15,125,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_block_read()"]).
text('black',296,304,'Helvetica',0,17,1,1,0,1,90,15,132,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_low_read()"]).
text('black',296,320,'Helvetica',0,17,1,1,0,1,98,15,136,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_sec2_read()"]).
text('black',296,336,'Helvetica',0,17,1,1,0,1,33,15,140,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"read()"]).
text('black',664,208,'Helvetica',0,17,1,1,0,1,106,15,146,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_stride_copy()"]).
text('black',664,176,'Helvetica',0,17,1,1,0,1,104,15,150,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5S_simp_mscat()"]).
text('black',664,272,'Helvetica',0,17,1,1,0,1,54,15,154,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"memcpy()"]).
text('black',672,368,'Helvetica',0,17,1,1,0,1,106,15,158,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_stride_copy()"]).
text('black',672,336,'Helvetica',0,17,1,1,0,1,105,15,162,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5S_simp_mgath()"]).
text('black',672,432,'Helvetica',0,17,1,1,0,1,54,15,166,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"memcpy()"]).
text('black',384,392,'Helvetica',0,17,1,1,0,1,105,15,170,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5T_conv_struct()"]).
poly('black',4,[
392,384,400,352,440,368,456,336],1,1,1,172,1,0,0,0,8,3,0,0,0,'1','8','3',
"6",[
]).
text('black',480,176,'Helvetica',0,17,1,1,0,1,44,15,176,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"TCONV"]).
text('black',480,416,'Helvetica',0,17,1,1,0,1,25,15,182,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"BKG"]).
box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[
]).
poly('black',5,[
72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"== Loop"]).
poly('black',3,[
48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',480,40,'Helvetica',0,24,1,1,0,1,404,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Fig 2: Partially Initialized Destination"]).
text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"A"]).
text('black',160,208,'Helvetica',0,17,1,1,0,1,8,15,207,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"B"]).
text('black',192,272,'Helvetica',0,17,1,1,0,1,9,15,211,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"C"]).
text('black',504,208,'Helvetica',0,17,1,1,0,1,8,15,215,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"E"]).
text('black',528,272,'Helvetica',0,17,1,1,0,1,8,15,223,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"F"]).
text('black',856,288,'Helvetica',0,17,1,1,0,1,9,15,225,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"G"]).
text('black',800,432,'Helvetica',0,17,1,1,0,1,9,15,229,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H"]).
text('black',464,304,'Helvetica',0,17,1,1,0,1,9,15,231,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"D"]).
poly('black',4,[
848,240,848,224,864,224,904,224],0,2,1,318,1,0,0,0,10,4,0,0,0,'2','10','4',
"6",[
]).
text('black',664,192,'Helvetica',0,17,1,1,0,1,107,15,326,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_hyper_copy()"]).
text('black',672,352,'Helvetica',0,17,1,1,0,1,107,15,334,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_hyper_copy()"]).

BIN
doc/html/pipe3.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 6.8 KiB

70
doc/html/pipe3.obj Normal file

@ -0,0 +1,70 @@
%TGIF 3.0-p5
state(1,33,100,0,0,0,8,1,9,1,1,0,0,0,0,1,1,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[
]).
box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[
]).
poly('black',2,[
128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',912,64,976,256,0,1,1,87,0,0,0,0,0,'1',[
]).
box('black',928,96,960,224,26,1,1,88,0,0,0,0,0,'1',[
]).
text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"File"]).
text('black',944,48,'Helvetica',0,17,1,1,0,1,64,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Application"]).
text('black',480,104,'Helvetica',0,17,1,1,0,1,65,15,99,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5D_read()"]).
text('black',480,88,'Helvetica',0,17,1,1,0,1,58,15,108,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5Dread()"]).
box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[
]).
poly('black',5,[
72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"== Loop"]).
poly('black',3,[
48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',480,40,'Helvetica',0,24,1,1,0,1,295,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Fig 3: No Type Conversion"]).
text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"A"]).
poly('black',5,[
152,160,136,224,152,272,168,224,152,160],1,2,1,273,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
text('black',480,120,'Helvetica',0,17,1,1,0,1,96,15,277,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5S_simp_read()"]).
text('black',480,136,'Helvetica',0,17,1,1,0,1,86,15,281,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_arr_read()"]).
poly('black',5,[
880,160,864,224,880,272,896,224,880,160],1,2,1,283,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',2,[
152,224,880,224],1,2,1,286,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
text('black',480,232,'Helvetica',0,17,1,1,0,1,101,15,291,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_block_read()"]).
text('black',480,248,'Helvetica',0,17,1,1,0,1,90,15,293,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_low_read()"]).
text('black',480,264,'Helvetica',0,17,1,1,0,1,98,15,309,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_sec2_read()"]).
text('black',480,280,'Helvetica',0,17,1,1,0,1,33,15,311,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"read()"]).
text('black',176,208,'Helvetica',0,17,1,1,0,1,8,15,418,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"B"]).

BIN
doc/html/pipe4.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 8.2 KiB

92
doc/html/pipe4.obj Normal file

@ -0,0 +1,92 @@
%TGIF 3.0-p5
state(1,33,100,0,0,0,8,1,9,1,1,1,2,1,0,1,0,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[
]).
box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[
]).
poly('black',2,[
128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',912,96,944,224,26,1,1,88,0,0,0,0,0,'1',[
]).
text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"File"]).
text('black',928,72,'Helvetica',0,17,1,1,0,1,32,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Buffer"]).
box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[
]).
poly('black',5,[
72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"== Loop"]).
poly('black',3,[
48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',480,40,'Helvetica',0,24,1,1,0,1,372,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Fig 4: Regularly Chunked Storage"]).
text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"A"]).
text('black',480,104,'Helvetica',0,17,1,1,0,1,86,15,281,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_arr_read()"]).
text('black',480,120,'Helvetica',0,17,1,1,0,1,102,15,349,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_istore_read()"]).
text('black',480,136,'Helvetica',0,17,1,1,0,1,167,15,351,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_istore_copy_hyperslab()"]).
poly('black',5,[
160,160,144,224,160,272,176,224,160,160],1,2,1,362,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
poly('black',5,[
880,160,864,224,880,272,896,224,880,160],1,2,1,363,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
box('black',448,192,512,256,26,1,1,364,0,0,0,0,0,'1',[
]).
text('black',480,176,'Helvetica',0,17,1,1,0,1,43,15,367,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"CHUNK"]).
poly('black',2,[
160,224,448,224],1,2,1,372,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
poly('black',2,[
512,224,880,224],1,2,1,373,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
text('black',288,224,'Helvetica',0,17,1,1,0,1,101,15,385,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_block_read()"]).
text('black',288,240,'Helvetica',0,17,1,1,0,1,90,15,387,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_low_read()"]).
text('black',288,256,'Helvetica',0,17,1,1,0,1,98,15,391,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_sec2_read()"]).
text('black',288,272,'Helvetica',0,17,1,1,0,1,33,15,395,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"read()"]).
poly('black',5,[
456,256,448,296,480,320,512,296,504,256],1,2,1,401,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
text('black',184,208,'Helvetica',0,17,1,1,0,1,8,15,422,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"B"]).
text('black',520,208,'Helvetica',0,17,1,1,0,1,9,15,434,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"D"]).
text('black',440,272,'Helvetica',0,17,1,1,0,1,9,15,440,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"C"]).
text('black',480,320,'Helvetica',0,17,1,1,0,1,107,15,444,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5Z_uncompress()"]).
text('black',672,224,'Helvetica',0,17,1,1,0,1,107,15,454,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_hyper_copy()"]).
text('black',672,240,'Helvetica',0,17,1,1,0,1,106,15,464,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5V_stride_copy()"]).
text('black',672,256,'Helvetica',0,17,1,1,0,1,54,15,466,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"memcpy()"]).
text('black',168,488,'Helvetica',0,17,1,0,0,1,282,15,471,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"NOTE: H5Z_uncompress() is not implemented yet."]).

BIN
doc/html/pipe5.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 6.1 KiB

52
doc/html/pipe5.obj Normal file

@ -0,0 +1,52 @@
%TGIF 3.0-p5
state(1,33,100,0,0,0,8,1,9,1,1,1,2,1,0,1,0,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[
]).
box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[
]).
poly('black',2,[
128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4',
"0",[
]).
box('black',912,96,944,224,26,1,1,88,0,0,0,0,0,'1',[
]).
text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"File"]).
text('black',928,72,'Helvetica',0,17,1,1,0,1,32,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Buffer"]).
box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[
]).
text('black',480,40,'Helvetica',0,24,1,1,0,1,333,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Fig 5: Reading a Single Chunk"]).
text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"A"]).
text('black',480,112,'Helvetica',0,17,1,1,0,1,86,15,281,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_arr_read()"]).
text('black',480,128,'Helvetica',0,17,1,1,0,1,102,15,349,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_istore_read()"]).
text('black',480,144,'Helvetica',0,17,1,1,0,1,167,15,351,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_istore_copy_hyperslab()"]).
text('black',480,160,'Helvetica',0,17,1,1,0,1,101,15,385,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_block_read()"]).
text('black',480,176,'Helvetica',0,17,1,1,0,1,90,15,387,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_low_read()"]).
text('black',480,192,'Helvetica',0,17,1,1,0,1,98,15,391,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5F_sec2_read()"]).
text('black',480,208,'Helvetica',0,17,1,1,0,1,33,15,395,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"read()"]).
text('black',864,240,'Helvetica',0,17,1,1,0,1,107,15,444,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"H5Z_uncompress()"]).
text('black',56,488,'Helvetica',0,17,1,0,0,1,282,15,471,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"NOTE: H5Z_uncompress() is not implemented yet."]).
poly('black',5,[
912,176,864,176,840,208,872,232,912,216],1,2,1,490,2,0,0,0,10,4,0,0,0,'2','10','4',
"",[
]).
text('black',896,184,'Helvetica',0,17,1,0,0,1,8,15,491,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"B"]).

283
doc/html/review1.html Normal file

@ -0,0 +1,283 @@
<html><head><title>
HDF5 Draft Revised API Example Code
</title></head><body>
<center>
<h1>HDF5: Revised API Example Code</h1>
</center>
<P>Example programs/sections of code below:
<dl COMPACT>
<dt><a href="#Example1">#1</a>
<dd>A simple example showing how to create a file.
<dt><a href="#Example2">#2</a>
<dd>A example showing how to check if a file is an HDF5 file and list it's contents.
<dt><a href="#Example3">#3</a>
<dd>A example showing how to create a homogenous multi-dimensional dataset.
<dt><a href="#Example4">#4</a>
<dd>A example showing how to create a compound 1-D dataset.
<dt><a href="#Example5">#5</a>
<dd>A example showing how to create a compound multi-dimensional dataset.
<dt><a href="#Example6">#6</a>
<dd>A example showing how to read a generic dataset.
</dl>
<hr>
<h2><a name="Example1">Simple Example showing how to create a file.</a></h2>
<P>Notes:<br>
This example creates a new HDF5 file and allows write access.
If the file exists already, the H5F_ACC_TRUNC flag would also be necessary to
overwrite the previous file's information.
<P>Code:
<code> <pre>
hid_t file_id;
file_id=<A HREF="H5.apiv3.html#File-Create">H5Fcreate</a>("example1.h5",0);
<A HREF="H5.apiv3.html#File-Close">H5Fclose</a>(file_id);
</pre> </code>
<hr>
<h2><a name="Example2">Example showing how check if a file is an HDF5 file and list it's contents.</a></h2>
<P>Notes:<br>
This example checks if a file is an HDF5 file and lists the contents of the top
level (file level) group.
<P>Code:
<code> <pre>
hid_t file_id; /* File ID */
uint32 num_items; /* number of items in top-level group */
intn i; /* counter */
char *obj_name; /* object's name as string atom */
uintn name_len; /* object name's length in chars */
uintn buf_len=0; /* buffer length for names */
char *buf=NULL; /* buffer for names */
if(<A HREF="H5.apiv3.html#File-IsHDF5">H5Fis_hdf5</a>("example2.h5")==TRUE)
{
file_id=<A HREF="H5.apiv3.html#File-Open">H5Fopen</a>("example2.h5",H5F_ACC_RDWR|H5ACC_CREATE);
num_items=<A HREF="H5.apiv3.html#Group-GetNumContents">H5GgetNumContents</a>(file_id);
for(i=0; i&lt;num_items; i++)
{
obj_name=<A HREF="H5.apiv3.html#Group-GetNameByIndex">H5GgetNameByIndex</a>(file_id,i,NULL,0);
printf("object #%d is: %s\n",i,buf);
HDfree(obj_name);
}
<A HREF="H5.apiv3.html#File-Close">H5Fclose</a>(file_id);
}
</pre> </code>
<hr>
<h2><a name="Example3">Example showing how create a homogenous multi-dimensional dataset.</a></h2>
<P>Notes:<br>
This example creates a 4-dimensional dataset of 32-bit floating-point
numbers, corresponding to the current Scientific Dataset functionality.
This example assumes that the datatype and dataspace of the dataset will not
be re-used.
<P>Code:
<code> <pre>
hid_t file_id; /* File's ID */
uint32 dims[4]={6,5,4,3}; /* the size of each dimension */
hid_t dataset_id; /* new object's ID */
float32 obj_data[6][5][4][3]; /* storage for the dataset's data */
if((file_id=<A HREF="H5.apiv3.html#File-Create">H5Fcreate</a>("example3.h5",H5F_ACC_TRUNC))>=0)
{
/* Create & initialize the dataset object */
dataset_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATASET,"Simple Object");
/* Create & initialize a datatype object */
<A HREF="H5.apiv3.html#Datatype-SetType">H5TsetType</a>(dataset_id,H5TYPE_FLOAT,4,H5T_BIGENDIAN);
/* Initialize dimensionality of dataset */
<A HREF="H5.apiv3.html#Dimensionality-SetSpace">H5SsetSpace</a>(dataset_id,rank,dims);
&lt;initialize data array&gt;
/* Write the entire dataset out */
<A HREF="H5.apiv3.html#Dataset-Write">H5Dwrite</a>(dataset_id,H5S_SCALAR,data);
&lt;or&gt;
<A HREF="H5.apiv3.html#Dataset-Write">H5Dwrite</a>(dataset_id,dataset_id,data);
/* Release the atoms we've created */
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(dataset_id);
/* close the file */
<A HREF="H5.apiv3.html#File-Close">H5Fclose</a>(file_id);
}
</pre> </code>
<hr>
<h2><a name="Example4">Example showing how create a compound 1-D dataset.</a></h2>
<P>Notes:<br>
This example creates a 1-dimensional dataset of compound datatype records,
corresponding to the current Vdata functionality. This example also assumes
that the datatype and dataspace will not be re-used.
<P>Code:
<code> <pre>
hid_t file_id; /* File's ID */
uint32 dims[1]={45}; /* the size of the dimension */
hid_t dataset_id; /* object's ID */
void *obj_data; /* pointer to the dataset's data */
if((file_id=<A HREF="H5.apiv3.html#File-Create">H5Fcreate</a>("example4.h5",H5F_ACC_TRUNC))>=0)
{
/* Create & initialize the dataset object */
dataset_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATASET,"Compound Object");
/* Initialize datatype */
<A HREF="H5.apiv3.html#Datatype-SetType">H5TsetType</a>(dataset_id,H5TYPE_STRUCT);
<A HREF="H5.apiv3.html#Datatype-AddField">H5TaddField</a>(dataset_id,H5TYPE_FLOAT32,"Float32 Scalar Field",H5SPACE_SCALAR);
<A HREF="H5.apiv3.html#Datatype-AddField">H5TaddField</a>(dataset_id,H5TYPE_CHAR,"Char Field",H5SPACE_SCALAR);
<A HREF="H5.apiv3.html#Datatype-AddField">H5TaddField</a>(dataset_id,H5TYPE_UINT16,"UInt16 Field",H5SPACE_SCALAR);
<A HREF="H5.apiv3.html#Datatype-EndDefine">H5TendDefine</a>(dataset_id);
/* Initialize dimensionality */
<A HREF="H5.apiv3.html#Dimensionality-SetSpace">H5SsetSpace</a>(dataset_id,1,dims);
&lt;initialize data array&gt;
/* Write the entire dataset out */
<A HREF="H5.apiv3.html#Dataset-Write">H5Dwrite</a>(dataset_id,H5S_SCALAR,data);
/* Release the atoms we've created */
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(dataset_id);
/* close the file */
<A HREF="H5.apiv3.html#File-Close">H5Fclose</a>(file_id);
}
</pre> </code>
<hr>
<h2><a name="Example5">Example showing how create a compound multi-dimensional dataset.</a></h2>
<P>Notes:<br>
This example creates a 3-dimensional dataset of compound datatype records,
roughly corresponding to a multi-dimensional Vdata functionality. This
example also shows the use of multi-dimensional fields in the compound datatype.
This example uses "stand-alone" datatypes and dataspaces.
<P>Code:
<code> <pre>
hid_t file_id; /* File's ID */
hid_t type_id; /* datatype's ID */
hid_t dim_id; /* dimensionality's ID */
uint32 dims[3]={95,67,5}; /* the size of the dimensions */
hid_t field_dim_id; /* dimensionality ID for fields in the structure */
uint32 field_dims[4]; /* array for field dimensions */
hid_t dataset_id; /* object's ID */
void *obj_data; /* pointer to the dataset's data */
if((file_id=<A HREF="H5.apiv3.html#File-Create">H5Fcreate</a>("example5.h5",H5F_ACC_TRUNC))>=0)
{
/* Create & initialize a datatype object */
type_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATATYPE,"Compound Type #1");
<A HREF="H5.apiv3.html#Datatype-SetType">H5TsetType</a>(type_id,H5TYPE_STRUCT);
/* Create each multi-dimensional field in structure */
field_dim_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATASPACE,"Lat/Long Dims");
field_dims[0]=360;
field_dims[1]=720;
<A HREF="H5.apiv3.html#Dimensionality-SetSpace">H5SsetSpace</a>(field_dim_id,2,field_dims);
<A HREF="H5.apiv3.html#Datatype-AddField">H5TaddField</a>(type_id,H5TYPE_FLOAT32,"Lat/Long Locations",field_dim_id);
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(field_dim_id);
field_dim_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATASPACE,"Browse Dims");
field_dims[0]=40;
field_dims[1]=40;
<A HREF="H5.apiv3.html#Dimensionality-SetSpace">H5SsetSpace</a>(field_dim_id,2,field_dims);
<A HREF="H5.apiv3.html#Datatype-AddField">H5TaddField</a>(type_id,H5TYPE_CHAR,"Browse Image",field_dim_id);
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(field_dim_id);
field_dim_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATASPACE,"Multispectral Dims");
field_dims[0]=80;
field_dims[1]=60;
field_dims[2]=40;
<A HREF="H5.apiv3.html#Dimensionality-SetSpace">H5SsetSpace</a>(field_dim_id,3,field_dims);
<A HREF="H5.apiv3.html#Datatype-AddField">H5TaddField</a>(type_id,H5TYPE_UINT16,"Multispectral Scans",field_dim_id);
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(field_dim_id);
<A HREF="H5.apiv3.html#Datatype-EndDefine">H5TendDefine</a>(type_id);
/* Create & initialize a dimensionality object */
dim_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATASPACE,"3-D Dim");
<A HREF="H5.apiv3.html#Dimensionality-SetSpace">H5SsetSpace</a>(dim_id,3,dims);
/* Create & initialize the dataset object */
dataset_id=<A HREF="H5.apiv3.html#Meta-Create">H5Mcreate</a>(file_id,H5OBJ_DATASET,"Compound Multi-Dim Object");
<A HREF="H5.apiv3.html#Dataset-SetInfo">H5DsetInfo</a>(dataset_id,type_id,dim_id);
&lt;initialize data array&gt;
/* Write the entire dataset out */
<A HREF="H5.apiv3.html#Dataset-Write">H5Dwrite</a>(dataset_id,H5S_SCALAR,data);
/* Release the atoms we've created */
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(type_id);
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(dim_id);
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(dataset_id);
/* close the file */
<A HREF="H5.apiv3.html#File-Close">H5Fclose</a>(file_id);
}
</pre> </code>
<hr>
<h2><a name="Example6">Example showing how read a generic dataset.</a></h2>
<P>Notes:<br>
This example shows how to get the information for and display a generic
dataset.
<P>Code:
<code> <pre>
hid_t file_id; /* File's ID */
hid_t dataset_id; /* dataset's ID in memory */
uintn elem_size; /* size of each element */
uintn nelems; /* number of elements in array */
void *obj_data; /* pointer to the dataset's data */
if((file_id=<A HREF="H5.apiv3.html#File-Open">H5Fopen</a>("example6.h5",0))>=0)
{
/* Attach to a datatype object */
dataset_id=<A HREF="H5.apiv3.html#Meta-Access">H5MaccessByIndex</a>(obj_oid,0);
if(<A HREF="H5.apiv3.html#Datatype-BaseType">H5TbaseType</a>(dataset_id)==H5T_COMPOUND)
{
&lt;set up for compound object&gt;
}
else
{
&lt;set up for homogenous object&gt;
}
elem_size=<A HREF="H5.apiv3.html#Datatype-Size">H5Tsize</a>(dataset_id);
nelems=<A HREF="H5.apiv3.html#Dataspace-NElem">H5Snelem</a>(dataset_id);
&lt;allocate space based on element size and number of elements &gt;
/* Read in the dataset */
<A HREF="H5.apiv3.html#Dataset-Read">H5Dwrite</a>(dataset_id,H5S_SCALAR,data);
&lt;or&gt;
<A HREF="H5.apiv3.html#Dataset-Read">H5Dwrite</a>(dataset_id,dataset_id,data);
/* Release the atoms we've accessed */
<A HREF="H5.apiv3.html#Meta-Release">H5Mrelease</a>(dataset_id);
/* close the file */
<A HREF="H5.apiv3.html#File-Close">H5Fclose</a>(file_id);
}
</pre> </code>

252
doc/html/review1a.html Normal file

@ -0,0 +1,252 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Group Examples</title>
</head>
<body>
<center><h1>Group Examples</h1></center>
<hr>
<h1>Background</h1>
<p>Directories (or now <i>Groups</i>) are currently implemented as
a directed graph with a single entry point into the graph which
is the <i>Root Object</i>. The root object is usually a
group. All objects have at least one predecessor (the <i>Root
Object</i> always has the HDF5 file boot block as a
predecessor). The number of predecessors of a group is also
known as the <i>hard link count</i> or just <i>link count</i>.
Unlike Unix directories, HDF5 groups have no ".." entry since
any group can have multiple predecessors. Given the handle or
id of some object and returning a full name for that object
would be an expensive graph traversal.
<p>A special optimization is that a file may contain a single
non-group object and no group(s). The object has one
predecessor which is the file boot block. However, once a root
group is created it never dissappears (although I suppose it
could if we wanted).
<p>A special object called a <i>Symbolic Link</i> is simply a
name. Usually the name refers to some (other) object, but that
object need not exist. Symbolic links in HDF5 will have the
same semantics as symbolic links in Unix.
<p>The symbol table graph contains "entries" for each name. An
entry contains the file address for the object header and
possibly certain messages cached from the object header.
<p>The H5G package understands the notion of <i>opening</i> and object
which means that given the name of the object, a handle to the
object is returned (this isn't an API function). Objects can be
opened multiple times simultaneously through the same name or,
if the object has hard links, through other names. The name of
an object cannot be removed from a group if the object is opened
through that group (although the name can change within the
group).
<p>Below the API, object attributes can be read without opening
the object; object attributes cannot change without first
opening that object. The one exception is that the contents of a
group can change without opening the group.
<hr>
<h1>Building a hierarchy from a flat namespace</h1>
<p>Assuming we have a flat name space (that is, the root object is
a group which contains names for all other objects in the file
and none of those objects are groups), then we can build a
hierarchy of groups that also refer to the objects.
<p>The file initially contains `foo' `bar' `baz' in the root
group. We wish to add groups `grp1' and `grp2' so that `grp1'
contains objects `foo' and `baz' and `grp2' contains objects
`bar' and `baz' (so `baz' appears in both groups).
<p>In either case below, one might want to move the flat objects
into some other group (like `flat') so their names don't
interfere with the rest of the hierarchy (or move the hierarchy
into a directory called `/hierarchy').
<h2>with symbolic links</h2>
<p>Create group `grp1' and add symbolic links called `foo' whose
value is `/foo' and `baz' whose value is `/baz'. Similarly for
`grp2'.
<p>Accessing `grp1/foo' involves searching the root group for
the name `grp1', then searching that group for `foo', then
searching the root directory for `foo'. Alternatively, one
could change working groups to the grp1 group and then ask for
`foo' which searches `grp1' for the name `foo', then searches
the root group for the name `foo'.
<p>Deleting `/grp1/foo' deletes the symbolic link without
affecting the `/foo' object. Deleting `/foo' leaves the
`/grp1/foo' link dangling.
<h2>with hard links</h2>
<p>Creating the hierarchy is the same as with symbolic links.
<p>Accessing `/grp1/foo' searches the root group for the name
`grp1', then searches that group for the name `foo'. If the
current working group is `/grp1' then we just search for the
name `foo'.
<p>Deleting `/grp1/foo' leaves `/foo' and vice versa.
<h2>the code</h2>
<p>Depending on the eventual API...
<code><pre>
H5Gcreate (file_id, "/grp1");
H5Glink (file_id, H5G_HARD, "/foo", "/grp1/foo");
</pre></code>
or
<code><pre>
group_id = H5Gcreate (root_id, "grp1");
H5Glink (file_id, H5G_HARD, root_id, "foo", group_id, "foo");
H5Gclose (group_id);
</pre></code>
<hr>
<h1>Building a flat namespace from a hierarchy</h1>
<p>Similar to abvoe, but in this case we have to watch out that
we don't get two names which are the same: what happens to
`/grp1/baz' and `/grp2/baz'? If they really refer to the same
object then we just have `/baz', but if they point to two
different objects what happens?
<p>The other thing to watch out for cycles in the graph when we
traverse it to build the flat namespace.
<hr>
<h1>Listing the Group Contents</h1>
<p>Two things to watch out for are that the group contents don't
appear to change in a manner which would confuse the
application, and that listing everything in a group is as
efficient as possible.
<h2>Method A</h2>
<p>Query the number of things in a group and then query each item
by index. A trivial implementation would be O(n*n) and wouldn't
protect the caller from changes to the directory which move
entries around and therefore change their indices.
<code><pre>
n = H5GgetNumContents (group_id);
for (i=0; i&lt;n; i++) {
H5GgetNameByIndex (group_id, i, ...); /*don't worry about args yet*/
}
</pre></code>
<h2>Method B</h2>
<p>The API contains a single function that reads all information
from the specified group and returns that info through an array.
The caller is responsible for freeing the array allocated by the
query and the things to which it points. This also makes it
clear the the returned value is a snapshot of the group which
doesn't change if the group is modified.
<code><pre>
n = H5Glist (file_id, "/grp1", info, ...);
for (i=0; i&lt;n; i++) {
printf ("name = %s\n", info[i].name);
free (info[i].name); /*and maybe other fields too?*/
}
free (info);
</pre></code>
Notice that it would be difficult to expand the info struct since
its definition is part of the API.
<h2>Method C</h2>
<p>The caller asks for a snapshot of the group and then accesses
items in the snapshot through various query-by-index API
functions. When finished, the caller notifies the library that
it's done with the snapshot. The word "snapshot" makes it clear
that subsequent changes to the directory will not be reflected in
the shapshot_id.
<code><pre>
snapshot_id = H5Gsnapshot (group_id); /*or perhaps group_name */
n = H5GgetNumContents (snapshot_id);
for (i=0; i&lt;n; i++) {
H5GgetNameByIndex (shapshot_id, i, ...);
}
H5Grelease (shapshot_id);
</pre></code>
In fact, we could allow the user to leave off the H5Gsnapshot and
H5Grelease and use group_id in the H5GgetNumContents and
H5GgetNameByIndex so they can choose between Method A and Method
C.
<hr>
<h1>An implementation of Method C</h1>
<dl>
<dt><code>hid_t H5Gshapshot (hid_t group_id)</code>
<dd>Opens every object in the specified group and stores the
handles in an array managed by the library (linear-time
operation). Open object handles are essentialy symbol table
entries with a little extra info (symbol table entries cache
certain things about the object which are also found in the
object header). Because the objects are open (A) they cannot be
removed from the group, (B) querying the object returns the
latest info even if something else has that object open, (C)
if the object is renamed within the group then its name with
<code>H5GgetNameByIndex</code> is changed. Adding new entries
to a group doesn't affect the snapshot.
<dt><code>char *H5GgetNameByIndex (hid_t shapshot_id, int
index)</code>
<dd>Uses the open object handle from entry <code>index</code> of
the snapshot array to get the object name. This is a
constant-time operation. The name is updated automatically if
the object is renamed within the group.
<dt><code>H5Gget&lt;whatever&gt;ByIndex...()</code>
<dd>Uses the open object handle from entry <code>index</code>,
which is just a symbol table entry, and reads the appropriate
object header message(s) which might be cached in the symbol
table entry. This is a constant-time operation if cached,
linear in the number of messages if not cached.
<dt><code>H5Grelease (hid_t snapshot_id)</code>
<dd>Closes each object refered to by the snapshot and then frees
the snapshot array. This is a linear-time operation.
</dl>
<hr>
<h1>To return <code>char*</code> or some HDF5 string type.</h1>
<p>In either case, the caller has to release resources associated
with the return value, calling free() or some HDF5 function.
<p>Names in the current implementation of the H5G package don't
contain embedded null characters and are always null terminated.
<p>Eventually the caller probably wants a <code>char*</code> so it
can pass it to some non-HDF5 function, does that require
strdup'ing the string again? Then the caller has to free() the
the char* <i>and</i> release the DHF5 string.
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Fri Sep 26 12:03:20 EST 1997 -->
<!-- hhmts start -->
Last modified: Fri Oct 3 09:32:10 EST 1997
<!-- hhmts end -->
</body>
</html>

274
doc/html/storage.html Normal file

@ -0,0 +1,274 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Raw Data Storage in HDF5</title>
</head>
<body>
<h1>Raw Data Storage in HDF5</h1>
<p>This document describes the various ways that raw data is
stored in an HDF5 file and the object header messages which
contain the parameters for the storage.
<p>Raw data storage has three components: the mapping from some
logical multi-dimensional element space to the linear address
space of a file, compression of the raw data on disk, and
striping of raw data across multiple files. These components
are orthogonal.
<p>Some goals of the storage mechanism are to be able to
efficently store data which is:
<dl>
<dt>Small
<dd>Small pieces of raw data can be treated as meta data and
stored in the object header. This will be achieved by storing
the raw data in the object header with message 0x0006.
Compression and striping are not supported in this case.
<dt>Complete Large
<dd>The library should be able to store large arrays
contiguously in the file provided the user knows the final
array size a priori. The array can then be read/written in a
single I/O request. This is accomplished by describing the
storage with object header message 0x0005. Compression and
striping are not supported in this case.
<dt>Sparse Large
<dd>A large sparse raw data array should be stored in a manner
that is space-efficient but one in which any element can still
be accessed in a reasonable amount of time. Implementation
details are below.
<dt>Dynamic Size
<dd>One often doesn't have prior knowledge of the size of an
array. It would be nice to allow arrays to grow dynamically in
any dimension. It might also be nice to allow the array to
grow in the negative dimension directions if convenient to
implement. Implementation details are below.
<dt>Subslab Access
<dd>Some multi-dimensional arrays are almost always accessed by
subslabs. For instance, a 2-d array of pixels might always be
accessed as smaller 1k-by-1k 2-d arrays always aligned on 1k
index values. We should be able to store the array in such a
way that striding though the entire array is not necessary.
Subslab access might also be useful with compression
algorithms where each storage slab can be compressed
independently of the others. Implementation details are below.
<dt>Compressed
<dd>Various compression algorithms can be applied to the entire
array. We're not planning to support separate algorithms (or a
single algorithm with separate parameters) for each chunk
although it would be possible to implement that in a manner
similar to the way striping across files is
implemented.
<dt>Striped Across Files
<dd>The array access functions should support arrays stored
discontiguously across a set of files.
</dl>
<h1>Implementation of Indexed Storage</h1>
<p>The Sparse Large, Dynamic Size, and Subslab Access methods
share so much code that they can be described with a single
message. The new Indexed Storage Message (<code>0x0008</code>)
will replace the old Chunked Object (<code>0x0009</code>) and
Sparse Object (<code>0x000A</code>) Messages.
<p>
<center>
<table border cellpadding=4 width="60%">
<caption align=bottom>
<b>The Format of the Indexed Storage Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4><br>Address of B-tree<br><br></td>
</tr>
<tr align=center>
<td>Number of Dimensions</td>
<td>Reserved</td>
<td>Reserved</td>
<td>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Reserved (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Alignment for Dimension 0 (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Alignment for Dimension 1 (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>...</td>
</tr>
<tr align=center>
<td colspan=4>Alignment for Dimension N (4 bytes)</td>
</tr>
</table>
</center>
<p>The alignment fields indicate the alignment in logical space to
use when allocating new storage areas on disk. For instance,
writing every other element of a 100-element one-dimensional
array (using one HDF5 I/O partial write operation per element)
that has unit storage alignment would result in 50
single-element, discontiguous storage segments. However, using
an alignment of 25 would result in only four discontiguous
segments. The size of the message varies with the number of
dimensions.
<p>A B-tree is used to point to the discontiguous portions of
storage which has been allocated for the object. All keys of a
particular B-tree are the same size and are a function of the
number of dimensions. It is therefore not possible to change the
dimensionality of an indexed storage array after its B-tree is
created.
<p>
<center>
<table border cellpadding=4 width="60%">
<caption align=bottom>
<b>The Format of a B-Tree Key</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4>External File Number or Zero (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Chunk Offset in Dimension 0 (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Chunk Offset in Dimension 1 (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>...</td>
</tr>
<tr align=center>
<td colspan=4>Chunk Offset in Dimension N (4 bytes)</td>
</tr>
</table>
</center>
<p>The keys within a B-tree obey an ordering based on the chunk
offsets. If the offsets in dimension-0 are equal, then
dimension-1 is used, etc. The External File Number field
contains a 1-origin offset into the External File List message
which contains the name of the external file in which that chunk
is stored.
<h1>Implementation of Striping</h1>
<p>The indexed storage will support arbitrary striping at the
chunk level; each chunk can be stored in any file. This is
accomplished by using the External File Number field of an
indexed storage B-tree key as a 1-origin offset into an External
File List Message (0x0009) which takes the form:
<p>
<center>
<table border cellpadding=4 width="60%">
<caption align=bottom>
<b>The Format of the External File List Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4><br>Name Heap Address<br><br></td>
</tr>
<tr align=center>
<td colspan=4>Number of Slots Allocated (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Number of File Names (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Byte Offset of Name 1 in Heap (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Byte Offset of Name 2 in Heap (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4>...</td>
</tr>
<tr align=center>
<td colspan=4><br>Unused Slot(s)<br><br></td>
</tr>
</table>
</center>
<p>Each indexed storage array that has all or part of its data
stored in external files will contain a single external file
list message. The size of the messages is determined when the
message is created, but it may be possible to enlarge the
message on demand by moving it. At this time, it's not possible
for multiple arrays to share a single external file list
message.
<dl>
<dt><code>
H5O_efl_t *H5O_efl_new (H5G_entry_t *object, intn
nslots_hint, intn heap_size_hint)
</code>
<dd>Adds a new, empty external file list message to an object
header and returns a pointer to that message. The message
acts as a cache for file descriptors of external files that
are open.
<p><dt><code>
intn H5O_efl_index (H5O_efl_t *efl, const char *filename)
</code>
<dd>Gets the external file index number for a particular file name.
If the name isn't in the external file list then it's added to
the H5O_efl_t struct and immediately written to the object
header to which the external file list message belongs. Name
comparison is textual. Each name should be relative to the
directory which contains the HDF5 file.
<p><dt><code>
H5F_low_t *H5O_efl_open (H5O_efl_t *efl, intn index, uintn mode)
</code>
<dd>Gets a low-level file descriptor for an external file. The
external file list caches file descriptors because we might
have many more external files than there are file descriptors
available to this process. The caller should not close this file.
<p><dt><code>
herr_t H5O_efl_release (H5O_efl_t *efl)
</code>
<dd>Releases an external file list, closes all files
associated with that list, and if the list has been modified
since the call to <code>H5O_efl_new</code> flushes the message
to disk.
</dl>
<hr>
<address><a href="mailto:robb@arborea.spizella.com">Robb Matzke</a></address>
<!-- Created: Fri Oct 3 09:52:32 EST 1997 -->
<!-- hhmts start -->
Last modified: Tue Nov 25 12:36:50 EST 1997
<!-- hhmts end -->
</body>
</html>

172
doc/html/study.html Normal file

@ -0,0 +1,172 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Testing the chunked layout of HDF5</title>
</head>
<body>
<h1>Testing the chunked layout of HDF5</h1>
<p>This is the results of studying the chunked layout policy in
HDF5. A 1000 by 1000 array of integers was written to a file
dataset extending the dataset with each write to create, in the
end, a 5000 by 5000 array of 4-byte integers for a total data
storage size of 100 million bytes.
<p>
<center>
<img alt="Order that data was written" src="study_p1.gif">
<br><b>Fig 1: Write-order of Output Blocks</b>
</center>
<p>After the array was written, it was read back in blocks that
were 500 by 500 bytes in row major order (that is, the top-left
quadrant of output block one, then the top-right quadrant of
output block one, then the top-left quadrant of output block 2,
etc.).
<p>I tried to answer two questions:
<ul>
<li>How does the storage overhead change as the chunk size
changes?
<li>What does the disk seek pattern look like as the chunk size
changes?
</ul>
<p>I started with chunk sizes that were multiples of the read
block size or k*(500, 500).
<p>
<center>
<table border>
<caption align=bottom>
<b>Table 1: Total File Overhead</b>
</caption>
<tr>
<th>Chunk Size (elements)</th>
<th>Meta Data Overhead (ppm)</th>
<th>Raw Data Overhead (ppm)</th>
</tr>
<tr align=center>
<td>500 by 500</td>
<td>85.84</td>
<td>0.00</td>
</tr>
<tr align=center>
<td>1000 by 1000</td>
<td>23.08</td>
<td>0.00</td>
</tr>
<tr align=center>
<td>5000 by 1000</td>
<td>23.08</td>
<td>0.00</td>
</tr>
<tr align=center>
<td>250 by 250</td>
<td>253.30</td>
<td>0.00</td>
</tr>
<tr align=center>
<td>499 by 499</td>
<td>85.84</td>
<td>205164.84</td>
</tr>
</table>
</center>
<hr>
<p>
<center>
<img alt="500x500" src="study_500x500.gif">
<br><b>Fig 2: Chunk size is 500x500</b>
</center>
<p>The first half of Figure 2 shows output to the file while the
second half shows input. Each dot represents a file-level I/O
request and the lines that connect the dots are for visual
clarity. The size of the request is not indicated in the
graph. The output block size is four times the chunk size which
results in four file-level write requests per block for a total
of 100 requests. Since file space for the chunks was allocated
in output order, and the input block size is 1/4 the output
block size, the input shows a staircase effect. Each input
request results in one file-level read request. The downward
spike at about the 60-millionth byte is probably the result of a
cache miss for the B-tree and the downward spike at the end is
probably a cache flush or file boot block update.
<hr>
<p>
<center>
<img alt="1000x1000" src="study_1000x1000.gif">
<br><b>Fig 2: Chunk size is 1000x1000</b>
</center>
<p>In this test I increased the chunk size to match the output
chunk size and one can see from the first half of the graph that
25 file-level write requests were issued, one for each output
block. The read half of the test shows that four times the
amount of data was read as written. This results from the fact
that HDF5 must read the entire chunk for any request that falls
within that chunk, which is done because (1) if the data is
compressed the entire chunk must be decompressed, and (2) the
library assumes that a chunk size was chosen to optimize disk
performance.
<hr>
<p>
<center>
<img alt="5000x1000" src="study_5000x1000.gif">
<br><b>Fig 3: Chunk size is 5000x1000</b>
</center>
<p>Increasing the chunk size further results in even worse
performance since both the read and write halves of the test are
re-reading and re-writing vast amounts of data. This proves
that one should be careful that chunk sizes are not much larger
than the typical partial I/O request.
<hr>
<p>
<center>
<img alt="250x250" src="study_250x250.gif">
<br><b>Fig 4: Chunk size is 250x250</b>
</center>
<p>If the chunk size is decreased then the amount of data
transfered between the disk and library is optimal for no
caching, but the amount of meta data required to describe the
chunk locations increases to 250 parts per million. One can
also see that the final downward spike contains more file-level
write requests as the meta data is flushed to disk just before
the file is closed.
<hr>
<p>
<center>
<img alt="499x499" src="study_499x499.gif">
<br><b>Fig 4: Chunk size is 499x499</b>
</center>
<p>This test shows the result of choosing a chunk size which is
close to the I/O block size. Because the total size of the
array isn't a multiple of the chunk size, the library allocates
an extra zone of chunks around the top and right edges of the
array which are only partially filled. This results in
20,516,484 extra bytes of storage, a 20% increase in the total
raw data storage size. But the amount of meta data overhead is
the same as for the 500 by 500 test. In addition, the mismatch
causes entire chunks to be read in order to update a few
elements along the edge or the chunk which results in a 3.6-fold
increase in the amount of data transfered.
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Fri Jan 30 21:04:49 EST 1998 -->
<!-- hhmts start -->
Last modified: Fri Jan 30 23:51:31 EST 1998
<!-- hhmts end -->
</body>
</html>

BIN
doc/html/study_1000x1000.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 6.4 KiB

BIN
doc/html/study_250x250.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 6.8 KiB

BIN
doc/html/study_499x499.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 10 KiB

BIN
doc/html/study_5000x1000.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 10 KiB

BIN
doc/html/study_500x500.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 6.7 KiB

BIN
doc/html/study_p1.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 6.4 KiB

113
doc/html/study_p1.obj Normal file

@ -0,0 +1,113 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,16,1,9,1,1,0,0,3,7,1,1,'Helvetica',0,24,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
box('black',64,64,384,384,0,1,1,22,0,0,0,0,0,'1',[
]).
poly('black',2,[
128,64,128,384],0,1,1,23,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
192,64,192,384],0,1,1,24,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
256,64,256,384],0,1,1,25,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
320,64,320,384],0,1,1,26,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
64,128,384,128],0,1,1,27,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
64,192,384,192],0,1,1,28,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
64,256,384,256],0,1,1,29,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
64,320,384,320],0,1,1,30,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',96,80,'Courier',0,17,1,1,0,1,7,14,37,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1"]).
text('black',160,80,'Courier',0,17,1,1,0,1,7,14,39,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"2"]).
text('black',224,80,'Courier',0,17,1,1,0,1,7,14,41,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"3"]).
text('black',288,80,'Courier',0,17,1,1,0,1,7,14,43,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"4"]).
text('black',352,80,'Courier',0,17,1,1,0,1,7,14,47,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"5"]).
text('black',96,144,'Courier',0,17,1,1,0,1,7,14,51,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"6"]).
text('black',160,144,'Courier',0,17,1,1,0,1,7,14,53,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"7"]).
text('black',224,144,'Courier',0,17,1,1,0,1,7,14,55,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"8"]).
text('black',288,144,'Courier',0,17,1,1,0,1,7,14,57,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"9"]).
text('black',352,144,'Courier',0,17,1,1,0,1,14,14,59,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"10"]).
text('black',96,208,'Courier',0,17,1,1,0,1,14,14,61,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"11"]).
text('black',160,208,'Courier',0,17,1,1,0,1,14,14,63,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"12"]).
text('black',224,208,'Courier',0,17,1,1,0,1,14,14,65,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"13"]).
text('black',288,208,'Courier',0,17,1,1,0,1,14,14,67,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"14"]).
text('black',352,208,'Courier',0,17,1,1,0,1,14,14,71,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"15"]).
text('black',96,272,'Courier',0,17,1,1,0,1,14,14,75,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"16"]).
text('black',160,272,'Courier',0,17,1,1,0,1,14,14,77,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"17"]).
text('black',224,272,'Courier',0,17,1,1,0,1,14,14,79,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"18"]).
text('black',288,272,'Courier',0,17,1,1,0,1,14,14,81,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"19"]).
text('black',352,272,'Courier',0,17,1,1,0,1,14,14,83,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"20"]).
text('black',96,336,'Courier',0,17,1,1,0,1,14,14,87,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"21"]).
text('black',160,336,'Courier',0,17,1,1,0,1,14,14,89,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"22"]).
text('black',224,336,'Courier',0,17,1,1,0,1,14,14,91,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"23"]).
text('black',288,336,'Courier',0,17,1,1,0,1,14,14,93,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"24"]).
text('black',352,336,'Courier',0,17,1,1,0,1,14,14,95,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"25"]).
poly('black',2,[
416,64,416,384],3,1,1,100,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
64,416,384,416],3,1,1,101,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',390,228,'Courier',0,17,1,0,0,1,14,35,102,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,1,0,[
390,228,390,228,425,242,0,-1000,1000,0,34,18,389,227,426,243],[
"5,000"]).
text('black',224,432,'Courier',0,17,1,1,0,1,35,14,116,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"5,000"]).
text('black',160,512,'Courier',0,17,1,0,0,1,105,14,131,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"= 1,000 x 1,000"]).
box('black',80,480,144,544,7,1,1,134,0,0,0,0,0,'1',[
]).
text('black',224,16,'Helvetica',0,24,1,1,0,1,296,29,144,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Order that data was written"]).
box('black',32,0,464,576,0,1,1,149,0,0,0,0,0,'1',[
]).

313
doc/html/symtab Normal file

@ -0,0 +1,313 @@
A number of issues involving caching of object header messages in
symbol table entries must be resolved.
What is the motivation for these changes?
If we make objects completely independent of object name it allows
us to refer to one object by multiple names (a concept called hard
links in Unix file systems), which in turn provides an easy way to
share data between datasets.
Every object in an HDF5 file has a unique, constant object header
address which serves as a handle (or OID) for the object. The
object header contains messages which describe the object.
HDF5 allows some of the object header messages to be cached in
symbol table entries so that the object header doesn't have to be
read from disk. For instance, an entry for a directory caches the
directory disk addresses required to access that directory, so the
object header for that directory is seldom read.
If an object has multiple names (that is, a link count greater than
one), then it has multiple symbol table entries which point to it.
All symbol table entries must agree on header messages. The
current mechanism is to turn off the caching of header messages in
symbol table entries when the header link count is more than one,
and to allow caching once the link count returns to one.
However, in the current implementation, a package is allowed to
copy a symbol table entry and use it as a private cache for the
object header. This doesn't work for a number of reasons (all but
one require a `delete symbol entry' operation).
1. If two packages hold copies of the same symbol table entry,
they don't notify each other of changes to the symbol table
entry. Eventually, one package reads a cached message and
gets the wrong value because the other package changed the
message in the object header.
2. If one package holds a copy of the symbol table entry and
some other part of HDF5 removes the object and replaces it
with some other object, then the original package will
continue to access the non-existent object using the new
object header.
3. If one package holds a copy of the symbol table entry and
some other part of HDF5 (re)moves the directory which
contains the object, then the package will be unable to
update the symbol table entry with the new cached
data. Packages that refer to the object by the new name will
use old cached data.
The basic problem is that there may be multiple copies of the object
symbol table entry floating around in the code when there should
really be at most one per hard link.
Level 0: A copy may exist on disk as part of a symbol table node, which
is a small 1d array of symbol table entries.
Level 1: A copy may be cached in memory as part of a symbol table node
in the H5Gnode.c file by the H5AC layer.
Level 2a: Another package may be holding a copy so it can perform
fast lookup of any header messages that might be cached in
the symbol table entry. It can't point directly to the
cached symbol table node because that node can dissappear
at any time.
Level 2b: Packages may hold more than one copy of a symbol table
entry. For instance, if H5D_open() is called twice for
the same name, then two copies of the symbol table entry
for the dataset exist in the H5D package.
How can level 2a and 2b be combined?
If package data structures contained pointers to symbol table
entries instead of copies of symbol table entries and if H5G
allocated one symbol table entry per hard link, then it's trivial
for Level 2a and 2b to benefit from one another's actions since
they share the same cache.
How does this work conceptually?
Level 2a and 2b must notify Level 1 of their intent to use (or stop
using) a symbol table entry to access an object header. The
notification of the intent to access an object header is called
`opening' the object and releasing the access is `closing' the
object.
Opening an object requires an object name which is used to locate
the symbol table entry to use for caching of object header
messages. The return value is a handle for the object. Figure 1
shows the state after Dataset1 opens Object with a name that maps
through Entry1. The open request created a copy of Entry1 called
Shadow1 which exists even if SymNode1 is preempted from the H5AC
layer.
______
Object / \
SymNode1 +--------+ |
+--------+ _____\ | Header | |
| | / / +--------+ |
+--------+ +---------+ \______/
| Entry1 | | Shadow1 | /____
+--------+ +---------+ \ \
: : \
+--------+ +----------+
| Dataset1 |
+----------+
FIGURE 1
The SymNode1 can appear and disappear from the H5AC layer at any
time without affecting the Object Header data cached in the Shadow.
The rules are:
* If the SymNode1 is present and is about to disappear and the
Shadow1 dirty bit is set, then Shadow1 is copied over Entry1, the
Entry1 dirty bit is set, and the Shadow1 dirty bit is cleared.
* If something requests a copy of Entry1 (for a read-only peek
request), and Shadow1 exists, then a copy (not pointer) of Shadow1
is returned instead.
* Entry1 cannot be deleted while Shadow1 exists.
* Entry1 cannot change directly if Shadow1 exists since this means
that some other package has opened the object and may be modifying
it. I haven't decided if it's useful to ever change Entry1
directly (except of course within the H5G layer itself).
* Shadow1 is created when Dataset1 `opens' the object through
Entry1. Dataset1 is given a pointer to Shadow1 and Shadow1's
reference count is incremented.
* When Dataset1 `closes' the Object the Shadow1 reference count is
decremented. When the reference count reaches zero, if the
Shadow1 dirty bit is set, then Shadow1's contents are copied to
Entry1, and the Entry1 dirty bit is set. Shadow1 is then deleted
if its reference count is zero. This may require reading SymNode1
back into the H5AC layer.
What happens when another Dataset opens the Object through Entry1?
If the current state is represented by the top part of Figure 2,
then Dataset2 will be given a pointer to Shadow1 and the Shadow1
reference count will be incremented to two. The Object header link
count remains at one so Object Header messages continue to be cached
by Shadow1. Dataset1 and Dataset2 benefit from one another
actions. The resulting state is represented by Figure 2.
_____
SymNode1 Object / \
+--------+ _____\ +--------+ |
| | / / | Header | |
+--------+ +---------+ +--------+ |
| Entry1 | | Shadow1 | /____ \_____/
+--------+ +---------+ \ \
: : _ \
+--------+ |\ +----------+
\ | Dataset1 |
\________ +----------+
\ \
+----------+ |
| Dataset2 | |- New Dataset
+----------+ |
/
FIGURE 2
What happens when the link count for Object increases while Dataset
has the Object open?
SymNode2
+--------+
SymNode1 Object | |
+--------+ ____\ +--------+ /______ +--------+
| | / / | header | \ `| Entry2 |
+--------+ +---------+ +--------+ +--------+
| Entry1 | | Shadow1 | /____ : :
+--------+ +---------+ \ \ +--------+
: : \
+--------+ +----------+ \________________/
| Dataset1 | |
+----------+ New Link
FIGURE 3
The current state is represented by the left part of Figure 3. To
create a new link the Object Header had to be located by traversing
through Entry1/Shadow1. On the way through, the Entry1/Shadow1
cache is invalidated and the Object Header link count is
incremented. Entry2 is then added to SymNode2.
Since the Object Header link count is greater than one, Object
header data will not be cached in Entry1/Shadow1.
If the initial state had been all of Figure 3 and a third link is
being added and Object is open by Entry1 and Entry2, then creation
of the third link will invalidate the cache in Entry1 or Entry2. It
doesn't matter which since both caches are already invalidated
anyway.
What happens if another Dataset opens the same object by another name?
If the current state is represented by Figure 3, then a Shadow2 is
created and associated with Entry2. However, since the Object
Header link count is more than one, nothing gets cached in Shadow2
(or Shadow1).
What happens if the link count decreases?
If the current state is represented by all of Figure 3 then it isn't
possible to delete Entry1 because the object is currently open
through that entry. Therefore, the link count must have
decreased because Entry2 was removed.
As Dataset1 reads/writes messages in the Object header they will
begin to be cached in Shadow1 again because the Object header link
count is one.
What happens if the object is removed while it's open?
That operation is not allowed.
What happens if the directory containing the object is deleted?
That operation is not allowed since deleting the directory requires
that the directory be empty. The directory cannot be emptied
because the open object cannot be removed from the directory.
What happens if the object is moved?
Moving an object is a process consisting of creating a new
hard-link with the new name and then deleting the old name.
This will fail if the object is open.
What happens if the directory containing the entry is moved?
The entry and the shadow still exist and are associated with one
another.
What if a file is flushed or closed when objects are open?
Flushing a symbol table with open objects writes correct information
to the file since Shadow is copied to Entry before the table is
flushed.
Closing a file with open objects will create a valid file but will
return failure.
How is the Shadow associated with the Entry?
A symbol table is composed of one or more symbol nodes. A node is a
small 1-d array of symbol table entries. The entries can move
around within a node and from node-to-node as entries are added or
removed from the symbol table and nodes can move around within a
symbol table, being created and destroyed as necessary.
Since a symbol table has an object header with a unique and constant
file offset, and since H5G contains code to efficiently locate a
symbol table entry given it's name, we use these two values as a key
within a shadow to associate the shadow with the symbol table
entry.
struct H5G_shadow_t {
haddr_t stab_addr; /*symbol table header address*/
char *name; /*entry name wrt symbol table*/
hbool_t dirty; /*out-of-date wrt stab entry?*/
H5G_entry_t ent; /*my copy of stab entry */
H5G_entry_t *main; /*the level 1 entry or null */
H5G_shadow_t *next, *prev; /*other shadows for this stab*/
};
The set of shadows will be organized in a hash table of linked
lists. Each linked list will contain the shadows associated with a
particular symbol table header address and the list will be sorted
lexicographically.
Also, each Entry will have a pointer to the corresponding Shadow or
null if there is no shadow.
When a symbol table node is loaded into the main cache, we look up
the linked list of shadows in the shadow hash table based on the
address of the symbol table object header. We then traverse that
list matching shadows with symbol table entries.
We assume that opening/closing objects will be a relatively
infrequent event compared with loading/flushing symbol table
nodes. Therefore, if we keep the linked list of shadows sorted it
costs O(N) to open and close objects where N is the number of open
objects in that symbol table (instead of O(1)) but it costs only
O(N) to load a symbol table node (instead of O(N^2)).
What about the root symbol entry?
Level 1 storage for the root symbol entry is always available since
it's stored in the hdf5_file_t struct instead of a symbol table
node. However, the contents of that entry can move from the file
handle to a symbol table node by H5G_mkroot(). Therefore, if the
root object is opened, we keep a shadow entry for it whose
`stab_addr' field is zero and whose `name' is null.
For this reason, the root object should always be read through the
H5G interface.
One more key invariant: The H5O_STAB message in a symbol table header
never changes. This allows symbol table entries to cache the H5O_STAB
message for the symbol table to which it points without worrying about
whether the cache will ever be invalidated.

192
doc/html/tracing.html Normal file

@ -0,0 +1,192 @@
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>API Tracing</title>
</head>
<body>
<h1>API Tracing</h1>
<h2>Introduction</h2>
<p>The HDF5 library is now able to trace API calls by printing the
function name, the argument names and their values, and the
return value. Some people like to see lots of output during
program execution instead of using a good symbolic debugger, and
this feature is intended for their consumption. For example,
the output from <code>h5ls foo</code> after turning on tracing,
includes:
<p>
<center>
<table border align=center width="100%">
<caption align=top><b>Sample Output</b></caption>
<tr>
<td>
<code><pre>
H5Tcopy(type=184549388) = 184549419 (type);
H5Tcopy(type=184549392) = 184549424 (type);
H5Tlock(type=184549424) = SUCCEED;
H5Tcopy(type=184549393) = 184549425 (type);
H5Tlock(type=184549425) = SUCCEED;
H5Fopen(filename="foo", flags=0, access=H5P_DEFAULT) = FAIL;
HDF5-DIAG: Error detected in thread 0. Back trace follows.
#000: H5F.c line 1245 in H5Fopen(): unable to open file
major(04): File interface
minor(10): Unable to open file
#001: H5F.c line 846 in H5F_open(): file does not exist
major(04): File interface
minor(10): Unable to open file
</pre></code>
</td>
</tr>
</table>
</center>
<h2>Configuation</h2>
<p>This all happens with some magic in the configuration script,
the makefiles, and macros. First, from the end-user point of
view, the library must be configured with the
<code>--enable-tracing</code> switch. This causes the library to
include the support necessary for API tracing.
<p>
<center>
<table border align=center width="100%">
<caption align=top><b>Configuration</b></caption>
<tr>
<td>
<code><pre>
$ make distclean
$ sh configure --enable-tracing
$ make
</pre></code>
</td>
</tr>
</table>
</center>
<h2>Execution</h2>
<p>In order to actually get tracing output one must turn tracing
on and specify a file descriptor where the tracing output should
be written. This is done by assigning a file descriptor number
to the <code>HDF5_TRACE</code> environment variable.
<p>
<center>
<table border align=center width="100%">
<caption align=top><b>Execution Examples</b></caption>
<tr>
<td>To display the trace on the standard error stream:
<code><pre>
$ export HDF5_TRACE=2
$ a.out
</pre></code>
</td>
</tr>
<tr>
<td>To send the trace to a file:
<code><pre>
$ export HDF5_TRACE=255
$ a.out 255>trace-output
</pre></code>
</td>
</tr>
</table>
</center>
<h2>Performance</h2>
<p>If the library was not configured for tracing then there is no
unnecessary overhead since all tracing code is
excluded.
<p>However, if tracing is enabled but not used there is a
small penalty. First, code size is larger because of extra
statically-declared character strings used to store argument
types and names and extra auto variable pointer in each
function. Also, execution is slower because each function sets
and tests a local variable and each API function calls the
<code>H5_trace()</code> function.
<p>If tracing is enabled and turned on then the penalties from the
previous paragraph apply plus the time required to format each
line of tracing information. There is also an extra call to
H5_trace() for each API function to print the return value.
<h2>Safety</h2>
<p>The tracing mechanism is invoked for each API function before
arguments are checked for validity. If bad arguments are passed
to an API function it could result in a segmentation fault.
However, the tracing output is line-buffered so all previous
output will appear.
<h2>Completeness</h2>
<p>There are two API functions that don't participate in
tracing. They are <code>H5Eprint()</code> and
<code>H5Eprint_cb()</code> because their participation would
mess up output during automatic error reporting.
<p>On the other hand, a number of API functions are called during
library initialization and they print tracing information.
<h2>Implementation</h2>
<p>For those interested in the implementation here is a
description. Each API function should have a call to one of the
<code>H5TRACE()</code> macros immediately after the
<code>FUNC_ENTER()</code> macro. The first argument is the
return type encoded as a string. The second argument is the
types of all the function arguments encoded as a string. The
remaining arguments are the function arguments. This macro was
designed to be as terse and unobtrousive as possible.
<p>In order to keep the <code>H5TRACE()</code> calls synchronized
with the source code we've written a perl script which gets
called automatically just before Makefile dependencies are
calculated for the file. However, this only works when one is
using GNU make. To reinstrument the tracing explicitly, invoke
the <code>trace</code> program from the hdf5 bin directory with
the names of the source files that need to be updated. If any
file needs to be modified then a backup is created by appending
a tilde to the file name.
<p>
<center>
<table border align=center width="100%">
<caption align=top><b>Explicit Instrumentation</b></caption>
<tr>
<td>
<code><pre>
$ ../bin/trace *.c
H5E.c: in function `H5Ewalk_cb':
H5E.c:336: warning: trace info was not inserted
</pre></code>
</td>
</tr>
</table>
</center>
<p>Note: The warning message is the result of a comment of the
form <code>/*NO TRACE*/</code> somewhere in the function
body. Tracing information will not be updated or inserted if
such a comment exists.
<p>Error messages have the same format as a compiler so that they
can be parsed from program development environments like
Emacs. Any function which generates an error will not be
modified.
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Wed Jun 17 12:29:12 EDT 1998 -->
<!-- hhmts start -->
Last modified: Wed Jun 17 15:45:50 EDT 1998
<!-- hhmts end -->
</body>
</html>

BIN
doc/html/version.gif Normal file

Binary file not shown.
Side by Side

After

(image error) Size: 4.7 KiB

96
doc/html/version.obj Normal file

@ -0,0 +1,96 @@
%TGIF 3.0-p5
state(0,33,100,0,0,0,8,1,9,1,1,0,2,1,0,1,0,'Courier',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880).
%
% @(#)$Header$
% %W%
%
unit("1 pixel/pixel").
page(1,"",1).
poly('black',2,[
128,128,128,448],0,3,1,0,0,0,0,0,12,5,0,0,0,'3','12','5',
"0",[
]).
poly('black',2,[
128,128,128,64],0,3,1,1,0,0,2,0,12,5,0,0,0,'3','12','5',
"0",[
]).
poly('black',2,[
128,448,128,512],0,3,1,4,0,0,2,0,12,5,0,0,0,'3','12','5',
"0",[
]).
text('black',144,112,'Courier',0,17,1,0,0,1,42,14,22,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.3.30"]).
text('black',144,144,'Courier',0,17,1,0,0,1,42,14,30,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.3.31"]).
text('black',144,176,'Courier',0,17,1,0,0,1,42,14,32,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.3.32"]).
poly('black',2,[
256,208,256,448],0,3,1,34,0,0,0,0,12,5,0,0,0,'3','12','5',
"0",[
]).
poly('black',2,[
256,448,256,512],0,3,1,36,0,0,2,0,12,5,0,0,0,'3','12','5',
"0",[
]).
poly('black',2,[
128,192,256,208],1,1,1,37,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',144,224,'Courier',0,17,1,0,0,1,42,14,41,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.3.33"]).
text('black',144,256,'Courier',0,17,1,0,0,1,42,14,43,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.3.34"]).
text('black',272,224,'Courier',0,17,1,0,0,1,35,14,45,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.5.0"]).
text('black',272,256,'Courier',0,17,1,0,0,1,35,14,47,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.5.1"]).
text('black',272,288,'Courier',0,17,1,0,0,1,35,14,49,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.5.2"]).
text('black',272,320,'Courier',0,17,1,0,0,1,35,14,51,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.5.3"]).
text('black',144,288,'Courier',0,17,1,0,0,1,42,14,53,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.3.35"]).
text('black',144,320,'Courier',0,17,1,0,0,1,35,14,57,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.4.0"]).
text('black',144,368,'Courier',0,17,1,0,0,1,35,14,59,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.4.1"]).
text('black',272,192,'Helvetica',0,17,1,0,0,1,144,15,67,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"New development branch"]).
text('black',144,64,'Helvetica',0,17,1,0,0,1,163,15,69,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Original development branch"]).
text('black',16,208,'Helvetica',0,17,2,0,0,1,87,30,71,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Feature Freeze",
"at this point."]).
text('black',16,320,'Helvetica',0,17,2,0,0,1,84,30,73,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Public Release",
"at this point."]).
poly('black',2,[
104,208,128,208],1,1,1,77,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
104,320,128,320],1,1,1,78,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
poly('black',2,[
256,336,128,352],1,1,1,79,0,0,0,0,8,3,0,0,0,'1','8','3',
"0",[
]).
text('black',320,368,'Helvetica',0,17,3,0,0,1,137,45,82,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"Merge a bug fix from the",
"development branch to",
"the release branch."]).
box('black',312,368,464,416,0,1,1,87,0,0,0,0,0,'1',[
]).
poly('black',4,[
312,392,240,384,296,344,232,344],1,1,1,90,1,0,0,0,8,3,0,0,0,'1','8','3',
"6",[
]).
box('black',8,208,104,240,0,1,1,95,0,0,0,0,0,'1',[
]).
box('black',8,320,104,352,0,1,1,98,0,0,0,0,0,'1',[
]).
text('black',144,408,'Courier',0,17,1,0,0,1,35,14,102,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[
"1.4.2"]).
box('black',0,40,480,528,0,1,1,104,0,0,0,0,0,'1',[
]).