mirror/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2024-11-27 02:10:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
5de00d5fb8
hdf5/doc/html/Performance.html

244 lines
10 KiB
HTML
Raw Normal View History

[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Performance</title>
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<!-- #BeginLibraryItem "/ed_libs/styles_UG.lbi" --><link href="ed_styles/UGelect.css" rel="stylesheet" type="text/css">
<!-- #EndLibraryItem --></head>
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<body bgcolor="#FFFFFF">
<!-- #BeginLibraryItem "/ed_libs/NavBar_UG.lbi" --><hr>
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
<center>
<table border=0 width=98%>
<tr><td valign=top align=left>
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<a href="index.html">HDF5 documents and links</a>&nbsp;<br>
<a href="H5.intro.html">Introduction to HDF5</a>&nbsp;<br>
<a href="RM_H5Front.html">HDF5 Reference Manual</a>&nbsp;<br>
<!--
<a href="Glossary.html">Glossary</a><br>
-->
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
</td>
<td valign=top align=right>
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
And in this document, the
<a href="H5.user.html"><strong>HDF5 User's Guide:</strong></a>&nbsp;&nbsp;&nbsp;&nbsp;
<br>
<a href="Files.html">Files</a>&nbsp;&nbsp;
<a href="Datasets.html">Datasets</a>&nbsp;&nbsp;
<a href="Datatypes.html">Datatypes</a>&nbsp;&nbsp;
<a href="Dataspaces.html">Dataspaces</a>&nbsp;&nbsp;
<a href="Groups.html">Groups</a>&nbsp;&nbsp;
<br>
<a href="References.html">References</a>&nbsp;&nbsp;
<a href="Attributes.html">Attributes</a>&nbsp;&nbsp;
<a href="Properties.html">Property Lists</a>&nbsp;&nbsp;
<a href="Errors.html">Error Handling</a>&nbsp;&nbsp;
<br>
<a href="Filters.html">Filters</a>&nbsp;&nbsp;
<a href="Caching.html">Caching</a>&nbsp;&nbsp;
<a href="Chunking.html">Chunking</a>&nbsp;&nbsp;
<a href="MountingFiles.html">Mounting Files</a>&nbsp;&nbsp;
<br>
<a href="Performance.html">Performance</a>&nbsp;&nbsp;
<a href="Debugging.html">Debugging</a>&nbsp;&nbsp;
<a href="Environment.html">Environment</a>&nbsp;&nbsp;
<a href="ddl.html">DDL</a>&nbsp;&nbsp;
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
</td></tr>
</table>
</center>
<hr>
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<!-- #EndLibraryItem --><h1>Performance Analysis and Issues</h1>
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
<h2>1. Introduction</h2>
<p>This section includes brief discussions of performance issues
in HDF5 and performance analysis tools for HDF5 or pointers to
such discussions.
<h2>2. Dataset Chunking</h2>
[svn-r4193] Purpose: New section -- "Freespace Management" Description: Added "Freespace Management" section. Minor formatting. Platforms tested: IE 5
2001-07-12 06:01:45 +08:00
Appropriate dataset chunking can make a siginificant difference
in HDF5 performance. This topic is discussed in
<a href="Chunking.html">Dataset Chunking Issues</a> elsewhere
in this <cite>User's Guide</cite>.
[svn-r4303] Purpose: Bringing 'last minute' release branch edits into development branch. Description: Chunking.html, Filters.html, and Performance.html Added links, with explanatory note, from the first 2 files above to the new freespace management section in Performance.html. Platforms tested: IE 5
2001-08-03 00:10:11 +08:00
<a name="Freespace">
[svn-r4193] Purpose: New section -- "Freespace Management" Description: Added "Freespace Management" section. Minor formatting. Platforms tested: IE 5
2001-07-12 06:01:45 +08:00
<h2>3. Freespace Management</h2>
[svn-r4303] Purpose: Bringing 'last minute' release branch edits into development branch. Description: Chunking.html, Filters.html, and Performance.html Added links, with explanatory note, from the first 2 files above to the new freespace management section in Performance.html. Platforms tested: IE 5
2001-08-03 00:10:11 +08:00
</a>
[svn-r4193] Purpose: New section -- "Freespace Management" Description: Added "Freespace Management" section. Minor formatting. Platforms tested: IE 5
2001-07-12 06:01:45 +08:00
<p>HDF5 does not yet manage freespace as effectively as it might.
While a file is opened, the library actively tracks and re-uses
<em>freespace</em>, i.e., space that is freed (or released)
during the run.
But the library does not yet manage freespace across the
closing and reopening of a file; when a file is closed,
all knowledge of available freespace is lost.
What was freespace becomes an unusable <em>hole</em> in the file.
<p>There are several circumstances that can result in freespace
in an HDF5 file:
<ul>
<li>Reading then rewriting a dataset or compressed dataset
chunk.<sup><a href="#footcchunk">1</a></sup>
<ul>
<li>If the rewritten dataset or compressed chunk is the same
size as or smaller than the original, it will be written
to the same file location.
<li>If, however, the dataset or compressed chunk is larger
than the original, it will be written contiguously elsewhere
in the file, leaving freespace at the original location.
<li>If the rewritten dataset or compressed chunk is
substantially smaller than the original, the remaining
space will be released and identified as freespace.
</ul>
<li>Deleting (or unlinking) a dataset or group.
<ul>
<li>If an object, such as a dataset, group, or named datatype,
is deleted (normally with <code>H5Gunlink</code>),
the space previously occupied by the object is released
and identified as freespace.
</ul>
</ul>
<p>As stated above, freespace is not managed across the
closing and reopening of an HDF5 file; file space that was
known freespace while the file remained open becomes an
inaccessible hole when the file is closed.
Thus, if a file is often closed and reopened, datasets
frequently rewritten, or groups and/or datasets frequently
added and deleted, that file can develop large numbers of
holes and grow unnecessarily large. This can, in turn,
seriously impair application or library performance
as the file ages.
<p>An <code>h5pack</code> utility would enable <em>packing</em>
a file to remove the holes, but writing such a utility to
universally pack the file correctly is a complex task and the
HDF5 development team has not to date had the resources to
complete the task.
<p>For application developers or researchers who find themselves
working with files that become bloated in this manner, there
are, at this time, two remedies:
<ul>
<li><code>H5view</code>, an HDF5 Java tool, allows the user
to open a file and, using the <code>Save As...</code> feature,
save the file under a new filename. The new file can then
be closed and will be a packed version of the original file.
This approach is reasonably reliable, but with two caveats:
<ul>
<li>It is not automated.
<li>This ability is a side-effect of the tool's design;
it was not designed for this purpose and this approach
to file packing has not been exhaustively tested.
</ul>
<li>An application developer or researcher can write a utility
that is tuned to their data and file structures. This
untility can then read in a file, copy the structures and
datasets to a new file, and write the new file to storage.
This will eliminate the holes, making the new file a
fully-packed version of the original file.
</ul>
<a name="footcchunk">
<p></a>
<sup>1</sup>
<font size=-1>
This is a problem only with compressed chunks.
The compression ratio of data is highly dependent on the data
itself; regardless of whether the <em>size</em> of the data
changes, the size of the compressed data change substantially
as the data changes. Uncompressed chunks do not vary in size,
so this issue does not arise.
</font>
<h2>4. Use of the Pablo Instrumentation of HDF5</h2>
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
Pablo HDF5 Trace software provides a means of measuring the
performance of programs using HDF5.
<p>The Pablo software consists
of an instrumented copy of the HDF5 library, the Pablo Trace and
Trace Extensions libraries, and some utilities for processing the
output. The instrumented version of the HDF5 library has hooks
inserted into the HDF5 code which call routines in the Pablo Trace
library just after entry to each instrumented HDF5 routine and
just prior to exit from the routine. The Pablo Trace Extension
library has programs that track the I/O activity between the
entry and exit of the HDF5 routine during execution.
<p>A few lines of code must be inserted in the user's main program
to enable tracing and to specify which HDF5 procedures are to be
traced. The program is linked with the special HDF5 and Pablo
libraries to produce an executable. Running this executable on
a single processor produces an output file called the trace file
which contains records, called Pablo Self-Defining Data Format
(SDDF) records, which can later be analyzed using the
HDF5 Analysis Utilities. The HDF5 Analysis Utilites can be used
to interpret the SDDF records in the trace files to produce a
report describing the HDF5 IO activity that occurred during
execution.
<p>For further instructions, see the file <code>READ_ME</code>
in the <code> $(toplevel)/hdf5/pablo/ </code> subdirectory of
the HDF5 source code distribution.
<p>For further information about Pablo and the
Self-Defining Data Format, visit the Pablo website at
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<code><a href="http://www-pablo.cs.uiuc.edu/">http://www-pablo.cs.uiuc.edu/</a></code>.</p>
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<!-- #BeginLibraryItem "/ed_libs/NavBar_UG.lbi" --><hr>
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
<center>
<table border=0 width=98%>
<tr><td valign=top align=left>
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<a href="index.html">HDF5 documents and links</a>&nbsp;<br>
<a href="H5.intro.html">Introduction to HDF5</a>&nbsp;<br>
<a href="RM_H5Front.html">HDF5 Reference Manual</a>&nbsp;<br>
<!--
<a href="Glossary.html">Glossary</a><br>
-->
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
</td>
<td valign=top align=right>
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
And in this document, the
<a href="H5.user.html"><strong>HDF5 User's Guide:</strong></a>&nbsp;&nbsp;&nbsp;&nbsp;
<br>
<a href="Files.html">Files</a>&nbsp;&nbsp;
<a href="Datasets.html">Datasets</a>&nbsp;&nbsp;
<a href="Datatypes.html">Datatypes</a>&nbsp;&nbsp;
<a href="Dataspaces.html">Dataspaces</a>&nbsp;&nbsp;
<a href="Groups.html">Groups</a>&nbsp;&nbsp;
<br>
<a href="References.html">References</a>&nbsp;&nbsp;
<a href="Attributes.html">Attributes</a>&nbsp;&nbsp;
<a href="Properties.html">Property Lists</a>&nbsp;&nbsp;
<a href="Errors.html">Error Handling</a>&nbsp;&nbsp;
<br>
<a href="Filters.html">Filters</a>&nbsp;&nbsp;
<a href="Caching.html">Caching</a>&nbsp;&nbsp;
<a href="Chunking.html">Chunking</a>&nbsp;&nbsp;
<a href="MountingFiles.html">Mounting Files</a>&nbsp;&nbsp;
<br>
<a href="Performance.html">Performance</a>&nbsp;&nbsp;
<a href="Debugging.html">Debugging</a>&nbsp;&nbsp;
<a href="Environment.html">Environment</a>&nbsp;&nbsp;
<a href="ddl.html">DDL</a>&nbsp;&nbsp;
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
</td></tr>
</table>
</center>
<hr>
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
<!-- #EndLibraryItem --><!-- #BeginLibraryItem "/ed_libs/Footer.lbi" --><address>
<a href="mailto:hdfhelp@ncsa.uiuc.edu">HDF Help Desk</a>
<br>
Describes HDF5 Release 1.5, Unreleased Development Branch
</address><!-- #EndLibraryItem -->
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
<!-- Created: Thu Oct 14 16:46:00 CDT 1999 -->
<!-- hhmts start -->
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
Last modified: 2 August 2001
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
<!-- hhmts end -->
[svn-r6478] Purpose: DreamWeaver (DW) adaptation of the HDF5 doc set, introducing the use of stylesheets and DW libraries. Stylesheets -- for consistent format management. DW libraries -- for managing common elements across documents. Platforms tested: IE 5 and 6 Netscape
2003-03-15 03:12:34 +08:00
</body>
[svn-r3188] Purpose: Checking in MountingFiles.html and Performance.html. These documents were originally created in 1.2 branch and never checked in to the development branch. Platforms tested: IE 5
2000-12-22 21:32:40 +08:00
</html>
Reference in New Issue Copy Permalink