hdf5/doc/html/Caching.html

217 lines
8.1 KiB
HTML
Raw Normal View History

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Data Caching</title>
</head>
[svn-r985] Checking in changes from "HDF5 Release 1.0" CVS branch that have not yet been checked into "HDF5 Development" branch. General ======= Various "copy edit" types of repairs. Revisions to document cross-linking. Added structural links tying all docs together (banners at top and bottom of documents, similar to Reference Manual banners of the Beta release). Set background color to white in all documents. NCSAfooterlogo.gif hdf2.jpg New image files to make the documents more self-contained (i.e., to prevent loading images from NCSA and HDF home servers). index.html Redesigned to isolate links external to the installation in a single location. Intro to HDF5 ============= H5.intro.html Fixed banner linking Intro to other docs. Set all URLs to be relative within the distribution; nothing points back to the HDF server. Updates to "Limits of the Current Release" and "Changes in the Current Release." HDF5 User's Guide ================= Changed several User Guide section titles such that all sections that are primarily about a particular interface are now titled in the format "The xxxxx Interface (H5x)". H5.user.html Commented out links to developer docs since they are marked in MANIFEST as not being for distribution in the release. Removed 2nd and 3rd indices from page. Changed "freeform" lists of sections (TOCs) to aligned tables. Datatypes.html Removed the sentence "I'm deferring definition until later since they're probably not as important as the other data types." from Section 3.3, "Properties of Date and Time Atomic Types." Added info regarding 'char' versus 'string' datatypes. Added as Section 3.7, "Character and String Datatype Issues." References.html Commented out substantial material (at end of document) from References planning document that is not appropriate for the User Guide but that is worth keeping around. Groups.html Final edits from elimination of "current working group." HDF5 Reference Manual ===================== Removed "Draft" from the <title>__</title> lines. RM_H5Front.html Removed 2nd and 3rd indices from page. Changed "freeform" lists of sections (TOCs) to aligned tables. RM_H5F.html RM_H5P.html Add file mounting information.
1998-12-22 06:52:56 +08:00
<body bgcolor="#FFFFFF">
<hr>
<center>
<table border=0 width=98%>
<tr><td valign=top align=left>
<a href="H5.intro.html">Introduction to HDF5</a>&nbsp;<br>
<a href="RM_H5Front.html">HDF5 Reference Manual</a>&nbsp;<br>
<a href="index.html">Other HDF5 documents and links</a>&nbsp;<br>
<!--
<a href="Glossary.html">Glossary</a><br>
-->
</td>
<td valign=top align=right>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:&nbsp;&nbsp;&nbsp;&nbsp;
<a href="Files.html">Files</a>&nbsp;&nbsp;
<br>
<a href="Datasets.html">Datasets</a>&nbsp;&nbsp;
<a href="Datatypes.html">Data Types</a>&nbsp;&nbsp;
<a href="Dataspaces.html">Dataspaces</a>&nbsp;&nbsp;
<a href="Groups.html">Groups</a>&nbsp;&nbsp;
<a href="References.html">References</a>&nbsp;&nbsp;
<br>
<a href="Attributes.html">Attributes</a>&nbsp;&nbsp;
<a href="Properties.html">Property Lists</a>&nbsp;&nbsp;
<a href="Errors.html">Error Handling</a>&nbsp;&nbsp;
<a href="Filters.html">Filters</a>&nbsp;&nbsp;
Caching&nbsp;&nbsp;
<br>
<a href="Chunking.html">Chunking</a>&nbsp;&nbsp;
<a href="Debugging.html">Debugging</a>&nbsp;&nbsp;
<a href="Environment.html">Environment</a>&nbsp;&nbsp;
<a href="ddl.html">DDL</a>&nbsp;&nbsp;
<a href="Ragged.html">Ragged Arrays</a>&nbsp;&nbsp;
<!--
<hr>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:&nbsp;&nbsp;&nbsp;&nbsp;
<a href="Attributes.html">H5A</a>&nbsp;&nbsp;
<a href="Datasets.html">H5D</a>&nbsp;&nbsp;
<a href="Errors.html">H5E</a>&nbsp;&nbsp;
<a href="Files.html">H5F</a>&nbsp;&nbsp;
<a href="Groups.html">H5G</a>&nbsp;&nbsp;
<a href="Properties.html">H5P</a>&nbsp;&nbsp;
<a href="References.html">H5R & H5I</a>&nbsp;&nbsp;
<a href="Ragged.html">H5RA</a>&nbsp;&nbsp;
<a href="Dataspaces.html">H5S</a>&nbsp;&nbsp;
<a href="Datatypes.html">H5T</a>&nbsp;&nbsp;
<a href="Filters.html">H5Z</a>&nbsp;&nbsp;
<a href="Caching.html">Caching</a>&nbsp;&nbsp;
<a href="Chunking.html">Chunking</a>&nbsp;&nbsp;
<a href="Debugging.html">Debugging</a>&nbsp;&nbsp;
<a href="Environment.html">Environment</a>&nbsp;&nbsp;
<a href="ddl.html">DDL</a>&nbsp;&nbsp;
-->
</td></tr>
</table>
</center>
<hr>
<h1>Meta Data Caching</h1>
<p>The HDF5 library caches two types of data: meta data and raw
data. The meta data cache holds file objects like the file
header, symbol table nodes, global heap collections, object
headers and their messages, etc. in a partially decoded
state. The cache has a fixed number of entries which is set with
the file access property list (defaults to 10k) and each entry
can hold a single meta data object. Collisions between objects
are handled by preempting the older object in favor of the new
one.
<h1>Raw Data Chunk Caching</h1>
<p>Raw data chunks are cached because I/O requests at the
application level typically don't map well to chunks at the
storage level. The chunk cache has a maximum size in bytes
set with the file access property list (defaults to 1MB) and
when the limit is reached chunks are preempted based on the
following set of heuristics.
<ul>
<li>Chunks which have not been accessed for a long time
relative to other chunks are penalized.
<li>Chunks which have been accessed frequently in the recent
past are favored.
<li>Chunks which are completely read and not written, completely
written but not read, or completely read and completely
written are penalized according to <em>w0</em>, an
application-defined weight between 0 and 1 inclusive. A weight
of zero does not penalize such chunks while a weight of 1
penalizes those chunks more than all other chunks. The
default is 0.75.
<li>Chunks which are larger than the maximum cache size do not
participate in the cache.
</ul>
<p>One should choose large values for <em>w0</em> if I/O requests
typically do not overlap but smaller values for <em>w0</em> if
the requests do overlap. For instance, reading an entire 2d
array by reading from non-overlapping "windows" in a row-major
order would benefit from a high <em>w0</em> value while reading
a diagonal accross the dataset where each request overlaps the
previous request would benefit from a small <em>w0</em>.
<h1>The API</h1>
<p>The cache parameters for both caches are part of a file access
property list and are set and queried with this pair of
functions:
<dl>
<dt><code>herr_t H5Pset_cache(hid_t <em>plist</em>, unsigned int
<em>mdc_nelmts</em>, size_t <em>rdcc_nbytes</em>, double
<em>w0</em>)</code>
<dt><code>herr_t H5Pget_cache(hid_t <em>plist</em>, unsigned int
*<em>mdc_nelmts</em>, size_t *<em>rdcc_nbytes</em>, double
<em>w0</em>)</code>
<dd>Sets or queries the meta data cache and raw data chunk cache
parameters. The <em>plist</em> is a file access property
list. The number of elements (objects) in the meta data cache
is <em>mdc_nelmts</em>. The total size of the raw data chunk
cache and the preemption policy is <em>rdcc_nbytes</em> and
<em>w0</em>. For <code>H5Pget_cache()</code> any (or all) of
the pointer arguments may be null pointers.
</dl>
[svn-r985] Checking in changes from "HDF5 Release 1.0" CVS branch that have not yet been checked into "HDF5 Development" branch. General ======= Various "copy edit" types of repairs. Revisions to document cross-linking. Added structural links tying all docs together (banners at top and bottom of documents, similar to Reference Manual banners of the Beta release). Set background color to white in all documents. NCSAfooterlogo.gif hdf2.jpg New image files to make the documents more self-contained (i.e., to prevent loading images from NCSA and HDF home servers). index.html Redesigned to isolate links external to the installation in a single location. Intro to HDF5 ============= H5.intro.html Fixed banner linking Intro to other docs. Set all URLs to be relative within the distribution; nothing points back to the HDF server. Updates to "Limits of the Current Release" and "Changes in the Current Release." HDF5 User's Guide ================= Changed several User Guide section titles such that all sections that are primarily about a particular interface are now titled in the format "The xxxxx Interface (H5x)". H5.user.html Commented out links to developer docs since they are marked in MANIFEST as not being for distribution in the release. Removed 2nd and 3rd indices from page. Changed "freeform" lists of sections (TOCs) to aligned tables. Datatypes.html Removed the sentence "I'm deferring definition until later since they're probably not as important as the other data types." from Section 3.3, "Properties of Date and Time Atomic Types." Added info regarding 'char' versus 'string' datatypes. Added as Section 3.7, "Character and String Datatype Issues." References.html Commented out substantial material (at end of document) from References planning document that is not appropriate for the User Guide but that is worth keeping around. Groups.html Final edits from elimination of "current working group." HDF5 Reference Manual ===================== Removed "Draft" from the <title>__</title> lines. RM_H5Front.html Removed 2nd and 3rd indices from page. Changed "freeform" lists of sections (TOCs) to aligned tables. RM_H5F.html RM_H5P.html Add file mounting information.
1998-12-22 06:52:56 +08:00
<hr>
<center>
<table border=0 width=98%>
<tr><td valign=top align=left>
<a href="H5.intro.html">Introduction to HDF5</a>&nbsp;<br>
<a href="RM_H5Front.html">HDF5 Reference Manual</a>&nbsp;<br>
<a href="index.html">Other HDF5 documents and links</a>&nbsp;<br>
<!--
<a href="Glossary.html">Glossary</a><br>
-->
</td>
<td valign=top align=right>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:&nbsp;&nbsp;&nbsp;&nbsp;
<a href="Files.html">Files</a>&nbsp;&nbsp;
<br>
<a href="Datasets.html">Datasets</a>&nbsp;&nbsp;
<a href="Datatypes.html">Data Types</a>&nbsp;&nbsp;
<a href="Dataspaces.html">Dataspaces</a>&nbsp;&nbsp;
<a href="Groups.html">Groups</a>&nbsp;&nbsp;
<a href="References.html">References</a>&nbsp;&nbsp;
<br>
<a href="Attributes.html">Attributes</a>&nbsp;&nbsp;
<a href="Properties.html">Property Lists</a>&nbsp;&nbsp;
<a href="Errors.html">Error Handling</a>&nbsp;&nbsp;
<a href="Filters.html">Filters</a>&nbsp;&nbsp;
Caching&nbsp;&nbsp;
<br>
<a href="Chunking.html">Chunking</a>&nbsp;&nbsp;
<a href="Debugging.html">Debugging</a>&nbsp;&nbsp;
<a href="Environment.html">Environment</a>&nbsp;&nbsp;
<a href="ddl.html">DDL</a>&nbsp;&nbsp;
<a href="Ragged.html">Ragged Arrays</a>&nbsp;&nbsp;
<!--
<hr>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:&nbsp;&nbsp;&nbsp;&nbsp;
<a href="Attributes.html">H5A</a>&nbsp;&nbsp;
<a href="Datasets.html">H5D</a>&nbsp;&nbsp;
<a href="Errors.html">H5E</a>&nbsp;&nbsp;
<a href="Files.html">H5F</a>&nbsp;&nbsp;
<a href="Groups.html">H5G</a>&nbsp;&nbsp;
<a href="Properties.html">H5P</a>&nbsp;&nbsp;
<a href="References.html">H5R & H5I</a>&nbsp;&nbsp;
<a href="Ragged.html">H5RA</a>&nbsp;&nbsp;
<a href="Dataspaces.html">H5S</a>&nbsp;&nbsp;
<a href="Datatypes.html">H5T</a>&nbsp;&nbsp;
<a href="Filters.html">H5Z</a>&nbsp;&nbsp;
<a href="Caching.html">Caching</a>&nbsp;&nbsp;
<a href="Chunking.html">Chunking</a>&nbsp;&nbsp;
<a href="Debugging.html">Debugging</a>&nbsp;&nbsp;
<a href="Environment.html">Environment</a>&nbsp;&nbsp;
<a href="ddl.html">DDL</a>&nbsp;&nbsp;
-->
</td></tr>
</table>
</center>
<!--
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
-->
<!-- Created: Tue May 26 15:20:14 EDT 1998 -->
<!-- hhmts start -->
<!--
Last modified: Tue May 26 15:38:27 EDT 1998
-->
<!-- hhmts end -->
<hr>
<address>
<a href="mailto:hdfhelp@ncsa.uiuc.edu">HDF Help Desk</a>
</address>
[svn-r985] Checking in changes from "HDF5 Release 1.0" CVS branch that have not yet been checked into "HDF5 Development" branch. General ======= Various "copy edit" types of repairs. Revisions to document cross-linking. Added structural links tying all docs together (banners at top and bottom of documents, similar to Reference Manual banners of the Beta release). Set background color to white in all documents. NCSAfooterlogo.gif hdf2.jpg New image files to make the documents more self-contained (i.e., to prevent loading images from NCSA and HDF home servers). index.html Redesigned to isolate links external to the installation in a single location. Intro to HDF5 ============= H5.intro.html Fixed banner linking Intro to other docs. Set all URLs to be relative within the distribution; nothing points back to the HDF server. Updates to "Limits of the Current Release" and "Changes in the Current Release." HDF5 User's Guide ================= Changed several User Guide section titles such that all sections that are primarily about a particular interface are now titled in the format "The xxxxx Interface (H5x)". H5.user.html Commented out links to developer docs since they are marked in MANIFEST as not being for distribution in the release. Removed 2nd and 3rd indices from page. Changed "freeform" lists of sections (TOCs) to aligned tables. Datatypes.html Removed the sentence "I'm deferring definition until later since they're probably not as important as the other data types." from Section 3.3, "Properties of Date and Time Atomic Types." Added info regarding 'char' versus 'string' datatypes. Added as Section 3.7, "Character and String Datatype Issues." References.html Commented out substantial material (at end of document) from References planning document that is not appropriate for the User Guide but that is worth keeping around. Groups.html Final edits from elimination of "current working group." HDF5 Reference Manual ===================== Removed "Draft" from the <title>__</title> lines. RM_H5Front.html Removed 2nd and 3rd indices from page. Changed "freeform" lists of sections (TOCs) to aligned tables. RM_H5F.html RM_H5P.html Add file mounting information.
1998-12-22 06:52:56 +08:00
Last modified: 30 October 1998
</body>
</html>