1998-07-08 22:54:54 +08:00
|
|
|
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
|
|
|
|
|
<html>
|
|
|
|
|
<head>
|
|
|
|
|
<title>Data Caching</title>
|
2003-03-15 03:12:34 +08:00
|
|
|
|
2003-07-03 05:28:11 +08:00
|
|
|
<!-- #BeginLibraryItem "/ed_libs/styles_UG.lbi" -->
|
|
|
|
|
<!--
|
|
|
|
|
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
|
|
|
|
|
* Copyright by the Board of Trustees of the University of Illinois. *
|
|
|
|
|
* All rights reserved. *
|
|
|
|
|
* *
|
|
|
|
|
* This file is part of HDF5. The full HDF5 copyright notice, including *
|
|
|
|
|
* terms governing use, modification, and redistribution, is contained in *
|
|
|
|
|
* the files COPYING and Copyright.html. COPYING can be found at the root *
|
|
|
|
|
* of the source code distribution tree; Copyright.html can be found at the *
|
|
|
|
|
* root level of an installed copy of the electronic HDF5 document set and *
|
|
|
|
|
* is linked from the top-level documents page. It can also be found at *
|
|
|
|
|
* http://hdf.ncsa.uiuc.edu/HDF5/doc/Copyright.html. If you do not have *
|
|
|
|
|
* access to either file, you may request a copy from hdfhelp@ncsa.uiuc.edu. *
|
|
|
|
|
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
|
|
|
|
|
-->
|
|
|
|
|
|
|
|
|
|
<link href="ed_styles/UGelect.css" rel="stylesheet" type="text/css">
|
2003-03-15 03:12:34 +08:00
|
|
|
<!-- #EndLibraryItem --></head>
|
1998-07-08 22:54:54 +08:00
|
|
|
|
1998-12-22 06:52:56 +08:00
|
|
|
<body bgcolor="#FFFFFF">
|
2003-03-15 03:12:34 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
<!-- #BeginLibraryItem "/ed_libs/NavBar_UG.lbi" --><hr>
|
1998-12-22 06:52:56 +08:00
|
|
|
<center>
|
|
|
|
|
<table border=0 width=98%>
|
|
|
|
|
<tr><td valign=top align=left>
|
2003-03-15 03:12:34 +08:00
|
|
|
<a href="index.html">HDF5 documents and links</a> <br>
|
|
|
|
|
<a href="H5.intro.html">Introduction to HDF5</a> <br>
|
|
|
|
|
<a href="RM_H5Front.html">HDF5 Reference Manual</a> <br>
|
2003-07-04 03:35:53 +08:00
|
|
|
<a href="http://hdf.ncsa.uiuc.edu/HDF5/doc/UG/index.html">HDF5 User's Guide for Release 1.6</a> <br>
|
2003-03-15 03:12:34 +08:00
|
|
|
<!--
|
|
|
|
|
<a href="Glossary.html">Glossary</a><br>
|
|
|
|
|
-->
|
1998-12-22 06:52:56 +08:00
|
|
|
</td>
|
|
|
|
|
<td valign=top align=right>
|
2003-03-15 03:12:34 +08:00
|
|
|
And in this document, the
|
2003-07-04 03:35:53 +08:00
|
|
|
<a href="H5.user.html"><strong>HDF5 User's Guide from Release 1.4.5:</strong></a>
|
2003-03-15 03:12:34 +08:00
|
|
|
<br>
|
|
|
|
|
<a href="Files.html">Files</a>
|
|
|
|
|
<a href="Datasets.html">Datasets</a>
|
|
|
|
|
<a href="Datatypes.html">Datatypes</a>
|
|
|
|
|
<a href="Dataspaces.html">Dataspaces</a>
|
|
|
|
|
<a href="Groups.html">Groups</a>
|
|
|
|
|
<br>
|
|
|
|
|
<a href="References.html">References</a>
|
|
|
|
|
<a href="Attributes.html">Attributes</a>
|
|
|
|
|
<a href="Properties.html">Property Lists</a>
|
|
|
|
|
<a href="Errors.html">Error Handling</a>
|
|
|
|
|
<br>
|
|
|
|
|
<a href="Filters.html">Filters</a>
|
|
|
|
|
<a href="Caching.html">Caching</a>
|
|
|
|
|
<a href="Chunking.html">Chunking</a>
|
|
|
|
|
<a href="MountingFiles.html">Mounting Files</a>
|
|
|
|
|
<br>
|
|
|
|
|
<a href="Performance.html">Performance</a>
|
|
|
|
|
<a href="Debugging.html">Debugging</a>
|
|
|
|
|
<a href="Environment.html">Environment</a>
|
|
|
|
|
<a href="ddl.html">DDL</a>
|
1998-12-22 06:52:56 +08:00
|
|
|
</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
</center>
|
|
|
|
|
<hr>
|
2003-03-15 03:12:34 +08:00
|
|
|
<!-- #EndLibraryItem --><h1>Data Caching</h1>
|
1999-12-14 04:39:48 +08:00
|
|
|
|
|
|
|
|
<h2>1. Meta Data Caching</h2>
|
1998-07-08 22:54:54 +08:00
|
|
|
|
|
|
|
|
<p>The HDF5 library caches two types of data: meta data and raw
|
|
|
|
|
data. The meta data cache holds file objects like the file
|
|
|
|
|
header, symbol table nodes, global heap collections, object
|
|
|
|
|
headers and their messages, etc. in a partially decoded
|
|
|
|
|
state. The cache has a fixed number of entries which is set with
|
|
|
|
|
the file access property list (defaults to 10k) and each entry
|
|
|
|
|
can hold a single meta data object. Collisions between objects
|
|
|
|
|
are handled by preempting the older object in favor of the new
|
|
|
|
|
one.
|
|
|
|
|
|
1999-12-14 04:39:48 +08:00
|
|
|
<h2>2. Raw Data Chunk Caching</h2>
|
1998-07-08 22:54:54 +08:00
|
|
|
|
|
|
|
|
<p>Raw data chunks are cached because I/O requests at the
|
|
|
|
|
application level typically don't map well to chunks at the
|
|
|
|
|
storage level. The chunk cache has a maximum size in bytes
|
|
|
|
|
set with the file access property list (defaults to 1MB) and
|
|
|
|
|
when the limit is reached chunks are preempted based on the
|
|
|
|
|
following set of heuristics.
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
<li>Chunks which have not been accessed for a long time
|
|
|
|
|
relative to other chunks are penalized.
|
|
|
|
|
<li>Chunks which have been accessed frequently in the recent
|
|
|
|
|
past are favored.
|
|
|
|
|
<li>Chunks which are completely read and not written, completely
|
|
|
|
|
written but not read, or completely read and completely
|
|
|
|
|
written are penalized according to <em>w0</em>, an
|
|
|
|
|
application-defined weight between 0 and 1 inclusive. A weight
|
|
|
|
|
of zero does not penalize such chunks while a weight of 1
|
|
|
|
|
penalizes those chunks more than all other chunks. The
|
|
|
|
|
default is 0.75.
|
|
|
|
|
<li>Chunks which are larger than the maximum cache size do not
|
|
|
|
|
participate in the cache.
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
<p>One should choose large values for <em>w0</em> if I/O requests
|
|
|
|
|
typically do not overlap but smaller values for <em>w0</em> if
|
|
|
|
|
the requests do overlap. For instance, reading an entire 2d
|
|
|
|
|
array by reading from non-overlapping "windows" in a row-major
|
|
|
|
|
order would benefit from a high <em>w0</em> value while reading
|
|
|
|
|
a diagonal accross the dataset where each request overlaps the
|
|
|
|
|
previous request would benefit from a small <em>w0</em>.
|
|
|
|
|
|
1999-12-14 04:39:48 +08:00
|
|
|
<h2>3. Data Caching Operations</h2>
|
1998-07-08 22:54:54 +08:00
|
|
|
|
|
|
|
|
<p>The cache parameters for both caches are part of a file access
|
|
|
|
|
property list and are set and queried with this pair of
|
|
|
|
|
functions:
|
|
|
|
|
|
|
|
|
|
<dl>
|
|
|
|
|
<dt><code>herr_t H5Pset_cache(hid_t <em>plist</em>, unsigned int
|
|
|
|
|
<em>mdc_nelmts</em>, size_t <em>rdcc_nbytes</em>, double
|
|
|
|
|
<em>w0</em>)</code>
|
|
|
|
|
<dt><code>herr_t H5Pget_cache(hid_t <em>plist</em>, unsigned int
|
|
|
|
|
*<em>mdc_nelmts</em>, size_t *<em>rdcc_nbytes</em>, double
|
|
|
|
|
<em>w0</em>)</code>
|
|
|
|
|
<dd>Sets or queries the meta data cache and raw data chunk cache
|
|
|
|
|
parameters. The <em>plist</em> is a file access property
|
|
|
|
|
list. The number of elements (objects) in the meta data cache
|
|
|
|
|
is <em>mdc_nelmts</em>. The total size of the raw data chunk
|
|
|
|
|
cache and the preemption policy is <em>rdcc_nbytes</em> and
|
|
|
|
|
<em>w0</em>. For <code>H5Pget_cache()</code> any (or all) of
|
|
|
|
|
the pointer arguments may be null pointers.
|
|
|
|
|
</dl>
|
|
|
|
|
|
1998-12-22 06:52:56 +08:00
|
|
|
|
2003-03-15 03:12:34 +08:00
|
|
|
<!-- #BeginLibraryItem "/ed_libs/NavBar_UG.lbi" --><hr>
|
1998-12-22 06:52:56 +08:00
|
|
|
<center>
|
|
|
|
|
<table border=0 width=98%>
|
|
|
|
|
<tr><td valign=top align=left>
|
2003-03-15 03:12:34 +08:00
|
|
|
<a href="index.html">HDF5 documents and links</a> <br>
|
|
|
|
|
<a href="H5.intro.html">Introduction to HDF5</a> <br>
|
|
|
|
|
<a href="RM_H5Front.html">HDF5 Reference Manual</a> <br>
|
2003-07-04 03:35:53 +08:00
|
|
|
<a href="http://hdf.ncsa.uiuc.edu/HDF5/doc/UG/index.html">HDF5 User's Guide for Release 1.6</a> <br>
|
2003-03-15 03:12:34 +08:00
|
|
|
<!--
|
|
|
|
|
<a href="Glossary.html">Glossary</a><br>
|
|
|
|
|
-->
|
1998-12-22 06:52:56 +08:00
|
|
|
</td>
|
|
|
|
|
<td valign=top align=right>
|
2003-03-15 03:12:34 +08:00
|
|
|
And in this document, the
|
2003-07-04 03:35:53 +08:00
|
|
|
<a href="H5.user.html"><strong>HDF5 User's Guide from Release 1.4.5:</strong></a>
|
2003-03-15 03:12:34 +08:00
|
|
|
<br>
|
|
|
|
|
<a href="Files.html">Files</a>
|
|
|
|
|
<a href="Datasets.html">Datasets</a>
|
|
|
|
|
<a href="Datatypes.html">Datatypes</a>
|
|
|
|
|
<a href="Dataspaces.html">Dataspaces</a>
|
|
|
|
|
<a href="Groups.html">Groups</a>
|
|
|
|
|
<br>
|
|
|
|
|
<a href="References.html">References</a>
|
|
|
|
|
<a href="Attributes.html">Attributes</a>
|
|
|
|
|
<a href="Properties.html">Property Lists</a>
|
|
|
|
|
<a href="Errors.html">Error Handling</a>
|
|
|
|
|
<br>
|
|
|
|
|
<a href="Filters.html">Filters</a>
|
|
|
|
|
<a href="Caching.html">Caching</a>
|
|
|
|
|
<a href="Chunking.html">Chunking</a>
|
|
|
|
|
<a href="MountingFiles.html">Mounting Files</a>
|
|
|
|
|
<br>
|
|
|
|
|
<a href="Performance.html">Performance</a>
|
|
|
|
|
<a href="Debugging.html">Debugging</a>
|
|
|
|
|
<a href="Environment.html">Environment</a>
|
|
|
|
|
<a href="ddl.html">DDL</a>
|
1998-12-22 06:52:56 +08:00
|
|
|
</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
</center>
|
1998-09-10 06:48:37 +08:00
|
|
|
<hr>
|
2003-03-15 03:12:34 +08:00
|
|
|
<!-- #EndLibraryItem --><!-- #BeginLibraryItem "/ed_libs/Footer.lbi" --><address>
|
|
|
|
|
<a href="mailto:hdfhelp@ncsa.uiuc.edu">HDF Help Desk</a>
|
|
|
|
|
<br>
|
2003-07-04 03:35:53 +08:00
|
|
|
Describes HDF5 Release 1.4.5, February 2003
|
2003-07-03 05:28:11 +08:00
|
|
|
</address><!-- #EndLibraryItem --><!-- Created: Tue May 26 15:20:14 EDT 1998 -->
|
1999-12-14 04:39:48 +08:00
|
|
|
<!-- hhmts start -->
|
2003-03-15 03:12:34 +08:00
|
|
|
Last modified: 13 December 1999
|
1999-12-14 04:39:48 +08:00
|
|
|
<!-- hhmts end -->
|
1998-09-10 06:48:37 +08:00
|
|
|
|
2000-12-09 06:12:36 +08:00
|
|
|
|
2003-03-15 03:12:34 +08:00
|
|
|
</body>
|
1998-07-08 22:54:54 +08:00
|
|
|
</html>
|
