mirror of
https://github.com/HDFGroup/hdf5.git
synced 2025-01-12 15:04:59 +08:00
b9e4deec10
* Sketch of the H5S life cycle. * Committing clang-format changes * Fix H5S_UNLIMITED snafu. * Updated RM template and RM page. * Added H5S life cycle. * Committing clang-format changes * Added H5T life cycle. * Committing clang-format changes * Cleaner layout (?) * Cleaned the H5F life cycle. Called out unfinished biz. * Committing clang-format changes * Remaining life cycle skeletons. * Committing clang-format changes * Committing clang-format changes * Added H5Z life cycle. * Committing clang-format changes * Added H5G life cycle. * Committing clang-format changes * H5 and H5I life cycle updates. * Committing clang-format changes * Added H5PL life cycle. * Committing clang-format changes * Added H5L life cycle. * Committing clang-format changes * Fix for Chris' comment. * Add a variable for Doxygen pre-processor definitions. * Forgot to add the H5M API. * Clarify the H5Z life cycle. * Committing clang-format changes * Add H5Zdevelop.h to Doxygen.in. Added H5I life cycle. * Committing clang-format changes * Clarified introduction and fixed missing label declaration. * Added H5O life cycle. * Committing clang-format changes * H5O cleanup, part 1. * Committing clang-format changes * Cleaned up some of the endless repetition in H5O. * Committing clang-format changes * Cookbook & RFC draft layouts. * Updated manifest. * Updated the manifest, the example paths, and sketched the 1st recipe. * Committing clang-format changes * Outlined two more recipes. * Committing clang-format changes * More recipes and RFCs. * Committing clang-format changes * Draft of templatized RFC references. * Another batch of RFC changes. * Another batch of RFCs. * Fixed reference. * RFCs in reverse chronological order. * First cut of RFCs. * Fixed reference. * Updated recipes. * Updated recipes. * More RFCs. * Updated D*PL comments. * Added H5P descriptions. * Committing clang-format changes * H5R life-cycle snapshot. * Committing clang-format changes * H5R life-cycle. Added line numbers to life-cycle examples. * Committing clang-format changes * Fixed formatting for H5Dchunk_iter(). * Added comment on collective mode requirement w/ compression. * Simplified API compat. macro dox. * More API vers. updates. * Hide the async macro entrails. * Latest VFD SWMR RFC. * Create a tag file for permalinks. * Added TODOs for metadoc. * Removed duplication. * Revised RM landing page. * Trimmed more duplication. * Committing clang-format changes * Revised H5D. * Committing clang-format changes * Updated survey link. * Added Doxygen RM entry template link. * Added the "Multi-Thread HDF5" RFC. * Added DOXYGEN_TAG_FILE. * Added selection I/O RFC. * Added the VFD Sub-filing RFC. * Updated meta-documentation and added two old presentations. * Added a few more RFCs (4). * Fixed MANIFEST. * Updated meta-documentation. * Added Filters technical note. * Fixed MANIFEST. * Restore the path stripper. * Experimental full-text search via Google. * Better full-text search integration. * Whoops. Forgot this one. * Oh boy. * Make CMake happy. * Added "Debugging HDF5 Applications" technical note. * Another batch of RFCs. Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
138 lines
5.4 KiB
HTML
138 lines
5.4 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
|
<!-- saved from url=(0064)https://gamma.hdfgroup.org/papers/HISS/030821.IOFlow/IOFlow.html -->
|
|
<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
|
<title>HDF5 Raw I/O Flow Notes</title>
|
|
|
|
<meta name="author" content="Quincey Koziol">
|
|
</head>
|
|
|
|
<body text="#000000" bgcolor="#FFFFFF">
|
|
|
|
<style type="text/css">
|
|
OL.loweralpha { list-style-type: lower-alpha }
|
|
OL.upperroman { list-style-type: upper-roman }
|
|
</style>
|
|
|
|
<center><h1>HDF5 Raw I/O Flow Notes</h1></center>
|
|
<center><h3>Quincey Koziol<br>
|
|
koziol@ncsa.uiuc.edu<br>
|
|
August 20, 2003
|
|
</h3></center>
|
|
|
|
<ol class="upperroman">
|
|
|
|
<li><h3><u>Document's Audience:</u></h3>
|
|
|
|
<ul>
|
|
<li>Current H5 library designers and knowledgable external developers.</li>
|
|
</ul>
|
|
|
|
</li><li><h3><u>Background Reading:</u></h3>
|
|
|
|
</li><li><h3><u>Introduction:</u></h3>
|
|
|
|
<dl>
|
|
<dt><strong>What is this document about?</strong></dt>
|
|
<dd>This document attempts to supplement the flow charts describing
|
|
the flow of control for raw data I/O in the library.
|
|
</dd> <br>
|
|
</dl>
|
|
|
|
</li><li><h3><u>Figures:</u></h3>
|
|
<p>The following figures provide the main information:</p>
|
|
<table>
|
|
<tr><td><img src="IOFlow.gif" alt="High-Level View of Writing Raw Data" style="height:50%;"></td></tr>
|
|
<tr><td><img src="IOFlow2.gif" alt="Perform Serial or Parallel I/O" style="height:50%;"></td></tr>
|
|
<tr><td><img src="IOFlow3.gif" alt="Gather/Convert/Scatter" style="height:50%;"></td></tr>
|
|
</table>
|
|
|
|
</li><li><h3><u>Notes From Accompanying Figures:</u></h3>
|
|
|
|
<p>This section provides notes to augment the information in the accompanying
|
|
figures.
|
|
</p>
|
|
|
|
<ol>
|
|
<li><b>Validate Parameters</b> - Resolve any H5S_ALL parameters
|
|
for dataspace selections to actual dataspaces, allocate
|
|
conversion buffers, etc.
|
|
</li>
|
|
|
|
<li><b>Space Allocated in File?</b> - Space may not have been allocated
|
|
in the file to store the dataset data, if "late allocation" was chosen
|
|
for the allocation time when the dataset was created.
|
|
</li>
|
|
|
|
<li><b>Allocate & Fill Space</b> - These operations allocate both contiguous
|
|
and chunked dataset's space in the file. The chunked dataset space
|
|
allocation iterates through all the chunks in the file and allocates
|
|
both the B-tree information and the raw data in the file. Because of
|
|
the way filters work, fill-values are written out for chunked datasets
|
|
as they are allocated, instead of as a separate step.
|
|
In parallel
|
|
I/O, the chunked dataset allocation can potentially be time-consuming,
|
|
since all the raw data in the dataset is allocated from one process.
|
|
</li>
|
|
|
|
<li><b>Datatype Conversion Needed?</b> - This currently is the deciding
|
|
factor between doing "direct I/O" (in serial or parallel) and needing
|
|
to perform gather/convert/scatter operations. I believe that MPI
|
|
is capable of performing a limited range of type conversions and if so,
|
|
we should add support to detect when they can be used. This will
|
|
allow more I/O operations to be performed collectively.
|
|
</li>
|
|
|
|
<li><b>Collective I/O Requested/Allowed?</b> - A user has to both request
|
|
that collective I/O occur and also their I/O operation must meet the
|
|
requirements that the library sets for supporting collective parallel
|
|
I/O:
|
|
<ul>
|
|
<li>The dataspace must be scalar or simple (which is a no-op really,
|
|
since we don't support "complex" dataspaces in the library
|
|
currently).
|
|
</li>
|
|
<li>The selection must be regular. "all" selections
|
|
and hyperslab selections that were
|
|
made with only one call to H5Sselect_hyperslab() (i.e. not a
|
|
hyperslab selection that has been aggregated over multiple
|
|
selection calls) are regular. Supporting point and
|
|
irregular hyperslab selections are on the "to do" list.
|
|
</li>
|
|
<li>The dataset must be stored contiguously on disk (as shown in the
|
|
figure also). Supporting chunked dataset storage is also
|
|
on the "to do" list.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li><b>Build "chunk map"</b> - This step still has some scalability issues
|
|
as it creates a data structure that is proportional to the number of
|
|
chunks which will be written to, which could potentially be very large.
|
|
Building the "chunk map" information incrementally is on the "to do"
|
|
list also.
|
|
</li>
|
|
|
|
<li><b>Perform Chunked I/O</b> - As the figure shows, there is no support
|
|
for collective parallel I/O on chunked datasets currently. As noted
|
|
earlier, this is on the "to do" list.
|
|
</li>
|
|
|
|
<li><b>Perform "Direct" Serial I/O</b> - "Direct" serial I/O writes data
|
|
from the application's buffer, without any intervening buffer or memory
|
|
copies. For maximum efficiency and performance, the elements in the
|
|
selections should be adjoining.
|
|
</li>
|
|
|
|
<li><b>Perform Collective Parallel I/O</b> - This step also writes data
|
|
directly from an application buffer, but additionally uses collective
|
|
MPI I/O operations to combine the data from each process in the parallel
|
|
application in an efficient manner.
|
|
</li>
|
|
</ol>
|
|
|
|
</li></ol>
|
|
|
|
|
|
|
|
</body></html>
|