mirror/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2025-01-24 15:25:00 +08:00
Code Issues Packages Projects Releases Wiki Activity
b5b1d7f713
hdf5/doc/html/TechNotes/DataTransformReport.htm
Albert Cheng 5106d9643f [svn-r9672] Purpose: 
New document committed.

Description:
Report of the Data Transform work.
2004-12-15 18:21:23 -05:00

878 lines
33 KiB
HTML
Raw Blame History

<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:w="urn:schemas-microsoft-com:office:word"
xmlns:st1="urn:schemas-microsoft-com:office:smarttags"
xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
<meta name=ProgId content=Word.Document>
<meta name=Generator content="Microsoft Word 10">
<meta name=Originator content="Microsoft Word 10">
<link rel=File-List href="Data_Transforms_Report_files/filelist.xml">
<title>Arithmetic Data Transforms</title>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="date"/>
<!--[if gte mso 9]><xml>
<o:DocumentProperties>
<o:Author>install</o:Author>
<o:Template>Normal</o:Template>
<o:LastAuthor>Albert Cheng</o:LastAuthor>
<o:Revision>4</o:Revision>
<o:TotalTime>212</o:TotalTime>
<o:LastPrinted>2004-12-10T21:28:00Z</o:LastPrinted>
<o:Created>2004-12-10T22:44:00Z</o:Created>
<o:LastSaved>2004-12-16T00:13:00Z</o:LastSaved>
<o:Pages>1</o:Pages>
<o:Words>1721</o:Words>
<o:Characters>9815</o:Characters>
<o:Company>UIUC</o:Company>
<o:Lines>81</o:Lines>
<o:Paragraphs>23</o:Paragraphs>
<o:CharactersWithSpaces>11513</o:CharactersWithSpaces>
<o:Version>10.2625</o:Version>
</o:DocumentProperties>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Print</w:View>
<w:Zoom>105</w:Zoom>
<w:PunctuationKerning/>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:ApplyBreakingRules/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:UseFELayout/>
</w:Compatibility>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
</w:WordDocument>
</xml><![endif]--><!--[if !mso]><object
classid="clsid:38481807-CA0E-42D2-BF39-B33AF135CC4D" id=ieooui></object>
<style>
st1\:*{behavior:url(#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;
mso-font-alt:"Courier New";
mso-font-charset:0;
mso-generic-font-family:modern;
mso-font-format:other;
mso-font-pitch:fixed;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;
mso-font-alt:\5B8B\4F53;
mso-font-charset:134;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:3 135135232 16 0 262145 0;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;
mso-font-charset:134;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:3 135135232 16 0 262145 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-parent:"";
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";
mso-fareast-language:EN-US;}
h1
{mso-style-next:Normal;
margin-top:12.0pt;
margin-right:0in;
margin-bottom:3.0pt;
margin-left:0in;
mso-pagination:widow-orphan;
page-break-after:avoid;
mso-outline-level:1;
font-size:16.0pt;
font-family:Arial;
mso-font-kerning:16.0pt;
mso-fareast-language:EN-US;}
h2
{mso-style-next:Normal;
margin-top:12.0pt;
margin-right:0in;
margin-bottom:3.0pt;
margin-left:0in;
mso-pagination:widow-orphan;
page-break-after:avoid;
mso-outline-level:2;
font-size:14.0pt;
font-family:Arial;
mso-fareast-language:EN-US;
font-style:italic;}
p.MsoFootnoteText, li.MsoFootnoteText, div.MsoFootnoteText
{mso-style-noshow:yes;
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";
mso-fareast-language:EN-US;}
p.MsoHeader, li.MsoHeader, div.MsoHeader
{margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
tab-stops:center 3.0in right 6.0in;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";
mso-fareast-language:EN-US;}
p.MsoFooter, li.MsoFooter, div.MsoFooter
{margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
tab-stops:center 3.0in right 6.0in;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";
mso-fareast-language:EN-US;}
span.MsoFootnoteReference
{mso-style-noshow:yes;
vertical-align:super;}
/* Page Definitions */
@page
{mso-footnote-separator:url("Data_Transforms_Report_files/header.htm") fs;
mso-footnote-continuation-separator:url("Data_Transforms_Report_files/header.htm") fcs;
mso-endnote-separator:url("Data_Transforms_Report_files/header.htm") es;
mso-endnote-continuation-separator:url("Data_Transforms_Report_files/header.htm") ecs;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;
mso-header-margin:.5in;
mso-footer-margin:.5in;
mso-even-footer:url("Data_Transforms_Report_files/header.htm") ef1;
mso-footer:url("Data_Transforms_Report_files/header.htm") f1;
mso-paper-source:0;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:270281622;
mso-list-template-ids:-1038041382;}
@list l1
{mso-list-id:332993863;
mso-list-type:hybrid;
mso-list-template-ids:1458465098 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2
{mso-list-id:550459489;
mso-list-type:hybrid;
mso-list-template-ids:1085822362 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l2:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3
{mso-list-id:756709709;
mso-list-template-ids:-1385235826;}
@list l4
{mso-list-id:1187134103;
mso-list-type:hybrid;
mso-list-template-ids:1324645902 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l4:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5
{mso-list-id:1433163999;
mso-list-type:hybrid;
mso-list-template-ids:-438374070 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l5:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6
{mso-list-id:1929534752;
mso-list-template-ids:-760440376;}
@list l7
{mso-list-id:2059475982;
mso-list-template-ids:658286432;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman";}
</style>
<![endif]--><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="3074"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1"/>
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US style='tab-interval:.5in'>
<div class=Section1>
<p class=MsoNormal align=center style='text-align:center'><b><span
style='font-size:16.0pt'>Arithmetic Data Transforms<o:p></o:p></span></b></p>
<p class=MsoNormal align=center style='text-align:center'>Leon Arber, Albert
Cheng, William Wendling<a style='mso-footnote-id:ftn1' href="#_ftn1"
name="_ftnref1" title=""><span class=MsoFootnoteReference><span
style='mso-special-character:footnote'><![if !supportFootnotes]><span
class=MsoFootnoteReference><span style='font-size:12.0pt;font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language:
EN-US;mso-bidi-language:AR-SA'>[1]</span></span><![endif]></span></span></a></p>
<p class=MsoNormal align=center style='text-align:center'><st1:date Year="2004"
Day="10" Month="12">December 10, 2004</st1:date></p>
<h1>Purpose</h1>
<p class=MsoNormal>Data can be stored and represented in many different
ways.<span style='mso-spacerun:yes'><EFBFBD> </span>In most fields of science, for
example, the metric system is used for storing all data.<span
style='mso-spacerun:yes'><EFBFBD> </span>However, many fields of engineering still use
the English system.<span style='mso-spacerun:yes'><EFBFBD> </span>In such scenarios,
there needs to be a way to easily perform arbitrary scaling of data.<span
style='mso-spacerun:yes'><EFBFBD> </span>The data transforms provide just such
functionality.<span style='mso-spacerun:yes'><EFBFBD> </span>They allow arbitrary
arithmetic expressions to be applied to a dataset during read and write
operations.<span style='mso-spacerun:yes'><EFBFBD> </span>This means that data can be
stored in Celsius in a data file, but read in and automatically converted to
Fahrenheit.<span style='mso-spacerun:yes'><EFBFBD> </span>Alternatively, data that is
obtained in Fahrenheit can be written out to the data file in Celsius.<span
style='mso-spacerun:yes'><EFBFBD> </span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Although a user can always manually modify the data they
read and write, having the data transform as a property means that the user
doesn<EFBFBD>t have to worry about forgetting to call the conversion function or even
writing it in the first place.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h1>Usage</h1>
<p class=MsoNormal>The data transform functionality is implemented as a
property that is set on a dataset transfer property list.<span
style='mso-spacerun:yes'><EFBFBD> </span>There are two functions available: one for
setting the transform and another for finding out what transform, if any, is
currently set.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>The function for setting the transform is:</p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>herr_t
H5Pset_data_transform(hid_t plist_id, const char* expression)<o:p></o:p></span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>plist_id
</span>is the identifier of the dataset transfer property list on which the
data transform property should be set.</p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>expression
</span>is a pointer to a string of the form <20>(5/9.0)*(x-32)<29> which describes
the transform.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>The function for getting the transform is: </p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>ssize_t
H5Pget_data_transform(hid_t plist_id, char* expression, size_t size)<o:p></o:p></span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>plist_id
</span>is the identifier of the dataset transfer property list which will be
queried for its data transform property.</p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>expression
</span>is either NULL or a pointer to memory where the data transform string,
if present, will be copied.</p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>size
</span>is the number of bytes to copy from the transform string into
expression. <span style='mso-spacerun:yes'><EFBFBD></span>H5Pget_data_transform will
never copy more than the length of the transform expression.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h1>Data Transform Expressions</h1>
<p class=MsoNormal>Data transforms are set by passing a pointer to a string,
which is the data transform expression.<span style='mso-spacerun:yes'><EFBFBD>
</span>This string describes what sort of arithmetic transform should be done
during data transfer of read or write.<span style='mso-spacerun:yes'><EFBFBD>
</span>The string is a standard mathematical expression, as would be entered
into a something like MATLAB.<span style='mso-spacerun:yes'><EFBFBD> </span></p>
<p class=MsoNormal>Expressions are defined by the following context-free
grammar:</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>expr:=<span style='mso-spacerun:yes'><EFBFBD> </span>term | term +
term | term - term</p>
<p class=MsoNormal>term := factor | factor * factor | factor / factor</p>
<p class=MsoNormal>factor :=<span style='mso-spacerun:yes'><EFBFBD> </span>number |
symbol | - factor | + factor | ( expr )</p>
<p class=MsoNormal>symbol := [a-zA-Z][a-zA-Z0-9]*</p>
<p class=MsoNormal>number := INT | FLOAT</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>where INT is interpreted as a C long int and FLOAT is interpreted
as a C double</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>This grammar allows for order of operations (multiplication
and dividision take precedence over addition and subtraction), floating and
integer constants, and grouping of terms by way of parentheses.<span
style='mso-spacerun:yes'><EFBFBD> </span>Although the grammar allows symbols to be
arbitrary strings, this documentation will always use <20>x<EFBFBD> for symbols.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Within a transform expression, the symbol represents a
variable which contains the data to be manipulated.<span
style='mso-spacerun:yes'><EFBFBD> </span>For this reason, the terms symbol and
variable will be used interchangeably.<span style='mso-spacerun:yes'><EFBFBD>
</span>Furthermore, in the current implementation of data transforms, all
symbols appearing in an expression are interpreted as referring to the same
dataset.<span style='mso-spacerun:yes'><EFBFBD> </span>So, an expression such as
<EFBFBD>alpha + 5<> is equivalent to <20>x+5<> and an expression such as <20>alpha + 3*beta +
5<EFBFBD> is equivalent to <20>alpha + 3*alpha + 5<> which is equivalent to <20>4*x +
5<EFBFBD>.<span style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD> </span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h1>Data Transform Implementation</h1>
<p class=MsoNormal>When the data transform property of a dataset transfer
property list is set, a parse tree of the expression is immediately generated
and its root is saved in the property list.<span style='mso-spacerun:yes'><EFBFBD>
</span>The generation of the parse involves several steps.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>First, the expression is reduced, so as to simply the final
parse and speed up the transform operations.<span style='mso-spacerun:yes'><EFBFBD>
</span>Expressions such as <20>(5/9.0) * (x-32)<29> will be reduced to
<EFBFBD>.555555*(x-32).<2E><span style='mso-spacerun:yes'><EFBFBD> </span>While further
simplification is algebraically possible, the data transform code will only
reduce simple trivial arithmetic operations.<span style='mso-spacerun:yes'><EFBFBD>
</span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Then, this reduced expression is parsed into a set of
tokens, from which the parse tree is generated.<span style='mso-spacerun:yes'><EFBFBD>
</span>From the expression <20>(5/9.0)*(x-32),<2C> for example, the following parse
tree would be created:</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal><span style='mso-tab-count:4'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>*</p>
<p class=MsoNormal><span style='mso-tab-count:4'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>/<span
style='mso-tab-count:1'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>\<span style='mso-spacerun:yes'><EFBFBD>
</span></p>
<p class=MsoNormal><span style='mso-tab-count:3'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>.555555<span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>-</p>
<p class=MsoNormal><span style='mso-tab-count:5'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>/<span
style='mso-spacerun:yes'><EFBFBD><EFBFBD> </span>\<span style='mso-tab-count:5'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span></p>
<p class=MsoNormal><span style='mso-tab-count:4'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>x<span style='mso-spacerun:yes'><EFBFBD>
</span>32</p>
<p class=MsoNormal><span style='mso-tab-count:1'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span></p>
<h2>HDread with Data Transform Expressions</h2>
<p class=MsoNormal>When a read is performed with a dataset transfer property
list that has the data transform property set, the following sequence of events
occurs:</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<ol style='margin-top:0in' start=1 type=1>
<li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>A
piece of the file is read into memory</li>
<li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>The
data transform is performed on this piece of memory</li>
<li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>This piece
of memory is then copied to the user</li>
<li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>Steps
1 <20> 3 are repeated until the read is complete.</li>
</ol>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Step 2 works like this:</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<ol style='margin-top:0in' start=1 type=1>
<li class=MsoNormal style='mso-list:l1 level1 lfo6;tab-stops:list .5in'>The
function responsible for doing the transform is passed a buffer and is
informed what type of data is inside this buffer and how many elements
there are.</li>
<li class=MsoNormal style='mso-list:l1 level1 lfo6;tab-stops:list .5in'>This
buffer is then treated as the variable in the data transform expression
and the transform expression is applied.</li>
<li class=MsoNormal style='mso-list:l1 level1 lfo6;tab-stops:list .5in'>The
transformed buffer is returned to the library.</li>
</ol>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>If the transform expression is <20>(5/9.0)*(x-32),<2C> with the
parse tree shown above and the buffer contains [-10 0 10 50 100], then the
intermediate steps involved in the transform are:</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<ol style='margin-top:0in' start=1 type=1>
<li class=MsoNormal style='mso-pagination:widow-orphan lines-together;
page-break-after:avoid;mso-list:l5 level1 lfo9;mso-hyphenate:none;
tab-stops:list .5in'>First, the (x-32) subexpression is evaluated.<span
style='mso-spacerun:yes'><EFBFBD> </span>Now the buffer would contain<span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> </span>[-42 -32 -22 18 68]</li>
<li class=MsoNormal style='mso-list:l5 level1 lfo9;tab-stops:list .5in'>Then,
the .55555 * part of the expression is evaluated.<span
style='mso-spacerun:yes'><EFBFBD> </span>Now the buffer would contain: [-23.3333
-17.7777 -12.2222 9.9999 37.7777]</li>
<li class=MsoNormal style='mso-list:l5 level1 lfo9;tab-stops:list .5in'>Now,
the transform would be completed and the resulting buffer returned.</li>
</ol>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Note that the original data in the file was not modified. </p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h2>HDwrite with Data Transform Expressions</h2>
<p class=MsoNormal>The process of a write works much the same way, but in the
reverse order.<span style='mso-spacerun:yes'><EFBFBD> </span>When a file is written
out with a dataset transfer property list that has the data transform property
set:</p>
<p class=MsoNormal><span style='mso-spacerun:yes'><EFBFBD> </span></p>
<ol style='margin-top:0in' start=1 type=1>
<li class=MsoNormal style='mso-list:l4 level1 lfo12;tab-stops:list .5in'>The
user passes a buffer to HDwrite, along with the type and number of
elements.</li>
<li class=MsoNormal style='mso-list:l4 level1 lfo12;tab-stops:list .5in'>The
data transform is performed on a copy of this piece of memory.</li>
<li class=MsoNormal style='mso-list:l4 level1 lfo12;tab-stops:list .5in'>This
copy with the transformed data is then written out to the file.</li>
</ol>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Step 2 works exactly as in the read example.<span
style='mso-spacerun:yes'><EFBFBD> </span>Note that the user<65>s data is not modified. <span
style='mso-spacerun:yes'><EFBFBD></span>Also, since the transform property is not
saved with the dataset, in order to recover the original data, a user must know
the inverse of the transform that was applied in order to recover it.<span
style='mso-spacerun:yes'><EFBFBD> </span>In the case of <20>(5/9.0)*(x-32)<29> this inverse
would be <20>(9/5.0)*x + 32<33>.<span style='mso-spacerun:yes'><EFBFBD> </span>Reading from
a data file that had previously been written out with a transform string of
<EFBFBD>(5/9.0)*(x-32)<29> with a transform string of <20>(9/5.0)*x + 32<33> would effectively
recover the original data the author of the file had been using.<a
style='mso-footnote-id:ftn2' href="#_ftn2" name="_ftnref2" title=""><span
class=MsoFootnoteReference><span style='mso-special-character:footnote'><![if !supportFootnotes]><span
class=MsoFootnoteReference><span style='font-size:12.0pt;font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language:
EN-US;mso-bidi-language:AR-SA'>[2]</span></span><![endif]></span></span></a></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h1>Mixed Mode and Truncation</h1>
<p class=MsoNormal>Because the data transform sits and modifies data between
the file space and the memory space, various effects can occur that are the
result of the typecasting that may be involved in the operations.<span
style='mso-spacerun:yes'><EFBFBD><EFBFBD> </span>In addition, because constants in the data
transform expression can be either INT or FLOAT, the data transform itself can
be a source of truncation.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>In the example above, the reason that the transform
expression is always written as <20>(5/9.0)*(x-32)<29> is because, if it were written
without a floating point constant, it would always evaluate to 0.<span
style='mso-spacerun:yes'><EFBFBD> </span>The expression <20>(5/9)*(x-32)<29> would, when
set, get reduced to <20>0*(x-32)<29> because both 5 and 9 would get read as C long
ints and, when divided, the result would get truncated to 0.<span
style='mso-spacerun:yes'><EFBFBD> </span>This resulting expression, <20>0*(x-32),<2C> would
cause any data read or written to be saved as an array of all 0<>s.<span
style='mso-spacerun:yes'><EFBFBD> </span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Another source of unpredictability caused by truncation
occurs when intermediate data is of a type that is more precise than the
destination memory type.<span style='mso-spacerun:yes'><EFBFBD> </span>For example, if
the transform expression <20>(1/2.0)*x<> is applied to data read from a file that
is being read into an integer memory buffer, the results can be
unpredictable.<span style='mso-spacerun:yes'><EFBFBD> </span>If the source array is [1
2 3 4], then the resulting array could be either [0 1 1 2] or [0 0 1 1],
depending on the floating point unit of the processors.<span
style='mso-spacerun:yes'><EFBFBD> </span>Note that this result is independent of the
source data type.<span style='mso-spacerun:yes'><EFBFBD> </span>It doesn<73>t matter if
the source data is integer or floating point because the 2.0 in the data
transform expression will cause everything to be evaluated in a floating-point
context.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>When setting transform expressions, care must be taken to
ensure that the truncation does not adversely affect the data.<span
style='mso-spacerun:yes'><EFBFBD> </span>A workaround for the possible effects of a
transform such as <20>(1/2.0) * x<> would be to used the transform expression
<EFBFBD>(1/2.0)*x + 0.5<EFBFBD> instead of the original.<span style='mso-spacerun:yes'><EFBFBD>
</span>This will ensure that all truncation rounds up, with the possible
exception of a boundary condition.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h1>Data Transform Example</h1>
<p class=MsoNormal>The following code snippet shows an example using data
transform, where the data transform property is set and a write is
performed.<span style='mso-spacerun:yes'><EFBFBD> </span>Then, a read is performed
with no data transform property set.<span style='mso-spacerun:yes'><EFBFBD> </span>It
is assumed that <span style='font-family:Courier'>dataset </span>is a dataset
that has been opened and <span style='font-family:Courier'>windchillF </span>and
<span style='font-family:Courier'>windchillC </span>are both arrays that hold
floating point data.<span style='mso-spacerun:yes'><EFBFBD> </span>The result of this
snippet is to fill <span style='font-family:Courier'>windchillC </span>with the
data in <span style='font-family:Courier'>windchillF</span>, converted to
Celcius.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal><span style='font-family:Courier'>hid_t dxpl_id_c_to_f;<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'>const char* c_to_f =
<EFBFBD>(9/5.0)*x + 32<33>;<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><o:p>&nbsp;</o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'>/* Create the dataset
transfer property list */<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD> </span>dxpl_id_c_to_f =
H5Pcreate(H5P_DATASET_XFER);<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><o:p>&nbsp;</o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'>/* Set the data transform
to be used on the read*/<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD> </span>H5Pset_data_transform(dxpl_id_c_to_f,
c_to_f);<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><o:p>&nbsp;</o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD> </span><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'>/*<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'>* Write the data to the
dataset using the f_to_c transform<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'>*/<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD> </span>status = H5Dwrite(dataset, H5T_NATIVE_FLOAT,
H5S_ALL, H5S_ALL, dxpl_id_f_to_c, windchillF);<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD> </span><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'>/* Read the data with the
c_to_f data transform */<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:Courier'><span
style='mso-spacerun:yes'><EFBFBD><EFBFBD><EFBFBD> </span>H5Dread(dataset, H5T_NATIVE_FLOAT, H5S_ALL,
H5S_ALL, H5P_DEFAULT, windchillC);<o:p></o:p></span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h1>H5Pget_data_transform Details</h1>
<p class=MsoNormal>Querying the data transform string of a dataset transfer
property list requires the use of the H5Pget_data_transform function.<span
style='mso-spacerun:yes'><EFBFBD> </span>This function provides the ability to both
query the size of the string stored and retrieve part or all of it.<span
style='mso-spacerun:yes'><EFBFBD> </span>Note that H5Pget_data_transform will return
the expression that was set by H5Pset_data_transform.<span
style='mso-spacerun:yes'><EFBFBD> </span>The reduced transform string, computed when
H5Pset_data_transform is called, is not stored in string form and is not
available to the user.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>In order to ascertain the size of the string, a NULL <span
style='font-family:Courier'>expression</span> should be passed to the
function.<span style='mso-spacerun:yes'><EFBFBD> </span>This will make the function
return the length of the transform string (not including the terminated <20>\0<>
character).</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>To actually retrieve the string, a pointer to a valid memory
location should be passed in for <span style='font-family:Courier'>expression </span>and
the number of bytes from the string that should be copied to that memory
location should be passed in as <span style='font-family:Courier'>size</span>.</p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<h1>Further Work</h1>
<p class=MsoNormal>Some additional functionality can still be added to the data
transform.<span style='mso-spacerun:yes'><EFBFBD> </span>Currently the most important
feature lacking is the addition of operators, such as exponentiation and the
trigonometric functions.<span style='mso-spacerun:yes'><EFBFBD> </span>Although
exponentiation can be explicitly carried with a transform expression such as
<EFBFBD>x*x*x<> it may be easier to support expression like <20>x^3.<2E> Also lacking are the
commonly used trigonometric functions, such as sin, cos, and tan.<span
style='mso-spacerun:yes'><EFBFBD> </span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>Popular constants could also be added, such as &#960; or
e.<span style='mso-spacerun:yes'><EFBFBD> </span></p>
<p class=MsoNormal><o:p>&nbsp;</o:p></p>
<p class=MsoNormal>More advanced functionality, such as the ability to perform
a transform on multiple datasets is also a possibility, but is a feature is
more a completely new addition than an extension to data transforms.<span
style='mso-spacerun:yes'><EFBFBD> </span></p>
</div>
<div style='mso-element:footnote-list'><![if !supportFootnotes]><br clear=all>
<hr align=left size=1 width="33%">
<![endif]>
<div style='mso-element:footnote' id=ftn1>
<p class=MsoFootnoteText><a style='mso-footnote-id:ftn1' href="#_ftnref1"
name="_ftn1" title=""><span class=MsoFootnoteReference><span style='mso-special-character:
footnote'><![if !supportFootnotes]><span class=MsoFootnoteReference><span
style='font-size:10.0pt;font-family:"Times New Roman";mso-fareast-font-family:
"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language:EN-US;
mso-bidi-language:AR-SA'>[1]</span></span><![endif]></span></span></a> Mr.
Wendling, who involved in the initial design and implemented the expression
parser, has left NCSA.</p>
</div>
<div style='mso-element:footnote' id=ftn2>
<p class=MsoFootnoteText><a style='mso-footnote-id:ftn2' href="#_ftnref2"
name="_ftn2" title=""><span class=MsoFootnoteReference><span style='mso-special-character:
footnote'><![if !supportFootnotes]><span class=MsoFootnoteReference><span
style='font-size:10.0pt;font-family:"Times New Roman";mso-fareast-font-family:
"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language:EN-US;
mso-bidi-language:AR-SA'>[2]</span></span><![endif]></span></span></a> See the
h5_dtransform.c example in the examples directory of the hdf5 library for just
such an illustration.</p>
</div>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink