mirror/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2024-12-21 07:51:46 +08:00
Code Issues Packages Projects Releases Wiki Activity
901a8db2a2
hdf5/doxygen/dox/ViewToolsJPSS.dox
Allen Byrne 3fad116502
Use aliases for URLs (#4409)
2024-04-16 13:51:20 -05:00

764 lines
31 KiB
Plaintext
Raw Blame History

/** @page ViewToolsJPSS Use Case: Examining a JPSS NPP File With HDF5 Tools
Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand
<hr>
\section secViewToolsJPSSTOC Contents
<ul>
<li>\ref secViewToolsJPSSDeter</li>
<li>\ref secViewToolsJPSSView</li>
<li>\ref secViewToolsJPSSExam</li>
</ul>
This tutorial illustrates how to use the HDF5 tools to examine NPP files from the JPSS project. The following files are discussed:
\code
SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 (<a href="https://\AEXURL/files/tutorial/SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5.gz">gzipped file</a>)
SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 (<a href="https://\AEXURL/files/tutorial/SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5.gz">gzipped file</a>)
\endcode
\section secViewToolsJPSSDeter Determining File Contents
The first thing you may want to do is determine what is in your file. You can use the command-line tools or HDFView to do this:
\li @ref subsecViewToolsJPSSDeter_h5dump
\li @ref subsecViewToolsJPSSDeter_h5ls
\li @ref subsecViewToolsJPSSDeter_HDFView
JPSS NPP files all contain two root level groups:
<table>
<tr>
<th>Group</th><th>Description</th>
</tr>
<tr>
<td>/All_Data
</td>
<td>Contains the raw data and optional geo-location information.
</td>
</tr>
<tr>
<td>/Data_Products
</td>
<td>Contains a dataset ending in <code style="background-color:whitesmoke;">Aggr</code> with
references to objects in the <code style="background-color:whitesmoke;">/All_Data</code> group.
Contains granules (datasets with a name ending in <code style="background-color:whitesmoke;">Gran_#</code>)
with references to selected regions in datasets under <code style="background-color:whitesmoke;">/All_Data</code>.
</td>
</tr>
</table>
\subsection subsecViewToolsJPSSDeter_h5dump h5dump
With <code style="background-color:whitesmoke;">h5dump</code> you can see a list of the objects
in the file using the <code style="background-color:whitesmoke;">-n</code> option:
\code
h5dump -n <file>
\endcode
For example:
\code
$ h5dump -n SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5
HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" {
FILE_CONTENTS {
group /
group /All_Data
group /All_Data/VIIRS-M9-SDR_All
dataset /All_Data/VIIRS-M9-SDR_All/ModeGran
dataset /All_Data/VIIRS-M9-SDR_All/ModeScan
dataset /All_Data/VIIRS-M9-SDR_All/NumberOfBadChecksums
dataset /All_Data/VIIRS-M9-SDR_All/NumberOfDiscardedPkts
dataset /All_Data/VIIRS-M9-SDR_All/NumberOfMissingPkts
dataset /All_Data/VIIRS-M9-SDR_All/NumberOfScans
dataset /All_Data/VIIRS-M9-SDR_All/PadByte1
dataset /All_Data/VIIRS-M9-SDR_All/QF1_VIIRSMBANDSDR
dataset /All_Data/VIIRS-M9-SDR_All/QF2_SCAN_SDR
dataset /All_Data/VIIRS-M9-SDR_All/QF3_SCAN_RDR
dataset /All_Data/VIIRS-M9-SDR_All/QF4_SCAN_SDR
dataset /All_Data/VIIRS-M9-SDR_All/QF5_GRAN_BADDETECTOR
dataset /All_Data/VIIRS-M9-SDR_All/Radiance
dataset /All_Data/VIIRS-M9-SDR_All/RadianceFactors
dataset /All_Data/VIIRS-M9-SDR_All/Reflectance
dataset /All_Data/VIIRS-M9-SDR_All/ReflectanceFactors
group /Data_Products
group /Data_Products/VIIRS-M9-SDR
dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Aggr
dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0
dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_1
dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_2
dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_3
}
}
\endcode
In the output above you can see that there are four granules (ending in
<code style="background-color:whitesmoke;">Gran_#</code>) in the
<code style="background-color:whitesmoke;">/Data_Products/VIIRS-M9-SDR/</code> group.
\subsection subsecViewToolsJPSSDeter_h5ls h5ls
With <code style="background-color:whitesmoke;">h5ls</code> you can see a list of the objects in the
file using the <code style="background-color:whitesmoke;">-lr</code>
options. The <code style="background-color:whitesmoke;">h5ls</code> utility also shows shape and size
(dataspace) information about datasets.
\code
h5ls -lr <file>
\endcode
For example:
\code
$ h5ls -lr SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5
/ Group
/All_Data Group
/All_Data/VIIRS-M9-SDR_All Group
/All_Data/VIIRS-M9-SDR_All/ModeGran Dataset {4/Inf}
/All_Data/VIIRS-M9-SDR_All/ModeScan Dataset {192/Inf}
/All_Data/VIIRS-M9-SDR_All/NumberOfBadChecksums Dataset {192/Inf}
/All_Data/VIIRS-M9-SDR_All/NumberOfDiscardedPkts Dataset {192/Inf}
/All_Data/VIIRS-M9-SDR_All/NumberOfMissingPkts Dataset {192/Inf}
/All_Data/VIIRS-M9-SDR_All/NumberOfScans Dataset {4/Inf}
/All_Data/VIIRS-M9-SDR_All/PadByte1 Dataset {12/Inf}
/All_Data/VIIRS-M9-SDR_All/QF1_VIIRSMBANDSDR Dataset {3072/Inf, 3200/Inf}
/All_Data/VIIRS-M9-SDR_All/QF2_SCAN_SDR Dataset {192/Inf}
/All_Data/VIIRS-M9-SDR_All/QF3_SCAN_RDR Dataset {192/Inf}
/All_Data/VIIRS-M9-SDR_All/QF4_SCAN_SDR Dataset {3072/Inf}
/All_Data/VIIRS-M9-SDR_All/QF5_GRAN_BADDETECTOR Dataset {64/Inf}
/All_Data/VIIRS-M9-SDR_All/Radiance Dataset {3072/Inf, 3200/Inf}
/All_Data/VIIRS-M9-SDR_All/RadianceFactors Dataset {8/Inf}
/All_Data/VIIRS-M9-SDR_All/Reflectance Dataset {3072/Inf, 3200/Inf}
/All_Data/VIIRS-M9-SDR_All/ReflectanceFactors Dataset {8/Inf}
/Data_Products Group
/Data_Products/VIIRS-M9-SDR Group
/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Aggr Dataset {16/Inf}
/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0 Dataset {16/Inf}
/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_1 Dataset {16/Inf}
/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_2 Dataset {16/Inf}
/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_3 Dataset {16/Inf}
\endcode
Note that the <code style="background-color:whitesmoke;">Inf</code> indicates that those datasets are appendable or unlimited in size.
\subsection subsecViewToolsJPSSDeter_HDFView HDFView
If you open the file in HDFView, it will display the file and the root level groups within
it in the TreeView on the left. An HDF5 file is a folder with a "5" in the middle, followed
by the file name. There are two folders (groups) within the JPSS file
(<code style="background-color:whitesmoke;">All_Data/</code> and <code style="background-color:whitesmoke;">Data Products/</code>),
which you can select to see their contents:
<table>
<tr>
<td>
\image html hdfview-tree.png
</td>
</tr>
</table>
If you click twice with the left-mouse button on a folder or group in the TreeView, the contents
of the folder will be listed. If you click twice on an object such as a dataset, a window with
the object's values will be displayed.
Underneath the <code style="background-color:whitesmoke;">VIIRS-M1-SDR</code> folder are what HDF5
calls datasets. The scarlet letter <code style="background-color:whitesmoke;">"A"</code> attached
to the group and datasets under <code style="background-color:whitesmoke;">Data_Products/</code>
indicates that there are attributes associated with them.
\section secViewToolsJPSSView Viewing the User Block
All JPSS files contain a user block in XML with information about the file. The user block is an
optional space allocated at the beginning of an HDF5 file that is not interpreted by the HDF5
library. Its size is a multiple of 512.
Since the user block in JPSS files is stored in ASCII and it is stored at the beginning of an
HDF5 file, you could use a text editor or viewer to examine it. However, there are HDF5 utilities
that can help with this:
<table>
<tr>
<th>Utility</th><th>Description</th>
</tr>
<tr>
<td>h5unjam
</td>
<td>Extracts a user block from an HDF5 file
</td>
</tr>
<tr>
<td>h5dump
</td>
<td>The -B (--superblock) option displays the size of the user block in an HDF5 file
</td>
</tr>
</table>
\subsection subsecViewToolsJPSSView_h5unjam h5unjam
The \ref secViewToolsEditAdd tutorial topic discusses the use of the
<code style="background-color:whitesmoke;">h5jam</code> and <code style="background-color:whitesmoke;">h5unjam</code>
utilities for adding or removing a user block from a file. An input HDF5 file
(<code style="background-color:whitesmoke;">-i</code>), output HDF5 file
(<code style="background-color:whitesmoke;">-o</code>), and user block text file
(<code style="background-color:whitesmoke;">-u</code>) can be specified with these tools. You can use the
<code style="background-color:whitesmoke;">h5unjam</code> tool to extract and view the user block in a JPSS file:
\code
h5unjam -i <Input HDF5 File> -o <Output HDF5 File> -u <User Block File>
\endcode
For example this command will extract the user block into the file UB.xml:
\code
$ h5unjam -i SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5
-o svm09-noUB.h5 -u UB.xml
\endcode
The input HDF5 file remains unchanged. The output HDF5 file will not contain the user block.
The <code style="background-color:whitesmoke;">UB.xml</code> file contains the user block
which can be viewed with a browser.
\subsection subsecViewToolsJPSSView_h5dump h5dump
The h5dump utility has the <code style="background-color:whitesmoke;">-B (--superblock)</code> option for displaying the superblock in an HDF5 file.
The superblock contains information about the file such as the file signature, file consistency flags,
the number of bytes to store addresses and size of an object, as well as the size of the user block:
\code
h5dump -B (--superblock)
\endcode
Below is an example (Unix):
\code
$ h5dump -B -H SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 | more
HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" {
SUPER_BLOCK {
SUPERBLOCK_VERSION 0
FREELIST_VERSION 0
SYMBOLTABLE_VERSION 0
OBJECTHEADER_VERSION 0
OFFSET_SIZE 8
LENGTH_SIZE 8
BTREE_RANK 16
BTREE_LEAF 4
ISTORE_K 32
USER_BLOCK {
USERBLOCK_SIZE 1024
}
}
\endcode
Once you have the size of the user block, you can extract it from the file using system commands.
For example, on Unix platforms you can use the head command-line tool:
\code
head -c <USERBLOCK_SIZE> <JPSS File> >& USERBLOCK.xml
\endcode
There are Unix tools for Windows that may work, such as <a href="http://gnuwin32.sourceforge.net/packages/coreutils.htm">CoreUtils for Windows</a>.
\section secViewToolsJPSSExam Examining a Granule
<ul>
<li>@ref subsecViewToolsJPSSExam_h5dump<br />
<ul>
<li>@ref subsubsecViewToolsJPSSExam_h5dumpRegRef</li>
<li>@ref subsubsecViewToolsJPSSExam_h5dumpQuality</li>
<li>@ref subsubsecViewToolsJPSSExam_h5dumpProps</li>
</ul></li>
<li>@ref subsecViewToolsJPSSExamr_HDFView</li>
</ul>
\subsection subsecViewToolsJPSSExam_h5dump h5dump
There are several options that you may first want to use when examining a granule with h5dump:
<table>
<tr>
<th>Option</th><th>Description</th>
</tr>
<tr>
<td>-H, --header
</td>
<td>Prints header (metadata) information only
</td>
</tr>
<tr>
<td>-d D, --dataset=D
</td>
<td>Specifies the granule dataset
</td>
</tr>
<tr>
<td>-A 0, --onlyattr=0
</td>
<td>Suppresses attributes
</td>
</tr>
<tr>
<td>-p, --properties
</td>
<td>Show properties of datasets
(See Properties)
</td>
</tr>
</table>
You would specify the dataset (<code style="background-color:whitesmoke;">-d D</code>) and the
<code style="background-color:whitesmoke;">-H</code> options to view the metadata associated with
a specific granule. There are many attributes associated with a granule and
<code style="background-color:whitesmoke;">-A 0</code> can be used to suppress those.
For example:
\code
h5dump -H -A 0 -d "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0"
SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5
\endcode
This command displays:
\code
HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" {
DATASET "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0" {
DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) }
}
}
\endcode
To see the actual contents of the granule remove the <code style="background-color:whitesmoke;">-H</code> option:
\code
h5dump -A 0 -d "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0"
SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5
\endcode
The above command displays:
\code
HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" {
DATASET "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0" {
DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) }
DATA {
DATASET /All_Data/VIIRS-M9-SDR_All/Radiance {(0,0)-(767,3199)},
DATASET /All_Data/VIIRS-M9-SDR_All/Reflectance {(0,0)-(767,3199)},
DATASET /All_Data/VIIRS-M9-SDR_All/ModeScan {(0)-(47)},
DATASET /All_Data/VIIRS-M9-SDR_All/ModeGran {(0)-(0)},
DATASET /All_Data/VIIRS-M9-SDR_All/PadByte1 {(0)-(2)},
DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfScans {(0)-(0)},
DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfMissingPkts {(0)-(47)},
DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfBadChecksums {(0)-(47)},
DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfDiscardedPkts {(0)-(47)},
DATASET /All_Data/VIIRS-M9-SDR_All/QF1_VIIRSMBANDSDR {(0,0)-(767,3199)},
DATASET /All_Data/VIIRS-M9-SDR_All/QF2_SCAN_SDR {(0)-(47)},
DATASET /All_Data/VIIRS-M9-SDR_All/QF3_SCAN_RDR {(0)-(47)},
DATASET /All_Data/VIIRS-M9-SDR_All/QF4_SCAN_SDR {(0)-(767)},
DATASET /All_Data/VIIRS-M9-SDR_All/QF5_GRAN_BADDETECTOR {(0)-(15)},
DATASET /All_Data/VIIRS-M9-SDR_All/RadianceFactors {(0)-(1)},
DATASET /All_Data/VIIRS-M9-SDR_All/ReflectanceFactors {(0)-(1)}
}
}
}
\endcode
As you can see in the output above, the datatype for this dataset is:
\code
DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
\endcode
This indicates that it is a dataset specifically for storing references to regions (or subsets)
in other datasets. The dataset contains 16 such references, and more can be added to it, as
indicated by the dataspace (in other words it is unlimited):
\code
DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) }
\endcode
\subsubsection subsubsecViewToolsJPSSExam_h5dumpRegRef Viewing a Region Reference
What if we wanted to look at the <code style="background-color:whitesmoke;">NumberOfScans</code> data for a specific granule in a file?
First, we may be interested in determining whether the scans were done at night or in the day. If a scan was at night, there will be no data.
The attribute <code style="background-color:whitesmoke;">N_Day_Night_Flag</code> is used to determine when the scan was done. If you don't know where this attribute is located, you can use the <code style="background-color:whitesmoke;">-N</code> option to search for it in the file. If you were to run this command on the <code style="background-color:whitesmoke;">SVM09</code> file used above, you would see that the <code style="background-color:whitesmoke;">N_Day_Night_Flag</code> attribute has a value of <code style="background-color:whitesmoke;">Night</code> for the four granules in the file. Indeed, if you actually examine the <code style="background-color:whitesmoke;">NumberOfScans</code> data, you will see that only fill values are written.
For that reason we will examine the <code style="background-color:whitesmoke;">NumberOfScans</code> data for the <code style="background-color:whitesmoke;">SVMO1</code> file below, as it was obtained during the day:
\code
h5dump -N N_Day_Night_Flag SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
\endcode
It displays:
\code
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
ATTRIBUTE "N_Day_Night_Flag" {
DATATYPE H5T_STRING {
STRSIZE 4;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1, 1 ) / ( 1, 1 ) }
DATA {
(0,0): "Day"
}
}
}
\endcode
There is just one granule in this <code style="background-color:whitesmoke;">SVM01</code> file, as shown below:
\code
$ h5dump -n SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
FILE_CONTENTS {
group /
group /All_Data
group /All_Data/VIIRS-M1-SDR_All
dataset /All_Data/VIIRS-M1-SDR_All/ModeGran
dataset /All_Data/VIIRS-M1-SDR_All/ModeScan
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfBadChecksums
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfDiscardedPkts
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfMissingPkts
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfScans
dataset /All_Data/VIIRS-M1-SDR_All/PadByte1
dataset /All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR
dataset /All_Data/VIIRS-M1-SDR_All/QF2_SCAN_SDR
dataset /All_Data/VIIRS-M1-SDR_All/QF3_SCAN_RDR
dataset /All_Data/VIIRS-M1-SDR_All/QF4_SCAN_SDR
dataset /All_Data/VIIRS-M1-SDR_All/QF5_GRAN_BADDETECTOR
dataset /All_Data/VIIRS-M1-SDR_All/Radiance
dataset /All_Data/VIIRS-M1-SDR_All/RadianceFactors
dataset /All_Data/VIIRS-M1-SDR_All/Reflectance
dataset /All_Data/VIIRS-M1-SDR_All/ReflectanceFactors
group /Data_Products
group /Data_Products/VIIRS-M1-SDR
dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Aggr
dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0
}
}
\endcode
Now examine the references in the <code style="background-color:whitesmoke;">VIIRS-M1-SDR_Gran_0</code> granule
\code
$ h5dump -A 0 -d "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0"
SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
DATASET "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" {
DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) }
DATA {
DATASET /All_Data/VIIRS-M1-SDR_All/Radiance {(0,0)-(767,3199)},
DATASET /All_Data/VIIRS-M1-SDR_All/Reflectance {(0,0)-(767,3199)},
DATASET /All_Data/VIIRS-M1-SDR_All/ModeScan {(0)-(47)},
DATASET /All_Data/VIIRS-M1-SDR_All/ModeGran {(0)-(0)},
DATASET /All_Data/VIIRS-M1-SDR_All/PadByte1 {(0)-(2)},
DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfScans {(0)-(0)},
DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfMissingPkts {(0)-(47)},
DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfBadChecksums {(0)-(47)},
DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfDiscardedPkts {(0)-(47)},
DATASET /All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR {(0,0)-(767,3199)},
DATASET /All_Data/VIIRS-M1-SDR_All/QF2_SCAN_SDR {(0)-(47)},
DATASET /All_Data/VIIRS-M1-SDR_All/QF3_SCAN_RDR {(0)-(47)},
DATASET /All_Data/VIIRS-M1-SDR_All/QF4_SCAN_SDR {(0)-(767)},
DATASET /All_Data/VIIRS-M1-SDR_All/QF5_GRAN_BADDETECTOR {(0)-(15)},
DATASET /All_Data/VIIRS-M1-SDR_All/RadianceFactors {(0)-(1)},
DATASET /All_Data/VIIRS-M1-SDR_All/ReflectanceFactors {(0)-(1)}
}
}
}
\endcode
In the output above, you can see that the <code style="background-color:whitesmoke;">NumberOfScans</code>
reference is the sixth reference in the granule counting from the top.
The list of references shown above is a 0-based index to the dataset. Therefore, to specify
<code style="background-color:whitesmoke;">NumberOfScans</code>, enter a start offset of
<code style="background-color:whitesmoke;">5</code> for the <code style="background-color:whitesmoke;">-s</code>
option (the sixth reference minus 1). To see the region reference data, use the <code style="background-color:whitesmoke;">-R</code> option.
This command will display the data in the <code style="background-color:whitesmoke;">NumberOfScans</code> region reference:
\code
h5dump -A 0 -d "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" -s 5 -R
SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
\endcode
It displays the number of scans (48):
\code
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
DATASET "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" {
DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) }
SUBSET {
START ( 5 );
STRIDE ( 1 );
COUNT ( 1 );
BLOCK ( 1 );
DATA {
(5): DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfScans {
(5): REGION_TYPE BLOCK (0)-(0)
(5): DATATYPE H5T_STD_I32BE
(5): DATASPACE SIMPLE { ( 1 ) / ( H5S_UNLIMITED ) }
(5): DATA {
(0): 48
(5): }
(5): }
}
}
}
}
\endcode
The <code style="background-color:whitesmoke;">-s</code> option may be familiar as one of the options
that was described in the \ref secViewToolsViewSub tutorial topic. The other subsetting options are not included,
indicating that the default values are used.
If you leave off the <code style="background-color:whitesmoke;">-R</code> option, you will see the subset selection, but not the data:
\code
$ h5dump -A 0 -d "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" -s 5
SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
DATASET "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" {
DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) }
SUBSET {
START ( 5 );
STRIDE ( 1 );
COUNT ( 1 );
BLOCK ( 1 );
DATA {
DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfScans {(0)-(0)}
}
}
}
}
\endcode
\subsubsection subsubsecViewToolsJPSSExam_h5dumpQuality Viewing a Quality Flag
The quality flags in an NPP file can be viewed with h5dump using the <code style="background-color:whitesmoke;">-M</code>
option. Quality flags are packed into each integer value in a quality flag dataset. Quality flag datasets in NPP
files begin with the letters <code style="background-color:whitesmoke;">QF</code>.
In the following NPP file, there are five Quality Flag datasets
(<code style="background-color:whitesmoke;">/All_Data/VIIRS-M1-SDR_All/QF*</code>):
\code
$ h5dump -n SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
FILE_CONTENTS {
group /
group /All_Data
group /All_Data/VIIRS-M1-SDR_All
dataset /All_Data/VIIRS-M1-SDR_All/ModeGran
dataset /All_Data/VIIRS-M1-SDR_All/ModeScan
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfBadChecksums
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfDiscardedPkts
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfMissingPkts
dataset /All_Data/VIIRS-M1-SDR_All/NumberOfScans
dataset /All_Data/VIIRS-M1-SDR_All/PadByte1
dataset /All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR
dataset /All_Data/VIIRS-M1-SDR_All/QF2_SCAN_SDR
dataset /All_Data/VIIRS-M1-SDR_All/QF3_SCAN_RDR
dataset /All_Data/VIIRS-M1-SDR_All/QF4_SCAN_SDR
dataset /All_Data/VIIRS-M1-SDR_All/QF5_GRAN_BADDETECTOR
dataset /All_Data/VIIRS-M1-SDR_All/Radiance
dataset /All_Data/VIIRS-M1-SDR_All/RadianceFactors
dataset /All_Data/VIIRS-M1-SDR_All/Reflectance
dataset /All_Data/VIIRS-M1-SDR_All/ReflectanceFactors
group /Data_Products
group /Data_Products/VIIRS-M1-SDR
dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Aggr
dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0
}
}
\endcode
The flags in this particular dataset happen to be stored in every two bits of each quality flag dataset
element, and the values range from 0 to 2. In other words, to see the quality flag values for this
dataset, these bits would be examined: 0 and 1, 2 and 3, 4 and 5, or 6 and 7 (This information was
obtained from the Product Profile XML File.)
For example, bits 0 and 1 in the <code style="background-color:whitesmoke;">VQF1_VIIRSMBANDSDR</code> dataset specify the flag that
"Indicates calibration quality due to bad space view offsets, OBC view offsets, etc or use of a
previous calibration view". It has 3 values: Good (0), Poor (1), or No Calibration (2).
The <code style="background-color:whitesmoke;">-M</code> option is used to specify the quality
flag bit offset (<code style="background-color:whitesmoke;">O</code>) and length (<code style="background-color:whitesmoke;">L</code>):
\code
h5dump -d DATASET -M O,L FILE
\endcode
To view the first quality flag (0-1) in a 5 x 6 subset of the <code style="background-color:whitesmoke;">QF1_VIIRSMBANDSDR</code> dataset, specify:
\code
h5dump -d "/All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR[0,0;;5,6;]"
-M 0,2 SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
\endcode
This outputs:
\code
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
DATASET "/All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR" {
DATATYPE H5T_STD_U8BE
DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) }
PACKED_BITS OFFSET=0 LENGTH=2
SUBSET {
START ( 0, 0 );
STRIDE ( 1, 1 );
COUNT ( 5, 6 );
BLOCK ( 1, 1 );
DATA {
(0,0): 2, 2, 2, 2, 2, 2,
(1,0): 2, 2, 2, 2, 2, 2,
(2,0): 0, 0, 0, 0, 0, 0,
(3,0): 0, 0, 0, 0, 0, 0,
(4,0): 0, 0, 0, 0, 0, 0
}
}
}
}
\endcode
To view more than one quality flag at a time simply add the bit offset and length values to
<code style="background-color:whitesmoke;">-M</code>, separated by commas. For example, this
<code style="background-color:whitesmoke;">-M</code> option specifies bits 0-1 and 2-3:
\code
h5dump -d DATASET -M 0,2,2,2 FILE
\endcode
\subsubsection subsubsecViewToolsJPSSExam_h5dumpProps Properties
To view properties of a specific dataset with <code style="background-color:whitesmoke;">h5dump</code>
use the <code style="background-color:whitesmoke;">-p</code> option along with the
<code style="background-color:whitesmoke;">-d</code> option. Depending on the number of attributes
and the amount of data, the <code style="background-color:whitesmoke;">-A 0</code> and
<code style="background-color:whitesmoke;">-H</code> options can also be specified to suppress
printing of attributes and data values:
\code
h5dump -p -H -A 0 -d DATASET
\endcode
The <code style="background-color:whitesmoke;">-p</code> option shows any compression filters
associated with a dataset, as well as layout and fill value information. This option can be helpful
in diagnosing performance and other issues.
As an example, examine the <code style="background-color:whitesmoke;">/All_Data/VIIRS-M1-SDR_All/Radiance</code>
dataset in the <code style="background-color:whitesmoke;">SVM01</code> file:
\code
$ h5dump -p -H -A 0 -d "/All_Data/VIIRS-M1-SDR_All/Radiance"
SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" {
DATASET "/All_Data/VIIRS-M1-SDR_All/Radiance" {
DATATYPE H5T_STD_U16BE
DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) }
STORAGE_LAYOUT {
CHUNKED ( 768, 3200 )
SIZE 4915200
}
FILTERS {
NONE
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_IFSET
VALUE 65529
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
\endcode
We can see that the chunk size for this dataset is 768 x 3200, and the storage size is 4915200.
What if the chunk size were smaller?
The dataset was modified to have a chunk size of 1 x 10, using the
<code style="background-color:whitesmoke;">h5repack</code> utility, as shown below.
\code
$ h5repack -l /All_Data/VIIRS-M1-SDR_All/Radiance:CHUNK=1x10
SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 SVM01repack.h5
$ h5dump -p -H -A 0 -d "/All_Data/VIIRS-M1-SDR_All/Radiance" SVM01repack.h5
HDF5 "SVM01repack.h5" {
DATASET "/All_Data/VIIRS-M1-SDR_All/Radiance" {
DATATYPE H5T_STD_U16BE
DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) }
STORAGE_LAYOUT {
CHUNKED ( 1, 10 )
SIZE 4915200
}
FILTERS {
NONE
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_IFSET
VALUE 65529
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
\endcode
In this case, the storage size of the dataset is the same, but the size of the file almost doubled!:
\code
$ ls -1sh
total 35M
12M SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5
23M SVM01repack.h5
\endcode
In general, the smaller the chunk size, the more chunks that HDF5 has to keep track of, which increases
the size of the file and can affect performance.
\subsection subsecViewToolsJPSSExamr_HDFView HDFView
As mentioned previously, the structure of an HDF5 file is displayed in the TreeView on the left side of the HDFView screen,
and you can click on objects and have metadata information displayed on the right side.
To discover more about the granule <code style="background-color:whitesmoke;">/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0</code>
in the <code style="background-color:whitesmoke;">SVM01</code> file shown below in the TreeView, position
the mouse over the granule and click to select. Properties for the object is displayed on the right side of the HDFView screen.
You can see Datatype and Dataspace information on the <code style="background-color:whitesmoke;">General Object Info</code>
tab, any Attributes associated with the granulewill be on the
<code style="background-color:whitesmoke;">Object Attribute Info</code> tab. In the
<code style="background-color:whitesmoke;">General Object Info</code>, you can see that the dataset is a
Region Reference dataset, and that there are sixteen Region References in this dataset:
<table>
<tr>
<td>
\image html hdfview-prop.png
</td>
</tr>
</table>
To examine the data in the granule, click twice on it with the left mouse button in the TreeView,
and it will open in a new window.:
<table>
<tr>
<td>
\image html hdfview-regref.png
</td>
</tr>
</table>
If you click twice with the left mouse button on the fifth Region Reference
<code style="background-color:whitesmoke;">/All_Data/VIIRS-M1-SDR_All/NumberOfScans</code> a window
will pop up with the value(s) of the reference:
<table>
<tr>
<td>
\image html hdfview-regref2.png
</td>
</tr>
</table>
You can also set a user option to automatically show the value(s) in a Region Reference. Under the
<code style="background-color:whitesmoke;">Tools</code> pull-down menu, select
<code style="background-color:whitesmoke;">User Options</code> and then select
<code style="background-color:whitesmoke;">HDF Settings</code> and then select
<code style="background-color:whitesmoke;">Show RegRef Values</code> in the
<code style="background-color:whitesmoke;">Data</code> section (see the middle of the image below):
<table>
<tr>
<td>
\image html hdfview-regrefval.png
</td>
</tr>
</table>
Then you will automatically see the values of the Region Reference when you open it and select an entry:
<table>
<tr>
<td>
\image html hdfview-regref1.png
</td>
</tr>
</table>
You can view and set quality flags by clicking the right mouse button over a quality flags dataset under
<code style="background-color:whitesmoke;">All_Data</code> and selecting
<code style="background-color:whitesmoke;">Open As</code> from the pop-up menu. In the middle of
the window that pops up, you will see where you can specify <code style="background-color:whitesmoke;">Bitmask</code> options.
<table>
<tr>
<td>
\image html hdfview-qf.png
</td>
</tr>
</table>
<hr>
Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand
*/
Reference in New Issue View Git Blame Copy Permalink