mirror of
https://github.com/Unidata/netcdf-c.git
synced 2024-11-21 03:13:42 +08:00
Initial copy of files into NUG directory.
This commit is contained in:
Ward Fisher
parent
3aaf092b59
commit
e6708aaa82
259
NUG/DAP4.dox
Normal file
259
NUG/DAP4.dox
Normal file
@ -0,0 +1,259 @@
|
||||
/*!
|
||||
\page dap4 DAP4 Protocol Support
|
||||
|
||||
|
||||
<!-- Note that this file has the .dox extension, but is mostly markdown -->
|
||||
<!-- Begin MarkDown -->
|
||||
|
||||
\tableofcontents
|
||||
|
||||
# DAP4 Introduction {#dap4_introduction}
|
||||
|
||||
Beginning with netCDF version 4.5.0, optional support is provided for
|
||||
accessing data from servers supporting the DAP4 protocol.
|
||||
|
||||
DAP4 support is enabled if the _--enable-dap_ option
|
||||
is used with './configure'. If DAP4 support is enabled, then
|
||||
a usable version of _libcurl_ must be specified
|
||||
using the _LDFLAGS_ environment variable (similar to the way
|
||||
that the _HDF5_ libraries are referenced).
|
||||
Refer to the installation manual for details.
|
||||
By default DAP4 support is enabled if _libcurl_ is found.
|
||||
DAP4 support can be disabled using the _--disable-dap_.
|
||||
|
||||
DAP4 uses a data model that is, by design, similar to,
|
||||
but -- for historical reasons -- not identical with,
|
||||
the netCDF-Enhanced (aka netcdf-4) model.
|
||||
Generically, the DAP4 data model is encoded in XML document
|
||||
called a DMR.
|
||||
For detailed information about the DAP4 DMR, refer to
|
||||
the DAP4 specification Volume 1:
|
||||
http://docs.opendap.org/index.php/DAP4:_Specification_Volume_1
|
||||
|
||||
# Accessing Data Using the DAP4 Prototocol {#dap4_accessing_data}
|
||||
|
||||
In order to access a DAP4 data source through the netCDF API, the
|
||||
file name normally used is replaced with a URL with a specific
|
||||
format. The URL is composed of three parts.
|
||||
- URL - this is a standard form URL with specific markers to indicate that
|
||||
it refers to a DAP4 encoded dataset. The markers are of the form
|
||||
"dap4", "mode=dap4", or "/thredds/dap4". The following
|
||||
examples show how they are specified. Note that the "/thredds/dap4"
|
||||
case will only work when accessing a Thredds-based server.
|
||||
+ [dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
|
||||
+ [mode=dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
|
||||
+ http://remotetest.unidata.ucar.edu/d4ts/test.01#dap4
|
||||
+ http://remotetest.unidata.ucar.edu/d4ts/test.01#mode=dap4
|
||||
+ http://thredds.ucar.edu/thredds/dap4/...
|
||||
|
||||
- Constraints - these are suffixed to the URL and take the form
|
||||
“?dap4.ce=\<expression\>”. The form of the constraint expression
|
||||
is somewhat complicated, and the specification should be consulted.
|
||||
|
||||
- Client parameters - these may be specified in either of
|
||||
two ways. The older, deprecated form prefixes text to the
|
||||
front of the url and is of the the general form [\<name>]
|
||||
or [\<name>=value]. Examples include [show=fetch] or [noprefetch].
|
||||
The newer, preferred form prefixes the
|
||||
parameters to the end of the url using the semi-standard '#'
|
||||
format: e.g. http://....#show=fetch&noprefetch.
|
||||
|
||||
It is possible to see what the translation does to a particular DAP4
|
||||
data source in two steps. First, one can examine the DMR
|
||||
source through a web browser and then second, one can examine
|
||||
the translation using the "ncdump -h" command to see the
|
||||
corresponding netCDF-4 representation.
|
||||
|
||||
For example, if a web browser is given the following (fictional) URL,
|
||||
it will return the DMR for the specified dataset (in XML format).
|
||||
\code
|
||||
http://remotetest.unidata.ucar.edu/d4ts/test.01.dmr.xml#dap4
|
||||
\endcode
|
||||
|
||||
By using the following ncdump command, it is possible to see the
|
||||
equivalent netCDF-4 translation.
|
||||
|
||||
\code
|
||||
ncdump -h '[dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01'
|
||||
\endcode
|
||||
|
||||
# Defined Client Parameters {#dap4_defined_params}
|
||||
|
||||
Currently, a limited set of client parameters is
|
||||
recognized. Parameters not listed here are
|
||||
ignored, but no error is signalled.
|
||||
|
||||
Parameter Name Legal Values Semantics
|
||||
- "log" | "log=<file>" - Turn on logging and send the log output to
|
||||
the specified file. If no file is specified, then output is sent to standard
|
||||
error.
|
||||
- "show=fetch" - This parameter causes the netCDF code to log a copy
|
||||
of the complete url for every HTTP get request. If logging is
|
||||
enabled, then this can be helpful in checking to see the access
|
||||
behavior of the netCDF code.
|
||||
- "translate=nc4" - This parameter causes the netCDF code to look
|
||||
for specially named elements in the DMR XML in order to
|
||||
achieve a better translation of the DAP4 meta-data to NetCDF enhanced
|
||||
metadata.
|
||||
- "opaquesize=<integer>" - This parameter causes the netCDF code to
|
||||
convert DAP4 variable size OPAQUE objects to netcdf-4 fixed size
|
||||
objects and forces all of them to be of the size specified.
|
||||
- "fillmismatch" - Unfortunately, a number of servers sometimes
|
||||
fail to make sure that the type of the "_FillValue" attribute of a variable
|
||||
is the same as the type of the containing variable. Setting this tag
|
||||
caused the netcdf translation to attempt to fix this mismatch. If not set,
|
||||
then an error will occur.
|
||||
|
||||
# Notes on Debugging DAP4 Access {#dap4_debug}
|
||||
|
||||
The DAP4 support has a logging facility.
|
||||
Turning on this logging can
|
||||
sometimes give important information. Logging can be enabled by
|
||||
using the client parameter "log" or "log=filename",
|
||||
where the first case will send log output to standard error and the
|
||||
second will send log output to the specified file.
|
||||
|
||||
Users should also be aware that if one is
|
||||
accessing data over an NFS mount, one may see some .nfsxxxxx files;
|
||||
those can be ignored.
|
||||
|
||||
## HTTP Configuration. {#dap4_http2_config}
|
||||
|
||||
Limited support for configuring the http connection is provided via
|
||||
parameters in the “.daprc” configuration file (aka ".dodsrc").
|
||||
The relevant .daprc file is
|
||||
located by first looking in the current working directory, and if not
|
||||
found, then looking in the directory specified by the “$HOME”
|
||||
environment variable.
|
||||
|
||||
Entries in the .daprc file are of the form:
|
||||
````
|
||||
['['<url>']']<key>=<value>
|
||||
````
|
||||
|
||||
That is, it consists of a key name and value pair and optionally
|
||||
preceded by a url enclosed in square brackets.
|
||||
|
||||
For given KEY and URL strings, the value chosen is as follows:
|
||||
|
||||
If URL is null, then look for the .daprc entry that has no url prefix
|
||||
and whose key is same as the KEY for which we are looking.
|
||||
|
||||
If the URL is not null, then look for all the .daprc entries that
|
||||
have a url, URL1, say, and for which URL1 has the same host and port
|
||||
as URL. All parts of the url's except host and port are ignored.
|
||||
For example, if URL = http//x.y/a, then it will match
|
||||
entries of the form
|
||||
_[http//x.y/a]KEY=VALUE_ or _[http//x.y/b]KEY=VALUE_.
|
||||
It will not match an entry of the form _[http//x.y:8080]KEY=VALUE
|
||||
because the second has a port number (8080) different than the URL.
|
||||
Finally from the set so constructed, choose the first matching entry.
|
||||
|
||||
Currently, the supported set of keys (with descriptions) are as
|
||||
follows.
|
||||
|
||||
-# HTTP.VERBOSE
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Produce verbose output, especially using SSL.
|
||||
Related CURL Flags: CURLOPT_VERBOSE
|
||||
|
||||
-# HTTP.DEFLATE
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Allow use of compression by the server.
|
||||
Related CURL Flags: CURLOPT_ENCODING
|
||||
|
||||
-# HTTP.COOKIEJAR
|
||||
Type: String representing file path
|
||||
Description: Specify the name of file into which to store cookies. Defaults to in-memory storage.
|
||||
Related CURL Flags:CURLOPT_COOKIEJAR
|
||||
|
||||
-# HTTP.CREDENTIALS.USER
|
||||
Type: String representing user name
|
||||
Description: Specify the user name for Digest and Basic authentication.
|
||||
Related CURL Flags:
|
||||
|
||||
-# HTTP.CREDENTIALS.PASSWORD
|
||||
Type: String representing password
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Specify the password for Digest and Basic authentication.
|
||||
Related CURL Flags:
|
||||
|
||||
-# HTTP.SSL.CERTIFICATE
|
||||
Type: String representing file path
|
||||
Description: Path to a file containing a PEM cerficate.
|
||||
Related CURL Flags: CURLOPT_CERT
|
||||
|
||||
-# HTTP.SSL.KEY
|
||||
Type: String representing file path
|
||||
Description: Same as HTTP.SSL.CERTIFICATE, and should usually have the same value.
|
||||
Related CURL Flags: CURLOPT_SSLKEY
|
||||
|
||||
-# HTTP.SSL.KEYPASSWORD
|
||||
Type: String representing password
|
||||
Description: Password for accessing the HTTP.SSL.KEY/HTTP.SSL.CERTIFICATE
|
||||
Related CURL Flags: CURLOPT_KEYPASSWORD
|
||||
|
||||
-# HTTP.SSL.CAPATH
|
||||
Type: String representing directory
|
||||
Description: Path to a directory containing trusted certificates for validating server certificates.
|
||||
Related CURL Flags: CURLOPT_CAPATH
|
||||
|
||||
-# HTTP.SSL.VALIDATE
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Cause the client to verify the server's presented certificate.
|
||||
Related CURL Flags: CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST
|
||||
|
||||
-# HTTP.TIMEOUT
|
||||
Type: String ("dddddd")
|
||||
Description: Specify the maximum time in seconds that you allow the http transfer operation to take.
|
||||
Related CURL Flags: CURLOPT_TIMEOUT, CURLOPT_NOSIGNAL, CURLOPT_CONNECTIONTIMEOUT
|
||||
|
||||
-# HTTP.PROXY_SERVER
|
||||
Type: String representing url to access the proxy: (e.g.http://[username:password@]host[:port])
|
||||
Description: Specify the needed information for accessing a proxy.
|
||||
Related CURL Flags: CURLOPT_PROXY, CURLOPT_PROXYHOST, CURLOPT_PROXYUSERPWD
|
||||
|
||||
-# HTTP.READ.BUFFERSIZE
|
||||
Type: String ("dddddd")
|
||||
Description: Specify the the internal buffer size for curl reads.
|
||||
Related CURL Flags: CURLOPT_BUFFERSIZE, CURL_MAX_WRITE_SIZE (16kB),
|
||||
CURL_MAX_READ_SIZE (512kB).
|
||||
|
||||
-# HTTP.KEEPALIVE
|
||||
Type: String ("on|n/m")
|
||||
Description: Specify that TCP KEEPALIVE should be enabled and that the associated idle wait time is n and that the associated repeat interval is m. If the value is of the form is the string "on", then turn on keepalive, but do not set idle or interval.
|
||||
Related CURL Flags: CURLOPT_TCP_KEEPALIVE, CURLOPT_TCP_KEEPIDLE,
|
||||
CURLOPT_TCP_KEEPINTVL.
|
||||
|
||||
-# HTTP.CONNECTIONTIMEOUT
|
||||
Type: String ("dddddd")
|
||||
Description: Specify the maximum time in seconds that you allow the http connection to complete.
|
||||
Related CURL Flags: CURLOPT_TIMEOUT, CURLOPT_NOSIGNAL
|
||||
|
||||
The related curl flags line indicates the curl flags modified by this
|
||||
key. See the libcurl documentation of the _curl_easy_setopt()_ function
|
||||
for more detail (http://curl.haxx.se/libcurl/c/curl_easy_setopt.html).
|
||||
|
||||
For servers that require client authentication, as a rule, the following
|
||||
.daprc entries should be specified.
|
||||
````
|
||||
HTTP.SSL.VALIDATE
|
||||
HTTP.COOKIEJAR
|
||||
HTTP.SSL.CERTIFICATE
|
||||
HTTP.SSL.KEY
|
||||
HTTP.SSL.CAPATH
|
||||
````
|
||||
|
||||
Additionally, the _HTTP.SSL.CERTIFICATE_ and _HTTP.SSL.KEY_
|
||||
entries should have same value, which is the file path for a
|
||||
certificate. The HTTP.SSL.CAPATH entry should
|
||||
be the path to a directory containing validation "certificates".
|
||||
|
||||
# Point of Contact {#dap4_poc}
|
||||
|
||||
__Author__: Dennis Heimbigner<br>
|
||||
__Email__: dmh at ucar dot edu<br>
|
||||
__Initial Version__: 6/5/2017<br>
|
||||
__Last Revised__: 11/7/2018
|
||||
|
||||
*/
|
||||
2489
NUG/Doxyfile
Normal file
2489
NUG/Doxyfile
Normal file
File diff suppressed because it is too large
Load Diff
730
NUG/OPeNDAP.dox
Normal file
730
NUG/OPeNDAP.dox
Normal file
@ -0,0 +1,730 @@
|
||||
/*!
|
||||
\page dap2 DAP2 Support
|
||||
|
||||
\tableofcontents
|
||||
|
||||
\section dap_introduction DAP2 (OPeNDAP) Introduction
|
||||
|
||||
Beginning with netCDF version 4.1, optional support is provided for
|
||||
accessing data through OPeNDAP servers using the DAP protocol.
|
||||
Currently, only DAP protocol version 2 is supported; DAP protocol
|
||||
version 4 support is under development.
|
||||
|
||||
DAP support is automatically enabled if a usable curl library can be
|
||||
set using the LDFLAGS environment variable (similar to the way
|
||||
that the HDF5 libraries are referenced).
|
||||
DAP support can forcibly be enabled or disabled using the --enable-dap
|
||||
flag or the --disable-dap flag, respectively. If enabled,
|
||||
then DAP2 support requires access to the curl library.
|
||||
Refer to the installation manual for details
|
||||
|
||||
DAP uses a data model that is different from that supported by netCDF,
|
||||
either classic or enhanced. Generically, the DAP data model is encoded
|
||||
textually in a DDS (Dataset Descriptor Structure). There is a second
|
||||
data model for DAP attributes, which is encoded textually in a DAS
|
||||
(Dataset Attribute Structure). For detailed information about the DAP
|
||||
DDS and DAS, refer to the OPeNDAP web site http://opendap.org.
|
||||
|
||||
\subsection dap_dap_information OPeNDAP Documentation
|
||||
|
||||
See the following pages for more information.
|
||||
|
||||
- \subpage dap_accessing_data
|
||||
- <a href="auth.md">netCDF Authorization Support</a>
|
||||
- \subpage dap_to_netcdf
|
||||
- \subpage dap2_reserved_keywords
|
||||
- \subpage var_dim_trans
|
||||
- \subpage var_name_trans
|
||||
|
||||
\section dap_accessing_data Accessing OPeNDAP Data
|
||||
|
||||
In order to access an OPeNDAP data source through the netCDF API, the
|
||||
file name normally used is replaced with a URL with a specific
|
||||
format. The URL is composed of three parts.
|
||||
- URL - this is a standard form URL such as
|
||||
http://remotetest.unidata.ucar.edu/dts/test.01
|
||||
|
||||
- Constraints - these are suffixed to the URL and take the form
|
||||
“?\<projections>&\<selections>”. The meaning of the terms projection
|
||||
and selection is somewhat complicated; and the OPeNDAP web site,
|
||||
http://www.opendap.org , should be consulted. The interaction of DAP
|
||||
constraints with netCDF is complex and at the moment requires an
|
||||
understanding of how DAP is translated to netCDF.
|
||||
|
||||
- Client parameters - these may be specified in either of
|
||||
two ways. The older, deprecated form prefixes text to the
|
||||
front of the url and is of the the general form [\<name>]
|
||||
or [\<name>=value]. Examples include [show=fetch] and
|
||||
[noprefetch]. The newer, preferred form prefixes the
|
||||
parameters to the end of the url using the semi-standard '#'
|
||||
format: e.g. http://....#show=fetch&noprefetch.
|
||||
|
||||
It is possible to see what the translation does to a particular DAP
|
||||
data source in either of two ways. First, one can examine the DDS
|
||||
source through a web browser and then examine the translation using
|
||||
the ncdump -h command to see the netCDF Classic translation. The
|
||||
ncdump output will actually be the union of the DDS with the DAS, so
|
||||
to see the complete translation, it is necessary to view both.
|
||||
|
||||
For example, if a web browser is given the following, the first URL
|
||||
will return the DDS for the specified dataset, and the second URL will
|
||||
return the DAS for the specified dataset.
|
||||
|
||||
\code
|
||||
http://remotetest.unidata.ucar.edu/dts/test.01.dds
|
||||
http://remotetest.unidata.ucar.edu/dts/test.01.das
|
||||
\endcode
|
||||
|
||||
Then by using the following ncdump command, it is possible to see the
|
||||
equivalent netCDF Classic translation.
|
||||
|
||||
\code
|
||||
ncdump -h http://remotetest.unidata.ucar.edu/dts/test.01
|
||||
\endcode
|
||||
|
||||
The DDS output from the web server should look like this.
|
||||
|
||||
\code
|
||||
Dataset {
|
||||
Byte b;
|
||||
Int32 i32;
|
||||
UInt32 ui32;
|
||||
Int16 i16;
|
||||
UInt16 ui16;
|
||||
Float32 f32;
|
||||
Float64 f64;
|
||||
String s;
|
||||
Url u;
|
||||
} SimpleTypes;
|
||||
\endcode
|
||||
|
||||
The DAS output from the web server should look like this.
|
||||
|
||||
\code
|
||||
Attributes {
|
||||
Facility {
|
||||
String PrincipleInvestigator ``Mark Abbott'', ``Ph.D'';
|
||||
String DataCenter ``COAS Environmental Computer Facility'';
|
||||
String DrifterType ``MetOcean WOCE/OCM'';
|
||||
}
|
||||
b {
|
||||
String Description ``A test byte'';
|
||||
String units ``unknown'';
|
||||
}
|
||||
i32 {
|
||||
String Description ``A 32 bit test server int'';
|
||||
String units ``unknown'';
|
||||
}
|
||||
}
|
||||
\endcode
|
||||
|
||||
The output from ncdump should look like this.
|
||||
|
||||
\code
|
||||
netcdf test {
|
||||
dimensions:
|
||||
stringdim64 = 64 ;
|
||||
variables:
|
||||
byte b ;
|
||||
b:Description = "A test byte" ;
|
||||
b:units = "unknown" ;
|
||||
int i32 ;
|
||||
i32:Description = "A 32 bit test server int" ;
|
||||
i32:units = "unknown" ;
|
||||
int ui32 ;
|
||||
short i16 ;
|
||||
short ui16 ;
|
||||
float f32 ;
|
||||
double f64 ;
|
||||
char s(stringdim64) ;
|
||||
char u(stringdim64) ;
|
||||
}
|
||||
\endcode
|
||||
|
||||
Note that the fields of type String and type URL have suddenly
|
||||
acquired a dimension. This is because strings are translated to arrays
|
||||
of char, which requires adding an extra dimension. The size of the
|
||||
dimension is determined in a variety of ways and can be specified. It
|
||||
defaults to 64 and when read, the underlying string is either padded
|
||||
or truncated to that length.
|
||||
|
||||
Also note that the Facility attributes do not appear in the
|
||||
translation because they are neither global nor associated with a
|
||||
variable in the DDS.
|
||||
|
||||
Alternately, one can get the text of the DDS as a global attribute by
|
||||
using the client parameters mechanism . In this case, the parameter
|
||||
“show=dds” can be used, and the data retrieved using
|
||||
the following command
|
||||
|
||||
\code
|
||||
ncdump -h http://remotetest.unidata.ucar.edu/dts/test.01.dds#show=dds
|
||||
\endcode
|
||||
|
||||
The ncdump -h command will then show both the translation and the
|
||||
original DDS. In the above example, the DDS would appear as the global
|
||||
attribute “_DDS” as follows.
|
||||
|
||||
\code
|
||||
netcdf test {
|
||||
...
|
||||
variables:
|
||||
:_DDS = "Dataset { Byte b; Int32 i32; UInt32 ui32; Int16 i16;
|
||||
UInt16 ui16; Float32 f32; Float64 f64;
|
||||
Strings; Url u; } SimpleTypes;"
|
||||
|
||||
byte b ;
|
||||
...
|
||||
}
|
||||
\endcode
|
||||
|
||||
\section dap_to_netcdf DAP to NetCDF Translation Rules
|
||||
|
||||
Currently only one translation available: DAP 2 Protocol to netCDF-3.
|
||||
There used to be a DAP 2 Protocol to netCDF-4 translation
|
||||
but that has been removed until the DAP4 protocol is available.
|
||||
|
||||
\subsection nc3_trans_rules netCDF-3 Translation Rules
|
||||
|
||||
The current default translation code translates the OPeNDAP protocol
|
||||
to netCDF-3 (classic). This netCDF-3 translation converts an OPeNDAP
|
||||
DAP protocol version 2 DDS to netCDF-3 and is designed to mimic as
|
||||
closely as possible the translation provided by the libnc-dap
|
||||
system, except that some errors in that older translation have
|
||||
been fixed.
|
||||
|
||||
For illustrative purposes, the following example will be used.
|
||||
|
||||
\code
|
||||
Dataset {
|
||||
Int32 f1;
|
||||
Structure {
|
||||
Int32 f11;
|
||||
Structure {
|
||||
Int32 f1[3];
|
||||
Int32 f2;
|
||||
} FS2[2];
|
||||
} S1;
|
||||
Structure {
|
||||
Grid {
|
||||
Array:
|
||||
Float32 temp[lat=2][lon=2];
|
||||
Maps:
|
||||
Int32 lat[lat=2];
|
||||
Int32 lon[lon=2];
|
||||
} G1;
|
||||
} S2;
|
||||
Grid {
|
||||
Array:
|
||||
Float32 G2[lat=2][lon=2];
|
||||
Maps:
|
||||
Int32 lat[2];
|
||||
Int32 lon[2];
|
||||
} G2;
|
||||
Int32 lat[lat=2];
|
||||
Int32 lon[lon=2];
|
||||
} D1;
|
||||
\endcode
|
||||
|
||||
\subsection var_def Variable Definition
|
||||
|
||||
The set of netCDF variables is derived from the fields with primitive
|
||||
base types as they occur in Sequences, Grids, and Structures. The
|
||||
field names are modified to be fully qualified initially. For the
|
||||
above, the set of variables are as follows. The coordinate variables
|
||||
within grids are left out in order to mimic the behavior of libnc-dap.
|
||||
|
||||
\code
|
||||
f1
|
||||
S1.f11
|
||||
S1.FS2.f1
|
||||
S1.FS2.f2
|
||||
S2.G1.temp
|
||||
S2.G2.G2
|
||||
lat
|
||||
lon
|
||||
\endcode
|
||||
|
||||
\section dap2_reserved_keywords DAP2 Reserved Keywords
|
||||
|
||||
In the OPeNDAP DAP2 protocol, there are a number of reserved keywords. These keywords are case insensitive and if you use one as a netCDF variable name, you may encounter odd behavior such as case changes (depending on the client DDS/DAS parser). The list of reserved keywords as used by the netCDF-C library parser are as follows:
|
||||
|
||||
- alias
|
||||
- array
|
||||
- attributes
|
||||
- byte
|
||||
- dataset
|
||||
- error
|
||||
- float32
|
||||
- float64
|
||||
- grid
|
||||
- int16
|
||||
- int32
|
||||
- maps
|
||||
- sequence
|
||||
- string
|
||||
- structure
|
||||
- uint16
|
||||
- uint32
|
||||
- url
|
||||
- code
|
||||
- message
|
||||
- program_type
|
||||
- program
|
||||
|
||||
|
||||
\section var_dim_trans Variable Dimension Translation
|
||||
|
||||
A variable's rank is determined from three sources.
|
||||
- The variable has the dimensions associated with the field it
|
||||
represents (e.g. S1.FS2.f1[3] in the above example).
|
||||
- The variable inherits the dimensions associated with any containing
|
||||
structure that has a rank greater than zero. These dimensions precede
|
||||
those of case 1. Thus, we have in our example, f1[2][3], where the
|
||||
first dimension comes from the containing Structure FS2[2].
|
||||
- The variable's set of dimensions are altered if any of its
|
||||
containers is a DAP DDS Sequence. This is discussed more fully below.
|
||||
|
||||
If the type of the netCDF variable is char, then an extra string
|
||||
dimension is added as the last dimension.
|
||||
|
||||
\subsection dim_trans Dimension translation
|
||||
|
||||
For dimensions, the rules are as follows.
|
||||
|
||||
Fields in dimensioned structures inherit the dimension of the
|
||||
structure; thus the above list would have the following dimensioned
|
||||
variables.
|
||||
|
||||
\code
|
||||
S1.FS2.f1 -> S1.FS2.f1[2][3]
|
||||
S1.FS2.f2 -> S1.FS2.f2[2]
|
||||
S2.G1.temp -> S2.G1.temp[lat=2][lon=2]
|
||||
S2.G1.lat -> S2.G1.lat[lat=2]
|
||||
S2.G1.lon -> S2.G1.lon[lon=2]
|
||||
S2.G2.G2 -> S2.G2.lon[lat=2][lon=2]
|
||||
S2.G2.lat -> S2.G2.lat[lat=2]
|
||||
S2.G2.lon -> S2.G2.lon[lon=2]
|
||||
lat -> lat[lat=2]
|
||||
lon -> lon[lon=2]
|
||||
\endcode
|
||||
|
||||
Collect all of the dimension specifications from the DDS, both named
|
||||
and anonymous (unnamed) For each unique anonymous dimension with value
|
||||
NN create a netCDF dimension of the form "XX_<i>=NN", where XX is the
|
||||
fully qualified name of the variable and i is the i'th (inherited)
|
||||
dimension of the array where the anonymous dimension occurs. For our
|
||||
example, this would create the following dimensions.
|
||||
|
||||
\code
|
||||
S1.FS2.f1_0 = 2 ;
|
||||
S1.FS2.f1_1 = 3 ;
|
||||
S1.FS2.f2_0 = 2 ;
|
||||
S2.G2.lat_0 = 2 ;
|
||||
S2.G2.lon_0 = 2 ;
|
||||
\endcode
|
||||
|
||||
If however, the anonymous dimension is the single dimension of a MAP
|
||||
vector in a Grid then the dimension is given the same name as the map
|
||||
vector This leads to the following.
|
||||
|
||||
\code
|
||||
S2.G2.lat_0 -> S2.G2.lat
|
||||
S2.G2.lon_0 -> S2.G2.lon
|
||||
\endcode
|
||||
|
||||
For each unique named dimension "<name>=NN", create a netCDF dimension
|
||||
of the form "<name>=NN", where name has the qualifications removed. If
|
||||
this leads to duplicates (i.e. same name and same value), then the
|
||||
duplicates are ignored. This produces the following.
|
||||
|
||||
\code
|
||||
S2.G2.lat -> lat
|
||||
S2.G2.lon -> lon
|
||||
\endcode
|
||||
|
||||
Note that this produces duplicates that will be ignored later.
|
||||
|
||||
At this point the only dimensions left to process should be named
|
||||
dimensions with the same name as some dimension from step number 3,
|
||||
but with a different value. For those dimensions create a dimension of
|
||||
the form "<name>M=NN" where M is a counter starting at 1. The example
|
||||
has no instances of this.
|
||||
|
||||
Finally and if needed, define a single UNLIMITED dimension named
|
||||
"unlimited" with value zero. Unlimited will be used to handle certain
|
||||
kinds of DAP sequences (see below).
|
||||
|
||||
This leads to the following set of dimensions.
|
||||
|
||||
\code
|
||||
dimensions:
|
||||
unlimited = UNLIMITED;
|
||||
lat = 2 ;
|
||||
lon = 2 ;
|
||||
S1.FS2.f1_0 = 2 ;
|
||||
S1.FS2.f1_1 = 3 ;
|
||||
S1.FS2.f2_0 = 2 ;
|
||||
\endcode
|
||||
|
||||
\section var_name_trans Variable Name Translation
|
||||
|
||||
The steps for variable name translation are as follows.
|
||||
|
||||
Take the set of variables captured above. Thus for the above DDS, the
|
||||
following fields would be collected.
|
||||
|
||||
\code
|
||||
f1
|
||||
S1.f11
|
||||
S1.FS2.f1
|
||||
S1.FS2.f2
|
||||
S2.G1.temp
|
||||
S2.G2.G2
|
||||
lat
|
||||
lon
|
||||
\endcode
|
||||
|
||||
All grid array variables are renamed to be the same as the containing
|
||||
grid and the grid prefix is removed. In the above DDS, this results in
|
||||
the following changes.
|
||||
|
||||
\code
|
||||
G1.temp -> G1
|
||||
G2.G2 -> G2
|
||||
\endcode
|
||||
|
||||
It is important to note that this process could produce duplicate
|
||||
variables (i.e. with the same name); in that case they are all assumed
|
||||
to have the same content and the duplicates are ignored. If it turns
|
||||
out that the duplicates have different content, then the translation
|
||||
will not detect this. YOU HAVE BEEN WARNED.
|
||||
|
||||
The final netCDF-3 schema (minus attributes) is then as follows.
|
||||
|
||||
\code
|
||||
netcdf t {
|
||||
dimensions:
|
||||
unlimited = UNLIMITED ;
|
||||
lat = 2 ;
|
||||
lon = 2 ;
|
||||
S1.FS2.f1_0 = 2 ;
|
||||
S1.FS2.f1_1 = 3 ;
|
||||
S1.FS2.f2_0 = 2 ;
|
||||
variables:
|
||||
int f1 ;
|
||||
int lat(lat) ;
|
||||
int lon(lon) ;
|
||||
int S1.f11 ;
|
||||
int S1.FS2.f1(S1.FS2.f1_0, S1.FS2.f1_1) ;
|
||||
int S1.FS2.f2(S1_FS2_f2_0) ;
|
||||
float S2.G1(lat, lon) ;
|
||||
float G2(lat, lon) ;
|
||||
}
|
||||
\endcode
|
||||
|
||||
In actuality, the unlimited dimension is dropped because it is unused.
|
||||
|
||||
There are differences with the original libnc-dap here because
|
||||
libnc-dap technically was incorrect. The original would have said
|
||||
this, for example.
|
||||
|
||||
\code
|
||||
int S1.FS2.f1(lat, lat) ;
|
||||
\endcode
|
||||
|
||||
Note that this is incorrect because it dimensions S1.FS2.f1(2,2)
|
||||
rather than S1.FS2.f1(2,3).
|
||||
|
||||
\subsection dap_translation Translating DAP DDS Sequences
|
||||
|
||||
Any variable (as determined above) that is contained directly or
|
||||
indirectly by a Sequence is subject to revision of its rank using the
|
||||
following rules.
|
||||
|
||||
Let the variable be contained in Sequence Q1, where Q1 is the
|
||||
innermost containing sequence. If Q1 is itself contained (directly or
|
||||
indirectly) in a sequence, or Q1 is contained (again directly or
|
||||
indirectly) in a structure that has rank greater than 0, then the
|
||||
variable will have an initial UNLIMITED dimension. Further, all
|
||||
dimensions coming from "above" and including (in the containment
|
||||
sense) the innermost Sequence, Q1, will be removed and replaced by
|
||||
that single UNLIMITED dimension. The size associated with that
|
||||
UNLIMITED is zero, which means that its contents are inaccessible
|
||||
through the netCDF-3 API. Again, this differs from libnc-dap, which
|
||||
leaves out such variables. Again, however, this difference is backward
|
||||
compatible.
|
||||
|
||||
If the variable is contained in a single Sequence (i.e. not nested)
|
||||
and all containing structures have rank 0, then the variable will have
|
||||
an initial dimension whose size is the record count for that
|
||||
Sequence. The name of the new dimension will be the name of the
|
||||
Sequence.
|
||||
|
||||
Consider this example.
|
||||
|
||||
\code
|
||||
Dataset {
|
||||
Structure {
|
||||
Sequence {
|
||||
Int32 f1[3];
|
||||
Int32 f2;
|
||||
} SQ1;
|
||||
} S1[2];
|
||||
Sequence {
|
||||
Structure {
|
||||
Int32 x1[7];
|
||||
} S2[5];
|
||||
} Q2;
|
||||
} D;
|
||||
\endcode
|
||||
|
||||
The corresponding netCDF-3 translation is pretty much as follows (the
|
||||
value for dimension Q2 may differ).
|
||||
|
||||
\code
|
||||
dimensions:
|
||||
unlimited = UNLIMITED ; // (0 currently)
|
||||
S1.SQ1.f1_0 = 2 ;
|
||||
S1.SQ1.f1_1 = 3 ;
|
||||
S1.SQ1.f2_0 = 2 ;
|
||||
Q2.S2.x1_0 = 5 ;
|
||||
Q2.S2.x1_1 = 7 ;
|
||||
Q2 = 5 ;
|
||||
variables:
|
||||
int S1.SQ1.f1(unlimited, S1.SQ1.f1_1) ;
|
||||
int S1.SQ1.f2(unlimited) ;
|
||||
int Q2.S2.x1(Q2, Q2.S2.x1_0, Q2.S2.x1_1) ;
|
||||
\endcode
|
||||
|
||||
Note that for example S1.SQ1.f1_0 is not actually used because it has
|
||||
been folded into the unlimited dimension.
|
||||
|
||||
Note that for sequences without a leading unlimited dimension, there
|
||||
is a performance cost because the translation code has to walk the
|
||||
data to determine how many records are associated with the
|
||||
sequence. Since libnc-dap did essentially the same thing, it can be
|
||||
assumed that the cost is not prohibitive.
|
||||
|
||||
\section dap_caching Caching
|
||||
|
||||
In an effort to provide better performance for some access patterns,
|
||||
client-side caching of data is available. The default is no caching,
|
||||
but it may be enabled by prefixing the URL with the parameter "cache".
|
||||
|
||||
Caching operates basically as follows.
|
||||
|
||||
When a URL is first accessed using nc_open(), netCDF automatically
|
||||
does a pre-fetch of selected variables. These include all variables
|
||||
smaller than a specified (and user definable) size. This allows, for
|
||||
example, quick access to coordinate variables. This can be suppressed
|
||||
with the parameter "noprefetch".
|
||||
|
||||
Whenever a request is made using some variant of the nc_get_var() API
|
||||
procedures, the complete variable is fetched and stored in the cache
|
||||
as a new cache entry. Subsequence requests for any part of that
|
||||
variable will access the cache entry to obtain the data.
|
||||
|
||||
The cache may become too full, either because there are too many
|
||||
entries or because it is taking up too much disk space. In this case
|
||||
cache entries are purged until the cache size limits are reached. The
|
||||
cache purge algorithm is LRU (least recently used) so that variables
|
||||
that are repeatedly referenced will tend to stay in the cache.
|
||||
|
||||
The cache is completely purged when nc_close() is invoked.
|
||||
|
||||
In order to decide if you should enable caching, you will need to have
|
||||
some understanding of the access patterns of your program.
|
||||
|
||||
The ncdump program always dumps one or more whole variables so it
|
||||
turns on caching.
|
||||
|
||||
If your program accesses only parts of a number of variables, then
|
||||
caching should probably not be used since fetching whole variables
|
||||
will probably slow down your program for no purpose.
|
||||
|
||||
Unfortunately, caching is currently an all or nothing proposition, so
|
||||
for more complex access patterns, the decision to cache or not may not
|
||||
have an obvious answer. Probably a good rule of thumb is to avoid
|
||||
caching initially and later turn it on to see its effect on
|
||||
performance.
|
||||
|
||||
\section dap_defined_params Defined Client Parameters
|
||||
|
||||
Currently, a limited set of client parameters is
|
||||
recognized. Parameters not listed here are ignored, but no error is
|
||||
signalled. All names are case insensitive.
|
||||
|
||||
Parameter Name Legal Values Semantics
|
||||
- "log" | "log=<file>" - Turn on logging and send the log output to
|
||||
the specified file. If no file is specified, then output is sent to standard
|
||||
error.
|
||||
- "show=... das|dds|url" - This causes information to appear as
|
||||
specific global attributes. The currently recognized tags are "dds"
|
||||
to display the underlying DDS, "das" similarly, and "url" to display
|
||||
the url used to retrieve the data. This parameter may be specified
|
||||
multiple times (e.g. “show=dds&show=url”).
|
||||
- "show=fetch" - This parameter causes the netCDF code to log a copy
|
||||
of the complete url for every HTTP get request. If logging is
|
||||
enabled, then this can be helpful in checking to see the access
|
||||
behavior of the netCDF code.
|
||||
- "stringlength=NN" - Specify the default string length to use for
|
||||
string dimensions. The default is 64. The name "maxstrlen" is an
|
||||
alias for "stringlength".
|
||||
- "stringlength_<var>=NN" - Specify the default string length to use
|
||||
for a string dimension for the specified variable. The default is
|
||||
64. The name "maxstrlen_<var>" is an alias for "stringlength_<var>".
|
||||
- "cache" - This enables caching.
|
||||
- "nocache" - This disbles caching.
|
||||
- "cachelimit=NN" - Specify the maximum amount of space allowed for
|
||||
the cache.
|
||||
- "cachecount=NN" - Specify the maximum number of entries in the
|
||||
cache.
|
||||
- "prefetch" - This enables prefetch of small variables (default).
|
||||
- "noprefetch" - This disables prefetch of small variables.
|
||||
- "fillmismatch" - This enables _FillValue/Variable type mismatch.
|
||||
- "nofillmismatch" - This disables _FillValue/Variable type mismatch (default).
|
||||
|
||||
\section dap_debug Notes on Debugging OPeNDAP Access
|
||||
|
||||
The OPeNDAP support makes use of the logging facility of the
|
||||
underlying oc system (see http://www.opendap.org/oc).
|
||||
Note that this is currently separate from the
|
||||
existing netCDF logging facility. Turning on this logging can
|
||||
sometimes give important information. Logging can be enabled by
|
||||
using the client parameter "log" or "log=filename",
|
||||
where the first case will send log output to standard error and the
|
||||
second will send log output to the specified file.
|
||||
|
||||
Users should also be aware that if one is
|
||||
accessing data over an NFS mount, one may see some .nfsxxxxx files;
|
||||
those can be ignored.
|
||||
|
||||
\subsection http_config HTTP Configuration.
|
||||
|
||||
Limited support for configuring the http connection is provided via
|
||||
parameters in the “.dodsrc” configuration file. The relevant .dodsrc file is
|
||||
located by first looking in the current working directory, and if not
|
||||
found, then looking in the directory specified by the “$HOME”
|
||||
environment variable.
|
||||
|
||||
Entries in the .dodsrc file are of the form:
|
||||
|
||||
\code
|
||||
['['<url>']']<key>=<value>
|
||||
\endcode
|
||||
|
||||
That is, it consists of a key name and value pair and optionally
|
||||
preceded by a url enclosed in square brackets.
|
||||
|
||||
For given KEY and URL strings, the value chosen is as follows:
|
||||
|
||||
If URL is null, then look for the .dodsrc entry that has no url prefix
|
||||
and whose key is same as the KEY for which we are looking.
|
||||
|
||||
If the URL is not null, then look for all the .dodsrc entries that
|
||||
have a url, URL1, say, and for which URL1 is a prefix (in the string
|
||||
sense) of URL. For example, if URL = http//x.y/a, then it will match
|
||||
entries of the form
|
||||
|
||||
\code
|
||||
1. [http//x.y/a]KEY=VALUE
|
||||
2. [http//x.y/a/b]KEY=VALUE
|
||||
\endcode
|
||||
|
||||
It will not match an entry of the form
|
||||
|
||||
\code
|
||||
[http//x.y/b]KEY=VALUE
|
||||
\endcode
|
||||
|
||||
because “http://x.y/b” is not a string prefix of
|
||||
“http://x.y/a”. Finally from the set so constructed, choose the entry
|
||||
with the longest url prefix: “http//x.y/a/b]KEY=VALUE” in this case.
|
||||
|
||||
Currently, the supported set of keys (with descriptions) are as
|
||||
follows.
|
||||
|
||||
<pre>
|
||||
HTTP.VERBOSE
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Produce verbose output, especially using SSL.
|
||||
Related CURL Flags: CURLOPT_VERBOSE
|
||||
HTTP.DEFLATE
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Allow use of compression by the server.
|
||||
Related CURL Flags: CURLOPT_ENCODING
|
||||
HTTP.COOKIEJAR
|
||||
Type: String representing file path
|
||||
Description: Specify the name of file into which to store cookies. Defaults to in-memory storage.
|
||||
Related CURL Flags:CURLOPT_COOKIEJAR
|
||||
HTTP.CREDENTIALS.USER
|
||||
Type: String representing user name
|
||||
Description: Specify the user name for Digest and Basic authentication.
|
||||
Related CURL Flags:
|
||||
HTTP.CREDENTIALS.PASSWORD
|
||||
Type: String representing password
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Specify the password for Digest and Basic authentication.
|
||||
Related CURL Flags:
|
||||
HTTP.SSL.CERTIFICATE
|
||||
Type: String representing file path
|
||||
Description: Path to a file containing a PEM cerficate.
|
||||
Related CURL Flags: CURLOPT_CERT
|
||||
HTTP.SSL.KEY
|
||||
Type: String representing file path
|
||||
Description: Same as HTTP.SSL.CERTIFICATE, and should usually have the same value.
|
||||
Related CURL Flags: CURLOPT_SSLKEY
|
||||
HTTP.SSL.KEYPASSWORD
|
||||
Type: String representing password
|
||||
Description: Password for accessing the HTTP.SSL.KEY/HTTP.SSL.CERTIFICATE
|
||||
Related CURL Flags: CURLOPT_KEYPASSWORD
|
||||
HTTP.SSL.CAPATH
|
||||
Type: String representing directory
|
||||
Description: Path to a directory containing trusted certificates for validating server certificates.
|
||||
Related CURL Flags: CURLOPT_CAPATH
|
||||
HTTP.SSL.VALIDATE
|
||||
Type: boolean ("1"/"0")
|
||||
Description: Cause the client to verify the server's presented certificate.
|
||||
Related CURL Flags: CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST
|
||||
HTTP.TIMEOUT
|
||||
Type: String ("dddddd")
|
||||
Description: Specify the maximum time in seconds that you allow the http transfer operation to take.
|
||||
Related CURL Flags: CURLOPT_TIMEOUT, CURLOPT_NOSIGNAL
|
||||
HTTP.PROXY_SERVER
|
||||
Type: String representing url to access the proxy: (e.g.http://[username:password@]host[:port])
|
||||
Description: Specify the needed information for accessing a proxy.
|
||||
Related CURL Flags: CURLOPT_PROXY, CURLOPT_PROXYHOST, CURLOPT_PROXYUSERPWD
|
||||
HTTP.READ.BUFFERSIZE
|
||||
Type: String ("dddddd")
|
||||
Description: Specify the the internal buffer size for curl reads.
|
||||
Related CURL Flags: CURLOPT_BUFFERSIZE, CURL_MAX_WRITE_SIZE (16kB),
|
||||
CURL_MAX_READ_SIZE (512kB).
|
||||
|
||||
HTTP.KEEPALIVE
|
||||
Type: String ("on|n/m")
|
||||
Description: Specify that TCP KEEPALIVE should be enabled and that the associated idle wait time is n and that the associated repeat interval is m. If the value is of the form is the string "on", then turn on keepalive, but do not set idle or interval.
|
||||
Related CURL Flags: CURLOPT_TCP_KEEPALIVE, CURLOPT_TCP_KEEPIDLE,
|
||||
CURLOPT_TCP_KEEPINTVL.
|
||||
</pre>
|
||||
|
||||
The related curl flags line indicates the curl flags modified by this
|
||||
key. See the libcurl documentation of the curl_easy_setopt() function
|
||||
for more detail (http://curl.haxx.se/libcurl/c/curl_easy_setopt.html).
|
||||
|
||||
For ESG client side key support, the following entries must be specified:
|
||||
|
||||
\code
|
||||
HTTP.SSL.VALIDATE
|
||||
HTTP.COOKIEJAR
|
||||
HTTP.SSL.CERTIFICATE
|
||||
HTTP.SSL.KEY
|
||||
HTTP.SSL.CAPATH
|
||||
\endcode
|
||||
|
||||
Additionally, for ESG, the HTTP.SSL.CERTIFICATE and HTTP.SSL.KEY
|
||||
entries should have same value, which is the file path for the
|
||||
certificate produced by MyProxyLogon. The HTTP.SSL.CAPATH entry should
|
||||
be the path to the "certificates" directory produced by MyProxyLogon.
|
||||
|
||||
*/
|
||||
353
NUG/bestpractices.md
Normal file
353
NUG/bestpractices.md
Normal file
@ -0,0 +1,353 @@
|
||||
Writing NetCDF Files: Best Practices {#BestPractices}
|
||||
====================================
|
||||
|
||||
[TOC]
|
||||
|
||||
Best Practices {#bp_Best_Practices}
|
||||
=====================================
|
||||
|
||||
## Conventions {#bp_Conventions}
|
||||
|
||||
While netCDF is intended for "self-documenting data", it is often
|
||||
necessary for data writers and readers to agree upon attribute
|
||||
conventions and representations for discipline-specific data structures.
|
||||
These agreements are written up as human readable documents called
|
||||
***netCDF conventions***.
|
||||
|
||||
Use an existing Convention if possible. See the list of [registered
|
||||
conventions](/software/netcdf/conventions.html).
|
||||
|
||||
The CF Conventions are recommended where applicable, especially for
|
||||
gridded (model) datasets.
|
||||
|
||||
Document the convention you are using by adding the global attribute
|
||||
"Conventions" to each netCDF file, for example:
|
||||
|
||||
This document refers to conventions for the netCDF *classic* data model.
|
||||
For recommendations about conventions for the netCDF-4 *enhanced* data
|
||||
model, see [Developing Conventions for
|
||||
NetCDF-4](/netcdf/papers/nc4_conventions.html).
|
||||
|
||||
## Coordinate Systems {#bp_Coordinate-Systems}
|
||||
|
||||
A ***coordinate variable*** is a one-dimensional variable with the same
|
||||
name as a dimension, which names the coordinate values of the dimension.
|
||||
It must not have any missing data (for example, no `_FillValue` or
|
||||
`missing_value` attributes) and must be strictly monotonic (values
|
||||
increasing or decreasing). A two-dimensional variable of type char is a
|
||||
***string-valued coordinate variable*** if it has the same name as its
|
||||
first dimension, e.g.: **char time( time, time\_len);** all of its
|
||||
strings must be unique. A variable's ***coordinate system*** is the set
|
||||
of coordinate variables used by the variable. Coordinates that refer to
|
||||
physical space are called ***spatial coordinates***, ones that refer to
|
||||
physical time are called ***time coordinates***, ones that refer to
|
||||
either physical space or time are called ***spatio-temporal
|
||||
coordinates.***
|
||||
|
||||
- Make coordinate variables for every dimension possible (except for
|
||||
string length dimensions).
|
||||
- Give each coordinate variable at least `unit` and `long_name`
|
||||
attributes to document its meaning.
|
||||
- Use an existing netCDF [Convention](#Conventions) for your
|
||||
coordinate variables, especially to identify
|
||||
spatio-temporal coordinates.
|
||||
- Use shared dimensions to indicate that two variables use the same
|
||||
coordinates along that dimension. If two variables' dimensions are
|
||||
not related, create separate dimensions for them, even if they
|
||||
happen to have the same length.
|
||||
|
||||
## Variable Grouping {#bp_Variable-Grouping}
|
||||
|
||||
You may structure the data in a netCDF file in different ways, for
|
||||
example putting related parameters into a single variable by adding an
|
||||
extra dimension. Standard visualization and analysis software may have
|
||||
trouble breaking that data out, however. On the other extreme, it is
|
||||
possible to create different variables e.g. for different vertical
|
||||
levels of the same parameter. However, standard visualization and
|
||||
analysis software may have trouble grouping that data back together.
|
||||
Here are some guidelines for deciding how to group your data into
|
||||
variables:
|
||||
|
||||
- All of the data in a variable must be of the same type and should
|
||||
have the same units of measurement.
|
||||
- A variable's attributes should be applicable to all its data.
|
||||
- If possible, all of the coordinate variables should be
|
||||
spatio-temporal, with no extra dimensions.
|
||||
- Use 4D spatio-temporal coordinate systems in preference to 3D. Use
|
||||
3D spatio-temporal coordinate systems in preference to 2D.
|
||||
- Vector valued (e.g. wind) parameters are legitimate uses of extra
|
||||
dimensions. There are trade-offs between putting vectors in the same
|
||||
variables vs. putting each component of a vector in a
|
||||
different variable. Check that any visualization software you plan
|
||||
to use can deal with the structure you choose.
|
||||
- Think in terms of complete coordinate systems (especially
|
||||
spatio-temporal), and organize your data into variables accordingly.
|
||||
Variables with the same coordinate system implicitly form a group.
|
||||
|
||||
## Variable Attributes {#bp_Variable-Attributes}
|
||||
|
||||
|
||||
- For each variable where it makes sense, add a **units** attribute,
|
||||
using the [udunits](/software/udunits/index.html) conventions,
|
||||
if possible.
|
||||
- For each variable where it makes sense, add a **long\_name ****
|
||||
attribute, which is a human-readable descriptive name for
|
||||
the variable. This could be used for labeling plots, for example.
|
||||
|
||||
## Strings and Variables of type char {#bp_Strings-and-Variables-of-type-char}
|
||||
|
||||
NetCDF-3 does not have a primitive **String** type, but does have arrays
|
||||
of type **char**, which are 8 bits in size. The main difference is that
|
||||
Strings are variable length arrays of chars, while char arrays are fixed
|
||||
length. Software written in C usually depends on Strings being zero
|
||||
terminated, while software in Fortran and Java do not. Both C
|
||||
(*nc\_get\_vara\_text*) and Java (*ArrayChar.getString*) libraries have
|
||||
convenience routines that read char arrays and convert to Strings.
|
||||
|
||||
- Do not use char type variables for numeric data, use byte type
|
||||
variables instead.
|
||||
- Consider using a global Attribute instead of a Variable to store a
|
||||
String applicable to the whole dataset.
|
||||
- When you want to store arrays of Strings, use a multidimensional
|
||||
char array. All of the Strings will be the same length.
|
||||
- There are 3 strategies for writing variable length Strings and
|
||||
zero-byte termination:
|
||||
1. *Fortran convention*: pad with blanks and never terminate with a
|
||||
zero byte.
|
||||
2. *C convention*: pad with zeros and always terminate with a
|
||||
zero byte.
|
||||
3. *Java convention*: You don't need to store a trailing zero byte,
|
||||
but pad trailing unused characters with zero bytes.
|
||||
- When reading, trim zeros and blanks from the end of the char array
|
||||
and if in C, add a zero byte terminator.
|
||||
|
||||
## Calendar Date/Time {#bp_Calendar-Date-Time}
|
||||
|
||||
Time as a fundamental unit means a time interval, measured in seconds. A
|
||||
Calendar date/time is a specific instance in real, physical time. Dates
|
||||
are specified as an interval from some ***reference time*** e.g. "days
|
||||
elapsed since Greenwich mean noon on 1 January 4713 BCE". The reference
|
||||
time implies a system of counting time called a ***calendar*** (e.g.
|
||||
Gregorian calendar) and a textual representation (e.g. [ISO
|
||||
8601](http://www.cl.cam.ac.uk/%7Emgk25/iso-time.html)).
|
||||
|
||||
There are two strategies for storing a date/time into a netCDF variable.
|
||||
One is to encode it as a numeric value and a unit that includes the
|
||||
reference time, e.g. "seconds since 2001-1-1 0:0:0" or"days since
|
||||
2001-1-1 0:0:0" . The other is to store it as a String using a standard
|
||||
encoding and Calendar. The former is more compact if you have more than
|
||||
one date, and makes it easier to compute intervals between two dates.
|
||||
|
||||
Unidata's [udunits](/software/udunits/) package provides a convenient
|
||||
way to implement the first strategy. It uses the ISO 8601 encoding and a
|
||||
hybrid Gregorian/Julian calendar, but udunits does not support use of
|
||||
other Calendars or encodings for the reference time. However the ncdump
|
||||
"-T" option can display numeric times that use udunits (and optionally
|
||||
climate calendars) as ISO 8601 strings that are easy for humans to
|
||||
interpret.
|
||||
|
||||
- If your data uses real, physical time that is well represented using
|
||||
the Gregorian/Julian calendar, encode it as an interval from a
|
||||
reference time, and add a units attribute which uses a
|
||||
udunits-compatible time unit. If the data assumes one of the
|
||||
non-standard calendars mentioned in the CF Conventions, specify that
|
||||
with a Calendar attribute. Readers can then use the udunits package
|
||||
to manipulate or format the date values, and the ncdump utility can
|
||||
display them with either numeric or string representation.
|
||||
- If your data uses a calendar not supported by the CF Conventions,
|
||||
make it compatible with existing date manipulation packages if
|
||||
possible (for example, java.text.SimpleDateFormat).
|
||||
- Add multiple sets of time encodings if necessary to allow different
|
||||
readers to work as well as possible.\
|
||||
|
||||
## Unsigned Data {#bp_Unsigned-Data}
|
||||
|
||||
NetCDF-3 does not have unsigned integer primitive types.
|
||||
|
||||
- To be completely safe with unknown readers, widen the data type, or
|
||||
use floating point.
|
||||
- You can use the corresponding signed types to store unsigned data
|
||||
only if all client programs know how to interpret this correctly.
|
||||
- A new proposed convention is to create a variable attribute
|
||||
`_Unsigned = "true"` to indicate that integer data should be treated
|
||||
as unsigned.
|
||||
|
||||
## Packed Data Values {#bp_Packed-Data-Values}
|
||||
|
||||
Packed data is stored in a netCDF file by limiting precision and using a
|
||||
smaller data type than the original data, for example, packing
|
||||
double-precision (64-bit) values into short (16-bit) integers. The
|
||||
C-based netCDF libraries do not do the packing and unpacking. (The
|
||||
[netCDF Java library](/software/netcdf-java/) will do automatic
|
||||
unpacking when the
|
||||
[VariableEnhanced](/software/netcdf-java/v4.1/javadocAll/ucar/nc2/dataset/VariableEnhanced.html)
|
||||
Interface is used. For details see
|
||||
[EnhancedScaleMissing](/software/netcdf-java/v4.1/javadocAll/ucar/nc2/dataset/EnhanceScaleMissing.html)).
|
||||
|
||||
- Each variable with packed data has two attributes called
|
||||
**scale\_factor** and **add\_offset**, so that the packed data may
|
||||
be read and unpacked using the formula:
|
||||
|
||||
> ***unpacked\_data\_value = packed\_data\_value \* scale\_factor +
|
||||
> add\_offset***
|
||||
|
||||
- The type of the stored variable is the packed data type, typically
|
||||
byte, short or int.
|
||||
- The type of the scale\_factor and add\_offset attributes should be
|
||||
the type that you want the unpacked data to be, typically float
|
||||
or double.
|
||||
- To avoid introducing a bias into the unpacked values due to
|
||||
truncation when packing, the data provider should round to the
|
||||
nearest integer rather than just truncating towards zero before
|
||||
writing the data:
|
||||
|
||||
> ***packed\_data\_value = nint((unpacked\_data\_value -
|
||||
> add\_offset) / scale\_factor)***
|
||||
|
||||
Depending on whether the packed data values are intended to be
|
||||
interpreted by the reader as signed or unsigned integers, there are
|
||||
alternative ways for the data provider to compute the *scale\_factor*
|
||||
and *add\_offset* attributes. In either case, the formulas above apply
|
||||
for unpacking and packing the data.
|
||||
|
||||
A conventional way to indicate whether a byte, short, or int variable is
|
||||
meant to be interpreted as unsigned, even for the netCDF-3 classic model
|
||||
that has no external unsigned integer type, is by providing the special
|
||||
variable attribute `_Unsigned` with value `"true"`. However, most
|
||||
existing data for which packed values are intended to be interpreted as
|
||||
unsigned are stored without this attribute, so readers must be aware of
|
||||
packing assumptions in this case. In the enhanced netCDF-4 data model,
|
||||
packed integers may be declared to be of the appropriate unsigned type.
|
||||
|
||||
Let *n* be the number of bits in the packed type, and assume *dataMin*
|
||||
and *dataMax* are the minimum and maximum values that will be used for a
|
||||
variable to be packed.
|
||||
|
||||
- If the packed values are intended to be interpreted as signed
|
||||
integers (the default assumption for classic model data), you may
|
||||
use:
|
||||
|
||||
> *scale\_factor =(dataMax - dataMin) / (2^n^ - 1)*
|
||||
|
||||
> *add\_offset = dataMin + 2^n\\ -\\ 1^ \* scale\_factor*
|
||||
|
||||
- If the packed values are intended to be interpreted as unsigned (for
|
||||
example, when read in the C interface using the `nc_get_var_uchar()`
|
||||
function), use:
|
||||
|
||||
> *scale\_factor =(dataMax - dataMin) / (2^n^ - 1)*
|
||||
|
||||
> *add\_offset = dataMin*
|
||||
|
||||
- In either the signed or unsigned case, an alternate formula may be
|
||||
used for the add\_offset and scale\_factor packing parameters that
|
||||
reserves a packed value for a special value, such as an indicator of
|
||||
missing data. For example, to reserve the minimum packed value
|
||||
(-2^n\\ -\\ 1^) for use as a special value in the case of signed
|
||||
packed values:
|
||||
|
||||
> *scale\_factor =(dataMax - dataMin) / (2^n^ - 2)*
|
||||
|
||||
> *add\_offset = (dataMax + dataMin) / 2*
|
||||
|
||||
- If the packed values are unsigned, then the analogous formula that
|
||||
reserves 0 as the packed form of a special value would be:
|
||||
|
||||
> *scale\_factor =(dataMax - dataMin) / (2^n^ - 2)*
|
||||
|
||||
> *add\_offset = dataMin - scale\_factor*
|
||||
|
||||
- Example, packing 32-bit floats into 16-bit shorts:
|
||||
|
||||
variables:
|
||||
short data( z, y, x);
|
||||
data:scale_offset = 34.02f;
|
||||
data:add_offset = 1.54f;
|
||||
|
||||
- The `units` attribute applies to unpacked values.
|
||||
|
||||
## Missing Data Values {#bp_Missing-Data-Values}
|
||||
|
||||
***Missing data*** is a general name for data values that are invalid,
|
||||
never written, or missing. The netCDF library itself does not handle
|
||||
these values in any special way, except that the value of a `_FillValue`
|
||||
attribute, if any, is used in pre-filling unwritten data. (The
|
||||
Java-netCDF library will assist in recognizing these values when
|
||||
reading, see class **VariableStandardized**).
|
||||
|
||||
- Default fill values for each type are available in the C-based
|
||||
interfaces, and are defined in the appropriate header files. For
|
||||
example, in the C interface, NC\_FILL\_FLOAT and NC\_FILL\_DOUBLE
|
||||
are numbers near 9.9692e+36 that are returned when you try to read
|
||||
values that were never written. Writing, reading, and testing for
|
||||
equality with these default fill values works portably on the
|
||||
platforms on which netCDF has been tested.
|
||||
- The `_FillValue` attribute should have the same data type as the
|
||||
variable it describes. If the variable is packed using
|
||||
`scale_factor` and `add_offset` attributes, the `_FillValue`
|
||||
attribute should have the data type of the packed data.
|
||||
- Another way of indicating missing values for real type data is to
|
||||
store an IEEE **NaN** floating point value. The advantage of this is
|
||||
that any computation using a NaN results in a NaN. Client software
|
||||
must know to look for NaNs, however, and detection of NaNs is
|
||||
tricky, since any comparison with a NaN is required to return
|
||||
*false*.
|
||||
- In Java, you can use **Double.NaN** and **Float.NaN** constants.
|
||||
- In many C compilers, you can generate a NaN value using **double
|
||||
nan = 0.0 / 0.0;**
|
||||
- Alternatively or in addition, set the **valid\_range** attribute for
|
||||
each variable that uses missing values, and make sure all valid data
|
||||
is within that range, and all missing or invalid data is outside of
|
||||
that range. Again, the client software must recognize and make use
|
||||
of this information. Example:
|
||||
|
||||
variables:
|
||||
float data( z, y, x);
|
||||
data:valid_range = -999.0f, 999.0f;
|
||||
|
||||
|
||||
If the variable is packed using `scale_factor` and `add_offset`
|
||||
attributes, the `valid_range` attribute should have the data type of
|
||||
the packed data.
|
||||
|
||||
If the variable is unsigned the `valid_range` values should be
|
||||
widened if needed and stored as unsigned integers.
|
||||
|
||||
## Miscellaneous tips {#bp_Miscellaneous-tips}
|
||||
|
||||
- To define a file whose structure is known in advance, write a CDL
|
||||
file and create the netCDF file using
|
||||
[ncgen](/cgi-bin/man-cgi?ncgen). Then write the data into the netCDF
|
||||
file using your program. This is typically much easier than
|
||||
programming all of the create calls yourself.
|
||||
- For the netCDF classic or 64-bit-offset formats, it's possible to
|
||||
reserve extra space in the file when it is created so that you can
|
||||
later add additional attributes or non-record variables without
|
||||
copying all the data. (This is not necessary for netCDF-4 files,
|
||||
because metadata can be added efficiently to such files.) See the [C
|
||||
man-page reference documentation](/cgi-bin/man-cgi?netcdf+-s3) (or
|
||||
the [Fortran reference documentation](/cgi-bin/man-cgi?netcdf+-s3f))
|
||||
for `nc__create` and `nc__enddef` (`nf__create` and `nf__enddef`
|
||||
for Fortran) for more details on reserving extra space in
|
||||
the header.
|
||||
|
||||
## Spelling netCDF: Best Practices {#bp_Spelling-netCDF-Best-Practices}
|
||||
|
||||
There are only 3 correct spellings of "netCDF":
|
||||
|
||||
1. **netCDF:** The original spelling of the name of the data model,
|
||||
API, and format. The acronym stands for network Common Data Form
|
||||
(not Format), and the "CDF" part was capitalized in part to pay
|
||||
homage to the NASA "CDF" data model which the netCDF data
|
||||
model extended.
|
||||
2. **netcdf:** Used in certain file names, such as:
|
||||
|
||||
#include <netcdf.h>
|
||||
|
||||
3. **NetCDF**: Used in titles and at the beginning of sentences, where
|
||||
"netCDF" is awkward or violates style guidelines.
|
||||
|
||||
All other forms, and most especially "Netcdf", are considered vulgar and
|
||||
a sign of ill-breeding or misspent youth, analogous to the egregious but
|
||||
common misspelling "JAVA" used by those who are new to the language or
|
||||
who mistakenly think it is an acronym.\
|
||||
6
NUG/cleanup.sh
Executable file
6
NUG/cleanup.sh
Executable file
@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
|
||||
rm -rf html
|
||||
rm -rf latex
|
||||
rm -rf *~
|
||||
|
||||
605
NUG/filters.md
Normal file
605
NUG/filters.md
Normal file
@ -0,0 +1,605 @@
|
||||
NetCDF-4 Filter Support
|
||||
============================
|
||||
<!-- double header is needed to workaround doxygen bug -->
|
||||
|
||||
NetCDF-4 Filter Support {#filters}
|
||||
============================
|
||||
|
||||
[TOC]
|
||||
|
||||
# Introduction {#filters_intro}
|
||||
|
||||
The HDF5 library (1.8.11 and later)
|
||||
supports a general filter mechanism to apply various
|
||||
kinds of filters to datasets before reading or writing.
|
||||
The netCDF enhanced (aka netCDF-4) library inherits this
|
||||
capability since it depends on the HDF5 library.
|
||||
|
||||
Filters assume that a variable has chunking
|
||||
defined and each chunk is filtered before
|
||||
writing and "unfiltered" after reading and
|
||||
before passing the data to the user.
|
||||
|
||||
The most common kind of filter is a compression-decompression
|
||||
filter, and that is the focus of this document.
|
||||
|
||||
HDF5 supports dynamic loading of compression filters using the following
|
||||
process for reading of compressed data.
|
||||
|
||||
1. Assume that we have a dataset with one or more variables that
|
||||
were compressed using some algorithm. How the dataset was compressed
|
||||
will be discussed subsequently.
|
||||
|
||||
2. Shared libraries or DLLs exist that implement the compress/decompress
|
||||
algorithm. These libraries have a specific API so that the HDF5 library
|
||||
can locate, load, and utilize the compressor.
|
||||
These libraries are expected to installed in a specific
|
||||
directory.
|
||||
|
||||
# Enabling A Compression Filter {#filters_enable}
|
||||
|
||||
In order to compress a variable, the netcdf-c library
|
||||
must be given three pieces of information:
|
||||
(1) some unique identifier for the filter to be used,
|
||||
(2) a vector of parameters for
|
||||
controlling the action of the compression filter, and
|
||||
(3) a shared library implementation of the filter.
|
||||
|
||||
The meaning of the parameters is, of course,
|
||||
completely filter dependent and the filter
|
||||
description [3] needs to be consulted. For
|
||||
bzip2, for example, a single parameter is provided
|
||||
representing the compression level.
|
||||
It is legal to provide a zero-length set of parameters.
|
||||
Defaults are not provided, so this assumes that
|
||||
the filter can operate with zero parameters.
|
||||
|
||||
Filter ids are assigned by the HDF group. See [4]
|
||||
for a current list of assigned filter ids.
|
||||
Note that ids above 32767 can be used for testing without
|
||||
registration.
|
||||
|
||||
The first two pieces of information can be provided in one of three ways:
|
||||
using __ncgen__, via an API call, or via command line parameters to __nccopy__.
|
||||
In any case, remember that filtering also requires setting chunking, so the
|
||||
variable must also be marked with chunking information.
|
||||
|
||||
## Using The API {#filters_API}
|
||||
The necessary API methods are included in __netcdf.h__ by default.
|
||||
One API method is for setting the filter to be used
|
||||
when writing a variable. The relevant signature is
|
||||
as follows.
|
||||
````
|
||||
int nc_def_var_filter(int ncid, int varid, unsigned int id, size_t nparams, const unsigned int* parms);
|
||||
````
|
||||
This must be invoked after the variable has been created and before
|
||||
__nc_enddef__ is invoked.
|
||||
|
||||
A second API methods makes it possible to query a variable to
|
||||
obtain information about any associated filter using this signature.
|
||||
````
|
||||
int nc_inq_var_filter(int ncid, int varid, unsigned int* idp, size_t* nparams, unsigned int* params);
|
||||
|
||||
````
|
||||
The filter id will be returned in the __idp__ argument (if non-NULL),
|
||||
the number of parameters in __nparamsp__ and the actual parameters in
|
||||
__params__. As is usual with the netcdf API, one is expected to call
|
||||
this function twice. The first time to get __nparams__ and the
|
||||
second to get the parameters in client-allocated memory.
|
||||
|
||||
## Using ncgen {#filters_NCGEN}
|
||||
|
||||
In a CDL file, compression of a variable can be specified
|
||||
by annotating it with the following attribute:
|
||||
|
||||
* ''_Filter'' -- a string containing a comma separated list of
|
||||
constants specifying (1) the filter id to apply, and (2)
|
||||
a vector of constants representing the
|
||||
parameters for controlling the operation of the specified filter.
|
||||
See the section on the <a href="#Syntax">parameter encoding syntax</a>
|
||||
for the details on the allowable kinds of constants.
|
||||
|
||||
This is a "special" attribute, which means that
|
||||
it will normally be invisible when using
|
||||
__ncdump__ unless the -s flag is specified.
|
||||
|
||||
### Example CDL File (Data elided)
|
||||
|
||||
````
|
||||
netcdf bzip2 {
|
||||
dimensions:
|
||||
dim0 = 4 ; dim1 = 4 ; dim2 = 4 ; dim3 = 4 ;
|
||||
variables:
|
||||
float var(dim0, dim1, dim2, dim3) ;
|
||||
var:_Filter = "307,9" ;
|
||||
var:_Storage = "chunked" ;
|
||||
var:_ChunkSizes = 4, 4, 4, 4 ;
|
||||
data:
|
||||
...
|
||||
}
|
||||
````
|
||||
|
||||
## Using nccopy {#filters_NCCOPY}
|
||||
|
||||
When copying a netcdf file using __nccopy__ it is possible
|
||||
to specify filter information for any output variable by
|
||||
using the "-F" option on the command line; for example:
|
||||
````
|
||||
nccopy -F "var,307,9" unfiltered.nc filtered.nc
|
||||
````
|
||||
Assume that __unfiltered.nc__ has a chunked but not bzip2 compressed
|
||||
variable named "var". This command will create that variable in
|
||||
the __filtered.nc__ output file but using filter with id 307
|
||||
(i.e. bzip2) and with parameter(s) 9 indicating the compression level.
|
||||
See the section on the <a href="#Syntax">parameter encoding syntax</a>
|
||||
for the details on the allowable kinds of constants.
|
||||
|
||||
The "-F" option can be used repeatedly as long as the variable name
|
||||
part is different. A different filter id and parameters can be
|
||||
specified for each occurrence.
|
||||
|
||||
It can be convenient to specify that the same compression is to be
|
||||
applied to more than one variable. To support this, two additional
|
||||
*-F* cases are defined.
|
||||
|
||||
1. ````-F *,...``` means apply the filter to all variables in the dataset.
|
||||
2. ````-F v1|v2|..,...``` means apply the filter to a multiple variables.
|
||||
|
||||
Note that the characters '*' and '|' are bash reserved characters,
|
||||
so you will probably need to escape or quote the filter spec in
|
||||
that environment.
|
||||
|
||||
As a rule, any input filter on an input variable will be applied
|
||||
to the equivalent output variable -- assuming the output file type
|
||||
is netcdf-4. It is, however, sometimes convenient to suppress
|
||||
output compression either totally or on a per-variable basis.
|
||||
Total suppression of output filters can be accomplished by specifying
|
||||
a special case of "-F", namely this.
|
||||
````
|
||||
nccopy -F none input.nc output.nc
|
||||
````
|
||||
The expression ````-F *,none```` is equivalent to ````-F none````.
|
||||
|
||||
Suppression of output filtering for a specific set of variables
|
||||
can be accomplished using these formats.
|
||||
````
|
||||
nccopy -F "var,none" input.nc output.nc
|
||||
nccopy -F "v1|v2|...,none" input.nc output.nc
|
||||
````
|
||||
where "var" and the "vi" are the fully qualified name of a variable.
|
||||
|
||||
The rules for all possible cases of the "-F none" flag are defined
|
||||
by this table.
|
||||
|
||||
<table>
|
||||
<tr><th>-F none<th>-Fvar,...<th>Input Filter<th>Applied Output Filter
|
||||
<tr><td>true<td>unspecified<td>NA<td>unfiltered
|
||||
<tr><td>true<td>-Fvar,none<td>NA<td>unfiltered
|
||||
<tr><td>true<td>-Fvar,...<td>NA<td>use output filter
|
||||
<tr><td>false<td>unspecified<td>defined<td>use input filter
|
||||
<tr><td>false<td>-Fvar,none<td>NA<td>unfiltered
|
||||
<tr><td>false<td>-Fvar,...<td>NA<td>use output filter
|
||||
<tr><td>false<td>unspecified<td>none<td>unfiltered
|
||||
</table>
|
||||
|
||||
# Parameter Encode/Decode {#filters_paramcoding}
|
||||
|
||||
The parameters passed to a filter are encoded internally as a vector
|
||||
of 32-bit unsigned integers. It may be that the parameters
|
||||
required by a filter can naturally be encoded as unsigned integers.
|
||||
The bzip2 compression filter, for example, expects a single
|
||||
integer value from zero thru nine. This encodes naturally as a
|
||||
single unsigned integer.
|
||||
|
||||
Note that signed integers and single-precision (32-bit) float values
|
||||
also can easily be represented as 32 bit unsigned integers by
|
||||
proper casting to an unsigned integer so that the bit pattern
|
||||
is preserved. Simple integer values of type short or char
|
||||
(or the unsigned versions) can also be mapped to an unsigned
|
||||
integer by truncating to 16 or 8 bits respectively and then
|
||||
zero extending.
|
||||
|
||||
Machine byte order (aka endian-ness) is an issue for passing
|
||||
some kinds of parameters. You might define the parameters when
|
||||
compressing on a little endian machine, but later do the
|
||||
decompression on a big endian machine. Byte order is not an
|
||||
issue for 32-bit values because HDF5 takes care of converting
|
||||
them between the local machine byte order and network byte
|
||||
order.
|
||||
|
||||
Parameters whose size is larger than 32-bits present a byte order problem.
|
||||
This specifically includes double precision floats and (signed or unsigned)
|
||||
64-bit integers. For these cases, the machine byte order issue must be
|
||||
handled, in part, by the compression code. This is because HDF5 will treat,
|
||||
for example, an unsigned long long as two 32-bit unsigned integers
|
||||
and will convert each to network order separately. This means that
|
||||
on a machine whose byte order is different than the machine in which
|
||||
the parameters were initially created, the two integers will be separately
|
||||
endian converted. But this will be incorrect for 64-bit values.
|
||||
|
||||
So, we have this situation:
|
||||
|
||||
1. the 8 bytes come in as native machine order for the machine
|
||||
doing the call to *nc_def_var_filter*.
|
||||
2. HDF5 divides the 8 bytes into 2 four byte pieces and ensures that each piece
|
||||
is in network (big) endian order.
|
||||
3. When the filter is called, the two pieces are returned in the same order
|
||||
but with the bytes in each piece consistent with the native machine order
|
||||
for the machine executing the filter.
|
||||
|
||||
## Encoding Algorithms
|
||||
|
||||
In order to properly extract the correct 8-byte value, we need to ensure
|
||||
that the values stored in the HDF5 file have a known format independent of
|
||||
the native format of the creating machine.
|
||||
|
||||
The idea is to do sufficient manipulation so that HDF5
|
||||
will store the 8-byte value as a little endian value
|
||||
divided into two 4-byte integers.
|
||||
Note that little-endian is used as the standard
|
||||
because it is the most common machine format.
|
||||
When read, the filter code needs to be aware of this convention
|
||||
and do the appropriate conversions.
|
||||
|
||||
This leads to the following set of rules.
|
||||
|
||||
### Encoding
|
||||
|
||||
1. Encode on little endian (LE) machine: no special action is required.
|
||||
The 8-byte value is passed to HDF5 as two 4-byte integers. HDF5 byte
|
||||
swaps each integer and stores it in the file.
|
||||
2. Encode on a big endian (BE) machine: several steps are required:
|
||||
|
||||
1. Do an 8-byte byte swap to convert the original value to little-endian
|
||||
format.
|
||||
2. Since the encoding machine is BE, HDF5 will just store the value.
|
||||
So it is necessary to simulate little endian encoding by byte-swapping
|
||||
each 4-byte integer separately.
|
||||
3. This doubly swapped pair of integers is then passed to HDF5 and is stored
|
||||
unchanged.
|
||||
|
||||
### Decoding
|
||||
|
||||
1. Decode on LE machine: no special action is required.
|
||||
HDF5 will get the two 4-bytes values from the file and byte-swap each
|
||||
separately. The concatenation of those two integers will be the expected
|
||||
LE value.
|
||||
2. Decode on a big endian (BE) machine: the inverse of the encode case must
|
||||
be implemented.
|
||||
|
||||
1. HDF5 sends the two 4-byte values to the filter.
|
||||
2. The filter must then byte-swap each 4-byte value independently.
|
||||
3. The filter then must concatenate the two 4-byte values into a single
|
||||
8-byte value. Because of the encoding rules, this 8-byte value will
|
||||
be in LE format.
|
||||
4. The filter must finally do an 8-byte byte-swap on that 8-byte value
|
||||
to convert it to desired BE format.
|
||||
|
||||
To support these rules, some utility programs exist and are discussed in
|
||||
<a href="#AppendixA">Appendix A</a>.
|
||||
|
||||
# Filter Specification Syntax {#filters_syntax}
|
||||
|
||||
Both of the utilities
|
||||
<a href="#NCGEN">__ncgen__</a>
|
||||
and
|
||||
<a href="#NCCOPY">__nccopy__</a>
|
||||
allow the specification of filter parameters in text format.
|
||||
These specifications consist of a sequence of comma
|
||||
separated constants. The constants are converted
|
||||
within the utility to a proper set of unsigned int
|
||||
constants (see the <a href="#ParamEncode">parameter encoding section</a>).
|
||||
|
||||
To simplify things, various kinds of constants can be specified
|
||||
rather than just simple unsigned integers. The utilities will encode
|
||||
them properly using the rules specified in
|
||||
the section on <a href="#filters_paramcoding">parameter encode/decode</a>.
|
||||
|
||||
The currently supported constants are as follows.
|
||||
<table>
|
||||
<tr halign="center"><th>Example<th>Type<th>Format Tag<th>Notes
|
||||
<tr><td>-17b<td>signed 8-bit byte<td>b|B<td>Truncated to 8 bits and zero extended to 32 bits
|
||||
<tr><td>23ub<td>unsigned 8-bit byte<td>u|U b|B<td>Truncated to 8 bits and zero extended to 32 bits
|
||||
<tr><td>-25S<td>signed 16-bit short<td>s|S<td>Truncated to 16 bits and zero extended to 32 bits
|
||||
<tr><td>27US<td>unsigned 16-bit short<td>u|U s|S<td>Truncated to 16 bits and zero extended to 32 bits
|
||||
<tr><td>-77<td>implicit signed 32-bit integer<td>Leading minus sign and no tag<td>
|
||||
<tr><td>77<td>implicit unsigned 32-bit integer<td>No tag<td>
|
||||
<tr><td>93U<td>explicit unsigned 32-bit integer<td>u|U<td>
|
||||
<tr><td>789f<td>32-bit float<td>f|F<td>
|
||||
<tr><td>12345678.12345678d<td>64-bit double<td>d|D<td>LE encoding
|
||||
<tr><td>-9223372036854775807L<td>64-bit signed long long<td>l|L<td>LE encoding
|
||||
<tr><td>18446744073709551615UL<td>64-bit unsigned long long<td>u|U l|L<td>LE encoding
|
||||
</table>
|
||||
Some things to note.
|
||||
|
||||
1. In all cases, except for an untagged positive integer,
|
||||
the format tag is required and determines how the constant
|
||||
is converted to one or two unsigned int values.
|
||||
The positive integer case is for backward compatibility.
|
||||
2. For signed byte and short, the value is sign extended to 32 bits
|
||||
and then treated as an unsigned int value.
|
||||
3. For double, and signed|unsigned long long, they are converted
|
||||
as specified in the section on
|
||||
<a href="#filters_paramcoding">parameter encode/decode</a>.
|
||||
|
||||
Dynamic Loading Process {#filters_Process}
|
||||
==========
|
||||
|
||||
The documentation[1,2] for the HDF5 dynamic loading was (at the time
|
||||
this was written) out-of-date with respect to the actual HDF5 code
|
||||
(see HDF5PL.c). So, the following discussion is largely derived
|
||||
from looking at the actual code. This means that it is subject to change.
|
||||
|
||||
Plugin directory {#filters_Plugindir}
|
||||
----------------
|
||||
|
||||
The HDF5 loader expects plugins to be in a specified plugin directory.
|
||||
The default directory is:
|
||||
* "/usr/local/hdf5/lib/plugin” for linux/unix operating systems (including Cygwin)
|
||||
* “%ALLUSERSPROFILE%\\hdf5\\lib\\plugin” for Windows systems, although the code
|
||||
does not appear to explicitly use this path.
|
||||
|
||||
The default may be overridden using the environment variable
|
||||
__HDF5_PLUGIN_PATH__.
|
||||
|
||||
Plugin Library Naming {#filters_Pluginlib}
|
||||
---------------------
|
||||
|
||||
Given a plugin directory, HDF5 examines every file in that
|
||||
directory that conforms to a specified name pattern
|
||||
as determined by the platform on which the library is being executed.
|
||||
<table>
|
||||
<tr halign="center"><th>Platform<th>Basename<th>Extension
|
||||
<tr halign="left"><td>Linux<td>lib*<td>.so*
|
||||
<tr halign="left"><td>OSX<td>lib*<td>.so*
|
||||
<tr halign="left"><td>Cygwin<td>cyg*<td>.dll*
|
||||
<tr halign="left"><td>Windows<td>*<td>.dll
|
||||
</table>
|
||||
|
||||
Plugin Verification {#filters_Pluginverify}
|
||||
-------------------
|
||||
For each dynamic library located using the previous patterns,
|
||||
HDF5 attempts to load the library and attempts to obtain information
|
||||
from it. Specifically, It looks for two functions with the following
|
||||
signatures.
|
||||
|
||||
1. __H5PL_type_t H5PLget_plugin_type(void)__ --
|
||||
This function is expected to return the constant value
|
||||
__H5PL_TYPE_FILTER__ to indicate that this is a filter library.
|
||||
2. __const void* H5PLget_plugin_info(void)__ --
|
||||
This function returns a pointer to a table of type __H5Z_class2_t__.
|
||||
This table contains the necessary information needed to utilize the
|
||||
filter both for reading and for writing. In particular, it specifies
|
||||
the filter id implemented by the library and if must match that id
|
||||
specified for the variable in __nc_def_var_filter__ in order to be used.
|
||||
|
||||
If plugin verification fails, then that plugin is ignored and
|
||||
the search continues for another, matching plugin.
|
||||
|
||||
Debugging {#filters_Debug}
|
||||
-------
|
||||
Debugging plugins can be very difficult. You will probably
|
||||
need to use the old printf approach for debugging the filter itself.
|
||||
|
||||
One case worth mentioning is when you have a dataset that is
|
||||
using an unknown filter. For this situation, you need to
|
||||
identify what filter(s) are used in the dataset. This can
|
||||
be accomplished using this command.
|
||||
````
|
||||
ncdump -s -h <dataset filename>
|
||||
````
|
||||
Since ncdump is not being asked to access the data (the -h flag), it
|
||||
can obtain the filter information without failures. Then it can print
|
||||
out the filter id and the parameters (the -s flag).
|
||||
|
||||
Test Case {#filters_TestCase}
|
||||
-------
|
||||
Within the netcdf-c source tree, the directory
|
||||
__netcdf-c/nc_test4__ contains a test case (__test_filter.c__) for
|
||||
testing dynamic filter writing and reading using
|
||||
bzip2. Another test (__test_filter_misc.c__) validates
|
||||
parameter passing. These tests are disabled if __--enable-shared__
|
||||
is not set or if __--enable-netcdf-4__ is not set.
|
||||
|
||||
Example {#filters_Example}
|
||||
-------
|
||||
A slightly simplified version of the filter test case is also
|
||||
available as an example within the netcdf-c source tree
|
||||
directory __netcdf-c/examples/C. The test is called __filter_example.c__
|
||||
and it is executed as part of the __run_examples4.sh__ shell script.
|
||||
The test case demonstrates dynamic filter writing and reading.
|
||||
|
||||
The files __example/C/hdf5plugins/Makefile.am__
|
||||
and __example/C/hdf5plugins/CMakeLists.txt__
|
||||
demonstrate how to build the hdf5 plugin for bzip2.
|
||||
|
||||
Notes
|
||||
==========
|
||||
|
||||
Memory Allocation Issues
|
||||
-----------
|
||||
|
||||
Starting with HDF5 version 1.10.x, the plugin code MUST be
|
||||
careful when using the standard *malloc()*, *realloc()*, and
|
||||
*free()* function.
|
||||
|
||||
In the event that the code is allocating, reallocating, for
|
||||
free'ing memory that either came from or will be exported to the
|
||||
calling HDF5 library, then one MUST use the corresponding HDF5
|
||||
functions *H5allocate_memory()*, *H5resize_memory()*,
|
||||
*H5free_memory()* [5] to avoid memory failures.
|
||||
|
||||
Additionally, if your filter code leaks memory, then the HDF5 library
|
||||
generates a failure something like this.
|
||||
````
|
||||
H5MM.c:232: H5MM_final_sanity_check: Assertion `0 == H5MM_curr_alloc_bytes_s' failed.
|
||||
````
|
||||
|
||||
One can look at the the code in plugins/H5Zbzip2.c and H5Zmisc.c to see this.
|
||||
|
||||
SZIP Issues
|
||||
-----------
|
||||
The current szip plugin code in the HDF5 library
|
||||
has some behaviors that can catch the unwary.
|
||||
Specifically, this filter may do two things.
|
||||
|
||||
1. Add extra parameters to the filter parameters: going from
|
||||
the two parameters provided by the user to four parameters
|
||||
for internal use. It turns out that the two parameters provided
|
||||
when calling nc_def_var_filter correspond to the first two
|
||||
parameters of the four parameters returned by nc_inq_var_filter.
|
||||
2. Change the values of some parameters: the value of the
|
||||
__options_mask__ argument is known to add additional flag bits,
|
||||
and the __pixels_per_block__ parameter may be modified.
|
||||
|
||||
The reason for these changes is has to do with the fact that
|
||||
the szip API provided by the underlying H5Pset_szip function
|
||||
is actually a subset of the capabilities of the real szip implementation.
|
||||
Presumably this is for historical reasons.
|
||||
|
||||
In any case, if the caller uses the __nc_inq_var_szip__, then
|
||||
the values returned may differ from those originally specified.
|
||||
If one used the __nc_inq_var_filter__ API calls, it may be the case that
|
||||
both the number of parameters and the values will differ from the original
|
||||
call to __nc_def_var_filter__.
|
||||
|
||||
Supported Systems
|
||||
-----------------
|
||||
The current matrix of OS X build systems known to work is as follows.
|
||||
<table>
|
||||
<tr><th>Build System<th>Supported OS
|
||||
<tr><td>Automake<td>Linux, Cygwin
|
||||
<tr><td>Cmake<td>Linux, Cygwin, Visual Studio
|
||||
</table>
|
||||
|
||||
Generic Plugin Build
|
||||
--------------------
|
||||
If you do not want to use Automake or Cmake, the following
|
||||
has been known to work.
|
||||
````
|
||||
gcc -g -O0 -shared -o libbzip2.so <plugin source files> -L${HDF5LIBDIR} -lhdf5_hl -lhdf5 -L${ZLIBDIR} -lz
|
||||
````
|
||||
|
||||
Appendix A. Support Utilities {#filters_AppendixA}
|
||||
==========
|
||||
|
||||
Two functions are exported from the netcdf-c library
|
||||
for use by client programs and by filter implementations.
|
||||
|
||||
1. ````int NC_parsefilterspec(const char* spec, unsigned int* idp, size_t* nparamsp, unsigned int** paramsp);````
|
||||
* idp will contain the filter id value from the spec.
|
||||
* nparamsp will contain the number of 4-byte parameters
|
||||
* paramsp will contain a pointer to the parsed parameters -- the caller
|
||||
must free.
|
||||
This function can parse filter spec strings as defined in
|
||||
the section on <a href="#filters_syntax">Filter Specification Syntax</a>.
|
||||
This function parses the first argument and returns several values.
|
||||
|
||||
2. ````int NC_filterfix8(unsigned char* mem8, int decode);````
|
||||
* mem8 is a pointer to the 8-byte value either to fix.
|
||||
* decode is 1 if the function should apply the 8-byte decoding algorithm
|
||||
else apply the encoding algorithm.
|
||||
This function implements the 8-byte conversion algorithms.
|
||||
Before calling *nc_def_var_filter* (unless *NC_parsefilterspec* was used),
|
||||
the client must call this function with the decode argument set to 0.
|
||||
Inside the filter code, this function should be called with the decode
|
||||
argument set to 1.
|
||||
|
||||
Examples of the use of these functions can be seen in the test program
|
||||
*nc_test4/tst_filterparser.c*.
|
||||
|
||||
Appendix B. Programmatic Filter Definition {#filters_programmatic}
|
||||
==========
|
||||
|
||||
HDF5 provides an API [6] to allow for the programmatic definition
|
||||
of filters -- as opposed to using the HDF5_PLUGIN_PATH environment variable.
|
||||
The idea is that instead of using dynamic shared libraries, the filter code
|
||||
is compiled into the application and the relevant information
|
||||
(namely an instance of *H5Z_class2_t*) is passed to the HDF5 library API.
|
||||
Because it is anticipated that in the future, other plugin formats
|
||||
will be used, this netcdf-c API is deliberately more general than
|
||||
strictly required by HDF5.
|
||||
|
||||
## API Concepts
|
||||
|
||||
Three concepts are used in this API.
|
||||
|
||||
1. Format - this is an integer defining the format of the plugin.
|
||||
Currently, only *NC_FILTER_FORMAT_HDF5* is defined and corresponds
|
||||
to the existing HDF5 plugin format.
|
||||
2. ID - this is an integer that is a unique identifier for the filter.
|
||||
This value is interpreted in the context of the format, so the same
|
||||
id might be assigned to different filters if the format is different.
|
||||
3. The structure *NC_FILTER_INFO* that provides generic information
|
||||
to the API and has a placeholder for format-specific information.
|
||||
|
||||
typedef struct NC_FILTER_INFO {
|
||||
int version; /* Of this structure */
|
||||
int format; /* Controls actual type of this structure */
|
||||
int id; /* Must be unique WRT format */
|
||||
void* info; /* The filter info as defined by the format. */
|
||||
} NC_FILTER_INFO;
|
||||
When the format is the value NC_FILTER_FORMAT_HDF5,
|
||||
then the info field is a pointer to an instance of
|
||||
H5Z_class2_t as define in H5Zpublic.h.
|
||||
The use of void* is, of course, to allow for passing arbitrary objects.
|
||||
|
||||
### NetCDF API
|
||||
|
||||
The following function signatures are provided (see *netcdf_filter.h*).
|
||||
|
||||
1. Register a filter
|
||||
|
||||
int nc_filter_register(NC_FILTER_INFO* filter_info);
|
||||
Register a filter whose format and ID are specified in the 'filter_info'
|
||||
argument.
|
||||
|
||||
2. Unregister a filter
|
||||
|
||||
int nc_filter_unregister(int format, int id);
|
||||
Unregister the filter specified by the id. Note that only
|
||||
filters registered using 'nc_filter_register' can be unregistered.
|
||||
|
||||
3. Inquire about a filter
|
||||
|
||||
int nc_filter_inq(int format, int id, NC_FILTER_INFO* filter_info);
|
||||
Unregister the filter specified by the id. Note that only
|
||||
filters registered using 'nc_filter_register' can be inquired.
|
||||
The 'filter_info' is filled with a copy of the original argument to
|
||||
'nc_filter_register'.
|
||||
|
||||
### Example
|
||||
|
||||
static const H5Z_class2_t H5Z_REG[1] = {
|
||||
...
|
||||
};
|
||||
...
|
||||
NC_FILTER_INFO info;
|
||||
...
|
||||
info.version = NC_FILTER_INFO_VERSION;
|
||||
info.format = NC_FILTER_FORMAT_HDF5;
|
||||
info.id = FILTER_ID;
|
||||
info.info = (void*)&H5Z_REG[0];
|
||||
stat = nc_filter_register(&info);
|
||||
...
|
||||
memset(&info,0,sizeof(NC_FILTER_INFO));
|
||||
stat = nc_filter_inq(NC_FILTER_FORMAT_HDF5, FILTER_ID, &info);
|
||||
...
|
||||
stat = nc_filter_unregister(NC_FILTER_FORMAT_HDF5, FILTER_ID);
|
||||
|
||||
# References {#filters_References}
|
||||
|
||||
1. https://support.hdfgroup.org/HDF5/doc/Advanced/DynamicallyLoadedFilters/HDF5DynamicallyLoadedFilters.pdf
|
||||
2. https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-CompressionTroubleshooting.pdf
|
||||
3. https://portal.hdfgroup.org/display/support/Contributions#Contributions-filters
|
||||
4. https://support.hdfgroup.org/services/contributions.html#filters
|
||||
5. https://support.hdfgroup.org/HDF5/doc/RM/RM_H5.html
|
||||
6. https://confluence.hdfgroup.org/display/HDF5/Filters
|
||||
|
||||
# Point of Contact
|
||||
|
||||
__Author__: Dennis Heimbigner<br>
|
||||
__Email__: dmh at ucar dot edu
|
||||
__Initial Version__: 1/10/2018<br>
|
||||
__Last Revised__: 2/5/2018
|
||||
|
||||
2932
NUG/guide.dox
Normal file
2932
NUG/guide.dox
Normal file
File diff suppressed because it is too large
Load Diff
2923
NUG/guide.md
Normal file
2923
NUG/guide.md
Normal file
File diff suppressed because it is too large
Load Diff
BIN
NUG/images/InstallTreeWindows.png
Normal file
BIN
NUG/images/InstallTreeWindows.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
4
NUG/images/Makefile.am
Normal file
4
NUG/images/Makefile.am
Normal file
@ -0,0 +1,4 @@
|
||||
EXTRA_DIST = aqua.jpg chunking2.png compatibility3.png compression.png \
|
||||
groups.png nc4-model.png ncatts.png nc-classic-uml.png nccoords.png \
|
||||
ncfile.png pnetcdf.png terra.jpg netcdf_architecture.png \
|
||||
deptree.jpg InstallTreeWindows.png
|
||||
BIN
NUG/images/aqua.jpg
Normal file
BIN
NUG/images/aqua.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 128 KiB |
BIN
NUG/images/chunking2.png
Normal file
BIN
NUG/images/chunking2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
BIN
NUG/images/compatibility3.png
Normal file
BIN
NUG/images/compatibility3.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 565 KiB |
BIN
NUG/images/compression.png
Normal file
BIN
NUG/images/compression.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
BIN
NUG/images/deptree.jpg
Normal file
BIN
NUG/images/deptree.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 98 KiB |
BIN
NUG/images/groups.png
Normal file
BIN
NUG/images/groups.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
BIN
NUG/images/nc-classic-uml.png
Normal file
BIN
NUG/images/nc-classic-uml.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
BIN
NUG/images/nc4-model.png
Normal file
BIN
NUG/images/nc4-model.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
BIN
NUG/images/ncatts.png
Normal file
BIN
NUG/images/ncatts.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
BIN
NUG/images/nccoords.png
Normal file
BIN
NUG/images/nccoords.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 7.2 KiB |
BIN
NUG/images/ncfile.png
Normal file
BIN
NUG/images/ncfile.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
NUG/images/netcdf_architecture.odg
Normal file
BIN
NUG/images/netcdf_architecture.odg
Normal file
Binary file not shown.
BIN
NUG/images/netcdf_architecture.png
Normal file
BIN
NUG/images/netcdf_architecture.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 129 KiB |
BIN
NUG/images/pnetcdf.png
Normal file
BIN
NUG/images/pnetcdf.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
BIN
NUG/images/terra.jpg
Normal file
BIN
NUG/images/terra.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
NUG/images/unidata_logo_cmyk.png
Normal file
BIN
NUG/images/unidata_logo_cmyk.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 5.9 KiB |
500
NUG/install.md
Normal file
500
NUG/install.md
Normal file
@ -0,0 +1,500 @@
|
||||
Getting and Building netCDF {#getting_and_building_netcdf}
|
||||
=============================
|
||||
|
||||
[TOC]
|
||||
|
||||
This document is for getting and building the netCDF C library and utilities for the most recent released version. Other libraries that depend on the netCDF C library, such as the Fortran, Python, Java, and C++ libraries, are available as separate distributions that can be optionally built and installed after the C library is successfully installed. The netCDF-Java library is independent of the netCDF C library unless writing netCDF-4 files from Java is required.
|
||||
|
||||
Getting netCDF-C {#getting}
|
||||
=========================
|
||||
|
||||
* For information regarding the netCDF-Fortran libraries, see \subpage building_netcdf_fortran.
|
||||
* Functionality to make it easier to build netcdf-fortran as part of the netcdf-c build for *non-MSVC* builds may be enabled at configure time by using the following **Highly Experimental** options:
|
||||
|
||||
* Autotools: `--enable-remote-fortran-bootstrap`
|
||||
* CMake: `-DENABLE_REMOTE_FORTRAN_BOOTSTRAP=ON`
|
||||
|
||||
For more details, see the <a href="https://github.com/Unidata/netcdf-c/blob/master/RELEASE_NOTES.md">draft instructions</a> in the Release Notes under the `4.3.3-rc3` section.
|
||||
|
||||
Getting pre-built netCDF-C libraries. {#sec_get_pre_built}
|
||||
-------------------------------------
|
||||
|
||||
The easiest way to get netCDF is through a package management program,
|
||||
such as rpm, yum, homebrew, macports, adept, and others. NetCDF is
|
||||
available from many different repositories, including the default Red
|
||||
Hat and Ubuntu repositories.
|
||||
|
||||
When getting netCDF from a software repository, you should get a
|
||||
development version that includes the netcdf.h header file. A
|
||||
development version will typically have a name such as "netcdf-devel"
|
||||
or "libnetcdf-dev".
|
||||
|
||||
Instructions for installing and using pre-built libraries for Windows may be found here: \ref winbin.
|
||||
|
||||
Getting the latest netCDF-C Source Code {#sec_get_source}
|
||||
----------------------------------------
|
||||
|
||||
The netCDF-C source code is hosted from the <a href="http://github.com/Unidata/netcdf-c" >Unidata GitHub repository</a>.
|
||||
|
||||
|
||||
Two options are available for building from source:
|
||||
|
||||
* The latest release.
|
||||
* The developer snapshot.
|
||||
|
||||
### The latest release {#sec_latest_release}
|
||||
|
||||
The latest full release may be <a href="http://github.com/Unidata/netcdf-c/releases" >downloaded from GitHub</a>.
|
||||
|
||||
Source files are available in `.tar.gz` and `.zip` formats.
|
||||
|
||||
### The developer snapshot {#sec_dev_snapshot}
|
||||
|
||||
The developer snapshot may be cloned from GitHub directly by using the `git` command.
|
||||
|
||||
> $ git clone http://github.com/Unidata/netcdf-c netcdf-c
|
||||
|
||||
**Note:**
|
||||
|
||||
*The developer snapshot release contains bug-fixes and new features added since the last full release, but may also contain new bugs, as it is not tested as extensively as the full release.*
|
||||
|
||||
Building netCDF-C {#building}
|
||||
===========================
|
||||
|
||||
The netCDF-C library and utilities require third-party libraries for
|
||||
full functionality. (See \ref architecture).
|
||||
* \ref build_default
|
||||
* \ref build_classic
|
||||
* \ref build_hdf4
|
||||
* \ref build_parallel
|
||||
* \ref building_netcdf_fortran
|
||||
* \ref configure_options
|
||||
|
||||
Requirements {#netcdf_requirements}
|
||||
----------------------------------
|
||||
|
||||
* For netCDF-4 support
|
||||
* HDF5 1.8.9 or later.
|
||||
* HDF5 1.10.1 or later.
|
||||
* zlib 1.2.5 or later (for netCDF-4 compression)
|
||||
* curl 7.18.0 or later (for DAP remote access client support)
|
||||
* For parallel I/O support on classic netCDF files
|
||||
* PnetCDF 1.6.0 or later
|
||||
|
||||
> **Important Note**: When building netCDF-C library versions older than 4.4.1, use only HDF5 1.8.x versions. Combining older netCDF-C versions with newer HDF5 1.10 versions will create superblock 3 files that are not readable by lots of older software. See <a href="http://www.unidata.ucar.edu/blogs/news/entry/netcdf-4-4-1">this announcement</a> for more details.
|
||||
|
||||
|
||||
CMake and Windows support {#sub}
|
||||
--------------------------------
|
||||
|
||||
* \ref netCDF-CMake
|
||||
* \subpage winbin
|
||||
|
||||
Building with netCDF-4 and the Remote Data Client {#build_default}
|
||||
--------------------------------
|
||||
|
||||
The usual way of building netCDF requires the HDF5, zlib, and curl libraries. Versions required are at least HDF5 1.8.9, zlib 1.2.5, and curl 7.18.0 or later.
|
||||
|
||||
HDF5 and zlib packages are available from the <a href="http://www.hdfgroup.org/downloads/">HDF5 downloads site</a> and the <a href="http://www.zlib.net/">zlib home site</a>. If you wish to use the remote data client code, then you will also need libcurl, which can be obtained from the <a href="http://curl.haxx.se/download.html">curl website</a>.
|
||||
|
||||
> Note that for building netCDF, it is not necessary to build the HDF5 Fortran, C++, or Java API's. Only the HDF5 C library is used, even for netCDF Fortran or C++ libraries.
|
||||
|
||||
### Optional: szip support {#op_szip_support}
|
||||
|
||||
*Optionally*, you can also build netCDF-4 with the szip library (a.k.a. szlib). If building with szlib, get szip 2.0 or later. Technically, we mean that
|
||||
the HDF5 library is built with szip support. The netcdf build will then
|
||||
inherit szip support from the HDF5 library.
|
||||
If you intend to write files with szip compression, then we suggest that you
|
||||
use [libaec](https://gitlab.dkrz.de/k202009/libaec.git)
|
||||
to avoid patent problems. That library can be used as a
|
||||
drop-in replacement for the standard szip library.
|
||||
If you plan to use the standard szip library,
|
||||
then determine whether license restrictions on the use of szip apply to your situation. See the <a href="http://www.hdfgroup.org/doc_resource/SZIP/">web page on szip compression in HDF products</a>.
|
||||
|
||||
If `make check` fails for either `zlib` or `HDF5`, the problem must be resolved before the netCDF-4 installation can continue. For HDF5 problems, see the <a href="http://www.hdfgroup.org/services/support.html">HDF5 help services</a>.
|
||||
|
||||
### Building zlib from source {#build_zlib_from_source}
|
||||
|
||||
To build zlib from source, specify where you want to install zlib in a shell variable you will also use later (ZDIR, for example), and build it like this from the top-level zlib source directory
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install zlib
|
||||
$ ZDIR=/usr/local
|
||||
$ ./configure --prefix=${ZDIR}
|
||||
$ make check
|
||||
$ make install # or sudo make install, if root permissions required
|
||||
~~~~
|
||||
|
||||
### Building hdf5 from source {#build_hdf5_from_source}
|
||||
|
||||
Next, specify where you want to install HDF5 in another shell variable, for example H5DIR, and build it from the HDF5 top-level source directory:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install HDF5
|
||||
$ H5DIR=/usr/local
|
||||
$ ./configure --with-zlib=${ZDIR} --prefix=${H5DIR} --enable-hl
|
||||
$ make check
|
||||
$ make install # or sudo make install, if root permissions required
|
||||
~~~~
|
||||
|
||||
If you are building HDF5 with the optional szip library, include the `--with-szlib=` option to specify where it was installed.
|
||||
|
||||
In all cases, the installation location specified with the `--prefix` option *must be different* from the source directory where the software is being built.
|
||||
|
||||
### Building netCDF-4 and the Remote Data Client from source {#build_nc4_dap_from_source}
|
||||
|
||||
Before building netCDF, you may need to add `${H5DIR}/lib` to the LD_LIBRARY_PATH environment variable if that lib directory is not searched by default. See the <a href="http://www.unidata.ucar.edu/netcdf/docs/faq.html#Shared%20Libraries">netCDF FAQ</a> for more details on using shared libraries.
|
||||
|
||||
Indicate where you want to install netCDF in another shell variable, for example NCDIR. Then run the netCDF configure script, specifying where HDF5 was installed using the CPPFLAGS and LDFLAGS environment variables. For example, from the top-level netCDF source directory:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install netCDF-4
|
||||
$ NCDIR=/usr/local
|
||||
$ CPPFLAGS='-I${H5DIR}/include -I${ZDIR}/include' LDFLAGS='-L${H5DIR}/lib -L${ZDIR}/lib' ./configure --prefix=${NCDIR}
|
||||
$ make check
|
||||
$ make install # or sudo make install
|
||||
~~~~
|
||||
|
||||
If you don't provide a `--prefix` option, installation will be in `/usr/local/`, in subdirectories lib/, include/, and bin/. The installation location specified with the `--prefix` option must be different from the source directory where the software is being built.
|
||||
|
||||
> WARNING: you should be able to use parallel 'make all'. But 'make check' will probably fail if you use parallel make. This is because historically, there are inter-dependencies between test programs. It is unlikely that this will be fixed any time soon, if ever.
|
||||
|
||||
Building netCDF with Classic Library Only {#build_classic}
|
||||
---------------------------------------
|
||||
|
||||
It is possible to build the netCDF C libraries and utilities so that
|
||||
only the netCDF classic and 64-bit offset formats are supported, or
|
||||
the remote data access client is not built. (See \ref netcdf_format
|
||||
for more information about the netCDF format variants. See the <a
|
||||
href="http://www.opendap.org/documentation">DAP documentation and
|
||||
support site</a> for more information about remote client access to
|
||||
data on OPeNDAP servers.)
|
||||
|
||||
If necessary, set the NCDIR shell variable to indicate where netCDF should be
|
||||
installed. Then to build a netCDF-3 library without support for the
|
||||
netCDF-4 formats or functions, but with remote client access, use:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install netCDF-3 from netCDF-4 source
|
||||
$ ./configure --prefix=${NCDIR} --disable-netcdf-4
|
||||
$ make check install
|
||||
~~~~
|
||||
|
||||
To build with full support for netCDF-4 API's and format but without
|
||||
remote client access, use:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install netCDF-4 without DAP client support
|
||||
$ ./configure --prefix=${NCDIR} --disable-dap
|
||||
$ make check install
|
||||
~~~~
|
||||
|
||||
To build without netCDF-4 support or remote client access, use:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install minimal netCDF-3 with no DAP client support
|
||||
$ ./configure --prefix=${NCDIR} --disable-netcdf-4 --disable-dap
|
||||
$ make check install
|
||||
~~~~
|
||||
|
||||
If you get the message that netCDF installed correctly, then you are
|
||||
done!
|
||||
|
||||
Building with HDF4 Support {#build_hdf4}
|
||||
---------------------
|
||||
|
||||
The netCDF-4 library can read HDF4 data files, if they were created
|
||||
with the SD (Scientific Data) API.
|
||||
|
||||
For this to work, you must build the HDF4 library with the
|
||||
configure option `--disable-netcdf`
|
||||
to prevent it from building an HDF4 version of the netCDF-2 library
|
||||
that conflicts with the netCDF-2 functions that are built into the Unidata
|
||||
netCDF library.
|
||||
|
||||
Then, when building netCDF-4, use the `--enable-hdf4`
|
||||
option to configure. The location for the HDF4 header files and
|
||||
library must be specified in the CPPFLAGS and LDFLAGS environment variables
|
||||
or configure options.
|
||||
|
||||
For HDF4 access to work, the library must be built with netCDF-4
|
||||
features.
|
||||
|
||||
Here's an example, assuming the HDF5 library has been built and
|
||||
installed in H5DIR and you will build and install the HDF4 library in
|
||||
H4DIR (which could be the same as H5DIR). From the top-level HDF4
|
||||
source directory:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install HDF4
|
||||
$ ./configure --enable-shared --disable-netcdf --disable-fortran --prefix=${H4DIR}
|
||||
$ make check
|
||||
$ make install
|
||||
~~~~
|
||||
|
||||
Then from the top-level netCDF directory:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install netCDF-4 with HDF4 access enabled
|
||||
$ CPPFLAGS="-I${H5DIR}/include -I${H4DIR}/include" \
|
||||
LDFLAGS="-L${H5DIR}/lib -L${H4DIR}/lib" \
|
||||
./configure --enable-hdf4 --enable-hdf4-file-tests
|
||||
$ make check
|
||||
$ make install
|
||||
~~~~
|
||||
|
||||
Building with Parallel I/O Support {#build_parallel}
|
||||
--------------
|
||||
|
||||
For parallel I/O to work, HDF5 must be installed with
|
||||
`--enable-parallel`, and an MPI library (and related libraries) must be
|
||||
made available to the HDF5 configure. This can be accomplished with
|
||||
an mpicc wrapper script.
|
||||
|
||||
The following works from the top-level HDF5 source directory to build
|
||||
HDF5 with parallel I/O:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install HDF5 with parallel support
|
||||
$ CC=mpicc ./configure --enable-parallel --prefix=${H5DIR}
|
||||
$ make check
|
||||
$ make install
|
||||
~~~~
|
||||
|
||||
If the HDF5 used by netCDF has been built with parallel I/O, then netCDF will also be built with inherited support for parallel I/O. This allows parallel I/O access to netCDF-4/HDF5 files. (See /ref netcdf_formats for more information about the netCDF format variants.)
|
||||
|
||||
From the top-level netCDF-4 source directory, the following builds netCDF-4 with parallel I/O, assuming H5DIR specifies where parallel HDF5 was installed:
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build, test, and install netCDF-4 with HDF5 parallel support
|
||||
$ CC=mpicc CPPFLAGS=-I${H5DIR}/include LDFLAGS=-L${H5DIR}/lib \
|
||||
./configure --disable-shared --enable-parallel-tests --prefix=${NCDIR}
|
||||
$ make check
|
||||
$ make install
|
||||
~~~~
|
||||
|
||||
### Building PnetCDF from source {#build_pnetcdf_from_source}
|
||||
|
||||
To enable parallel I/O support for classic netCDF files, i.e. CDF-1, 2 and 5
|
||||
formats, [PnetCDF library](https://parallel-netcdf.github.io) must also be
|
||||
installed. First specify where you want to install PnetCDF in a shell
|
||||
variable, for example PNDIR, and build it from the PnetCDF top-level source
|
||||
directory. If you would like to build the shared library, include
|
||||
`--enable-shared` option at the configure command line. By default, only a
|
||||
static library is built.
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build and install PnetCDF
|
||||
$ PNDIR=/usr/local
|
||||
$ ./configure --prefix=${PNDIR} --with-mpi=/path/to/MPI/compilers
|
||||
$ make check
|
||||
$ make install # or sudo make install, if root permissions required
|
||||
~~~~
|
||||
|
||||
To build netCDF-4 with PnetCDF support, from the top-level netCDF-4 source
|
||||
directory, configure netCDF with the "--enable-pnetcdf" option. If PnetCDF
|
||||
is built with static library only, add "--disable-shared" option.
|
||||
|
||||
~~~~{.py}
|
||||
$ # Build, test, and install netCDF-4 with PnetCDF support
|
||||
$ CC=mpicc CPPFLAGS="-I${H5DIR}/include -I${PNDIR}/include" \
|
||||
LDFLAGS="-L${H5DIR}/lib -L${PNDIR}/lib" ./configure \
|
||||
--enable-pnetcdf --enable-parallel-tests \
|
||||
--prefix=${NCDIR}
|
||||
$ make check
|
||||
$ make install
|
||||
~~~~
|
||||
|
||||
Linking to netCDF-C {#linking}
|
||||
-------------------
|
||||
|
||||
For static builds of applications that use netCDF-4 you must link to all the
|
||||
libraries, netCDF, HDF5, zlib, szip (if used with HDF5 build), PnetCDF (if used
|
||||
with PnetCDF build), and curl (if the remote access client has not been
|
||||
disabled). This will require -L options to your build for the locations of the
|
||||
libraries, and -l (lower-case L) for the names of the libraries.
|
||||
|
||||
For example, you might build other applications with netCDF-4 by
|
||||
setting the LIBS environment variable, assuming NCDIR, H5DIR, PNDIT, and ZDIR
|
||||
indicate where netCDF, HDF5, PnetCDF, and zlib are installed:
|
||||
|
||||
~~~~{.py}
|
||||
LIBS="-L${NCDIR}/lib -lnetcdf -L${H5DIR}/lib -lhdf5_hl -lhdf5 -L${PNDIR} -lpnetcdf -L${ZDIR}/lib -lz -lm"
|
||||
~~~~
|
||||
|
||||
For shared builds, only `-L${NCDIR}/lib -lnetcdf` is
|
||||
needed. All other libraries will be found automatically.
|
||||
|
||||
The `pkg-config` or `nc-config` utilities can be
|
||||
used to specify build options for software that uses netCDF. For
|
||||
example, to compile and link an application named myapp.c with a
|
||||
netCDF-C libraries, whether shared or static, you can use
|
||||
|
||||
~~~~{.py}
|
||||
$ cc -o myapp myapp.c `nc-config --cflags --libs`
|
||||
~~~~
|
||||
|
||||
or
|
||||
|
||||
~~~~{.py}
|
||||
$ PKG_CONFIG_PATH=${NCDIR}/lib/pkgconfig:$PKG_CONFIG_PATH
|
||||
$ export PKG_CONFIG_PATH
|
||||
$ cc -o myapp myapp.c `pkg-config --cflags --libs netcdf`
|
||||
~~~~
|
||||
|
||||
configure options {#configure_options}
|
||||
-----------------------------
|
||||
|
||||
These options are used for `autotools`-based builds.yup
|
||||
|
||||
Note: `--disable` prefix indicates that the option is normally enabled.
|
||||
<table>
|
||||
<tr><th>Option<th>Description<th>Dependencies
|
||||
<tr><td>--disable-doxygen<td>Disable generation of documentation.<td>doxygen
|
||||
<tr><td>--disable-fsync<td>disable fsync support<td>kernel fsync support
|
||||
<tr><td>--disable-netcdf-4<td>build netcdf-3 without HDF5 and zlib<td>
|
||||
<tr><td>--disable-netcdf4<td>synonym for disable-netcdf-4<td>
|
||||
<tr><td>--enable-hdf4<td>build netcdf-4 with HDF4 read capability<td>HDF4, HDF5 and zlib
|
||||
<tr><td>--enable-hdf4-file-tests<td>test ability to read HDF4 files<td>selected HDF4 files from Unidata ftp site
|
||||
<tr><td>--disable-parallel4<td>build netcdf-4 without parallel I/O support<td>
|
||||
<tr><td>--disable-cdf5<td>build netcdf-4 without support of classic CDF-5 file format<td>
|
||||
<tr><td>--enable-pnetcdf<td>build netcdf-4 with parallel I/O for classic files (CDF-1, 2, and 5 formats) using PnetCDF<td>PnetCDF
|
||||
<tr><td>--enable-extra-example-tests<td>Run extra example tests<td>--enable-netcdf-4,GNU sed
|
||||
<tr><td>--disable-filter-testing<td>Run filter example<td>--enable-shared --enable-netcdf-4
|
||||
<tr><td>--enable-parallel-tests <td>run extra parallel IO tests<td>--enable-netcdf-4 or --enable-pnetcdf, parallel IO support
|
||||
<tr><td>--enable-logging<td>enable logging capability<td>--enable-netcdf-4
|
||||
<tr><td>--disable-dap<td>build without DAP client support.<td>libcurl
|
||||
<tr><td>--disable-dap-remote-tests<td>disable dap remote tests<td>--enable-dap
|
||||
<tr><td>--enable-dap-long-tests<td>enable dap long tests<td>
|
||||
<tr><td>--enable-extra-tests<td>run some extra tests that may not pass because of known issues<td>
|
||||
<tr><td>--enable-ffio<td>use ffio instead of posixio (ex. on the Cray)<td>
|
||||
<tr><td>--disable-examples<td>don't build the netCDF examples during make check
|
||||
(examples are treated as extra tests by netCDF)<td>
|
||||
<tr><td>--disable-v2<td>turn off the netCDF version 2 API<td>
|
||||
<tr><td>--disable-utilities<td>don't build netCDF utilities ncgen, ncdump, and nccopy<td>
|
||||
<tr><td>--disable-testsets<td>don't build or run netCDF tests<td>
|
||||
<tr><td>--enable-large-file-tests <td>Run tests which create very large data
|
||||
files<td>~13 GB disk space required, but recovered when
|
||||
tests are complete). See option --with-temp-large to
|
||||
specify temporary directory
|
||||
<tr><td>--enable-benchmarks<td>Run benchmarks. This is an experimental feature.
|
||||
The benchmarks are extra tests, used to check netCDF performance.
|
||||
<td>sample data files from the Unidata ftp site
|
||||
<tr><td>--disable-extreme-numbers
|
||||
<td>don't use extreme numbers during testing, such as MAX_INT - 1<td>
|
||||
<tr><td>--disable-shared<td>don't build shared libraries<td>
|
||||
<tr><td>--disable-static<td>don't build static libraries<td>
|
||||
<tr><td>--disable-largefile<td>omit support for files larger than 2GB<td>
|
||||
<tr><td>--enable-mmap<td>Use mmap to implement NC_DISKLESS<td>System-provided `mmap` or `mremap` functions
|
||||
<tr><td>--enable-valgrind-tests <td>build with valgrind-tests; static builds only<td>valgrind
|
||||
</table>
|
||||
|
||||
Build Instructions for netCDF-C using CMake {#netCDF-CMake}
|
||||
===========================================
|
||||
|
||||
## Overview {#cmake_overview}
|
||||
|
||||
Starting with netCDF-C 4.3.0, we are happy to announce the inclusion of CMake support. CMake will allow for building netCDF on a wider range of platforms, include Microsoft Windows with Visual Studio. CMake support also provides robust unit and regression testing tools. We will also maintain the standard autotools-based build system in parallel.
|
||||
|
||||
In addition to providing new build options for netCDF-C, we will also provide pre-built binary downloads for the shared versions of netCDF for use with Visual Studio.
|
||||
|
||||
|
||||
## Requirements {#cmake_requirements}
|
||||
The following packages are required to build netCDF-C using CMake.
|
||||
|
||||
* netCDF-C Source Code
|
||||
* CMake version 2.8.12 or greater.
|
||||
* Optional Requirements:
|
||||
* HDF5 Libraries for netCDF4/HDF5 support.
|
||||
* libcurl for DAP support.
|
||||
* PnetCDF libraries for parallel I/O support to classic netCDF files
|
||||
|
||||
<center>
|
||||
<img src="deptree.jpg" height="250px" />
|
||||
</center>
|
||||
|
||||
## The CMake Build Process {#cmake_build}
|
||||
|
||||
There are four steps in the Build Process when using CMake
|
||||
|
||||
1. Configuration: Before compiling, the software is configured based on the desired options.
|
||||
2. Building: Once configuration is complete, the libraries are compiled.
|
||||
3. Testing: Post-build, it is possible to run tests to ensure the functionality of the netCDF-C libraries.
|
||||
4. Installation: If all tests pass, the libraries can be installed in the location specified during configuration.
|
||||
|
||||
For users who prefer pre-built binaries, installation packages are available at \ref winbin
|
||||
|
||||
### Configuration {#cmake_configuration}
|
||||
|
||||
The output of the configuration step is a project file based on the appropriate configurator specified. Common configurators include:
|
||||
|
||||
* Unix Makefiles
|
||||
* Visual Studio
|
||||
* CodeBlocks
|
||||
* ... and others
|
||||
|
||||
### Common CMake Options {#cmake_common_options}
|
||||
|
||||
| **Option** | **Autotools** | **CMake** |
|
||||
| :------- | :---- | :----- |
|
||||
Specify Install Location | --prefix=PREFIX | -D"CMAKE_INSTALL_PREFIX=PREFIX"
|
||||
Enable/Disable netCDF-4 | --enable-netcdf-4<br>--disable-netcdf-4 | -D"ENABLE_NETCDF_4=ON" <br> -D"ENABLE_NETCDF_4=OFF"
|
||||
Enable/Disable DAP | --enable-dap <br> --disable-dap | -D"ENABLE_DAP=ON" <br> -D"ENABLE_DAP=OFF"
|
||||
Enable/Disable Utilities | --enable-utilities <br> --disable-utilities | -D"BUILD_UTILITIES=ON" <br> -D"BUILD_UTILITIES=OFF"
|
||||
Specify shared/Static Libraries | --enable-shared <br> --enable-static | -D"BUILD_SHARED_LIBS=ON" <br> -D"BUILD_SHARED_LIBS=OFF"
|
||||
Enable/Disable Parallel netCDF-4 | --enable-parallel4 <br> --disable-parallel4 | -D"ENABLE_PARALLEL4=ON" <br> -D"ENABLE_PARALLEL4=OFF"
|
||||
Enable/Disable PnetCDF | --enable-pnetcdf<br>--disable-pnetcdf | -D"ENABLE_PNETCDF=ON" <br> -D"ENABLE_PNETCDF=OFF"
|
||||
Enable/Disable CDF5 | --enable-cdf5 <br> --disable-cdf5 | -D"ENABLE_CDF5=ON" <br> -D"ENABLE_CDF5=OFF"
|
||||
Enable/Disable Tests | --enable-testsets <br> --disable-testsets | -D"ENABLE_TESTS=ON" <br> -D"ENABLE_TESTS=OFF"
|
||||
Enable/Disable Parallel Tests | --enable-parallel-tests<br> --disable-parallel-tests | -D"ENABLE_PARALLEL_TESTS=ON" <br> -D"ENABLE_PARALLEL_TESTS=OFF"
|
||||
Specify a custom library location | Use *CFLAGS* and *LDFLAGS* | -D"CMAKE_PREFIX_PATH=/usr/custom_libs/"
|
||||
|
||||
A full list of *basic* options can be found by invoking `cmake [Source Directory] -L`. To enable a list of *basic* and *advanced* options, one would invoke `cmake [Source Directory] -LA`.
|
||||
|
||||
### Configuring your build from the command line. {#cmake_command_line}
|
||||
|
||||
The easiest configuration case would be one in which all of the dependent libraries are installed on the system path (in either Unix/Linux or Windows) and all the default options are desired. From the build directory (often, but not required to be located within the source directory):
|
||||
|
||||
> $ cmake [Source Directory]
|
||||
|
||||
If you have libraries installed in a custom directory, you may need to specify the **CMAKE\_PREFIX_PATH** variable to tell cmake where the libraries are installed. For example:
|
||||
|
||||
> $ cmake [Source Directory] -DCMAKE\_PREFIX\_PATH=/usr/custom_libraries/
|
||||
|
||||
## Building {#cmake_building}
|
||||
|
||||
The compiler can be executed directly with 'make' or the appropriate command for the configurator which was used.
|
||||
|
||||
> $ make
|
||||
|
||||
Building can also be executed indirectly via cmake:
|
||||
|
||||
> $ cmake --build [Build Directory]
|
||||
|
||||
## Testing {#cmake_testing}
|
||||
|
||||
Testing can be executed several different ways:
|
||||
|
||||
> $ make test
|
||||
|
||||
or
|
||||
|
||||
> $ ctest
|
||||
|
||||
or
|
||||
|
||||
> $ cmake --build [Build Directory] --target test
|
||||
|
||||
### Installation {#cmake_installation}
|
||||
|
||||
Once netCDF has been built and tested, it may be installed using the following commands:
|
||||
|
||||
> $ make install
|
||||
|
||||
or
|
||||
|
||||
> $ cmake --build [Build Directory] --target install
|
||||
|
||||
## See Also {#cmake_see_also}
|
||||
|
||||
For further information regarding netCDF and CMake, see \ref cmake_faq.
|
||||
266
NUG/nug.tag
Normal file
266
NUG/nug.tag
Normal file
@ -0,0 +1,266 @@
|
||||
<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>
|
||||
<tagfile>
|
||||
<compound kind="file">
|
||||
<name>bestpractices.md</name>
|
||||
<path>/home/wfisher/Desktop/gitprojects/netcdf-c/NUG/</path>
|
||||
<filename>bestpractices_8md</filename>
|
||||
</compound>
|
||||
<compound kind="file">
|
||||
<name>DAP4.dox</name>
|
||||
<path>/home/wfisher/Desktop/gitprojects/netcdf-c/NUG/</path>
|
||||
<filename>_d_a_p4_8dox</filename>
|
||||
</compound>
|
||||
<compound kind="file">
|
||||
<name>filters.md</name>
|
||||
<path>/home/wfisher/Desktop/gitprojects/netcdf-c/NUG/</path>
|
||||
<filename>filters_8md</filename>
|
||||
</compound>
|
||||
<compound kind="file">
|
||||
<name>guide.md</name>
|
||||
<path>/home/wfisher/Desktop/gitprojects/netcdf-c/NUG/</path>
|
||||
<filename>guide_8md</filename>
|
||||
</compound>
|
||||
<compound kind="file">
|
||||
<name>install.md</name>
|
||||
<path>/home/wfisher/Desktop/gitprojects/netcdf-c/NUG/</path>
|
||||
<filename>install_8md</filename>
|
||||
</compound>
|
||||
<compound kind="file">
|
||||
<name>OPeNDAP.dox</name>
|
||||
<path>/home/wfisher/Desktop/gitprojects/netcdf-c/NUG/</path>
|
||||
<filename>_o_pe_n_d_a_p_8dox</filename>
|
||||
</compound>
|
||||
<compound kind="file">
|
||||
<name>user_defined_formats.md</name>
|
||||
<path>/home/wfisher/Desktop/gitprojects/netcdf-c/NUG/</path>
|
||||
<filename>user__defined__formats_8md</filename>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>nug_netcdf_introduction</name>
|
||||
<title>An Introduction to NetCDF</title>
|
||||
<filename>nug_netcdf_introduction</filename>
|
||||
<docanchor file="nug_netcdf_introduction" title="The netCDF Interface">nug_netcdf_interface</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="The netCDF File Format">nug_netcdf_format</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="How to Select the Format">nug_select_format</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="NetCDF Classic Format (CDF-1)">nug_classic_format</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="NetCDF 64-bit Offset Format (CDF-2)">nug_netcdf_64bit_offset_format</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="NetCDF 64-bit Data Format (CDF-5)">nug_netcdf_64bit_data_format</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="NetCDF-4 Format">nug_netcdf_4_format</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="NetCDF Library Architecture">nug_architecture</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="What about Performance?">nug_performance</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="Creating Self-Describing Data conforming to Conventions">nug_creating_self</docanchor>
|
||||
<docanchor file="nug_netcdf_introduction" title="Limitations of netCDF">nug_limitations</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>nug_netcdf_data_set_components</name>
|
||||
<title>The Components of a NetCDF Data Set</title>
|
||||
<filename>nug_netcdf_data_set_components</filename>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="The Data Model">nug_data_model</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Data Model in NetCDF-4/HDF5 Files">nug_Enhanced</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Dimensions">nug_dimensions</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Variables">nug_variables</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Coordinate Variables">nug_coordinate_variables</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Attributes">nug_attributes</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Differences between Attributes and Variables">nug_differences_atts_vars</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="NetCDF Names">nug_object_name</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Characters in NetCDF Names">nug_Permitted</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Length">nug_Name</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Conventions">nug_NetCDF</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Is NetCDF a Good Archive Format?">nug_archival</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Background and Evolution of the NetCDF Interface">nug_background</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="The Remote Data Access Client">nug_remote_client</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Data Access">nug_data_access</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="Forms of Data Access">nug_forms_of_data_access</docanchor>
|
||||
<docanchor file="nug_netcdf_data_set_components" title="A C Example of Array-Section Access">nug_c_array_section_access</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>nug_file_structure_and_performance</name>
|
||||
<title>File Structure and Performance</title>
|
||||
<filename>nug_file_structure_and_performance</filename>
|
||||
<docanchor file="nug_file_structure_and_performance" title="Parts of a NetCDF Classic File">nug_classic_file_parts</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="Parts of a NetCDF-4 HDF5 File">nug_parts_of_netcdf4</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="The Extended XDR Layer">nug_xdr_layer</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="Large File Support">nug_large_file_support</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="NetCDF 64-bit Offset Format Limitations">nug_offset_format_limitations</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="NetCDF Classic Format Limitations">nug_classic_format_limitations</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="The NetCDF-3 I/O Layer">nug_netcdf_3_io</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="Parallel Access with NetCDF-4">nug_parallel_access</docanchor>
|
||||
<docanchor file="nug_file_structure_and_performance" title="Interoperability with HDF5">nug_interoperability_with_hdf5</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>nug_netcdf_perf_chunking</name>
|
||||
<title>Improving Performance with Chunking</title>
|
||||
<filename>nug_netcdf_perf_chunking</filename>
|
||||
<docanchor file="nug_netcdf_perf_chunking" title="The Chunk Cache">nug_chunk_cache</docanchor>
|
||||
<docanchor file="nug_netcdf_perf_chunking" title="The Default Chunking Scheme">nug_default_chunking_4_1</docanchor>
|
||||
<docanchor file="nug_netcdf_perf_chunking" title="Chunking and Parallel I/O">nug_chunking_parallel_io</docanchor>
|
||||
<docanchor file="nug_netcdf_perf_chunking" title="A Utility to Help Benchmark Results: bm_file">nug_bm_file</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>nug_netcdf_utilities_guide</name>
|
||||
<title>NetCDF Utilities</title>
|
||||
<filename>nug_netcdf_utilities_guide</filename>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="CDL Guide">nug_cdl_guide</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="CDL Syntax">nug_cdl_syntax</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="CDL Data Types">nug_cdl_data_types</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="CDL Notation for Data Constants">nug_cdl_constants</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="ncdump">nug_ncdump_guide</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="ncdump synopsis">nug_ncdump_SYNOPSIS</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="ncdump description">nug_ncdump_DESCRIPTION</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="ncdump options">nug_ncdump_OPTIONS</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="ncdump examples">nug_ncdump_EXAMPLES</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="SEE ALSO">nug_see_also_ncdump</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="NOTE ON STRING OUTPUT">nug_ncdump_string_note</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="nccopy">nug_guide_nccopy</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="nccopy synopsis">nug_nccopy_SYNOPSIS</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="nccopy description">nug_nccopy_DESCRIPTION</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="nccopy options">nug_nccopy_OPTIONS</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="nccopy examples">nug_nccopy_EXAMPLES</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="SEE ALSO">nug_see_also_nccopy</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="ncgen">nug_guide_ncgen</docanchor>
|
||||
<docanchor file="nug_netcdf_utilities_guide" title="ncgen3">nug_guide_ncgen3</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>nug_users_guide_appendices</name>
|
||||
<title>Appendices</title>
|
||||
<filename>nug_users_guide_appendices</filename>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>BestPractices</name>
|
||||
<title>Writing NetCDF Files: Best Practices</title>
|
||||
<filename>_best_practices</filename>
|
||||
<docanchor file="_best_practices" title="Best Practices ">bp_Best_Practices</docanchor>
|
||||
<docanchor file="_best_practices" title="Conventions">bp_Conventions</docanchor>
|
||||
<docanchor file="_best_practices" title="Coordinate Systems">bp_Coordinate-Systems</docanchor>
|
||||
<docanchor file="_best_practices" title="Variable Grouping">bp_Variable-Grouping</docanchor>
|
||||
<docanchor file="_best_practices" title="Variable Attributes">bp_Variable-Attributes</docanchor>
|
||||
<docanchor file="_best_practices" title="Strings and Variables of type char">bp_Strings-and-Variables-of-type-char</docanchor>
|
||||
<docanchor file="_best_practices" title="Calendar Date/Time">bp_Calendar-Date-Time</docanchor>
|
||||
<docanchor file="_best_practices" title="Unsigned Data">bp_Unsigned-Data</docanchor>
|
||||
<docanchor file="_best_practices" title="Packed Data Values">bp_Packed-Data-Values</docanchor>
|
||||
<docanchor file="_best_practices" title="Missing Data Values">bp_Missing-Data-Values</docanchor>
|
||||
<docanchor file="_best_practices" title="Miscellaneous tips">bp_Miscellaneous-tips</docanchor>
|
||||
<docanchor file="_best_practices" title="Spelling netCDF: Best Practices">bp_Spelling-netCDF-Best-Practices</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>getting_and_building_netcdf</name>
|
||||
<title>Getting and Building netCDF</title>
|
||||
<filename>getting_and_building_netcdf</filename>
|
||||
<docanchor file="getting_and_building_netcdf" title="Getting netCDF-C ">getting</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Getting pre-built netCDF-C libraries. ">sec_get_pre_built</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Getting the latest netCDF-C Source Code ">sec_get_source</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="The latest release">sec_latest_release</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="The developer snapshot">sec_dev_snapshot</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building netCDF-C ">building</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Requirements ">netcdf_requirements</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="CMake and Windows support ">sub</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building with netCDF-4 and the Remote Data Client ">build_default</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Optional: szip support">op_szip_support</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building zlib from source">build_zlib_from_source</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building hdf5 from source">build_hdf5_from_source</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building netCDF-4 and the Remote Data Client from source">build_nc4_dap_from_source</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building netCDF with Classic Library Only ">build_classic</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building with HDF4 Support ">build_hdf4</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building with Parallel I/O Support ">build_parallel</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building PnetCDF from source">build_pnetcdf_from_source</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Linking to netCDF-C ">linking</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="configure options ">configure_options</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Build Instructions for netCDF-C using CMake ">netCDF-CMake</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Overview">cmake_overview</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Requirements">cmake_requirements</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="The CMake Build Process">cmake_build</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Configuration">cmake_configuration</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Common CMake Options">cmake_common_options</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Configuring your build from the command line.">cmake_command_line</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Building">cmake_building</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Testing">cmake_testing</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="Installation">cmake_installation</docanchor>
|
||||
<docanchor file="getting_and_building_netcdf" title="See Also">cmake_see_also</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>dap4</name>
|
||||
<title>DAP4 Protocol Support</title>
|
||||
<filename>dap4</filename>
|
||||
<docanchor file="dap4" title="DAP4 Introduction">dap4_introduction</docanchor>
|
||||
<docanchor file="dap4" title="Accessing Data Using the DAP4 Prototocol">dap4_accessing_data</docanchor>
|
||||
<docanchor file="dap4" title="Defined Client Parameters">dap4_defined_params</docanchor>
|
||||
<docanchor file="dap4" title="Notes on Debugging DAP4 Access">dap4_debug</docanchor>
|
||||
<docanchor file="dap4" title="HTTP Configuration.">dap4_http2_config</docanchor>
|
||||
<docanchor file="dap4" title="Point of Contact">dap4_poc</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>dap2</name>
|
||||
<title>DAP2 Support</title>
|
||||
<filename>dap2</filename>
|
||||
<docanchor file="dap2" title="DAP2 (OPeNDAP) Introduction">dap_introduction</docanchor>
|
||||
<docanchor file="dap2" title="OPeNDAP Documentation">dap_dap_information</docanchor>
|
||||
<docanchor file="dap2" title="Accessing OPeNDAP Data">dap_accessing_data</docanchor>
|
||||
<docanchor file="dap2" title="DAP to NetCDF Translation Rules">dap_to_netcdf</docanchor>
|
||||
<docanchor file="dap2" title="netCDF-3 Translation Rules">nc3_trans_rules</docanchor>
|
||||
<docanchor file="dap2" title="Variable Definition">var_def</docanchor>
|
||||
<docanchor file="dap2" title="DAP2 Reserved Keywords">dap2_reserved_keywords</docanchor>
|
||||
<docanchor file="dap2" title="Variable Dimension Translation">var_dim_trans</docanchor>
|
||||
<docanchor file="dap2" title="Dimension translation">dim_trans</docanchor>
|
||||
<docanchor file="dap2" title="Variable Name Translation">var_name_trans</docanchor>
|
||||
<docanchor file="dap2" title="Translating DAP DDS Sequences">dap_translation</docanchor>
|
||||
<docanchor file="dap2" title="Caching">dap_caching</docanchor>
|
||||
<docanchor file="dap2" title="Defined Client Parameters">dap_defined_params</docanchor>
|
||||
<docanchor file="dap2" title="Notes on Debugging OPeNDAP Access">dap_debug</docanchor>
|
||||
<docanchor file="dap2" title="HTTP Configuration.">http_config</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>user_defined_formats</name>
|
||||
<title>User-Defined Formats for NetCDF</title>
|
||||
<filename>user_defined_formats</filename>
|
||||
<docanchor file="user_defined_formats" title="User-Defined Formats ">udf_user_defined_formats</docanchor>
|
||||
<docanchor file="user_defined_formats" title="Introduction">udf_Introduction</docanchor>
|
||||
<docanchor file="user_defined_formats" title="Magic Numbers">autotoc_md1</docanchor>
|
||||
<docanchor file="user_defined_formats" title="Using User-Defined Formats from C Programs">udf_With_C</docanchor>
|
||||
<docanchor file="user_defined_formats" title="Building NetCDF C Library with a User-Defined Format Library">udf_Build_NetCDF_With_UDF</docanchor>
|
||||
<docanchor file="user_defined_formats" title="Creating a User-Defined Format">udf_Create_UDF</docanchor>
|
||||
<docanchor file="user_defined_formats" title="Read-Only User-Defined Formats">autotoc_md2</docanchor>
|
||||
<docanchor file="user_defined_formats" title="Examples">udf_Examples</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>md_filters</name>
|
||||
<title>NetCDF-4 Filter Support</title>
|
||||
<filename>md_filters</filename>
|
||||
<docanchor file="md_filters" title="NetCDF-4 Filter Support ">filters</docanchor>
|
||||
<docanchor file="md_filters" title="Introduction">filters_intro</docanchor>
|
||||
<docanchor file="md_filters" title="Enabling A Compression Filter">filters_enable</docanchor>
|
||||
<docanchor file="md_filters" title="Using The API">filters_API</docanchor>
|
||||
<docanchor file="md_filters" title="Using ncgen">filters_NCGEN</docanchor>
|
||||
<docanchor file="md_filters" title="Example CDL File (Data elided)">autotoc_md3</docanchor>
|
||||
<docanchor file="md_filters" title="Using nccopy">filters_NCCOPY</docanchor>
|
||||
<docanchor file="md_filters" title="Parameter Encode/Decode">filters_paramcoding</docanchor>
|
||||
<docanchor file="md_filters" title="Encoding Algorithms">autotoc_md4</docanchor>
|
||||
<docanchor file="md_filters" title="Encoding">autotoc_md5</docanchor>
|
||||
<docanchor file="md_filters" title="Decoding">autotoc_md6</docanchor>
|
||||
<docanchor file="md_filters" title="Filter Specification Syntax">filters_syntax</docanchor>
|
||||
<docanchor file="md_filters" title="Dynamic Loading Process ">filters_Process</docanchor>
|
||||
<docanchor file="md_filters" title="Plugin directory ">filters_Plugindir</docanchor>
|
||||
<docanchor file="md_filters" title="Plugin Library Naming ">filters_Pluginlib</docanchor>
|
||||
<docanchor file="md_filters" title="Plugin Verification ">filters_Pluginverify</docanchor>
|
||||
<docanchor file="md_filters" title="Debugging ">filters_Debug</docanchor>
|
||||
<docanchor file="md_filters" title="Test Case ">filters_TestCase</docanchor>
|
||||
<docanchor file="md_filters" title="Example ">filters_Example</docanchor>
|
||||
<docanchor file="md_filters" title="Notes">autotoc_md7</docanchor>
|
||||
<docanchor file="md_filters" title="Memory Allocation Issues">autotoc_md8</docanchor>
|
||||
<docanchor file="md_filters" title="SZIP Issues">autotoc_md9</docanchor>
|
||||
<docanchor file="md_filters" title="Supported Systems">autotoc_md10</docanchor>
|
||||
<docanchor file="md_filters" title="Generic Plugin Build">autotoc_md11</docanchor>
|
||||
<docanchor file="md_filters" title="Appendix A. Support Utilities ">filters_AppendixA</docanchor>
|
||||
<docanchor file="md_filters" title="Appendix B. Programmatic Filter Definition ">filters_programmatic</docanchor>
|
||||
<docanchor file="md_filters" title="API Concepts">autotoc_md12</docanchor>
|
||||
<docanchor file="md_filters" title="NetCDF API">autotoc_md13</docanchor>
|
||||
<docanchor file="md_filters" title="Example">autotoc_md14</docanchor>
|
||||
<docanchor file="md_filters" title="References">filters_References</docanchor>
|
||||
<docanchor file="md_filters" title="Point of Contact">autotoc_md15</docanchor>
|
||||
</compound>
|
||||
<compound kind="page">
|
||||
<name>index</name>
|
||||
<title></title>
|
||||
<filename>index</filename>
|
||||
<docanchor file="index" title="The Purpose of NetCDF">autotoc_md0</docanchor>
|
||||
</compound>
|
||||
</tagfile>
|
||||
304
NUG/types.dox
Normal file
304
NUG/types.dox
Normal file
@ -0,0 +1,304 @@
|
||||
/** \file types.dox Documentation related to NetCDF Types
|
||||
Documentation of types.
|
||||
|
||||
\page data_type Data Types
|
||||
|
||||
\tableofcontents
|
||||
|
||||
Data in a netCDF file may be one of the \ref external_types, or may be
|
||||
a user-defined data type (see \ref user_defined_types).
|
||||
|
||||
\section external_types External Data Types
|
||||
|
||||
The atomic external types supported by the netCDF interface are:
|
||||
- ::NC_BYTE 8-bit signed integer
|
||||
- ::NC_UBYTE 8-bit unsigned integer *
|
||||
- ::NC_CHAR 8-bit character
|
||||
- ::NC_SHORT 16-bit signed integer
|
||||
- ::NC_USHORT 16-bit unsigned integer *
|
||||
- ::NC_INT (or ::NC_LONG) 32-bit signed integer
|
||||
- ::NC_UINT 32-bit unsigned integer *
|
||||
- ::NC_INT64 64-bit signed integer *
|
||||
- ::NC_UINT64 64-bit unsigned integer *
|
||||
- ::NC_FLOAT 32-bit floating point
|
||||
- ::NC_DOUBLE 64-bit floating point
|
||||
- ::NC_STRING variable length character string +
|
||||
|
||||
\remark * These types are available only for CDF5 (NC_CDF5) and netCDF-4 format (NC_NETCDF4) files. All the unsigned ints and the 64-bit ints are for CDF5 or netCDF-4 files only.
|
||||
\remark + These types are available only for netCDF-4 (NC_NETCDF4) files.
|
||||
|
||||
These types were chosen to provide a reasonably wide range of
|
||||
trade-offs between data precision and number of bits required for each
|
||||
value. These external data types are independent from whatever
|
||||
internal data types are supported by a particular machine and language
|
||||
combination.
|
||||
|
||||
These types are called "external", because they correspond to the
|
||||
portable external representation for netCDF data. When a program reads
|
||||
external netCDF data into an internal variable, the data is converted,
|
||||
if necessary, into the specified internal type. Similarly, if you
|
||||
write internal data into a netCDF variable, this may cause it to be
|
||||
converted to a different external type, if the external type for the
|
||||
netCDF variable differs from the internal type.
|
||||
|
||||
The separation of external and internal types and automatic type
|
||||
conversion have several advantages. You need not be aware of the
|
||||
external type of numeric variables, since automatic conversion to or
|
||||
from any desired numeric type is available. You can use this feature
|
||||
to simplify code, by making it independent of external types, using a
|
||||
sufficiently wide internal type, e.g., double precision, for numeric
|
||||
netCDF data of several different external types. Programs need not be
|
||||
changed to accommodate a change to the external type of a variable.
|
||||
|
||||
If conversion to or from an external numeric type is necessary, it is
|
||||
handled by the library.
|
||||
|
||||
Converting from one numeric type to another may result in an error if
|
||||
the target type is not capable of representing the converted
|
||||
value. For example, an internal short integer type may not be able to
|
||||
hold data stored externally as an integer. When accessing an array of
|
||||
values, a range error is returned if one or more values are out of the
|
||||
range of representable values, but other values are converted
|
||||
properly.
|
||||
|
||||
Note that mere loss of precision in type conversion does not return an
|
||||
error. Thus, if you read double precision values into a
|
||||
single-precision floating-point variable, for example, no error
|
||||
results unless the magnitude of the double precision value exceeds the
|
||||
representable range of single-precision floating point numbers on your
|
||||
platform. Similarly, if you read a large integer into a float
|
||||
incapable of representing all the bits of the integer in its mantissa,
|
||||
this loss of precision will not result in an error. If you want to
|
||||
avoid such precision loss, check the external types of the variables
|
||||
you access to make sure you use an internal type that has adequate
|
||||
precision.
|
||||
|
||||
The names for the primitive external data types (char, byte, ubyte, short,
|
||||
ushort, int, uint, int64, uint64, float or real, double, string) are
|
||||
reserved words in CDL, so the names of variables, dimensions, and
|
||||
attributes must not be type names.
|
||||
|
||||
It is possible to interpret byte data as either signed (-128 to 127)
|
||||
or unsigned (0 to 255). However, when reading byte data to be
|
||||
converted into other numeric types, it is interpreted as signed.
|
||||
|
||||
For the correspondence between netCDF external data types and the data
|
||||
types of a language see \ref variables.
|
||||
|
||||
\section classic_structures Data Structures in Classic Files
|
||||
|
||||
The only kind of data structure directly supported by the netCDF
|
||||
classic abstraction, i.e. CDF-1, 2, and 5 formats, is a collection of named
|
||||
arrays with attached vector attributes. NetCDF is not particularly
|
||||
well-suited for storing linked lists, trees, sparse matrices, ragged
|
||||
arrays or other kinds of data structures requiring pointers.
|
||||
|
||||
It is possible to build other kinds of data structures in netCDF
|
||||
classic formats, from sets of arrays by adopting
|
||||
various conventions regarding the use of data in one array as pointers
|
||||
into another array. The netCDF library won't provide much help or
|
||||
hindrance with constructing such data structures, but netCDF provides
|
||||
the mechanisms with which such conventions can be designed.
|
||||
|
||||
The following netCDF classic example stores a ragged array ragged_mat
|
||||
using an attribute row_index to name an associated index variable
|
||||
giving the index of the start of each row. In this example, the first
|
||||
row contains 12 elements, the second row contains 7 elements (19 -
|
||||
12), and so on. (NetCDF-4 includes native support for variable length
|
||||
arrays. See below.)
|
||||
|
||||
\code
|
||||
float ragged_mat(max_elements);
|
||||
ragged_mat:row_index = "row_start";
|
||||
int row_start(max_rows);
|
||||
data:
|
||||
row_start = 0, 12, 19, ...
|
||||
\endcode
|
||||
|
||||
As another example, netCDF variables may be grouped within a netCDF
|
||||
classic dataset by defining attributes that list the
|
||||
names of the variables in each group, separated by a conventional
|
||||
delimiter such as a space or comma. Using a naming convention for
|
||||
attribute names for such groupings permits any number of named groups
|
||||
of variables. A particular conventional attribute for each variable
|
||||
might list the names of the groups of which it is a member. Use of
|
||||
attributes, or variables that refer to other attributes or variables,
|
||||
provides a flexible mechanism for representing some kinds of complex
|
||||
structures in netCDF datasets.
|
||||
|
||||
\section nc4_user_defined_types NetCDF-4 User Defined Data Types
|
||||
|
||||
NetCDF supported six data types through version 3.6.0 (char, byte,
|
||||
short, int, float, and double). Starting with version 4.0, many new
|
||||
data types are supported (unsigned int types, strings, compound types,
|
||||
variable length arrays, enums, opaque).
|
||||
|
||||
In addition to the new atomic types the user may define types.
|
||||
|
||||
Types are defined in define mode, and must be fully defined before
|
||||
they are used. New types may be added to a file by re-entering define
|
||||
mode.
|
||||
|
||||
Once defined the type may be used to create a variable or attribute.
|
||||
|
||||
Types may be nested in complex ways. For example, a compound type
|
||||
containing an array of VLEN types, each containing variable length
|
||||
arrays of some other compound type, etc. Users are cautioned to keep
|
||||
types simple. Reading data of complex types can be challenging for
|
||||
Fortran users.
|
||||
|
||||
Types may be defined in any group in the data file, but they are
|
||||
always available globally in the file.
|
||||
|
||||
Types cannot have attributes (but variables of the type may have
|
||||
attributes).
|
||||
|
||||
Only files created with the netCDF-4/HDF5 mode flag (::NC_NETCDF4) but
|
||||
without the classic model flag (::NC_CLASSIC_MODEL) may use
|
||||
user-defined types or the new atomic data types.
|
||||
|
||||
Once types are defined, use their ID like any other type ID when
|
||||
defining variables or attributes. Use functions
|
||||
|
||||
- nc_put_att() / nc_get_att()
|
||||
- nc_put_var() / nc_get_var()
|
||||
- nc_put_var1() / nc_get_var1()
|
||||
- nc_put_vara() / nc_get_vara()
|
||||
- nc_put_vars() / nc_get_vars()
|
||||
|
||||
functions to access attribute and variable data of user defined type.
|
||||
|
||||
\subsection types_compound_types Compound Types
|
||||
|
||||
Compound types allow the user to combine atomic and user-defined types
|
||||
into C-like structs. Since users defined types may be used within a
|
||||
compound type, they can contain nested compound types.
|
||||
|
||||
Users define a compound type, and (in their C code) a corresponding C
|
||||
struct. They can then use nc_put_vara() and related functions to write
|
||||
multi-dimensional arrays of these structs, and nc_get_vara() calls
|
||||
to read them.
|
||||
|
||||
While structs, in general, are not portable from platform to platform,
|
||||
the HDF5 layer (when installed) performs the magic required to figure
|
||||
out your platform's idiosyncrasies, and adjust to them. The end result
|
||||
is that HDF5 compound types (and therefore, netCDF-4 compound types),
|
||||
are portable.
|
||||
|
||||
For more information on creating and using compound types, see
|
||||
Compound Types in The NetCDF C Interface Guide.
|
||||
|
||||
\subsection vlen_types VLEN Types
|
||||
|
||||
Variable length arrays can be used to create a ragged array of data,
|
||||
in which one of the dimensions varies in size from point to point.
|
||||
|
||||
An example of VLEN use would the to store a 1-D array of dropsonde
|
||||
data, in which the data at each drop point is of variable length.
|
||||
|
||||
There is no special restriction on the dimensionality of VLEN
|
||||
variables. It's possible to have 2D, 3D, 4D, etc. data, in which each
|
||||
point contains a VLEN.
|
||||
|
||||
A VLEN has a base type (that is, the type that it is a VLEN of). This
|
||||
may be one of the atomic types (forming, for example, a variable
|
||||
length array of ::NC_INT), or it can be another user defined type,
|
||||
like a compound type.
|
||||
|
||||
With VLEN data, special memory allocation and deallocation procedures
|
||||
must be followed, or memory leaks may occur.
|
||||
|
||||
Compression is permitted but may not be effective for VLEN data,
|
||||
because the compression is applied to structures containing lengths
|
||||
and pointers to the data, rather than the actual data.
|
||||
|
||||
For more information on creating and using variable length arrays, see
|
||||
Variable Length Arrays in The NetCDF C Interface Guide.
|
||||
|
||||
\subsection types_opaque_types Opaque Types
|
||||
|
||||
Opaque types allow the user to store arrays of data blobs of a fixed size.
|
||||
|
||||
For more information on creating and using opaque types, see Opaque
|
||||
Type in The NetCDF C Interface Guide.
|
||||
|
||||
\subsection enum_types Enum Types
|
||||
|
||||
Enum types allow the user to specify an enumeration.
|
||||
|
||||
For more information on creating and using enum types, see Enum Type
|
||||
in The NetCDF C Interface Guide.
|
||||
|
||||
\section type_conversion Type Conversion
|
||||
|
||||
Each netCDF variable has an external type, specified when the variable
|
||||
is first defined. This external type determines whether the data is
|
||||
intended for text or numeric values, and if numeric, the range and
|
||||
precision of numeric values.
|
||||
|
||||
If the netCDF external type for a variable is char, only character
|
||||
data representing text strings can be written to or read from the
|
||||
variable. No automatic conversion of text data to a different
|
||||
representation is supported.
|
||||
|
||||
If the type is numeric, however, the netCDF library allows you to
|
||||
access the variable data as a different type and provides automatic
|
||||
conversion between the numeric data in memory and the data in the
|
||||
netCDF variable. For example, if you write a program that deals with
|
||||
all numeric data as double-precision floating point values, you can
|
||||
read netCDF data into double-precision arrays without knowing or
|
||||
caring what the external type of the netCDF variables are. On reading
|
||||
netCDF data, integers of various sizes and single-precision
|
||||
floating-point values will all be converted to double-precision, if
|
||||
you use the data access interface for double-precision values. Of
|
||||
course, you can avoid automatic numeric conversion by using the netCDF
|
||||
interface for a value type that corresponds to the external data type
|
||||
of each netCDF variable, where such value types exist.
|
||||
|
||||
The automatic numeric conversions performed by netCDF are easy to
|
||||
understand, because they behave just like assignment of data of one
|
||||
type to a variable of a different type. For example, if you read
|
||||
floating-point netCDF data as integers, the result is truncated
|
||||
towards zero, just as it would be if you assigned a floating-point
|
||||
value to an integer variable. Such truncation is an example of the
|
||||
loss of precision that can occur in numeric conversions.
|
||||
|
||||
Converting from one numeric type to another may result in an error if
|
||||
the target type is not capable of representing the converted
|
||||
value. For example, an integer may not be able to hold data stored
|
||||
externally as an IEEE floating-point number. When accessing an array
|
||||
of values, a range error is returned if one or more values are out of
|
||||
the range of representable values, but other values are converted
|
||||
properly.
|
||||
|
||||
Note that mere loss of precision in type conversion does not result in
|
||||
an error. For example, if you read double precision values into an
|
||||
integer, no error results unless the magnitude of the double precision
|
||||
value exceeds the representable range of integers on your
|
||||
platform. Similarly, if you read a large integer into a float
|
||||
incapable of representing all the bits of the integer in its mantissa,
|
||||
this loss of precision will not result in an error. If you want to
|
||||
avoid such precision loss, check the external types of the variables
|
||||
you access to make sure you use an internal type that has a compatible
|
||||
precision.
|
||||
|
||||
Whether a range error occurs in writing a large floating-point value
|
||||
near the boundary of representable values may be depend on the
|
||||
platform. The largest floating-point value you can write to a netCDF
|
||||
float variable is the largest floating-point number representable on
|
||||
your system that is less than 2 to the 128th power. The largest double
|
||||
precision value you can write to a double variable is the largest
|
||||
double-precision number representable on your system that is less than
|
||||
2 to the 1024th power.
|
||||
|
||||
The _uchar and _schar functions were introduced in netCDF-3 to
|
||||
eliminate an ambiguity, and support both signed and unsigned byte data.
|
||||
In netCDF-2, whether the external NC_BYTE type represented signed or
|
||||
unsigned values was left up to the user. In netcdf-3, we treat NC_BYTE
|
||||
as signed for the purposes of conversion to short, int, long, float, or
|
||||
double. (Of course, no conversion takes place when the internal type is
|
||||
signed char.) In the _uchar functions, we treat NC_BYTE as if it were
|
||||
unsigned. Thus, no NC_ERANGE error can occur converting between NC_BYTE
|
||||
and unsigned char.
|
||||
|
||||
*/
|
||||
113
NUG/user_defined_formats.md
Normal file
113
NUG/user_defined_formats.md
Normal file
@ -0,0 +1,113 @@
|
||||
User-Defined Formats for NetCDF {#user_defined_formats}
|
||||
===============================
|
||||
|
||||
[TOC]
|
||||
|
||||
User-Defined Formats {#udf_user_defined_formats}
|
||||
=====================================
|
||||
|
||||
## Introduction {#udf_Introduction}
|
||||
|
||||
User-defined formats allow users to write their own adaptors for the
|
||||
netCDF C library, so that it can read and (optionally) write a
|
||||
proprietary format through the netCDF API.
|
||||
|
||||
This capability is currently experimental. It involves the exposing of internal
|
||||
netcdf interfaces and data structures that were previously invisible to users.
|
||||
This means that it is unstable and the exposed interfaces are subject to change.
|
||||
Use with caution.
|
||||
|
||||
User-defined format code is packaged into a separate library, the
|
||||
user-defined format dispatch library. This library, when linked with
|
||||
the netCDF library, will allow user programs to read their proprietary
|
||||
format through the netCDF API. The proprietary format is treated as if
|
||||
it were one of the netCDF C library native binary formats.
|
||||
|
||||
Coding the user-defined format dispatch library requires knowledge of
|
||||
the netCDF library internals. User-defined format dispatch libraries
|
||||
must be written in C.
|
||||
|
||||
### Magic Numbers
|
||||
|
||||
Some file formats use the first few bytes of the file as an identifier
|
||||
for format. For example, HDF5 files have "HDF5" as the fist 4 bytes,
|
||||
and netCDF classic files have "CDF1" as the first four bytes. This is
|
||||
called the "magic number" of the file.
|
||||
|
||||
User-defined formats can optionally support magic numbers. If the
|
||||
user-defined format uses a magic number, and that magic number is
|
||||
associated with the user-defined format, then netCDF will be able to
|
||||
correctly identify those files from nc_open(). It will not be
|
||||
necessary for the user to know or specify the underlying format.
|
||||
|
||||
## Using User-Defined Formats from C Programs {#udf_With_C}
|
||||
|
||||
A user-defined format can be added dynamically in the case of C programs.
|
||||
|
||||
```
|
||||
/* Add our test user defined format. */
|
||||
if (nc_def_user_format(NC_UDF0, &tst_dispatcher, NULL)) ERR;
|
||||
```
|
||||
|
||||
The file can now be opened by netCDF:
|
||||
|
||||
```
|
||||
if (nc_open(FILE_NAME, NC_UDF0, &ncid)) ERR;
|
||||
```
|
||||
|
||||
If a magic number is used in the file, that may be passed to
|
||||
nc_def_user_format(). In that case, specifying the NC_UDF0 mode flag
|
||||
to nc_open() is optional. The nc_open() will check the file and find
|
||||
the magic number, and automatically associate the file with
|
||||
NC_UDF0. The user will not need to know the format in order to open
|
||||
the file with nc_open().
|
||||
|
||||
## Building NetCDF C Library with a User-Defined Format Library {#udf_Build_NetCDF_With_UDF}
|
||||
|
||||
Once a user-defined format library is created, it may built into a
|
||||
netCDF install. This allows the netCDF Fortran APIs, and the netCDF
|
||||
utilities (ncdump, ncgen, nccopy) to natively use the user-defined
|
||||
format.
|
||||
|
||||
First the user-defined dispatch library must be built and installed.
|
||||
|
||||
Then the netcdf-c package must be (re-)built. When building netcdf-c,
|
||||
add the location of the user-defined format dispatch library include
|
||||
file to the CPPFLAGS, and the location of the user-defined format
|
||||
dispatch library in LDFLAGS.
|
||||
|
||||
Configure netcdf-c with the option ````--with-udf0=<udf_lib_name>````.
|
||||
|
||||
If a magic number is associated with the user-defined format, it can
|
||||
be specified with the --with-udf0-magic-number= argument.
|
||||
|
||||
## Creating a User-Defined Format {#udf_Create_UDF}
|
||||
|
||||
Creators of user-defined format libraries will have to become familiar
|
||||
with the internals of the netCDF-4 code.
|
||||
|
||||
### Read-Only User-Defined Formats
|
||||
|
||||
Many users will find that a read-only user-defined formats meets most
|
||||
of their needs. With a read-only user-defined format, netCDF will be
|
||||
able to read files of the user-defined format. Tools like ncdump and
|
||||
nccopy can work on the files.
|
||||
|
||||
A read-only user-defined format can be implemented with only 6
|
||||
functions. The code in libhdf4 is an excellent example of a read-only
|
||||
dispatch layer.
|
||||
|
||||
## Examples {#udf_Examples}
|
||||
|
||||
The most simple-case example of a user-defined format is provided in
|
||||
test nc_test4/tst_udf.c.
|
||||
|
||||
A slightly more complex example, including the required
|
||||
autoconf/automake files to build a user-defined format library, can be
|
||||
found at the [sample user-defined format
|
||||
library](https://github.com/NOAA-GSD/sample-netcdf-dispatch). In this
|
||||
example, the HDF4 SD reader is re-implemented as an external
|
||||
user-defined format. (This is unnecessary if you just want to read
|
||||
HDF4 SD files, since the netCDF C library already includes an HDF4 SD
|
||||
file reader. This user-defined format library uses the same code. It
|
||||
is repackaged as a user-defined library to provide a working sample.)
|
||||
Loading…
Reference in New Issue
Block a user