mirror of
https://github.com/Unidata/netcdf-c.git
synced 2024-12-21 08:39:46 +08:00
481 lines
17 KiB
HTML
481 lines
17 KiB
HTML
<!- Copyright 2015, UCAR/Unidata and OPeNDAP, Inc. -->
|
|
<!- See the COPYRIGHT file for more information. -->
|
|
<html>
|
|
<head>
|
|
<style>
|
|
.break { page-break-before: always; }
|
|
body { counter-reset: H2; font-size: 12pt; }
|
|
h1.title {
|
|
font-size: 18pt;
|
|
text-decoration: underline;
|
|
}
|
|
div.subtitle {
|
|
}
|
|
.subtitle h1 {
|
|
font-size: 14pt;
|
|
margin-top: 0;
|
|
margin-bottom: 0;
|
|
}
|
|
|
|
h1.toc {
|
|
font-size: 16pt;
|
|
text-decoration: underline;
|
|
}
|
|
h1.appendix {
|
|
font-size: 14pt;
|
|
}
|
|
|
|
h2:before {
|
|
content: counter(H2) " ";
|
|
counter-increment: H2;
|
|
}
|
|
h2 { counter-reset: H3; text-decoration: underline; }
|
|
h3:before {
|
|
content: counter(H2) "." counter(H3) " ";
|
|
counter-increment:H3;
|
|
}
|
|
h3 { counter-reset: H4; }
|
|
h4:before {
|
|
content: counter(H2) "." counter(H3) "." counter(H4) " ";
|
|
counter-increment:H4;
|
|
}
|
|
|
|
</style>
|
|
</head>
|
|
<body>
|
|
|
|
<h1 class="title">netCDF Authorization Support</h1>
|
|
<div class="subtitle">
|
|
<h1>Author: Dennis Heimbigner</h1>
|
|
<h1>Address: http://www.unidata.ucar.edu/staff/dmh/</h1>
|
|
<h1>Draft: 11/21/2014</h1>
|
|
<h1>Last Revised: 10/24/2015</h1>
|
|
</div>
|
|
|
|
<h1 class="toc">Table of Contents</h1>
|
|
<ol>
|
|
<li> <a href="#Introduction">Introduction</a>
|
|
<li> <a href="#URL-AUTH">URL-Based Authentication</a>
|
|
<li> <a href="#DODSRC">RC File Authentication</a>
|
|
<li> <a href="#REDIR">Redirection-Based Authentication</a>
|
|
<li> <a href="#URLCONS">URL Constrained RC File Entries</a>
|
|
<li> <a href="#CLIENTCERTS">Client-Side Certificates</a>
|
|
<li> <a href="#allkeys">Appendix A. All RC-File Keys</a>
|
|
<li> <a href="#ESGDETAIL">Appendix B. ESG Access in Detail</a>
|
|
</ol>
|
|
|
|
<h2><a name="Introduction">Introduction</a></h2>
|
|
netCDF can support user authorization using the facilities provided by the curl
|
|
library. This includes basic password authentication as well as
|
|
certificate-based authorization.
|
|
<p>
|
|
At the moment, this document only applies to DAP2 and DAP4 access
|
|
because they are (for now) the only parts of the netCDF-C library
|
|
that uses libcurl.
|
|
<p>
|
|
With some exceptions (e.g. see the section on <a href="#REDIR">redirection</a>)
|
|
The libcurl authorization mechanisms can be accessed in two ways
|
|
<ol>
|
|
<li> Inserting the username and password into the url, or
|
|
<li> Accessing information from a so-called <i>rc</i> file named either
|
|
<i>.daprc</i> or <i>.dodsrc</i>
|
|
</ol>
|
|
|
|
<h2><a name="URL-AUTH">URL-Based Authentication</a></h2>
|
|
For simple password based authentication, it is possible to
|
|
directly insert the username and the password into a url in this form.
|
|
<pre>
|
|
http://username:password@host/...
|
|
</pre>
|
|
This username and password will be used if the server asks for
|
|
authentication. Note that only simple password authentication
|
|
is supported in this format.
|
|
Specifically note that <a href="#REDIR">redirection</a> based
|
|
authorization will not work with this because the username and password
|
|
will only be used on the initial request, not the redirection
|
|
|
|
<h2><a name="DODSRC">RC File Authentication</a></h2>
|
|
The netcdf library supports an <i>rc</i> file mechanism to allow the passing
|
|
of a number of parameters to libnetcdf and libcurl.
|
|
<p>
|
|
The file must be called one of the following names:
|
|
".daprc" or ".dodsrc"
|
|
If both .daprc and .dodsrc exist, then
|
|
the .daprc file will take precedence.
|
|
<p>
|
|
The rc file is searched for first in the current directory
|
|
and then in the home directory (as defined by the HOME environment
|
|
variable).
|
|
<p>
|
|
The rc file format is a series of lines of the general form:
|
|
<pre>
|
|
[<host:port>]<key>=<value>
|
|
</pre>
|
|
where the bracket-enclosed host:port is optional and will be discussed
|
|
subsequently.
|
|
<p>
|
|
The currently defined set of authorization-related keys are as follows.
|
|
The second column is the affected curl_easy_setopt option(s), if any.
|
|
<table>
|
|
<tr><th>Key<th>Affected curl_easy_setopt Options<th>Notes
|
|
<tr><td>HTTP.COOKIEJAR<td>CURLOPT_COOKIEJAR
|
|
<tr><td>HTTP.COOKIEFILE<td>CURLOPT_COOKIEJAR<td>Alias for CURLOPT_COOKIEJAR
|
|
<tr><td>HTTP.PROXY_SERVER<td>CURLOPT_PROXY, CURLOPT_PROXYPORT, CURLOPT_PROXYUSERPWD
|
|
<tr><td>HTTP.SSL.CERTIFICATE<td>CURLOPT_SSLCERT
|
|
<tr><td>HTTP.SSL.KEY<td>CURLOPT_SSLKEY
|
|
<tr><td>HTTP.SSL.KEYPASSWORD<td>CURLOPT_KEYPASSWORD
|
|
<tr><td>HTTP.SSL.CAINFO<td>CURLOPT_SSLCAINFO
|
|
<tr><td>HTTP.SSL.CAPATH<td>CURLOPT_SSLCAPATH
|
|
<tr><td>HTTP.SSL.VERIFYPEER<td>CURLOPT_SSL_VERIFYPEER
|
|
<tr><td>HTTP.SSL.VALIDATE<td>CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST
|
|
<tr><td>HTTP.CREDENTIALS.USERPASSWORD<td>CURLOPT_USERPASSWORD
|
|
<tr><td>HTTP.NETRC<td>N.A.<td>Specify path of the .netrc file
|
|
</table>
|
|
</ul>
|
|
|
|
<h3>Password Authentication</h3>
|
|
The key
|
|
HTTP.CREDENTIALS.USERPASSWORD
|
|
can be used to set the simple password authentication.
|
|
This is an alternative to setting it in the url.
|
|
The value must be of the form "username:password".
|
|
See <a href="#REDIR">redirection authorization</a>
|
|
for important additional information.
|
|
|
|
<h3>Cookie Jar</h3>
|
|
The HTTP.COOKIEJAR key
|
|
specifies the name of file from which
|
|
to read cookies (CURLOPT_COOKIEJAR) and also
|
|
the file into which to store cookies (CURLOPT_COOKIEFILE).
|
|
The same value is used for both CURLOPT values.
|
|
It defaults to in-memory storage.
|
|
See <a href="#REDIR">redirection authorization</a>
|
|
for important additional information.
|
|
|
|
<h3>Certificate Authentication</h3>
|
|
HTTP.SSL.CERTIFICATE
|
|
specifies a file path for a file containing a PEM cerficate.
|
|
This is typically used for client-side authentication.
|
|
<p>
|
|
HTTP.SSL.KEY is essentially the same as HTTP.SSL.CERTIFICATE
|
|
and should always have the same value.
|
|
<p>
|
|
HTTP.SSL.KEYPASSWORD
|
|
specifies the password for accessing the HTTP.SSL.CERTIFICAT/HTTP.SSL.key file.
|
|
<p>
|
|
HTTP.SSL.CAPATH
|
|
specifies the path to a directory containing
|
|
trusted certificates for validating server certificates.
|
|
<p>
|
|
HTTP.SSL.VALIDATE
|
|
is a boolean (1/0) value that if true (1)
|
|
specifies that the client should verify the server's presented certificate.
|
|
<p>
|
|
HTTP.PROXY_SERVER
|
|
specifies the url for accessing the proxy:
|
|
e.g. <i>http://[username:password@]host[:port]</i>
|
|
<p>
|
|
HTTP.NETRC
|
|
specifies the absolute path of the .netrc file.
|
|
See <a href="#REDIR">redirection authorization</a>
|
|
for information about using .netrc.
|
|
|
|
<h2><a name="REDIR">Redirection-Based Authentication</a> </h2>
|
|
Some sites provide authentication by using a third party site
|
|
to do the authentication. Examples include ESG and URS.
|
|
<p>
|
|
The process is usually as follows.
|
|
<ol>
|
|
<li>The client contacts the server of interest (SOI), the actual data provider
|
|
using, typically http protocol.
|
|
<li>The SOI sends a redirect to the client to connect to the e.g. URS system
|
|
using the 'https' protocol (note the use of https instead of http).
|
|
<li>The client authenticates with URS.
|
|
<li>URS sends a redirect (with authorization information) to send
|
|
the client back to the SOI to actually obtain the data.
|
|
</ol>
|
|
<p>
|
|
It turns out that libcurl uses the password in the .daprc file — or from the url —
|
|
only for the initial connection. This causes problems because
|
|
the redirected connection is the one that actually requires the password.
|
|
This is where .netrc comes in. Libcurl will use .netrc for
|
|
the redirected connection. It is possible to cause libcurl to use
|
|
the .daprc password always, but this introduces a security hole
|
|
because it may send the initial user+pwd to the redirection site.
|
|
In summary, if you are using redirection, then you must create a .netrc
|
|
file to hold the password for the site to which the redirection is sent.
|
|
<p>
|
|
The format of this .netrc file will contain content that
|
|
typically look like this.
|
|
<pre>
|
|
machine mmmmmm login xxxxxx password yyyyyy
|
|
</pre>
|
|
where the machine, mmmmmm, is the hostname of the machine to
|
|
which the client is redirected for authorization, and the
|
|
login and password are those needed to authenticate on that machine.
|
|
<p>
|
|
The .netrc file can be specified by
|
|
putting the following line in your .daprc/.dodsrc file.
|
|
<pre>
|
|
HTTP.NETRC=<path to netrc file>
|
|
</pre>
|
|
|
|
<p>
|
|
One final note. In using this, it is almost certain that you will
|
|
need to specify a real cookie jar file (HTTP.COOKIEJAR) so that the
|
|
redirect site can pass back authorization information.
|
|
|
|
<h2><a name="URLCONS">URL Constrained RC File Entries</a></h2>
|
|
Each line of the rc file can begin with
|
|
a host+port enclosed in square brackets.
|
|
The form is "host:port".
|
|
If the port is not specified
|
|
then the form is just "host".
|
|
The reason that more of the url is not used is that
|
|
libcurl's authorization grain is not any finer than host level.
|
|
<p>
|
|
Examples.
|
|
<pre>
|
|
[remotetest.unidata.ucar.edu]HTTP.VERBOSE=1
|
|
or
|
|
[fake.ucar.edu:9090]HTTP.VERBOSE=0
|
|
</pre>
|
|
If the url request from, say, the <i>netcdf_open</i> method
|
|
has a host+port matching one of the prefixes in the rc file, then
|
|
the corresponding entry will be used, otherwise ignored.
|
|
<p>
|
|
For example, the URL
|
|
<pre>
|
|
http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc
|
|
</pre>
|
|
will have HTTP.VERBOSE set to 1.
|
|
<p>
|
|
Similarly,
|
|
<pre>
|
|
http://fake.ucar.edu:9090/dts/test.01
|
|
</pre>
|
|
will have HTTP.VERBOSE set to 0.
|
|
|
|
<h2><a name="CLIENTCERTS">Client-Side Certificates</a></h2>
|
|
Some systems, notably ESG (Earth System Grid), requires
|
|
the use of client-side certificates, as well as being
|
|
<a href="#REDIR">re-direction based</a>.
|
|
This requires setting the following entries:
|
|
<ul>
|
|
<li>HTTP.COOKIEJAR — a file path for storing cookies across re-direction.
|
|
<li>HTTP.NETRC — the path to the netrc file.
|
|
<li>HTTP.SSL.CERTIFICATE — the file path for the client side certificate file.
|
|
<li>HTTP.SSL.KEY — this should have the same value as HTTP.SSL.CERTIFICATE.
|
|
<li>HTTP.SSL.CAPATH — the path to a "certificates" directory.
|
|
<li>HTTP.SSL.VALIDATE — force validation of the server certificate.
|
|
</ul>
|
|
Note that the first two are to support re-direction based authentication.
|
|
|
|
<h1 class="appendix><a name="allkeys">Appendix A. All RC-File Keys</a></h1>
|
|
For completeness, this is the list of all rc-file keys.
|
|
If this documentation is out of date with respect to the actual code,
|
|
the code is definitive.
|
|
<table>
|
|
<tr><th>Key<th>curl_easy_setopt Option
|
|
<tr valign="top"><td>HTTP.DEFLATE<td>CUROPT_DEFLATE<br>with value "deflate,gzip"
|
|
<tr><td>HTTP.VERBOSE <td>CUROPT_VERBOSE
|
|
<tr><td>HTTP.TIMEOUT<td>CUROPT_TIMEOUT
|
|
<tr><td>HTTP.USERAGENT<td>CUROPT_USERAGENT
|
|
<tr><td>HTTP.COOKIEJAR<td>CUROPT_COOKIEJAR
|
|
<tr><td>HTTP.COOKIE_JAR<td>CUROPT_COOKIEJAR
|
|
<tr valign="top"><td>HTTP.PROXY_SERVER<td>CURLOPT_PROXY,<br>CURLOPT_PROXYPORT,<br>CURLOPT_PROXYUSERPWD
|
|
<tr><td>HTTP.SSL.CERTIFICATE<td>CUROPT_SSLCERT
|
|
<tr><td>HTTP.SSL.KEY<td>CUROPT_SSLKEY
|
|
<tr><td>HTTP.SSL.KEYPASSWORD<td>CUROPT_KEYPASSWORD
|
|
<tr><td>HTTP.SSL.CAINFO<td>CUROPT_SSLCAINFO
|
|
<tr><td>HTTP.SSL.CAPATH<td>CUROPT_SSLCAPATH
|
|
<tr><td>HTTP.SSL.VERIFYPEER<td>CUROPT_SSL_VERIFYPEER
|
|
<tr><td>HTTP.CREDENTIALS.USERPASSWORD<td>CUROPT_USERPASSWORD
|
|
<tr><td>HTTP.NETRC<td>CURLOPT_NETRC,CURLOPT_NETRC_FILE
|
|
</table>
|
|
</ul>
|
|
|
|
<h1 class="appendix"><a name="URSDETAIL">Appendix B. URS Access in Detail</a></h1>
|
|
It is possible to use the NASA Earthdata Login System (URS)
|
|
with netcdf by using using the process specified in the
|
|
<a href="#REDIR">redirection</a> based authorization section.
|
|
In order to access URS controlled datasets, however, it is necessary to
|
|
register as a user with NASA at the
|
|
<i>https://uat.urs.earthdata.nasa.gov/</i>
|
|
website.
|
|
|
|
<h1 class="appendix"><a name="ESGDETAIL">Appendix C. ESG Access in Detail</a></h1>
|
|
It is possible to access Earth Systems Grid (ESG) datasets
|
|
from ESG servers through the netCDF API using the techniques
|
|
described in the section on <a href="#CLIENTCERTS">Client-Side Certificates</a>.
|
|
<p>
|
|
In order to access ESG datasets, however, it is necessary to
|
|
register as a user with ESG and to setup your environment
|
|
so that proper authentication is established between an netcdf
|
|
client program and the ESG data server. Specifically, it
|
|
is necessary to use what is called "client-side keys" to
|
|
enable this authentication. Normally, when a client accesses
|
|
a server in a secure fashion (using "https"), the server
|
|
provides an authentication certificate to the client.
|
|
With client-side keys, the client must also provide a
|
|
certificate to the server so that the server can know with
|
|
whom it is communicating.
|
|
<p>
|
|
The netcdf library uses the <i>curl</i> library and it is that
|
|
underlying library that must be properly configured.
|
|
|
|
<h3>Terminology</h3>
|
|
The key elements for client-side keys requires the constructions of
|
|
two "stores" on the client side.
|
|
<ul>
|
|
<li> Keystore - a repository to hold the client side key.
|
|
<li> Truststore - a repository to hold a chain of certificates
|
|
that can be used to validate the certificate
|
|
sent by the server to the client.
|
|
</ul>
|
|
The server actually has a similar set of stores, but the client
|
|
need not be concerned with those.
|
|
|
|
<h3>Initial Steps</h3>
|
|
|
|
The first step is to obtain authorization from ESG.
|
|
Note that this information may evolve over time, and
|
|
may be out of date.
|
|
This discussion is in terms of BADC and NCSA. You will need
|
|
to substitute as necessary.
|
|
<ol>
|
|
<li> Register at http://badc.nerc.ac.uk/register
|
|
to obtain access to badc and to obtain an openid,
|
|
which will looks something like:
|
|
<pre>https://ceda.ac.uk/openid/Firstname.Lastname</pre>
|
|
<li> Ask BADC for access to whatever datasets are of interest.
|
|
<p>
|
|
<li> Obtain short term credentials at
|
|
http://grid.ncsa.illinois.edu/myproxy/MyProxyLogon/
|
|
You will need to download and run the MyProxyLogon
|
|
program.
|
|
This will create a keyfile in, typically, the directory ".globus".
|
|
The keyfile will have a name similar to this: "x509up_u13615"
|
|
The other elements in ".globus" are certificates to use in
|
|
validating the certificate your client gets from the server.
|
|
<p>
|
|
<li> Obtain the program source ImportKey.java
|
|
from this location: http://www.agentbob.info/agentbob/79-AB.html
|
|
(read the whole page, it will help you understand the remaining steps).
|
|
</ol>
|
|
|
|
<h3>Building the KeyStore</h3>
|
|
You will have to modify the keyfile in the previous step
|
|
and then create a keystore and install the key and a certificate.
|
|
The commands are these:
|
|
<pre>
|
|
openssl pkcs8 -topk8 -nocrypt -in x509up_u13615 -inform PEM -out key.der -outform DER
|
|
|
|
openssl x509 -in x509up_u13615 -inform PEM -out cert.der -outform DER
|
|
|
|
java -classpath <path to ImportKey.class> -Dkeypassword="<password>" -Dkeystore=./<keystorefilename> key.der cert.der
|
|
</pre>
|
|
Note, the file names "key.der" and "cert.der" can be whatever you choose.
|
|
It is probably best to leave the .der extension, though.
|
|
|
|
<h3>Building the TrustStore</h3>
|
|
Building the truststore is a bit tricky because as provided, the
|
|
certificates in ".globus" need some massaging. See the script below
|
|
for the details. The primary command is this, which is executed for every
|
|
certificate, c, in globus. It sticks the certificate into the file
|
|
named "truststore"
|
|
<pre>
|
|
keytool -trustcacerts -storepass "password" -v -keystore "truststore" -importcert -file "${c}"
|
|
</pre>
|
|
|
|
<h3>Running the C Client</h3>
|
|
|
|
Refer to the section on <a href="#CLIENTCERTS">Client-Side Certificates</a>.
|
|
The keys specified there must be set in the rc file to support
|
|
ESG access.
|
|
<ul>
|
|
<li> HTTP.COOKIEJAR=~/.dods_cookies
|
|
<li> HTTP.NETRC=~/.netrc
|
|
<li> HTTP.SSL.CERTIFICATE=~/esgkeystore
|
|
<li> HTTP.SSL.KEY=~/esgkeystore
|
|
<li> HTTP.SSL.CAPATH=~/.globus
|
|
<li> HTTP.SSL.VALIDATE=1
|
|
</ul>
|
|
Of course, the file paths above are suggestions only;
|
|
you can modify as needed.
|
|
The HTTP.SSL.CERTIFICATE and HTTP.SSL.KEY
|
|
entries should have same value, which is the file path for the
|
|
certificate produced by MyProxyLogon. The HTTP.SSL.CAPATH entry
|
|
should be the path to the "certificates" directory produced by
|
|
MyProxyLogon.
|
|
<p>
|
|
As noted, also uses re-direction based authentication.
|
|
So, when it receives an initial connection from a client, it
|
|
redirects to a separate authentication server. When that
|
|
server has authenticated the client, it redirects back to
|
|
the original url to complete the request.
|
|
|
|
<h3>Script for creating Stores</h3>
|
|
The following script shows in detail how to actually construct the key
|
|
and trust stores. It is specific to the format of the globus file
|
|
as it was when ESG support was first added. It may have changed
|
|
since then, in which case, you will need to seek some help
|
|
in fixing this script. It would help if you communicated
|
|
what you changed to the author so this document can be updated.
|
|
<pre>
|
|
#!/bin/sh -x
|
|
KEYSTORE="esgkeystore"
|
|
TRUSTSTORE="esgtruststore"
|
|
GLOBUS="globus"
|
|
TRUSTROOT="certificates"
|
|
CERT="x509up_u13615"
|
|
TRUSTROOTPATH="$GLOBUS/$TRUSTROOT"
|
|
CERTFILE="$GLOBUS/$CERT"
|
|
PWD="password"
|
|
|
|
D="-Dglobus=$GLOBUS"
|
|
CCP="bcprov-jdk16-145.jar"
|
|
CP="./build:${CCP}"
|
|
JAR="myproxy.jar"
|
|
|
|
# Initialize needed directories
|
|
rm -fr build
|
|
mkdir build
|
|
rm -fr $GLOBUS
|
|
mkdir $GLOBUS
|
|
rm -f $KEYSTORE
|
|
rm -f $TRUSTSTORE
|
|
|
|
# Compile MyProxyCmd and ImportKey
|
|
javac -d ./build -classpath "$CCP" *.java
|
|
javac -d ./build ImportKey.java
|
|
|
|
# Execute MyProxyCmd
|
|
java -cp "$CP myproxy.MyProxyCmd
|
|
|
|
# Build the keystore
|
|
openssl pkcs8 -topk8 -nocrypt -in $CERTFILE -inform PEM -out key.der -outform DER
|
|
openssl x509 -in $CERTFILE -inform PEM -out cert.der -outform DER
|
|
java -Dkeypassword=$PWD -Dkeystore=./${KEYSTORE} -cp ./build ImportKey key.der cert.der
|
|
|
|
# Clean up the certificates in the globus directory
|
|
for c in ${TRUSTROOTPATH}/*.0 ; do
|
|
alias=`basename $c .0`
|
|
sed -e '0,/---/d' <$c >/tmp/${alias}
|
|
echo "-----BEGIN CERTIFICATE-----" >$c
|
|
cat /tmp/${alias} >>$c
|
|
done
|
|
|
|
# Build the truststore
|
|
for c in ${TRUSTROOTPATH}/*.0 ; do
|
|
alias=`basename $c .0`
|
|
echo "adding: $TRUSTROOTPATH/${c}"
|
|
echo "alias: $alias"
|
|
yes | keytool -trustcacerts -storepass "$PWD" -v -keystore ./$TRUSTSTORE -alias $alias -importcert -file "${c}"
|
|
done
|
|
exit
|
|
</pre>
|
|
|
|
</body>
|
|
</html>
|