6.3 KiB
NetCDF Byterange Support
[TOC]
Introduction
Suppose that you have the URL to a remote dataset which is a normal netcdf-3 or netcdf-4 file.
The netCDF-c library now supports read-only access to such datasets using the HTTP byte range capability [], assuming that the remote server supports byte-range access.
Two examples:
- A Thredds server supporting the Thredds "fileserver" Thredds protocol, and containing a netcdf classic file.
- An Amazon S3 dataset containing a netcdf enhanced file.
Other remote servers may also provide byte-range access in a similar form.
It is important to note that this is not intended as a true production capability because it is believed that this kind of access can be quite slow. In addition, the byte-range IO drivers do not currently do any sort of optimization or caching.
Configuration
This capability is enabled using the option --enable-byterange option to the ./configure command for Automake. For Cmake, the option flag is -DNETCDF_ENABLE_BYTERANGE=true.
This capability requires access to libcurl, and an error will occur if byterange is enabled, but libcurl could not be located. In this, it is similar to the DAP2 and DAP4 capabilities.
Run-time Usage
In order to use this capability at run-time, with ncdump for
example, it is necessary to provide a URL pointing to the basic
dataset to be accessed. The URL must be annotated to tell the
netcdf-c library that byte-range access should be used. This is
indicated by appending the phrase #mode=bytes
to the end of the URL.
The two examples above show how this will look.
In order to determine the kind of file being accessed, the netcdf-c library will read what is called the "magic number" from the beginning of the remote dataset. This magic number is a specific set of bytes that indicates the kind of file: classic, enhanced, cdf5, etc.
Architecture
Internally, this capability is implemented with the following drivers:
- libdispatch/dhttp.c -- wrap libcurl operations.
- libsrc/httpio.c -- provide byte-range reading to the netcdf-3 dispatcher.
- libhdf5/H5FDhttp.c -- provide byte-range reading to the netcdf-4 dispatcher for non-cloud storage.
- H5FDros3.c -- provide byte-range reading to the netcdf-4 dispatcher for cloud storage (Amazon S3 currently).
Both httpio.c and H5FDhttp.c are adapters that use dhttp.c to do the work. Testing for the magic number is also carried out by using the dhttp.c code. H5FDros3 is also an adapter, but specialized for cloud storage access.
NetCDF Classic Access
The netcdf-3 code in the directory libsrc is built using a secondary dispatch mechanism called ncio. This allows the netcdf-3 code be independent of the lowest level IO access mechanisms. This is how in-memory and mmap based access is implemented. The file httpio.c is the dispatcher used to provide byte-range IO for the netcdf-3 code. Note that httpio.c is mostly just an adapter between the ncio API and the dhttp.c code.
NetCDF Enhanced Access
Non-Cloud Access
Similar to the netcdf-3 code, the HDF5 library provides a secondary dispatch mechanism H5FD. This allows the HDF5 code to be independent of the lowest level IO access mechanisms. The netcdf-4 code in libhdf5 is built on the HDF5 library, so it indirectly inherits the H5FD mechanism.
The file H5FDhttp.c implements the H5FD dispatcher API and provides byte-range IO for the netcdf-4 code (and for the HDF5 library as a side effect). It only works for non-cloud servers such as the Unidata Thredds server.
Note that H5FDhttp.c is mostly just an adapter between the H5FD API and the dhttp.c code.
The dhttp.c Code
The core of all this is dhttp.c (and its header include/nchttp.c). It is a wrapper over libcurl and so exposes the libcurl handles -- albeit as void*.
The API for dhttp.c consists of the following procedures:
- int nc_http_open(const char* objecturl, void** curlp, fileoffset_t* filelenp);
- int nc_http_read(void* curl, const char* url, fileoffset_t start, fileoffset_t count, NCbytes* buf);
- int nc_http_close(void* curl);
- typedef long long fileoffset_t;
The type fileoffset_t is used to avoid use of off_t or off64_t which are too volatile. It is intended to be represent file lengths and offsets.
nc_http_open
The nc_http_open procedure creates a Curl handle and returns it in the curlp argument. It also obtains and searches the headers looking for two headers:
- "Accept-Ranges: bytes" -- to verify that byte-range access is supported.
- "Content-Length: ..." -- to obtain the size of the remote dataset.
The dataset length is returned in the filelenp argument.
nc_http_read
The nc_http_read procedure reads a specified set of contiguous bytes as specified by the start and count arguments. It takes the Curl handle produced by nc_http_open to indicate the server from which to read.
The buf argument is a pointer to an instance of type NCbytes, which is a dynamically expandable byte vector (see the file include/ncbytes.h).
This procedure reads count bytes from the remote dataset starting at the offset start position. The bytes are stored in buf.
nc_http_close
The nc_http_close function closes the Curl handle and does any necessary cleanup.
Cloud Access
The HDF5 library code-base also provides a Virtual File Drive (VFD) capable of providing byte-range access to cloud storage (Amazon S3 specifically).
This VFD is called H5FDros3. In order for the netcdf library to make use of it, the HDF5 library must be built using the --enable-ros3-vfd option. Netcdf can discover that this capability was enabled and can then make use of it to provide byte-range access to the cloud.
Point of Contact
Author: Dennis Heimbigner
Email: dmh at ucar dot edu
Initial Version: 12/30/2018
Last Revised: 3/11/2023