Three S3 storage drivers are available for accessing Amazon S3.
1.*H5FDros3* -- This is an HDF5 Virtual File Driver provided as part of the HDF5 library. It is specifically used by NetCDF-C to perform byte-range access to netcdf-4 files. In order for this functionality to be available to NetCDF-C,
it is necessary to build the HDF5 library with the *--enable-ros3-vfd* option.
2.*aws-sdk-cpp* -- This is the standard Amazon AWS S3 SDK. It is written in C++. It is used specifically for NCZarr access to Zarr formatted datasets on S3.
3.*nch5s3comms* -- This is an experimental SDK, written in C, that provides the minimum functionality necessary to access S3. As with *aws-sdk-cpp*, it is used specifically for NCZarr access to Zarr formatted datasets on S3.
All three S3 drivers use the AWS profile mechanism to provide configuration information, and especially to provide authorization information.
Specifically, the ''~/.aws/credentials'' file should contain something like this.
```
[default]
output = json
aws_access_key_id=XXXX...
aws_secret_access_key=YYYY...
```
## Byte-Range Access {#nccloud_byterange}
The NetCDF-C library contains a mechanism for accessing traditional netcdf-4 files stored on remote computers.
The idea is to treat the remote data as if it was one big file,
and to use the HTTP protocol to read a contiguous sequence of bytes
from that remote "file".
This is performed using the "byte-range" header in an HTTP request.
In the Amazon S3 context, a copy of a dataset, a netcdf-3 or netdf-4 file, is uploaded into a single object in some bucket.
Then using the key to this object, it is possible to tell the netcdf-c library to treat the object as a remote file and to use the HTTP Byte-Range protocol to access the contents of the object.
The dataset object is referenced using a URL with the trailing fragment containing the string ````#mode=bytes````.
An examination of the test program _nc_test/test_byterange.sh_ shows simple examples using the _ncdump_ program.
Note that for S3 access, it is expected that the URL is in what is called "path" format where the bucket, _noaa-goes16_ in this case, is part of the URL path instead of the host.
The _#mode=bytes_ mechanism generalizes to work with most servers that support byte-range access.
Specifically, Thredds servers support such access using the HttpServer access method as can be seen from this URL taken from the above test program.
3._--with-s3-testing_=yes|no|public -- "yes" means do all S3 tests, "no" means do no S3 testing, "public" means do only those tests that involve publically accessible S3 data.
2.*-DENABLE_S3_INTERNAL* -- Force use of the *nch5s3comms* SDK instead of the *aws-cpp-sdk*.
3. *-DWITH-S3-TESTING_=ON|OFF|PUBLIC -- "ON" means do all S3 tests, "OFF" means do no S3 testing, "PUBLIC" means do only those tests that involve publically accessible S3 data.
Note that unlike Automake, CMake can properly locate C++ libraries, so it should not be necessary to specify _-laws-cpp-sdk-s3_ assuming that the aws s3 libraries are installed in the default location.
For CMake with Visual Studio, the default location is here:
````
C:/Program Files (x86)/aws-cpp-sdk-all
````
It is possible to install the sdk library in another location.
In this case, one must add the following flag to the cmake command.
````
cmake ... -DAWSSDK_DIR=\<awssdkdir\>
````
where "awssdkdir" is the path to the sdk installation.
For example, this might be as follows.
````
cmake ... -DAWSSDK_DIR="c:\tools\aws-cpp-sdk-all"
````
This can be useful if blanks in path names cause problems in your build environment.
# Appendix B. Building the S3 SDKs {#nccloud_s3sdk}
As mentioned, three S3 storage drivers are available for accessing Amazon S3.
1.*H5FDros3*
2.*aws-sdk-cpp*
2.*nch5s3comms*
## Building *H5FDros3*
This driver is part of the HDF5 library codebase.
It must be enabled at the time that the HDF5 library is built
by using the *--enable-ros3=vfd* option.
If built, then the NetCDF-C build process should detect it and make use of it.
## Building *aws-sdk-cpp*
Amazon provides (thru AWS-labs) an SDK for accessing the Amazon S3 cloud.
This library, [aws-sdk-cpp library](https://github.com/aws/aws-sdk-cpp.git),
has a number of properties of interest:
* It is written in C++
* It is available on [GitHub](https://github.com/aws/aws-sdk-cpp.git),
* It uses CMake + ninja as its primary build system.
### *\*nix\** Build
For linux, the following context works. Of course your mileage may vary.
* It was constructed by heavily modifying the HDF5 *H5Fs3commons.c* file and combining it with crypto code wrappers provided by libcurl. The resulting file was then modified to fit into the netcdf coding style.
