NOTE: it is important that this fix gets into 4.9.3
re: Issue https://github.com/Unidata/netcdf-c/issues/2798
## Modifications
* This PR includes PR https://github.com/Unidata/netcdf-c/pull/2813
* Support the following AWS environment variables in the internal S3 library
(they are already supported by aws-sdk-cpp).
- AWS_REGION
- AWS_DEFAULT_REGION
- AWS_ACCESS_KEY_ID
- AWS_CONFIG_FILE
- AWS_PROFILE
- AWS_SECRET_ACCESS_KEY
- (source https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html).
* Support an empty region when specifying s3.amazonaws.com as the host.
* Move some S3/AWS related functions to ds3util.c
* Add a test case to test empty region and AWS_[DEFAULT]_REGION.
`options->file` is allocated on the stack and so will never be `NULL`,
making this conditional always true. Instead, we want to check the
value of `file`. As `options` is allocated with `calloc`, we know that
`file` will be zero-initialised, and it's safe to just check the first
element
`chunk_size` is only used if `PRINT_CHUNK_WASTE_REPORT` is
defined. Also move the declaration of `chunk_size` inside the `#ifdef`
to silence `set-but-unused` warning
The most critical bug is in nch5s3comms.c.
I for some reason assumed that signing keys
did not contain any zero bytes. But obviously
it can, so a test was removed.
Other fixes:
1. Guarantee allocated memory is initialized to all zeros.
2. Cleanup errmsg handling in libncpoco.
3. Fix processing of aws list-objects-v2 because I misread the syntax.
Mostly just add an explicit cast when calling `malloc` and its
variants. Sometimes instead change the type of a local variable if
this would silence multiple warnings.
Use memcpy to copy correctly even for unaligned memory. This was already done for some functions here, but not all.
Also took the oppurtunity to remove a bunch of seemingly obsolete/commented code.
