Commit Graph

30 Commits

Author SHA1 Message Date
Dennis Heimbigner
076da97aa4 Convert NCzarr meta-data to use only Zarr attributes
As discussed in a netcdf meeting, convert NCZarr V2 to store all netcdf-4 specific info as attributes. This improves interoperability with other Zarr implementations by no longer using non-standard keys.

## Other Changes
* Remove support for older NCZarr formats.
* Update anonymous dimension naming
* Begin the process of fixing the -Wconversion and -Wsign-compare warnings in libnczarr, nczarr_test, and v3_nczarr_test.
* Update docs/nczarr.md
* Rebuild using the .y and .l files
2024-06-19 18:09:29 -06:00
Dennis Heimbigner
9a478edb06 Fix duplicate definition when using aws-sdk-cpp.
re: Issue https://github.com/Unidata/netcdf-c/issues/2927

The NC_s3sdkinitialize NC_s3sdkfinalize functions were
misplaced. They should have been moved from ds3util.c to
ncs3sdk_h5.c.  When using ncs3sdl_aws.cpp, this resulted in a
duplicate definition.

Also, found and fixed a memory leak in the NCZarr S3 code.
2024-05-20 19:15:19 -06:00
Dennis Heimbigner
27f615bebc Properly handle missing regions in URLS
NOTE: it is important that this fix gets into 4.9.3

re: Issue https://github.com/Unidata/netcdf-c/issues/2798

## Modifications
* This PR includes PR https://github.com/Unidata/netcdf-c/pull/2813
* Support the following AWS environment variables in the internal S3 library
  (they are already supported by aws-sdk-cpp).
  - AWS_REGION
  - AWS_DEFAULT_REGION
  - AWS_ACCESS_KEY_ID
  - AWS_CONFIG_FILE
  - AWS_PROFILE
  - AWS_SECRET_ACCESS_KEY
  - (source https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html).
* Support an empty region when specifying s3.amazonaws.com as the host.
* Move some S3/AWS related functions to ds3util.c
* Add a test case to test empty region and AWS_[DEFAULT]_REGION.
2023-12-02 21:03:59 -07:00
Dennis Heimbigner
1552d894a2 Cleanup a number of issues.
re: Issue https://github.com/Unidata/netcdf-c/issues/2748

This PR fixes a number of issues and bugs.

## s3cleanup fixes
* Delete extraneous s3cleanup.sh related files.
* Remove duplicate s3cleanup.uids entries.

## Support the Google S3 API
* Add code to recognize "storage.gooleapis.com"
* Add extra code to track the kind of server being accessed: unknown, Amazon, Google.
* Add a new mode flag "gs3" (analog to "s3") to support this api.
* Modify the S3 URL code to support this case.
* Modify the listobjects result parsing because Google returns some non-standard XML elements.
* Change signature and calls for NC_s3urlrebuild.

## Handle corrupt Zarr files where shape is empty for a variable.
Modify behavior when a variable's "shape" dictionary entry.
Previously it returned an error, but now it suppresses such a variable.
This change makes it possible to read non-corrupt data from the file.
Also added a test case.

## Misc. Other Changes
* Fix the nclog level handling to suppress output by default.
* Fix de-duplicates code in ncuri.c
* Restore testing of iridl.ldeo.columbia.edu.
* Fix bug in define_vars() which did not always do a proper reclaim between variables.
2023-10-08 11:22:52 -06:00
Dennis Heimbigner
df3636b959 Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.

The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.

## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
   - "size" with an integer value representing the (current) size of the dimension.
   - "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.

For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.

Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.

## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
   AWS Transfer Utility mechanism. This is controlled by the
   ```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-26 16:56:48 -06:00
Dennis Heimbigner
98477b9f25 ## Addendum [5/9/23]
It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-09 21:13:49 -06:00
Dennis Heimbigner
681abc3fb1 s3-off 2023-04-30 18:41:31 -06:00
Dennis Heimbigner
75e3012ff3 debug5 2023-04-29 19:45:51 -06:00
Dennis Heimbigner
88bf095f26 intern2 2023-04-28 15:28:32 -06:00
Dennis Heimbigner
2eb131f516 intern1 2023-04-28 15:26:21 -06:00
Dennis Heimbigner
c2f3b6ee84 debug20 2023-04-28 15:11:42 -06:00
Dennis Heimbigner
2f693cf960 debug10 2023-04-28 14:52:20 -06:00
Dennis Heimbigner
613fd120d7 debug2 2023-04-28 14:40:46 -06:00
Dennis Heimbigner
20065682bb debug1 2023-04-28 14:30:48 -06:00
Dennis Heimbigner
7121588ac0 try10 2023-04-27 19:25:36 -06:00
Dennis Heimbigner
d0ef22360d try5 2023-04-27 19:15:07 -06:00
Dennis Heimbigner
b3fc253b58 try2 2023-04-27 19:04:06 -06:00
Dennis Heimbigner
3aaf5df99f try1 2023-04-27 18:58:35 -06:00
Dennis Heimbigner
579230d1a9 null1 2023-04-27 18:10:40 -06:00
Dennis Heimbigner
cd5199f51d ch1 2023-04-27 17:37:07 -06:00
Dennis Heimbigner
4bc1f1f3a3 ch0 2023-04-27 17:27:11 -06:00
Dennis Heimbigner
1cf6e3743b at1 2023-04-27 17:14:57 -06:00
Dennis Heimbigner
db2b59500f tag1 2023-04-27 17:02:36 -06:00
Dennis Heimbigner
c980b45617 region1 2023-04-27 16:49:43 -06:00
Dennis Heimbigner
b1899c05cb dump1 2023-04-27 16:40:09 -06:00
Dennis Heimbigner
174a50191a track3 2023-04-27 16:15:55 -06:00
Dennis Heimbigner
c548930e7d track2 2023-04-27 16:02:54 -06:00
Dennis Heimbigner
1c3b05a252 track1 2023-04-27 15:37:55 -06:00
Dennis Heimbigner
59c6d29394 segv1 2023-04-27 15:12:26 -06:00
Dennis Heimbigner
49737888ca Improve S3 Documentation and Support
## Improvements to S3 Documentation
* Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths.
* Modify *nczarr.md* to remove most of the S3 related text.
* Move the S3 text from *nczarr.md* to a new document *cloud.md*.
* Add some S3-related text to the *byterange.md* document.

Hopefully, this will make it easier for users to find the information they want.

## Rebuild NCZarr Testing
In order to avoid problems with running make check in parallel, two changes were made:
1. The *nczarr_test* test system was rebuilt. Now, for each test.
any generated files are kept in a test-specific directory, isolated
from all other test executions.
2. Similarly, since the S3 test bucket is shared, any generated S3 objects
are isolated using a test-specific key path.

## Other S3 Related Changes
* Add code to ensure that files created on S3 are reclaimed at end of testing.
* Used the bash "trap" command to ensure S3 cleanup even if the test fails.
* Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former.
* Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket.
* Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in.
* Merge partial S3 support into dhttp.c.
* Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it.
* Move some definitions from ncrc.h to ncs3sdk.h

## Other Changes
* Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-25 17:15:06 -06:00