Commit Graph

162 Commits

Author SHA1 Message Date
Dennis Heimbigner
9abe3cb961 Fix bug in run_newformat.sh
Problem was basically that the test file ref_oldformat.zip
was incorrect. Additionally, logic in zsync.c was incorrect.

### Misc. other fixes
1. Turn off accidental debug output
2024-08-13 20:53:04 -06:00
Dennis Heimbigner
06d2fe5b13 # Addendum
As suggested by Ward, I ensured that this PR supports
read backward compatibility with old key format.
This addition also adds a test case for this.

## Misc. Other Changes
* Remove some unused code
* Cleanup json error handling
* Fix some more unsigned/signed conversions warning
2024-06-25 17:13:02 -06:00
Dennis Heimbigner
076da97aa4 Convert NCzarr meta-data to use only Zarr attributes
As discussed in a netcdf meeting, convert NCZarr V2 to store all netcdf-4 specific info as attributes. This improves interoperability with other Zarr implementations by no longer using non-standard keys.

## Other Changes
* Remove support for older NCZarr formats.
* Update anonymous dimension naming
* Begin the process of fixing the -Wconversion and -Wsign-compare warnings in libnczarr, nczarr_test, and v3_nczarr_test.
* Update docs/nczarr.md
* Rebuild using the .y and .l files
2024-06-19 18:09:29 -06:00
Dennis Heimbigner
9a478edb06 Fix duplicate definition when using aws-sdk-cpp.
re: Issue https://github.com/Unidata/netcdf-c/issues/2927

The NC_s3sdkinitialize NC_s3sdkfinalize functions were
misplaced. They should have been moved from ds3util.c to
ncs3sdk_h5.c.  When using ncs3sdl_aws.cpp, this resulted in a
duplicate definition.

Also, found and fixed a memory leak in the NCZarr S3 code.
2024-05-20 19:15:19 -06:00
Dennis Heimbigner
f0f0f39950 Cleanup various Zarr-related build issues
# Description
Remove various obsolete build options. Also do some code movement.

## Specific Changes

* The remotetest server is sometimes unstable, so provide a mechanism
  to force disabling calls to remotetest.unidata.ucar.edu.
  This is enabled by adding a repository variable named
  REMOTETESTDOWN with the value "yes".
* Fix CMakeLists.txt to use the uname command as an alternate
  to using the hostname command (which does not work under cygwin).
* Remove the JNA stuff as obsolete
* Remove the ENABLE_CLIENTSIDE_FILTERS options since it has been
  disabled for a while.
* Fix bad option flag in some github action .yml files: change --disable-xml2 to --disable-libxml2
* Collect globalstate definitions into nc4internal.h
* Remove ENABLE_NCZARR_FILTERS_TESTING option as obsolete and replace
  with ENABLE_NCZARR_FILTERS
* Move some dispatcher independent functions from libsrc4/nc4internal.c to libdispatch/ddispatch.c
* As a long term goal, and because it is now the case that --enable-nczarr
    => USE_NETCDF4, make the external options --enable-netcdf-4 and
    --enable-netcdf4 obsolete in favor of --enable-hdf5
    We will do the following for one more release cycle.
        1. Make --enable-netcdf-4 be an alias for --enable-netcdf4.
        2. Make --enable-netcdf4 an alias for --enable-hdf5.
        3. Internally, convert most uses of USE_NETCDF_4 ad USE_NETCDF4 to USE_HDF5
    After the next release, --enable-netcdf-4 and --enable-netcdf4 will
    be removed.
2024-05-15 18:46:25 -06:00
Ward Fisher
c26f7eabf4 Refactor macro _FillValue to NC_FillValue in support of https://github.com/Unidata/netcdf-c/issues/2858 2024-04-24 11:38:07 -06:00
Ward Fisher
9639ba445d
Merge pull request #2875 from seanm/clang-tidy-fixes
Misc clang-tidy fixes, and added a .clang-tidy config file
2024-04-05 14:45:22 -06:00
Ward Fisher
38d3831319
Merge branch 'main' into clang-tidy-fixes 2024-04-02 15:30:47 -06:00
Peter Hill
3c6bd83330
Change format of backwards-loops
Enables using correct `size_t` type for loop counter, usually fixing
some conversion warnings
2024-03-25 14:25:38 +00:00
Kyle Shores
8fd6dcb979 Replace ENABLE_S3 with NETCDF_ENABLE_S3 2024-03-18 15:54:15 -05:00
Kyle Shores
d1334f7b88 Replace ENABLE_NCZARR with NETCDF_ENABLE_NCZARR 2024-03-18 15:52:19 -05:00
Kyle Shores
ece5a776cb Replace ENABLE_DLL with NETCDF_ENABLE_DLL 2024-03-18 15:51:21 -05:00
Ward Fisher
5973f3d683
Merge pull request #2847 from K20shores/packaging
Use cmake netCDF with target_* for many options
2024-03-11 15:55:36 -06:00
Ward Fisher
cc1494d988
Merge branch 'main' into awsdfalt.dmh 2024-03-05 12:50:08 -07:00
Sean McBride
42d0b48e8d Auto fixed clang-tidy readability-avoid-const-params-in-decls warnings 2024-03-01 20:46:58 -05:00
Sean McBride
c92381721e Manually fixed some bugprone-suspicious-semicolon warnings 2024-03-01 20:46:58 -05:00
Sean McBride
7516c21032 Manually fixed some bugprone-suspicious-string-compare warnings 2024-03-01 20:46:58 -05:00
Kyle Shores
c29db073eb setting dll export on each target 2024-02-29 11:36:47 -06:00
Peter Hill
907e5cc43f
CMake: Use target_link_libraries with HDF5::HDF5 target 2024-02-16 10:51:20 +00:00
Kyle Shores
f9e3247164 merging main, addressing some PR comments 2024-02-07 09:53:45 -06:00
Kyle Shores
f41178f6e0 adding hdf5 to nczarr 2024-01-26 16:48:50 -06:00
Ward Fisher
16bcb1ddb9 Merge branch 'silence-nclist-warnings' of https://github.com/ZedThree/netcdf-c into rebase-gh2812.wif 2024-01-19 11:11:21 -07:00
Ward Fisher
c1fb4b0bae
Merge pull request #2809 from ZedThree/silence-malloc-warnings
Silence conversion warnings from `malloc` arguments
2023-12-21 17:20:56 -07:00
Ward Fisher
6a628b9ca7 Rebased PR by hand against main. 2023-12-12 16:49:13 -07:00
wkliao
52198b3f12 count argument in H5Sselect_hyperslab
Argument 'count' in NetCDF is not exactly the same as the 'count' in
H5Sselect_hyperslabs(space_id, op, start, stride, count, block).
When the argument 'stride' is NULL, NetCDF's 'count' should be used in
argument 'block', for example,
   H5Sselect_hyperslabs(space_id, op, start, NULL, ones, count);
where 'one' is an array of all 1s. Although using NULL 'block' below
   H5Sselect_hyperslabs(space_id, op, start, NULL, count, NULL);
has the same effect, HDF5 internally stores the space of a subarray as a
list of single elements, instead of a "block", which can affect the
performance.
2023-12-12 16:45:19 -07:00
Ward Fisher
2616e2c411
Merge pull request #2745 from e-kwsm/chmod-x
chore: unset executable flag
2023-12-11 17:28:46 -07:00
Sean McBride
adc4dc1435 Replaced some sprintf with snprintf with aid of new variable containing size
One case required slightly complicated accounting of how much space is left in the buffer.
2023-12-08 13:30:38 -05:00
Dennis Heimbigner
b83ff8fc2b oops 2023-12-07 15:48:39 -07:00
Dennis Heimbigner
80dd2e95c6 Fix truncate code for S3 2023-12-07 15:38:46 -07:00
Dennis Heimbigner
d81d95a5cc unthrow 2023-12-02 21:03:25 -07:00
Peter Hill
4e1ff160e1
Change signature of nczm_sortenvv to take size_t
Always called with a `size_t` and passes `n` to `qsort` which expects
a `size_t` anyway
2023-11-28 16:28:31 +00:00
Peter Hill
653e09fd6d
Try to more consistently use size_t for nclistget index argument 2023-11-28 16:28:31 +00:00
Peter Hill
d07dac918c
Silence conversion warnings from malloc arguments
Mostly just add an explicit cast when calling `malloc` and its
variants. Sometimes instead change the type of a local variable if
this would silence multiple warnings.
2023-11-24 18:20:52 +00:00
Dennis Heimbigner
87497d79cf update 2023-11-04 16:08:59 -06:00
Dennis Heimbigner
5fa2defc7e Improve fetch performance of DAP4
Prior to this PR, DAP4 always fetched the whole (constrained) dataset
This PR changes the query processing so
1. It reads data on a per-variable request (equivalent to calling nc_get_var()).
2. It tracks a response for every query.

Most of the changes reflect having to do per-variable requests.
In any case, doing all this significantly reduces the amount of data transmitted and hence speeds up DAP4 requests.
2023-10-08 19:59:28 -06:00
Dennis Heimbigner
1552d894a2 Cleanup a number of issues.
re: Issue https://github.com/Unidata/netcdf-c/issues/2748

This PR fixes a number of issues and bugs.

## s3cleanup fixes
* Delete extraneous s3cleanup.sh related files.
* Remove duplicate s3cleanup.uids entries.

## Support the Google S3 API
* Add code to recognize "storage.gooleapis.com"
* Add extra code to track the kind of server being accessed: unknown, Amazon, Google.
* Add a new mode flag "gs3" (analog to "s3") to support this api.
* Modify the S3 URL code to support this case.
* Modify the listobjects result parsing because Google returns some non-standard XML elements.
* Change signature and calls for NC_s3urlrebuild.

## Handle corrupt Zarr files where shape is empty for a variable.
Modify behavior when a variable's "shape" dictionary entry.
Previously it returned an error, but now it suppresses such a variable.
This change makes it possible to read non-corrupt data from the file.
Also added a test case.

## Misc. Other Changes
* Fix the nclog level handling to suppress output by default.
* Fix de-duplicates code in ncuri.c
* Restore testing of iridl.ldeo.columbia.edu.
* Fix bug in define_vars() which did not always do a proper reclaim between variables.
2023-10-08 11:22:52 -06:00
Ward Fisher
80c746981d
Merge pull request #2758 from ZedThree/cmake-fix-linking-mpi
CMake: Ensure all libraries link against MPI if needed
2023-10-02 16:20:43 -06:00
Peter Hill
18c813b20b
CMake: Ensure all libraries link against MPI if needed 2023-10-02 10:31:24 +01:00
Dennis Heimbigner
df3636b959 Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.

The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.

## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
   - "size" with an integer value representing the (current) size of the dimension.
   - "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.

For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.

Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.

## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
   AWS Transfer Utility mechanism. This is controlled by the
   ```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-26 16:56:48 -06:00
Eisuke Kawashima
d755c4ff32
chore: unset executable flag 2023-08-23 13:31:42 +09:00
Dennis Heimbigner
9094d25409 Fix major bug in the NCZarr cache management
re: PR https://github.com/Unidata/netcdf-c/pull/2734
re: Issue https://github.com/Unidata/netcdf-c/issues/2733

As a result of an investigation by https://github.com/uweschulzweida,
I discovered a significant bug in the NCZarr cache management.
This PR extends the above PR to fix that bug.

## Change Overview
* Insert extra checks for cache overflow.
* Added test cases contingent on the --enable-large-file-tests option.
* The Columbia server is down, so it has been temporarily disabled.
2023-08-16 23:07:05 -06:00
Dennis Heimbigner
f1a3a64b65 Cleanup the handling of cache parameters.
re: https://github.com/Unidata/netcdf-c/issues/2733

When addressing the above issue, I noticed that there was a disconnect
in NCZarr between nc_set_chunk_cache and nc_set_var_chunk cache.
Specifically, setting nc_set_chunk_cache had no impact on the per-variable cache parameters when nc_set_var_chunk_cache was not used.

So, modified the NCZarr code so that the per-variable cache parameters are set in this order (#1 is first choice):
1. The values set by nc_set_var_chunk_cache
2. The values set by nc_set_chunk_cache
3. The defaults set by configure.ac
2023-08-10 16:57:57 -06:00
Dennis Heimbigner
db772ce34c Explicitly suppress variable length type compression
re: PR https://github.com/Unidata/netcdf-c/pull/2716).
re: Issue https://github.com/Unidata/netcdf-c/issues/2189

The basic change is to make use of the fact that HDF5 automatically suppresses optional filters when an attempt is made to apply them to variable-length typed arrays.
This means that e.g. ncdump or nccopy will properly see meaningful data.
Note that if a filter is defined as HDF5 mandatory, then the corresponding variable will be suppressed and will be invisible to ncdump and nccopy.
This functionality is also propagated to NCZarr.

This PR makes some minor changes to PR https://github.com/Unidata/netcdf-c/pull/2716 as follows:
* Move the test for filter X variable-length from dfilter.c down into the dispatch table functions.
* Make all filters for HDF5 optional rather than mandatory so that the built-in HDF5 test for filter X variable-length will be invoked.

The test case for this was expanded to verify that the filters are defined, but suppressed.
2023-08-03 15:47:28 -06:00
Dennis Heimbigner
8cab468169 Suppress filters on variables with non-fixed-size types.
re: Discussion https://github.com/Unidata/netcdf-c/discussions/2554
re: PR https://github.com/Unidata/netcdf-c/pull/2231
re: Issue https://github.com/Unidata/netcdf-c/issues/2189

After some discussion, the issue of applying filters on variables
whose type is not fixed size, was resolved as follows:
1. A call to nc_def_var_filter will ignore such filters, but will issue a log warning.
2. Loading (from an existing file) a variable whose type is not fixed-size and which has filters, will cause the variable to be suppressed.

This PR enforces those rules.

### Misc. Other changes
* Add a test case to test the vlen change.
* Make some minor clean-ups in various cmake and automake files.
* Remove unused test
2023-06-21 14:46:22 -06:00
Dennis Heimbigner
fb40a72b45 Improve performance of the nc_reclaim_data and nc_copy_data functions.
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179

As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.

However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.

This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.

First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.

The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.

The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.

## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-20 17:11:25 -06:00
Ward Fisher
bfb8a31aed
Merge pull request #2644 from ZhipengXue97/main
Fix potential dead store
2023-05-15 10:38:59 -06:00
Dennis Heimbigner
aeaf9e4bec notrace 2023-04-26 14:16:22 -06:00
Dennis Heimbigner
49737888ca Improve S3 Documentation and Support
## Improvements to S3 Documentation
* Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths.
* Modify *nczarr.md* to remove most of the S3 related text.
* Move the S3 text from *nczarr.md* to a new document *cloud.md*.
* Add some S3-related text to the *byterange.md* document.

Hopefully, this will make it easier for users to find the information they want.

## Rebuild NCZarr Testing
In order to avoid problems with running make check in parallel, two changes were made:
1. The *nczarr_test* test system was rebuilt. Now, for each test.
any generated files are kept in a test-specific directory, isolated
from all other test executions.
2. Similarly, since the S3 test bucket is shared, any generated S3 objects
are isolated using a test-specific key path.

## Other S3 Related Changes
* Add code to ensure that files created on S3 are reclaimed at end of testing.
* Used the bash "trap" command to ensure S3 cleanup even if the test fails.
* Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former.
* Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket.
* Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in.
* Merge partial S3 support into dhttp.c.
* Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it.
* Move some definitions from ncrc.h to ncs3sdk.h

## Other Changes
* Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-25 17:15:06 -06:00
Ward Fisher
dc6e392c9d
Merge branch 'main' into znotnc.dmh 2023-04-12 16:02:34 -06:00
Dennis Heimbigner
d7d216a3f5 Merge branch 'master' into dap4tests2.dmh 2023-03-16 14:03:29 -06:00