Specific changes:
1. Add dap4 code: libdap4 and dap4_test.
Note that until the d4ts server problem is solved, dap4 is turned off.
2. Modify various files to support dap4 flags:
configure.ac, Makefile.am, CMakeLists.txt, etc.
3. Add nc_test/test_common.sh. This centralizes
the handling of the locations of various
things in the build tree: e.g. where is
ncgen.exe located. See nc_test/test_common.sh
for details.
4. Modify .sh files to use test_common.sh
5. Obsolete separate oc2 by moving it to be part of
netcdf-c. This means replacing code with netcdf-c
equivalents.
5. Add --with-testserver to configure.ac to allow
override of the servers to be used for --enable-dap-remote-tests.
6. There were multiple versions of nctypealignment code. Try to
centralize in libdispatch/doffset.c and include/ncoffsets.h
7. Add a unit test for the ncuri code because of its complexity.
8. Move the findserver code out of libdispatch and into
a separate, self contained program in ncdap_test and dap4_test.
9. Move the dispatch header files (nc{3,4}dispatch.h) to
.../include because they are now shared by modules.
10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts.
11. Make use of MREMAP if available
12. Misc. minor changes e.g.
- #include <config.h> -> #include "config.h"
- Add some no-install headers to /include
- extern -> EXTERNL and vice versa as needed
- misc header cleanup
- clean up checking for misc. unix vs microsoft functions
13. Change copyright decls in some files to point to LICENSE file.
14. Add notes to RELEASENOTES.md
Update utf8proc.[ch] to use the version now
maintained by the Julia Language project
(https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md).
The license for the previous version was
unacceptable for the Debian and Ubuntu release
systems. The new version both updates the code
and addresses the license issue.
It turns out that the utf8proc software we are using
was turned over to the Julia Language developers
and the license terms changed to allow modification.
(https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md).
So the fix here is as follows:
1. Wrap the library with a fixed interface: libdispatch/dutf8.c
and include/ncutf8.h.
2. Replace the existing utf8proc code with the new version
from https://github.com/JuliaLang/utf8proc.
3. Add a couple more test cases: nc_test/tst_utf8_validate.c
and nc_test_utf8_phrases.c. If/when I can find a usable
normalization test, I will incorporate that later.
This consists of a persistent attribute named
_NCProperties plus two computed attributes
_IsNetcdf4 and _SuperblockVersion.
See the 'Provenance Attributes' section
of docs/attribute_conventions.md for details.
This supports better authorization
handling for DAP requests, especially redirection
based authorization. I also added a new test case
ncdap_tests/testauth.sh.
Specifically, suppose I have a netrc file /tmp/netrc
containing this.
machine uat.urs.earthdata.nasa.gov login xxxxxx password yyyyyy
Also suppose I have a .ocrc file containing these lines
HTTP.COOKIEJAR=/tmp/cookies
HTTP.NETRC=/tmp/netrc
Assume that .ocrc is in the local directory or HOME.
Then this command should work (assuming a valid login and password).
ncdump -h "https://54.86.135.31/opendap/data/nc/fnoc1.nc"
The pnetcdf support was not
properly being used to provide
mpi parallel io for netcdf-3 classic
files. The wrong dispatch table was being
used. The fix was to modify
dfile.c#NC_check_file_type to properly
specify the pnetcdf dispatch table when
use_parallel was true.
If the netCDF-C library is built with the
HDF5 library but without the HDF4 library and one attempts
to open an HDF4 file, an abort occurs rather than returning
a proper error code (NC_ENOTNC).
Fix is to modify dfile.c#NC_check_file_type to properly
#ifdef relevant tests.
extract info from libnetcdf.settings
API is below.
I have made this API public yet
by adding it to netcdf.h. I will
do that when everyone is agreed on the
proper API.
extern const char* nc_settings(const char* key); /*get value of a specific key */
extern const char** nc_settings_all(); /*get all settings in envv format */
extern void nc_settings_reclaim(); /* reclaim all space and clean up */
Envv format is
{key,value}*,NULL
Also added test: nc_test/tst_settings.c
Initial attempt to make libnsetcdf.settings
accessible programatically.
Added libdispatch/ncsettings.hdr
used to construct ncsettings.c
See end of libdispatch/Makefile.am to
see how ncsettings.c is constructed.
Initial split of Guide from rest of documentation. Using 'tagfiles' to cross-reference from one documentation set to the other works, but then it drags all of the 'external' related pages into the index of the other.
Modified visability of the external groups and pages.
Adding proper section names to 'Example' sections, to avoid Doxygen warnings.
Initial split of Guide from rest of documentation. Using 'tagfiles' to cross-reference from one documentation set to the other works, but then it drags all of the 'external' related pages into the index of the other.
Modified visability of the external groups and pages.
Adding proper section names to 'Example' sections, to avoid Doxygen warnings.
Reorganizing documentation into two separate doxygen projects.
Moved .texi documents into an old directory.
Pulled the tutorial document into a separate document, similar to the guide document.
Turned on treeview
was incomplete; complete the fix.
2. There is an inconsistency in the netcdfh interface
about whether rank (# dimensions) is an int or
size_t. I inadvertently assumed the latter and that
breaks some API calls; so revert back.
should be under ENABLE_DAP_REMOTE_TESTS.
Fixed to make sure that this is so.
Also attempted to fix ncdap_test/CMakeLists.txt,
but probably got it wrong.
HT to Nico Schlomer.
2. Attempted to reduce the number of conversion errors
when -Wconversion is set. Fixed oc2, but
rest of netcdf remains to be done.
HT to Nico Schlomer.
3. When doing #2, I discovered an error in ncgen.y
that has remained hidden. This required some other
test case fixes.
times, but returning the same ncid.
2. Separate out the dap auth tests
and make them disabled by default.
3. turn of ncdap_test/test_varm3 until
we can find a copy of coads_climatology.nc
Bumped revision to 4.3.1.2. There may not be a 4.3.1.2 release, but I didn't want to leave it as 4.3.1.1.
Removed CMake-specific doxygen/documentation template, made changes required so that
CMake and autotools-based builds can use the same template (Doxyfile.in).
Add a new function called nc_inq_format_extended that
returns more detailed format information (vis-a-vis
nc_inq_format) about an open dataset.
Note that the netcdf API will present the file as if it had
the format specified by nc_inq_format. The true file
format, however, may not even be a netcdf file; it might be
DAP, HDF4, or PNETCDF, for example. This function returns
that true file type. It also returns the effective mode for
the file.
signature: nc_inq_format_extended(int ncid, int* formatp, int* modep)
where
* ncid is the NetCDF ID from a previous call to nc_open() or
nc_create().
* formatp is a pointer to a location for returned true format.
* modep is a pointer to a location for returned mode flags.
Refer to the actual list in the file netcdf.h to see the
currently defined set.
Also added test cases (tst_formatx*).
group renaming. The primary API
is 'nc_rename_grp(int grpid, const char* name)'.
No test cases provided yet.
This also required adding a rename_grp entry
to the dispatch tables.
unlimited dimension hanging. Extending the size of an unlimited
dimension in HDF5 must be a collective operation, so now an error is
returned if trying to extend in independent access mode.
Quincey's bug fixes for parallel build portability, particularly
OpenMPI on MacOS-X.
netCDF classic or 64-bit offset files that have a UINT32_MAX flag for
large last record size of a variable that has values larger than 1
byte. This problem had previously been fixed for *writing* such data,
but was only tested with an ncbyte variable. Fixed test to
demonstrate problem and the fix.
More updates to chunking documentation, cosmetic fixes for some
"--option=" documentation that doxygen turns into mdash.
(primarily from libsrc4)
into its own dispatch library
called libsrc5.
2. Fixed part of Jira NCF-253
by removing the need for the
pnetcdf_ndims field.
For some reason, the original
code tried to cache the variable
ranks rather than computing them
as needed. Fixed by doing
an ...inq_varndims call as needed.
3. found some places where NC_MAX_DIMS
was being stack allocated and changed
to heap allocation.
Still some cases in nc_test4.
Problem was that the NC_create
code was not checking for the NC_CLASSIC_MODEL
mode flag in deciding what dispatch table to use.
This meant that it was then defaulting to use
the default format, and if that was changed
to e.g. NC_FORMAT_NETCDF4, then it would try
to create a netcdf-4 format file, even is
NC_CLASSIC_MODEL mode flag was set.
for JIRA issue NCF-241.
This is only temporary
until I can make pnetcdf
operate as a separate dispatch table.
Also, fix nc_test4/tst_pnetcdf
to open with nc_open_par;
this is necessary because a pnetcdf
created file cannot be opened
as a netcdf classic file.
to indicate that they only work for atomic types,
not user defined types.
2. modified NCDEFAULT_{get/put}_vars to no longer use
nc_get/put_varm. They now directly use nc_get/put_vara
directly. This means that nc_get/put_vars now work
properly for user defined types as well as atomic types.
3. Added test cases for get_vars/put_vars with a
user defined type. Tests placed into
nc_test/tst_compounds.c
read 4 bytes from a 3 byte file, Windows would fail with an
error 22, "Invalid Operation". This would happen when trying to
read the 4-byte magic number from a very small file.
This problem was uncovered by nc_test/tst_misc.c.
When fread returns 0 and errno is 22, NC_ENOTNC is returned.
CMake related changes in CMakeLists.txt files,
cmake_config.h.in. Other changes relate to
Windows-specific issues, and changes made
when regenerating generated source files.
o Changed variable names 'typeid' to 'typeid1' to avoid a namespace conflict
in visual studio.
o Cleaned up a handful of warnings in Visual Studio.
o Addressed a few Coverity-discovered issuesl
o Made changes to CMake-based builds.
contain as little file-type specific info as possible. It
modifies especially libsrc so that all of the netcdf-3 data
that used to be in struct NC is now kept in a separate chunk
of data pointed to by the struct NC. This makes all of
current protocols consistent: netcdf-3, netcdf-4, and dap.
reported by static analysis, including memory leak in ncdump, missing
size_t cast for chunk cache. Fixed various doc problems, including
byte vs. char issues, missing NC_UBYTE in type list, needed link to
"Building with Windows" page.
(on windows platforms), resulting in a downstream SIGSEGV upon memory
allocation of 0 bytes. Gave nvdims a default value of 1. This may not
be the best possible fix, but in the short term it has fixed the issue.
The problem would occur when NetCDF was compiled with NetCDF-4 support,
on Windows, in 64-bit mode, and when nctest/nctest.exe would run.
The in-memory files can be made persistent if nc_create is called with
NC_DISKLESS|NC_WRITE flags set. Initial test case also included.
- Modified ncio mechanism to support
multiple ncio packages; this is so we
can have posixio and memio operating
at the same time.
- cleanup up a bunch of lint issues (unused variables, etc).
- Fix NCF-157 to modify DAP code to support
partial variable retrieval.
- Fix of NCF-154 to solve problem of ncgen
improperly processing data lists for variables
of size greater than 2**18 bytes.
- Fix ncgen processing of char variables that have
multiple unlimited dimensions.
- Partly fix Jira issue: NCF-145 (vlen issues).
- Benchmark program nc_test4/tst_ar4_*) requires arguments
and should only be invoked inside a shell
script; fixed so that they terminate cleanly
if invoked with no arguments.
- Fix the Doxygen processing so it will work
with make distcheck.
- Begin switchover to using an alternative to ncio.
- Begin support for in-memory (diskless) files.
