strlcat provides better protection against buffer overflows.
Code is taken from the FreeBSD project source code. Specifically:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strlcat.c
License appears to be acceptable, but needs to be checked by e.g. Debian.
Step 1:
1. Add to netcdf-c/include/ncconfigure.h to use our version
if not already available as determined by HAVE_STRLCAT in config.h.
2. Add the strlcat code to libdispatch/dstring.c
3. Turns out that strlcat was already defined in several places.
So remove it from:
ncgen3/genlib.c
ncdump/dumplib.c
3. Define strlcat extern definition in ncconfigure.h.
4. Modify following directories to use strlcat:
libdap2 libdap4 ncdap_test dap4_test
Will do others in subsequent steps.
Some parameters like stringlength actually affect a dimension
named maxStrlen. So, add some aliasing so maxstrlen can be
specified as a parameter and as an alias for stringlength.
The affected parameters (case insensitive):
stringlength has alias maxstrlen
stringlength_<varname> has alias maxstrlen_<varname>
Also:
1. added a test case in ncdap_test/testurl.sh
2. added note to documentation
url parameter handling, the parameter
lookup code was ignoring parameters
because I was giving it a url stripped
of parameters instead of the one with
parameters.
This code change fixes the parameter processing
so it again used the correct url.
reference:
docs/OPeNDAP.dox#section Defined Client Parameters
re: https://github.com/conda-forge/libnetcdf-feedstock/pull/26
(Note: note conda-forge, not Unidata).
Revert dap2 code to 4.4.1.1. behavior so that attribute values
are forced into a specific range. Current behavior generates
error if an attribute value is out of range.
2. Factored out the parameter string parsing for ncgen and nccopy
int libdispatch/dfilter.c + include/ncfilter.h
3. Allow a parameter string to use constant types other than
unsigned int. See docs/filters.md for details.
4. Moved the old content of include/netcdf_filter.h into include/netcdf.h
and removed include/netcdf_filter.h as no longer needed.
5. Force the test filter (bzip2) in nc_test4/filter_test to
be built using BUILT_SOURCES.
dap code will create a real temporary file in which to store the
converted metadata for the DAP .dds or .dmr.
It was assumed that the nc_close code would reclaim the
temporary file. For DAP2, reclamation occurs in the ncio
code. For DAP4, it was assumed that the libsrc4 code would do
the reclamation, but for whatever reason, this is not happening.
Thus, in this situation, a temporary file is left in the file
system. Aside from being irritating to users, this screws up
'make distcheck'.
So the DAP4 code is fixed to ensure that the temporary file is
properly reclaimed independent of the libsrc4 code.
were added to provide a path name converter from e.g. cygwin
paths to e.g. windows paths. This is necessary because
the shell scripts may produce cygwin paths, but the code
may have been compiled with Visual Studio. Similar issues
arise with Mingw.
At appropriate places, and if using Visual Studio or Mingw,
I added calls to the path conversion code.
Apparently I forgot to find all the places where this
conversion was needed. So this pr does the following:
1. Push the calls to the converter to the various libXXX
directories and out of libdispatch/dfile.c.
2. Add conversion calls to other parts of the code like oc2.
I also turns out that conversion code in dapcvt.c
had a bug when handling DAP Byte type under visual studio.
Notes:
1. there may still be places I missed that need to do path conversion.
2. need to make sure that calls to e.g. H5open also use converted path.
The problem was that for opendap, it is possible to use keywords
as identifiers
when there is no ambiguity. However, the DAP2
parser lost the case of the identifier used the lower case version.
Fix is to use the actual text of the symbol when it is used as an identifier.
Also added a test case for this (kwcase.*).
Additionally cleaned up some misc. dap2 testing problems.
1. ncdap_test/tst_ncdap3.sh was using an empty test set.
restored the testing of datasets.
2. as a consequence of #1, some tests needed to be updated with minor
tweeks.
3. fix dapmerge to handle multiple DODS_EXTRAS attributes.
4. modify buildattribute to suppress nul characters and terminate
the name at the first nul.
5. clean up various test scripts to remove residual, unused
references to obsolete netcdf-4 translation.
6. export e.g. NCDUMP from test_common.in so that non-top-level
shell scripts can access it.
This relies on the HDF5 capability to
dynamically load compression filters.
Note that a compression filter is just
a subcase of filters.
The primary user-visible changes are as follows:
1. Add a standard header "netcdf_filter.h" that defines
the necessary API extensions
2. Modify ncgen to support two new special attributes
"_Filter_ID" and "_Filter_Parameters" so that compression
can be turned on when creating a file using ncgen.
4. Add a detailed description of filtering support
to the user's guide; see the file filters.md
5. Add a test case directory for this: nc_test4/filter_test.
It is fragile and a ./configure flags (-enable-filter-test)
is defined (default disabled) to shut this off this test
to avoid spurious 'make check' failures.
Note that the HDF5 documentation is not up-to-date, so
much of what is encoded here comes from examining the
actual code in the file H5PL.c in the HDF5 source code.
1. When running under windows (as opposed to cygwin)
we need to make sure to not user /cygdrive/ file paths.
This was ocurring in libdap4/d4read.c, but may occur
elsewhere.
2. Shell scripts in the git repo are not being checked-out
with the executable mode set. Had core.filemode set to false.
Was a major hassle to fix.
Github issue https://github.com/Unidata/netcdf-c/issues/152
requested that "orphaned" DAS attributes be included in the netcdf
metadata as global variables. The term orphaned here meant that
they were not connected to any variable in the DDS.
This was done in pull request https://github.com/Unidata/netcdf-c/pull/164
However, some servers (e.g. Thredds) include attributes for variables not
specified in a constraint expression, but which exist in the full DDS.
So I was adding these to the set of global attributes, but in retrospect
this should not have been done: they should have been elided.
Solution: modify oc2 code to be more distriminatory about
which orphaned attributes to include.
Specific changes:
1. Add dap4 code: libdap4 and dap4_test.
Note that until the d4ts server problem is solved, dap4 is turned off.
2. Modify various files to support dap4 flags:
configure.ac, Makefile.am, CMakeLists.txt, etc.
3. Add nc_test/test_common.sh. This centralizes
the handling of the locations of various
things in the build tree: e.g. where is
ncgen.exe located. See nc_test/test_common.sh
for details.
4. Modify .sh files to use test_common.sh
5. Obsolete separate oc2 by moving it to be part of
netcdf-c. This means replacing code with netcdf-c
equivalents.
5. Add --with-testserver to configure.ac to allow
override of the servers to be used for --enable-dap-remote-tests.
6. There were multiple versions of nctypealignment code. Try to
centralize in libdispatch/doffset.c and include/ncoffsets.h
7. Add a unit test for the ncuri code because of its complexity.
8. Move the findserver code out of libdispatch and into
a separate, self contained program in ncdap_test and dap4_test.
9. Move the dispatch header files (nc{3,4}dispatch.h) to
.../include because they are now shared by modules.
10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts.
11. Make use of MREMAP if available
12. Misc. minor changes e.g.
- #include <config.h> -> #include "config.h"
- Add some no-install headers to /include
- extern -> EXTERNL and vice versa as needed
- misc header cleanup
- clean up checking for misc. unix vs microsoft functions
13. Change copyright decls in some files to point to LICENSE file.
14. Add notes to RELEASENOTES.md
re: https://github.com/Unidata/netcdf-c/issues/365
1. Added to RELEASENOTES.md
2. Add a range check to more closely
mimic unix sscanf
3. locate and fix same sscanf problems in ncgen/cvt.c
Still need a stable url for a test case.
Following command fails under visual studio.
ncdump -h http://thredds.ucar.edu/thredds/dodsC/nexrad/composite/gini/n0r/1km/20170216/Level3_Composite_n0r_1km_20170216_1635.gini
The problem is that sscanf for windows does not appear to support
scanning 8bit integers: it appears to only allow scanning of characters.
Solution:
Scan the input as an integer (for type Byte) or unsigned int (for type UByte)
and then recast the result as a char or unsigned char.
Primary code fix is in libdap2/dapcvt.c#dapcvtattrval
Problem was in oc2/dap.y.
In definition of errormsg:, change WORD_WORD to WORD_STRING
since the msg field of an opendap error response is a quoted
string.
Also took the opportunity to modify ncgen to
transfer the logging level (-L flag) into the c-code
generated using -lc.
the multiple definitions of
typedef struct DCEparsestate
near lines 10 and 42 causes compiler
problems with some versions of gcc.
remove the second typedef.
(Re: [netcdfgroup] nccopy fails with corrupted double link list)
shows that ncdump/nccopy was returning EPERM instead of
NC_EDAPCONSTRAINT as an error when we have a malformed constraint.
Also clean up a potential bug that might occur if the user invokes
nc_set_default_format before calling nc_open on a dap url.
re: github netcdf-c issue #271
This occurs for several reasons, including:
1. using H5Aopen_name instead of H5Aexists to test if attribute exists.
2. using H5Eset_auto instead of H5Eset_auto2.
There are probably others that will have to be extinguished as encountered.
p.s Hope I did not overdo this and kill too much.
Re e-eupport VGQ-678069
It was noticed that an attribute value of "nan." was being treated as
legal (it should be "nan"). The reason is that sscanf was not be checked
to see that all the attribute value characters were being read.
Solution is to verify that all characters were being consumed.
It appears the problem is that synth9 was erroneously
included in testing. It involves a nested sequence which
is not translatable. Not sure why it was still there,
but fix is to suppress the test.
This supports better authorization
handling for DAP requests, especially redirection
based authorization. I also added a new test case
ncdap_tests/testauth.sh.
Specifically, suppose I have a netrc file /tmp/netrc
containing this.
machine uat.urs.earthdata.nasa.gov login xxxxxx password yyyyyy
Also suppose I have a .ocrc file containing these lines
HTTP.COOKIEJAR=/tmp/cookies
HTTP.NETRC=/tmp/netrc
Assume that .ocrc is in the local directory or HOME.
Then this command should work (assuming a valid login and password).
ncdump -h "https://54.86.135.31/opendap/data/nc/fnoc1.nc"
Jennifer Adams has requested a reversion in behavior so that
all dap requests include a constraint.
Problem is caused by change in prefetch where if all variables
are requested, then no constraint is generated.
Fix is to always generate a constraint in prefetch.
re: e support ZCL-340681 and CPW-270700
HDF4 supports compression (and chunking)
but the chunking was not being recorded
for HDF4 files. So, I modified the necessary
files to support HDF4 chunking.
