mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2024-11-21 03:13:42 +08:00
Code Issues Packages Projects Releases Wiki Activity
main
netcdf-c/plugins/Makefile.am
Dennis Heimbigner efdec07f3d Extend the netcdf API to support programmatic changes to the plugin search path 
Replaces PR https://github.com/Unidata/netcdf-c/pull/3024
         and PR https://github.com/Unidata/netcdf-c/pull/3033

re: https://github.com/Unidata/netcdf-c/issues/2753

As suggested by Ed Hartnett, This PR extends the netcdf.h API to support programmatic control over the search path used to locate plugins.

I created several different APIs, but finally settled on the following API as being the simplest possible. It does have the disadvantage that it requires use of a global lock (not implemented) if used in a threaded environment.

Specifically, note that modifying the plugin path must be done "atomically". That is, in a multi-threaded environment, it is important that the sequence of actions involved in setting up the plugin path must be done by a single processor or in some other way as to guarantee that two or more processors are not simultaneously accessing the plugin path get/set operations.

As an example, assume there exists a mutex lock called PLUGINLOCK. Then any processor accessing the plugin paths should operate as follows:
````
lock(PLUGINLOCK);
nc_plugin_path_get(...);
<rebuild plugin path>
nc_plugin_path_set(...);
unlock(PLUGINLOCK);
````
## Internal Architecture

It is assumed here that there only needs to be a single set of plugin path directories that is shared by all filter code and is independent of any file descriptor; it is global in other words. This means, for example, that the path list for NCZarr and for HDF5 will always be the same.

However internally, processing the set of plugin paths depends on the particular NC_FORMATX value (NC_FORMATX_NC_HDF5 and NC_FORMATX_NCZARR, currently). So the *nc_plugin_path_set* function, will take the paths it is given and propagate them to each of the NC_FORMATX dispatchers to store in a way that is appropriate to the given dispatcher.

There is a complication with respect to the *nc_plugin_path_get* function. It is possible for users to bypass the netcdf API and modify the HDF5 plugin paths directly. This can result in an inconsistent plugin path between the value used by HDF5 and the global value used by netcdf-c. Since there is no obvious fix for this, we warn the user of this possibility and otherwise ignore it.

## Test Changes
* New tests<br>
    a. unit_test/run_pluginpaths.sh -- was created to test this new capability.<br>
    b. A new test utility has been added as *unit_test/run_dfaltpluginpath.sh* to test the default plugin path list.
* New test support utilities<br>
    a. unit_test/ncpluginpath.c -- report current state of the plugin path<br>
    b. unit_test/tst_pluginpaths.c -- test program to support run_pluginpaths.sh

## Documentation
* A new file -- docs/pluginpath.md -- provides documentation of the new API. It includes some
  material taken fro filters.md.

## Other Major Changes
1. Cleanup the whole plugin path decision tree. This is described in the *docs/pluginpath.md* document and summarized in Addendum 2 below.
2. I noticed that the ncdump/testpathcvt.sh had been disabled, so fixed and re-enabled it. This necessitated some significant changes to dpathmgr.c.

## Misc. Changes
1. Add some path manipulation utilities to netcf_aux.h
2. Fix some minor bugs in netcdf_json.h
3. Convert netcdf_json.h and netcdf_proplist.h to BUILT_SOURCE.
4. Add NETCDF_ENABLE_HDF5 as synonym for USE_HDF5
5. Fix some size_t <-> int conversion warnings.
6. Encountered and fixed the Windows \r\n problem in tst_pluginpaths.c.
7. Cleanup some minor CMakeLists.txt problems.
8. Provide an implementation of echo -n since it appears to not be
   available on all platforms.
9. Add a property list mechanism to pass environmental information to filters.
10. Cleanup Doxyfile.in
11. Fixed a memory leak in libdap2; surprised that I did not find this earlier.

## Addendum 1: Proposed API

The API makes use of a counted vector of strings representing the sequence of directories in the path. The relevant type definition is as follows.
````
typedef struct NCPluginList {size_t ndirs; char** dirs;} NCPluginList;
````

The API proposed in this PR looks like this (from netcdf-c/include/netcdf_filter.h).

* ````int nc_plugin_path_ndirs(size_t* ndirsp);````
    Arguments: *ndirsp* -- store the number of directories in this memory.

    This function returns the number of directories in the sequence if internal directories of the internal plugin path list.

* ````int nc_plugin_path_get(NCPluginList* dirs);````
    Arguments:  *dirs* -- counted vector for storing the sequence of directies in the internal path list.

    This function returns the current sequence of directories from the internal plugin path list. Since this function does not modify the plugin path, it does not need to be locked; it is only when used to get the path to be modified that locking is required.  If the value of *dirs.dirs* is NULL (the normal case), then memory is allocated to hold the vector of directories. Otherwise, use the memory of *dirs.dirs* to hold the vector of directories.

* ````int nc_plugin_path_set(const NCPluginList* dirs);````
    Arguments: *dirs* -- counted vector for providing the new sequence of directories in the internal path list.

    This function empties the current internal path sequence and replaces it with the sequence of directories argument. Using an *ndirs* argument of 0 will clear the set of plugin paths.

## Addendum 2: Build-Time and Run-Time Constants.

### Build-Time Constants
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid;" align="center"><td colspan="4">Table showing the build-time computation of NETCDF_PLUGIN_INSTALL_DIR and NETCDF_PLUGIN_SEARCH_PATH.</td>
<tr style="outline: thin solid" ><th>--with-plugin-dir<th>--prefix<th>NETCDF_PLUGIN_INSTALL_DIR<th>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid" ><td>undefined<td>undefined<td>undefined<td>PLATFORMDEFALT
<tr style="outline: thin solid" ><td>undefined<td>&lt;abspath-prefix&gt;<td>&lt;abspath-prefix&gt;/hdf5/lib/plugin<td>&lt;abspath-prefix&gt;/hdf5/lib/plugin&lt;SEP&gt;PLATFORMDEFALT
<tr style="outline: thin solid" ><td>&lt;abspath-plugins&gt;<td>N.A.<td>&lt;abspath-plugins&gt;<td>&lt;abspath-plugins&gt;&lt;SEP&gt;PLATFORMDEFALT
</table>

<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid" align="center"><td colspan="2">Table showing the computation of the initial global plugin path</td>
<tr style="outline: thin solid"><th>HDF5_PLUGIN_PATH<th>Initial global plugin path
<tr style="outline: thin solid"><td>undefined<td>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid"><td>&lt;path1;...pathn&gt;<td>&lt;path1;...pathn&gt;
</table>
2024-10-18 19:23:59 -06:00

145 lines
4.4 KiB
Makefile
Raw Permalink Blame History

# Copyright 2018, UCAR/Unidata
# See netcdf/COPYRIGHT file for copying and redistribution conditions.
# Put Together AM_CPPFLAGS and AM_LDFLAGS
include $(top_srcdir)/lib_flags.am
# This linker flag specifies libtool version info.
# See http://www.gnu.org/software/libtool/manual/libtool.html#Libtool-versioning
# for information regarding incrementing `-version-info`.
plugin_version_info = -version-info 0:0:0
AM_LDFLAGS += -module -avoid-version -shared -export-dynamic $(NOUNDEFINED)
# Create an alternate directory if not installing.
ALTPLUGINDIR = ${abs_top_builddir}/plugins/plugindir
RPATH = -rpath $(abs_top_builddir)/.libs
# This is where the plugins are to be installed
if ENABLE_PLUGIN_DIR
plugindir = @NETCDF_PLUGIN_INSTALL_DIR@
else
plugindir = ${ALTPLUGINDIR}
AM_LDFLAGS += ${RPATH}
endif
plugin_LTLIBRARIES =
plugins_to_install =
# These libraries are for testing only
check_LTLIBRARIES =
if ISMINGW
LDADD = ${top_builddir}/liblib/libnetcdf.la
LIBADD = $(LDADD)
endif
if ISCYGWIN
LDADD = $(top_builddir)/liblib/libnetcdf.la
LIBADD = $(top_builddir)/liblib/libnetcdf.la
endif
EXTRA_DIST = CMakeLists.txt findplugin.in
# The HDF5 filter wrappers
EXTRA_DIST += \
H5Ztemplate.c H5Zmisc.c H5Zutil.c H5Znoop.c h5noop.h NCZmisc.c \
H5Zshuffle.c H5Zdeflate.c H5Zszip.c H5Zszip.h \
H5Zbzip2.c h5bzip2.h H5Zblosc.c H5Zblosc.h H5Zzstd.c H5Zzstd.h
# The Codec filter wrappers
EXTRA_DIST += NCZhdf5filters.c NCZstdfilters.c
# The Filter implementations
EXTRA_DIST += H5checksum.c
plugins_to_install += lib__nch5fletcher32.la lib__nch5shuffle.la
lib__nch5shuffle_la_SOURCES = H5Zshuffle.c
lib__nch5fletcher32_la_SOURCES = H5Zfletcher32.c H5checksum.c
if HAVE_DEFLATE
plugins_to_install += lib__nch5deflate.la
lib__nch5deflate_la_SOURCES = H5Zdeflate.c
endif
# Need our version of szip if libsz available and we are not using HDF5
if HAVE_SZ
plugins_to_install += lib__nch5szip.la
lib__nch5szip_la_SOURCES = H5Zszip.c H5Zszip.h
endif
if HAVE_BLOSC
lib__nch5blosc_la_SOURCES = H5Zblosc.c H5Zblosc.h
plugins_to_install += lib__nch5blosc.la
endif
if HAVE_ZSTD
lib__nch5zstd_la_SOURCES = H5Zzstd.c H5Zzstd.h
plugins_to_install += lib__nch5zstd.la
endif
if NETCDF_ENABLE_NCZARR_FILTERS
# The NCZarr codec libraries (they need libnetcdf)
lib__nczstdfilters_la_SOURCES = NCZstdfilters.c
lib__nczstdfilters_la_LIBADD = $(LIBADD) $(top_builddir)/liblib/libnetcdf.la
lib__nczhdf5filters_la_SOURCES = NCZhdf5filters.c
lib__nczhdf5filters_la_LIBADD = $(LIBADD) $(top_builddir)/liblib/libnetcdf.la
plugins_to_install += lib__nczhdf5filters.la
plugins_to_install += lib__nczstdfilters.la
endif
# The noop filter is to allow testing of multifilters and filter order
# Need two distinct instances
lib__nch5noop_la_SOURCES = H5Znoop.c H5Zutil.c h5noop.h
lib__nch5noop1_la_SOURCES = H5Znoop1.c H5Zutil.c h5noop.h
lib__nch5noop_la_LDFLAGS = $(AM_LDFLAGS) ${RPATH}
lib__nch5noop1_la_LDFLAGS = $(AM_LDFLAGS) ${RPATH}
if NETCDF_ENABLE_FILTER_TESTING
# The misc filter is to allow testing of filter arguments
lib__nch5misc_la_SOURCES = H5Zmisc.c H5Zutil.c h5misc.h
lib__nch5misc_la_LDFLAGS = ${top_builddir}/liblib/libnetcdf.la $(AM_LDFLAGS) ${RPATH}
lib__nczmisc_la_SOURCES = NCZmisc.c
lib__nczmisc_la_LDFLAGS = $(AM_LDFLAGS) ${RPATH}
# Provide a filter to test missing filter
lib__nch5unknown_la_SOURCES = H5Zunknown.c
lib__nch5unknown_la_LDFLAGS = $(AM_LDFLAGS) ${RPATH}
check_LTLIBRARIES += lib__nch5noop.la lib__nch5noop1.la lib__nch5unknown.la
check_LTLIBRARIES += lib__nch5misc.la lib__nczmisc.la
endif
# Bzip2 is used to test more complex filters
lib__nch5bzip2_la_SOURCES = H5Zbzip2.c h5bzip2.h
BZIP2SRC = blocksort.c huffman.c crctable.c randtable.c compress.c decompress.c bzlib.c bzlib.h bzlib_private.h
EXTRA_DIST += ${BZIP2SRC} BZIP2_LICENSE
if HAVE_LOCAL_BZ2
lib__nch5bzip2_la_SOURCES += ${BZIP2SRC}
endif
plugins_to_install += lib__nch5bzip2.la
if ENABLE_PLUGIN_DIR
plugin_LTLIBRARIES += $(plugins_to_install)
else
check_LTLIBRARIES += $(plugins_to_install)
endif
DISTCLEANFILES = H5Znoop1.c H5Znoop2.c ncjson.h findplugin.sh
BUILT_SOURCES = H5Znoop1.c
H5Znoop1.c: Makefile H5Znoop.c
echo '#define NOOP_INSTANCE 1' > $@
cat ${srcdir}/H5Znoop.c >> $@
# Record where bzip2 came from; may be out of date
BZIP2VER = 1.0.8
BZIP2DIR = bzip2-${BZIP2VER}
BZIP2URL = https://sourceware.org/pub/bzip2/${BZIP2DIR}.tar.gz
bzip2::
rm -fr ./${BZIP2DIR} ${BZIP2SRC} BZIP2_LICENSE
wget ${BZIP2URL}
tar -zxf ${BZIP2DIR}.tar.gz
cd ${BZIP2DIR}; cp ${BZIP2SRC} ..; cp LICENSE ../BZIP2_LICENSE ; cd ..
rm -fr ./${BZIP2DIR}
Reference in New Issue View Git Blame Copy Permalink