Commit Graph

91 Commits

Author SHA1 Message Date
Jacob Smith
075e74e9b2 Merge branch 'develop' of https://bitbucket.hdfgroup.org/scm/~jake.smith/hdf5 into feature/vfd_splitter_mirror_a 2020-04-07 10:29:00 -05:00
Quincey Koziol
b51585a911 Revise API for H5get_alloc_stats() to take a struct instead of separate values. 2020-03-20 16:39:55 -05:00
Jacob Smith
b65405439d Add Splitter VFD to library.
* "Simultaneous and equivalent" Read-Write and Write-Only channels for
  file I/O.
* Only supports drivers with the H5FD_FEAT_DEFAULT_VFD_COMPATIBLE flag for
  now, preventing issues with multi-file drivers.

Add Mirror VFD to library.

* Write-only operations over a network.
* Uses TCP/IP sockets.
* Server and auxiliary server-shutdown programs provided in a new directory,
  `utils/mirror_vfd`.
* Automated testing via loopback ("remote" of localhost).
2020-03-13 17:13:17 -05:00
David Young
ac7cd484d8 Filter all of the new _USED / _UNUSED type modifiers. 2020-02-18 16:31:35 -06:00
kmu
9be5b2842e merge and fix 2020-01-20 21:11:32 -06:00
kmu
628d267162 merge and fix conflict 2020-01-19 08:09:44 -06:00
Dana Robinson
a92c735c9b Squashed commit of the token_refactoring branch: 2020-01-16 13:29:34 -08:00
Quincey Koziol
58cf795321 Refactor all the 'H5VL_*_optional' callbacks to move the type of operation out
of the va_list, so it's at least possible for another connector to know what
the operation is and decide whether to implement it or not.

Added a new VOL sub-class called "introspect" where callbacks that report
information about the connector or container can be placed.  Added an
'opt_query' callback to this sub-class, for a connector to report back
to the library whether a particular optional callback operation is supported.
Also added a 'get_conn_cls' introspection callback, to retrieve the H5VL_class_t
of a connector (either the "current" connector, H5VL_GET_CONN_LVL_CURR, or
the terminal connector, H5VL_GET_CONN_LVL_TERM).

Moved the "post open" operation from a file 'specific' operation to a file
'optional' operation, now that it's possible to detect (with the 'opt_query'
introspection callback) whether a VOL connector implements an optional
operation, without just returning an error.

Added new internal VOL helper routines: H5VL_object_is_native, to determine
if an object is in (or is a) native file, and H5VL_file_is_same, to determine
if two objects are in (or are) the same terminal VOL connector's container.
(And moved the special handling for FILE_IS_EQUAL operation out of internal VOL
callback routine into H5VL_file_is_same)

Made new dataset 'get' operation for H5Dvlen_get_buf_size, aligning it better
with other 'get' operations in API.

Fixed several issues with pass-through connectors, which are now passing the
'make check-passthrough-vol' tests again.

A bunch of warning and style cleanups as well.
2019-12-19 22:41:37 -06:00
David Young
2dd5bbfe16 Merge pull request #1984 in HDFFV/hdf5 from ~DYOUNG/netbsd:develop to develop
* commit '0b721858e46a317c370a24115032d5be41688f67':
  Make these scripts relocatable again: derive a relative path for the original installation prefix from the examples prefix.  Use that relative path to locate the current installation prefix, always.  Fall back to an absolute installation prefix if the relative path cannot be derived.
  Get the path to prefix right: needs a ../ to back out of subdirectory c/.
  Make this script relocatable again: derive a relative path for the original installation prefix from the examples prefix.  Use that relative path to locate the current installation prefix, always.  Fall back to an absolute installation prefix if the relative path cannot be derived.
  Let us override the examples directory using --with-examplesdir=DIR. This is handy for NetBSD where HDF5 examples are installed by convention in $prefix/share/examples/hdf5/ rather than in ${prefix}/share/hdf5_examples/, which is the HDF5 default.
  Follow longstanding execv convention for compatibility with NetBSD.
  Under the examples directories, always find the installed HDF5 executables and scripts using @prefix@ instead of a relative path, because the number of ../ in the relative path will be different on NetBSD than on other systems.
  Make the HDF5 configure script grok NetBSD.
  For portability, insulate the HDF5 library from some system macros.
  Not every system has perl installed in /usr/bin/, so change the shebang (#!) line to `/usr/bin/env perl` to locate perl on the PATH.
  For portability, use the POSIX sh(1) string-comparison operator `=` instead of `==`.
2019-11-05 15:36:09 -06:00
Jerome Soumagne
ae490016b9 Add new H5R API that abstracts object, region and attribute reference types
Also support references to external files

Add new H5T_REF type and type conversion routines

Support conversion from H5T_REF_OBJ/DSET_REG to H5T_REF

Add H5Treclaim() API to reclaim memory of vlen/reference types

Deprecate H5Dvlen_reclaim()

Fix H5T_vlen_reclaim() and H5T_reclaim() to use private callback

Add H5T_ref_reclaim()

Move previous H5R APIs to H5Rdeprec.c

Clean up H5Ocopy

Separate H5O_copy_expand_ref() to H5Ocopy_ref()

Add support for copying new reference types

Clean up deprecated routines to go through VOL and same code path

Fix return codes in existing trefer.c test

Rename trefer.c to trefer_deprec.c

trefer.c is for new references

Add performance test for trefer

Add additional obj_copy_ref test

Make use of tokens and blobs to store references

Skip blob encoding for object references

Start adding new reference examples
2019-10-08 14:30:24 -05:00
Jerome Soumagne
fa6fdde1bf Fix H5VL_blob_get to return size of blob
Fix const in blob API

Add H5HG_HEAP_ID_SIZE macro to return native blob size

Add maximum size for blobs

Fix blob API callbacks to pass VOL file object

Add public wrappers for blob VOL API

Implement passthrough blob callbacks

Update H5Tvlen after callback changes

Update trace information for H5VL blob routines

Fix public header inclusion in native and passthru headers
2019-10-08 14:30:24 -05:00
David Young
7c82abc3ff Not every system has perl installed in /usr/bin/, so change the shebang
(#!) line to `/usr/bin/env perl` to locate perl on the PATH.

Everything after the first pathname in the shebang line is treated as
a single argument to the command interpreter (/usr/bin/env "perl -w"),
and there is not ordinarily any such program as "perl -w".  So if the
old shebang line used an option such as `-w`, add a `use warnings;`
statement to the script---note that the semantics change slightly.
`bin/destdep` uses a trick to pass `-p` to `/usr/bin/env perl`.  It
couldn't hurt to use the same trick to pass `-w`.

With these changes, `sh autogen.sh` runs on NetBSD.  It ought to still
work on every other system HDF5 supports, too.
2019-10-03 16:01:40 -05:00
Dana Robinson
5b6bd7c922 Added the map (H5M) API 2019-08-13 01:36:42 -07:00
Larry Knox
8008294578 Squashed commit of the following:
Merge changes from update_merged_S3_HDFS branch into develop.

commit d5034315aea88629929ac0c9c59ebfafd5f21a31
Merge: 9c48823 d3fdcd8
Author: Larry Knox <lrknox@hdfgroup.org>
Date:   Thu Jul 25 08:24:53 2019 -0500

    Merge branch 'develop' into update_merged_S3_HDFS
2019-07-25 11:47:12 -05:00
Quincey Koziol
2db4b6df13 Update API tracing info for VOL functions and regenerate trace macros for VOL
API calls.
2018-11-28 10:22:29 -06:00
Quincey Koziol
ca1b9025fe Switch switch remainder of API routines to use VOL callbacks. 2018-11-10 11:54:20 -06:00
Quincey Koziol
50dd616eb5 Correct error with parsing of "FUNC_ENTER_API_NOINIT" 2018-10-27 17:38:32 -05:00
Dana Robinson
e962df1591 VOL FEATURE 2018-10-10 08:10:15 -07:00
Dana Robinson
d4234d0a98 Major rework of H5PL package code before bringing VOL changes
over.

Brings coding standards in line with the rest of the library,
enforces better software engineering principles, and makes
everything more maintainable.
2017-07-14 09:18:33 -07:00
Larry Knox
89fbe00dec Merge pull request #426 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10 to hdf5_1_10
* commit '54957d37f5aa73912763dbb6e308555e863c43f4':
  Commit copyright header change for src/H5PLpkg.c which was added after running script to make changes.
  Add new files in release_docs to MANIFEST. Cimmit changes to Makefile.in(s) and H5PL.c that resulted from running autogen.sh.
  Merge pull request #407 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10_1 to hdf5_1_10_1
  Change copyright headers to replace url referring to file to be removed and replace it with new url for COPYING file.
2017-04-25 16:05:36 -05:00
Dana Robinson
b1a5ba4053 Updated bin/trace to emit errors and reordered autogen.sh processing
so that script failures leave the library in a non-compilable state,
making it harder to ignore errors.
2017-04-25 02:09:55 -04:00
Quincey Koziol
0313cbb91d Final merge of page buffering branch to develop 2017-03-13 21:30:37 -07:00
Dana Robinson
422fc7cc9c Whitespace changes in bin/trace. 2017-02-07 14:26:56 -05:00
Quincey Koziol
a28b65bb9e Clean up whitespace 2017-02-07 09:18:57 -08:00
Quincey Koziol
1949d5756d Cache image feature and testing 2017-01-28 23:19:37 -08:00
Dana Robinson
fe938a0d71 [svn-r28565] Normalization of bin/trace with revise_chunks in preparation
for merge.

Tested on: 64-bit Ubuntu 15.10 (Linux 4.2.0 x86_64)
           bin/trace run only
2015-12-10 20:09:28 -05:00
Dana Robinson
aa4e3e3985 [svn-r27191] Merged r26781 to r27185 from trunk
Tested on: 64-bit Ubuntu Linux VM w/ gcc 4.8.2
    (C++ and Fortran 2003 also tested)
2015-06-11 14:18:52 -05:00
Mohamad Chaarawi
4a2bee5cd8 [svn-r27158] missed rename if UNUSED to H5_ATTR_UNUSED in trace script. 2015-06-07 18:03:42 -05:00
Dana Robinson
8f74cc85d8 [svn-r27152] Updated bin/trace and src/H5trace.c to handle the H5D_vds_view_t
enum type.

Tested on: jam
2015-06-04 15:47:04 -05:00
Dana Robinson
4c2749fb1d [svn-r26324] Fixes another bug in bin/trace that prevents adding TRACE macros to API
calls that use a FUNC_ENTER macro that does not include parentheses.

This does not affect any source code at this time.

Part of: HDFFV-9141

Tested on: jam (bin/trace behavior only)
2015-02-26 19:49:32 -05:00
Dana Robinson
54a36600c0 [svn-r26249] Removed warnings about NO TRACE from bin/trace script.
Fixes: HDFFV-9098

Tested on: jam (bin/trace only - causes no source changes)
2015-02-20 09:48:18 -05:00
Dana Robinson
5867b6db9e [svn-r26246] Corrects a regular expression in the bin/trace Perl script that did not
match API calls which return a pointer.

Fixes: HDFFV-9141

Tested on: h5committest
2015-02-20 08:04:19 -05:00
Quincey Koziol
61a411b02f [svn-r25450] Description:
Initial tweaks to align with VOL branch.

Tested on:
    Mac OSX/64 10.9.3 (amazon) w/gcc 4.9
    (h5committest not required, too small)
2014-07-21 11:05:04 -05:00
Neil Fortner
0e3517d34f [svn-r23230] Purpose: Implement H5Dscatter and H5Dgather
Description:
Adds 2 new API functions, H5Dscatter and H5Dgather.  H5Dscatter retrieves data
from a specified callback function and scatters it into a selection, defined by
a supplied dataspace, within a supplied memory buffer.  H5Dgather gathers data
from a selection within a supplied memory buffer and passes it in a contiguous
form to a supplied callback function.  Added tests for these functions

Tested: jam, ostrich, koala (h5committest); ummon
2013-02-05 12:07:02 -05:00
Quincey Koziol
6a89177036 [svn-r22807] Description:
Add encode/decode API routines for property lists: H5Pencode/H5Pdecode.

Tested on:
        FreeBSD/32 8.2 (loyalty) w/gcc4.6, w/C++ & FORTRAN, in debug mode
        FreeBSD/64 8.2 (freedom) w/gcc4.6, w/C++ & FORTRAN, in debug mode
        Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
                w/C++ & FORTRAN, w/threadsafe, in debug mode
        Linux/64-amd64 2.6 (koala) w/Intel compilers, w/default API=1.6.x,
                w/C++ & FORTRAN, in production mode
        Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
                w/szip filter, w/threadsafe, in production mode
        Linux/PPC 2.6 (ostrich) w/C++ & FORTRAN, w/threadsafe, in debug mode
        Linux/64-ia64 2.6 (ember) w/Intel compilers, w/parallel, C++ & FORTRAN,
                in production mode
        Mac OS X/32 10.7.3 (amazon) in debug mode
        Mac OS X/32 10.7.3 (amazon) w/C++ & FORTRAN, w/threadsafe,
                in production mode
        Mac OS X/32 10.7.3 (amazon) w/parallel, in debug mode
2012-09-25 10:47:25 -05:00
Jonathan Kim
0aa8d4d88a [svn-r22735] Purpose:
HDFFV-8143 Provide a routine(s) for telling the user why the library broke collective data access

Description:
    Added H5Pget_mpio_no_collective_cause() function that retrive reasons why the collective I/O was broken during Read/Write IO access.
    Reasons to break collective I/O:
     - SET_INDEPENDENT
     - DATATYPE_CONVERSION
     - DATA_TRANSFORMS
     - MPIPOSIX
     - NOT_SIMPLE_OR_SCALAR_DATASPACES  (NULL Space)
     - POINT_SELECTIONS
     - NOT_CONTIGUOUS_OR_CHUNKED_DATASET  (Compact or External-Storage)
     - FILTERS  

Tested: 
    jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
2012-09-04 12:00:38 -05:00
Quincey Koziol
982f1ff8ff [svn-r22229] Description:
Add 'H5O_mcdt_search_cb_t' to bin/trace script and re-run the
bin/reconfigure script.

Tested on:
        FreeBSD/32 8.2 (loyalty) w/gcc4.6, w/C++ & FORTRAN, in debug mode
        FreeBSD/64 8.2 (freedom) w/gcc4.6, w/C++ & FORTRAN, in debug mode
        Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
                w/C++ & FORTRAN, w/threadsafe, in debug mode
        Linux/64-amd64 2.6 (koala) w/Intel compilers, w/default API=1.6.x,
                w/C++ & FORTRAN, in production mode
2012-04-01 17:32:04 -05:00
Quincey Koziol
c2a9dcb897 [svn-r22219] Description:
Merge "file image" changes from feature branch back to trunk.

Tested on:
    Mac OSX/64 10.7.3 (amazon) w/debug
    (h5committest upcoming)
2012-03-31 03:49:35 -05:00
Quincey Koziol
5f8a03a2bf [svn-r21529] Description:
Update tracing information with new types introduced in "actual I/O mode"
query routines.

Tested on:
    bin/reconfigure on jam
2011-10-12 14:16:32 -05:00
Dana Robinson
dc09f2117c [svn-r21302] Fixed the bin/trace Perl script so that it ignores a final ... parameter, which it can't process. This allows the correct H5TRACE8 trace function to be added to H5Epush2. Tested on jam (simple change). 2011-08-23 14:17:35 -05:00
Quincey Koziol
389c88e87c [svn-r20652] Description:
Teach the tracing script & routines about the 'unsigned long' and
'unsigned long long' types, so that the H5Pset_fapl_log() API routine gets
tracing information correct.

Tested on:
        FreeBSD/32 6.3 (duty) in debug mode
        FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
        FreeBSD/32 8.2 (loyalty) w/gcc4.6, w/C++ & FORTRAN, in debug mode
        FreeBSD/64 8.2 (freedom) w/gcc4.6, w/C++ & FORTRAN, in debug mode
        Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
                w/C++ & FORTRAN, w/threadsafe, in debug mode
        Linux/64-amd64 2.6 (koala) w/Intel compilers, w/default API=1.6.x,
                w/C++ & FORTRAN, in production mode
        Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
                w/szip filter, w/threadsafe, in production mode
        Linux/PPC 2.6 (heiwa) w/C++ & FORTRAN, w/threadsafe, in debug mode
        Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
                in production mode
        Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in debug mode
        Mac OS X/32 10.6.6 (amazon) in debug mode
        Mac OS X/32 10.6.6 (amazon) w/C++ & FORTRAN, w/threadsafe,
                in production mode
        Mac OS X/32 10.6.6 (amazon) w/parallel, in debug mode
2011-04-26 16:42:45 -05:00
Quincey Koziol
37ec6dc75e [svn-r17582] Description:
Bring changes from file free space branch back to the trunk.  *yay!*

Tested on:
        FreeBSD/32 6.3 (duty) in debug mode
        FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
        Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
                w/C++ & FORTRAN, w/threadsafe, in debug mode
        Linux/64-amd64 2.6 (smirom) w/Intel compilers, w/default API=1.6.x,
                w/C++ & FORTRAN, in production mode
        Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
                w/szip filter, in production mode
        Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
                in production mode
        Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in debug mode
        Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
        Mac OS X/32 10.5.8 (amazon) in debug mode
        Mac OS X/32 10.5.8 (amazon) w/C++ & FORTRAN, w/threadsafe,
                in production mode
2009-10-01 21:08:59 -05:00
Quincey Koziol
a365f0e6aa [svn-r16306] Description:
Bring r16305 back from revise_chunks branch:

	Add detection of C99 "designated initializers" to configure script and
use new H5_HAVE_C99_DESIGNATED_INITIALIZER macro to conditionally compile
default layout variables in src/H5Pdcpl.c

	Also, minor code cleanups, etc.

Tested on:
	FreeBSD/32 6.3 (duty) in debug mode
	(Other platforms tested on branch)
2009-01-14 08:09:42 -05:00
Quincey Koziol
7fc10c0ea7 [svn-r14413] Description:
Change H5P[gs]et_format_bounds() => H5P[gs]et_libver_bounds() and also
enumerated values H5F_FORMAT_{EARLIEST, LATEST} => H5F_LIBVER_{EARLIEST, LATEST}

Tested on:
        FreeBSD/32 6.2 (duty) in debug mode
        FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
        Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
                                in debug mode
        Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
                                in production mode
        Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
                                in production mode
        Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
                                w/szip filter, in production mode
        Mac OS X/32 10.4.10 (amazon) in debug mode
        Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
2008-01-14 18:49:12 -05:00
Quincey Koziol
c0b966ba1b [svn-r14374] Description:
Updates from running bin/reconfigure on recent changes, in preparation for
making private snapshot.  Also updated tracing information for recent types
added to public API routines.

Tested on:
    Linux/32 2.6.x (kagiso)
2008-01-04 12:43:56 -05:00
Quincey Koziol
cf56cd041d [svn-r14211] Description:
Remove --enable-hdfv1_6 configure flag, now that all dependencies on it
have been removed and new --with-default-api-version=[v16|v18] and
--disable-deprecated-symbols flags have taken its place.

	Update types in bin/trace so that bin/reconfigure runs cleanly.

	Run bin/reconfigure to regenerate configure script, along with
other associated files, etc.

Tested on:
        FreeBSD/32 6.2 (duty) in debug mode
        FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
        Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
                                in debug mode
        Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
                                in production mode
        Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
                                in production mode
        Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
                                w/szip filter, in production mode
        Mac OS X/32 10.4.10 (amazon) in debug mode
2007-10-18 13:57:08 -05:00
Quincey Koziol
4a5b97b930 [svn-r14022] Description:
Remove stray reference to "group revision" configure flag

    Remove Windows-specific EOL characters from src/H5FDwindows.c

    Add some missing types to bin/trace

    Re-run bin/reconfigure script to regenerate information after changes above.

Tested on:
    FreeBSD/32 6.2 (duty)
    FreeBSD/64 6.2 (liberty)
    Linux/32 2.6 (linux)
    Solaris/32 2.10 (linew)
2007-07-26 16:33:10 -05:00
Leon Arber
2618906fee [svn-r13438] Purpose: Bug Fix
Description:

Fixed some mistakes introduced yesterday due to a misunderstanding of what sort of datatypes the trace scripts prints out.  This commit mostly reverts those changes, adds some new datatypes, and fixes some errors with regards to how the trace script handles pointer datatypes.  Since the trace script scans the argument list for the "*" character and handles it appropriately, one should never need to explicitly specify a pointer type in the TypeString hash.
2007-02-28 20:00:16 -05:00
Leon Arber
1b2c200b10 [svn-r13429] Purpose: Bug Fix
Description:

Fix up the bin/trace script.  The script has been updated to include all of the new datatypes that have been added to the library.  It's output has also been made a bit clearer with regards to the NO TRACE comment scattered throughout the code which prevents tracing information from being added.

The script cannot handle functions that take a variable number of arguments nor can it deal with functions whose prototype differs based on certain preprocessor directives.  However, there are only 3 instances of this in the library.  All other warnings that the script used to generated have been fixed.

The script has not actually been run on the source files yet.

Tested:

smirom
2007-02-27 20:57:03 -05:00
Albert Cheng
5e1bee4f60 [svn-r13311] Updated copyright notice.
Test:
Just visual inspection as these are all comments changes.
2007-02-14 17:25:02 -05:00