Code cleanup (sorta)
Description:
Adjust H5G_stat_t to prepare for eventual "external link" information.
This also puts the information for regular objects and soft links into more
obviously separate places.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Purpose:
Bug fix
Description:
Modification time test (mtime) would die silently on some systems. This is
because the code is very system-dependant (it relies on getting the current
time and the timezone from the OS).
Solution:
mtime test now uses TEST_ERROR macro to print "FAILED" and to output where the
failure occurred. Configure script is a little smarter about whether
gettimeofday() function returns the timezone correctly.
Further bugs will need to be addressed on a system-by-system basis.
Platforms tested:
sleipnir, arabica, verbena, copper, windows (VC7)
Driver error
Description:
The tmtimeo.h5 file was supposed to have been generated from gen_old_mtime.c
which was compiled with a pre-1.5.x version of the library, however it appears
to have been created with a newer version of the library and did not contain
the older form of the mtime object header information that was being tested for.
Solution:
Run a version of gen_old_mtime that was compiled & linked against an older
version of the library to generate the tmtimeo.h5 file.
Had to update the modification time used for testing in the mtime.c file
also.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Update
Description:
Updated the Copyright statement
Platforms tested:
Linux (This change is only in the comments, so I just check that the
modules still compile)
Misc. update:
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
Update
Description:
Changed includes of the form:
#include <hdf5_file.h>
to
#include "hdf5_file.h"
so that gcc can pick them up easier without including the system
header files since we don't care about them.
Platforms tested:
Linux
Clean up warnings
Description:
The "FAILED" macro is defined by Windows and is causing warnings and
potential errors when compiled on that platform.
Solution:
Change our macro from FAILED to H5_FAILED.
Platforms tested:
FreeBSD 4.2 (hawkwind)
The "FILENAME" declared extern in h5test.h is not always used.
It was used in h5_cleanup to remove temporary files created
during tests. Not all tests codes have used this routine.
Indeed, quite a few of test programs do "#define FILENAME ".
Also, h5_cleanup needs to work in tandem with h5_fixname.
h5_fixname accepts an explicite base_name argument instead
of using the global variable FILENAME. That is cleaner.
Solution:
Added char *base_name[] as a new argument to h5_cleanup, in
the same style as h5_fixname. Removed "extern char *FILENAME..."
from use. Also, undo some unnecessary declaration of "char *FILENAME"
from some tests which don't use it at all (yet).
Platforms tested:
modi4-64(irix64), arabica(solari2.7), eirene(linux)
(arabica could not launch tests automatically. I had to hack
in LD_LIBRARY_PATH to make them run.)
----------------------
./config/irix64
The old (-32) compiler is now supported by setting envrionment
CC='cc -32'. The -64 compiler is the default or you can set
CC='cc -64'.
./src/H5A.c
./src/H5D.c
./src/H5F.c
./src/H5Fistore.c
./src/H5Flow.c
./src/H5G.c
./src/H5I.c
./src/H5Ocomp.c
./src/H5P.c
./src/H5R.c
./src/H5RA.c
./src/H5T.c
./src/H5Tbit.c
./src/H5Tconv.c
./src/H5Z.c
./src/H5detect.c
./test/big.c
./test/cmpd_dset.c
./test/dsets.c
./test/dtypes.c
./test/enum.c
./test/mtime.c
./test/ohdr.c
./tools/h5ls.c
Fixed lots of warnings on Irix64. Mailed a few remaining
warnings in H5S to Quincey and a few in the dumper to
Ruey-Hsia.
----------------------
./doc/html/Datatypes.html
./src/H5.c
./src/H5T.c
./src/H5Tconv.c
./src/H5Tpkg.h
./src/H5Tpublic.h
./src/H5detect.c
./test/chunk.c
./test/dtypes.c
./test/h5test.c
./test/mtime.c
./test/tattr.c
./test/th5s.c
./tools/h5dump.c
./tools/h5dumputil.c
./tools/h5import.c
./tools/h5ls.c
./tools/h5toh4.c
./tools/h5tools.c
Renamed the old H5T_NATIVE_CHAR type to H5T_NATIVE_SCHAR to
denote that it is always signed.
Added a new H5T_NATIVE_CHAR type which has the same range,
representation, and behavior as either H5T_NATIVE_SCHAR or
H5T_NATIVE_UCHAR depending on the compiler and its
command-line switches for the application source file which
references H5T_NATIVE_CHAR. If source files are compiled with
different switches then each source file will resolve
H5T_NATIVE_CHAR appropriately so it matches the C type `char'
in that source file.
NOTE: THERE ARE OTHER DOCUMENTATION FILES THAT I DIDN'T CHANGE
BECAUSE I CAN'T MODIFY THE SOURCE.
./test/extend.c
Swapped two lines to prevent diagnostic messages from messing
up the formatted output.
----------------------
./MANIFEST
./test/Makefile.in
./test/shtype.c [REMOVED]
Removed shtype.c because it was all commented out. Besides,
these tests are done in dtypes.c now anyway.
./test/external.c
./test/fillval.c
./test/flush1.c
./test/flush2.c
./test/links.c
./test/mount.c
./test/mtime.c
./test/unlink.c
The tests that check the HDF5 API use the h5test support
functions. For one thing, that means that you can specify the
file driver that thay use by the HDF5_DRIVER environment
variable. Possible values are:
HDF5_DRIVER='sec2' Use read() and write()
HDF5_DRIVER='stdio' Use fread() and fwrite()
HDF5_DRIVER='core' Use malloc() and free()
HDF5_DRIVER='split' Split meta and raw data
HDF5_DRIVER='family N' Use file families with each
member being N megabytes (N
can be fractional, defaults to
one).
Some tests might fail for certain drivers: for instance, the
mount and link tests fail for the `core' driver because
they must be able to close and then reopen a file.
----------------------
./test/flush2.c
./test/overhead.c
Removed carriage-returns inserted by a broken operating system.
./test/big.c
./test/mtime.c
./test/ragged.c
./tools/h5ls.c
Removed inclusion of <H5config.h>, system header files, and definition
of __unused__ since this all happens in <H5private.h>.
./test/chunk.c
./test/cmpd_dset.c
./test/dsets.c
./test/dtypes.c
./test/extend.c
./test/external.c
./test/fillval.c
./test/flush1.c
./test/flush2.c
./test/iopipe.c
./test/links.c
./test/mount.c
./test/overhead.c
./test/shtype.c
./test/unlink.c
./tools/h5import.c
./tools/h5repart.c
Removed inclusion of <H5config.h> since <hdf5.h> includes it.
./test/flush1.c
Includes <stdlib.h>, protects inclusion of <unistd.h> by using
HAVE_UNISTD_H instead of STDC_HEADERS.
----------------------
./MANIFEST
Added new Pablo files HDF5record_RT.h and ProcIDs.h
./acconfig.h
./configure [REGENERATED]
./configure.in
./src/H5.c
./src/H5Vprivate.h
./src/H5config.h.in [REGENERATED]
./src/H5private.h
./src/H5public.h
./test/big.c
Added more configuration stuff for the Win32 environment. Removed all
the #ifdef WIN32 from the source and replaced them with OS-independent
stuff. Specifics follow:
Check for non-Posix.1 `st_blocks' field in `struct stat' which is used
by the big file test to decide if the file system supports holes. If
the st_blocks field isn't present then we just skip the test.
Configure checks for <io.h> <sys/resource.h> <sys/time.h> and
<winsock.h> and defines HAVE_IO_H, HAVE_SYS_RESOURCE_H,
HAVE_SYS_TIME_H and HAVE_WINSOCK_H when they're found.
Configure checks whether both <sys/time.h> and <time.h> can be
included and defines SYS_TIME_WITH_TIME if so. Otherwise include only
<sys/time.h> or <time.h> even if both exist.
Configure checks sizeof(__int64) and defines SIZEOF___INT64 to the
result or to zero if __int64 isn't defined. The source uses `long
long' in preference to `__int64'.
Removed null WIN32 definition for `inline' since such a definition
already exists in H5config.h
Protected gettimeofday() calls in debugging code with
HAVE_GETTIMEOFDAY instead of WIN32.
./src/H5F.c
./src/H5Flow.c
./src/H5Fmpio.c
./src/H5Fsec2.c
./src/H5Fstdio.h
./src/H5P.c
./src/H5Tconv.c
./src/H5private.h
Removed #include of system files from library source files and
consolodated them into H5private.h where they're protected by various
configuration macros (most of them were duplicated there already
anyway).
./test/big.c
./test/chunk.c
./test/cmpd_dset.c
./test/dsets.c
./test/dtypes.c
./test/extend.c
./test/external.c
./test/fillval.c
./test/flush1.c
./test/flush2.c
./test/iopipe.c
./test/links.c
./test/mount.c
./test/mtime.c
./test/overhead.c
./test/ragged.c
./test/shtype.c
./test/unlink.c
Protected system #include's with #ifdef's from H5config.h.
Undefined NDEBUG since some of the tests rely on assert() to check
return values.
Removed WIN32 definitions for __unused__ since this can be controlled
by the definition of HAVE_ATTRIBUTE in H5config.h
./test/testhdf5.h
Removed the CLEAN_CMD definition because we no longer use it.
Albert's cleanup() functions replaced it.
./test/fillval.c
Initialized auto hid_t variables to fix warnings in error recovery
code when data flow analysis is turned on in compilers.
./test/h5tools.c
Initialized an auto variable to fix a compiler warning.
./test/chunk.c
./test/ragged.c
The WIN32 had some unsigned variables changed to signed because the
compiler generates warnings when coercing unsigned to double(?). I
changed them back to unsigned because they really are unsigned
quantities. If this the change was just to shut up extraneous warnings
then perhaps a compiler flag can do the same; otherwise if the
compiler generates bad code then we should supply a patch file instead
messing up source code with bug work-arounds.
./src/H5detect.c
Protected system #include's with #ifdef's from H5config.h thereby
removing a WIN32.
If getpwuid() doesn't exist (HAVE_GETPWUID) then we assume that
`struct passwd' doesn't exist either (we don't really need it in that
case).
The H5T_NATIVE_LLONG and H5T_NATIVE_ULLONG are defined in terms of
`long long' or else `__int64' or else `long' depending on what's
available.
./src/H5Flow.c
./src/H5Ofill.c
Added __unused__ to some function arguments that aren't used when
assertions are turned off.
./src/H5V.c
Changed an auto variable name in some hand-inlined code to get rid of
a warning about the variable shadowing a previous auto.
----------------------
./RELEASE
Updated with important changes I made since the second beta.
./src/H5A.c
./src/H5D.c
./src/H5Dprivate.h
./src/H5G.c
./src/H5Gprivate.h
./src/H5R.c
./src/H5Rprivate.h
./src/H5T.c
./src/H5Tprivate.h
Any API function that used to take an `hid_t loc_id' followed
by a `const char *name' can now take any type of object for
the loc_id as long as the object is somehow associated with a
file. Internally, H5G_loc() was modified to return an
H5G_entry_t* instead of an H5G_t* so it's more general.
Among other things, this allows one to retrieve information
about an object like a named type or dataset without knowing
the name of the type or dataset:
int
get_nlinks (hid_t obj)
{
H5G_stat_t sb;
if (H5Gstat(obj, ".", TRUE, &sb)<0) return -1;
return sb.nlink;
}
./test/gheap.c
./test/istore.c
These files needed a couple of changes because they call some
of the internal functions whose H5G_t arguments changed to
H5G_entry_t.
./src/H5A.c
Got rid of all the switch statements for getting symbol table
entries for varous objects and replaced them with a call to
H5G_loc() allowing attributes to automatically apply to any
type of object that belongs to a file.
./test/Makefile.in
Moved the ragged array tests from the normal list of tests to
the `make timings' target.
./test/ragged.c
Added rewrite tests -- rewrite the rows of a dataset changing
the number of rows and the length of each row.
./test/mtime.c
Added a test that checks that H5Gstat() can be called with a
dataset as the first argument.
./src/H5S.c
Added #ifdef HAVE_PARALLEL around code to check for the
HDF5_MPI_OPT_TYPES environment variable because the global
variable that gets set is #ifdef'd.
./bin/release
bzip2 uses .bz2 as the file extension.
----------------------
./src/H5G.c
./src/H5Gpublic.h
./test/links.c
Fixed a bug in H5Gstat() that caused the wrong information to
be returned sometimes. Added check for named data types.
./test/extend.c
./test/links.c
./test/mtime.c
Added `all tests passed' messages.
----------------------
./bin/release
Added ./Makefile to the distribution again -- it got lost in
the changes last week although it isn't all that important a
file since it gets clobbered by configure anyway.
./bin/trace
./doc/html/Filters.html
./doc/html/H5.format.html
./doc/html/H5.user.html
./src/H5.c
./src/H5D.c
./src/H5Dprivate.h
./src/H5E.c
./src/H5Epublic.h
./src/H5Farray.c
./src/H5Fistore.c
./src/H5Fprivate.h
./src/H5O.c
./src/H5Ocomp.c
./src/H5Oprivate.h
./src/H5P.c
./src/H5Ppublic.h
./src/H5Sall.c
./src/H5Shyper.c
./src/H5Spoint.c
./src/H5Sprivate.h
./src/H5Ssimp.c
./src/H5Z.c
./src/H5Zprivate.h
./src/H5Zpublic.h
./src/hdf5.h
./test/dsets.c
./tools/h5ls.c
Added the data filter pipeline, a generalization of the
compression stuff which allows things like checksums,
encryption, compression, performance monitoring, etc. See
./doc/html/Filters.html for details -- it replaces the
Compression.html doc.
./src/H5T.c
Cleaned up debugging output.
./config/linux
Added checks for egcs and pgcc and changed optimization flags
for the compilers.
./src/H5G.c
./tools/h5dump.c
Fixed compiler warnings in these files and others.
./configure.in
./src/H5private.h
./test/mtime.c
Added a check for difftime() and defined HDdifftime() to do
something else on systems that don't have difftime().
----------------------
./MANIFEST
./test/Makefile.in
./test/mtime.c [NEW]
Test for the modification time message.
./src/H5AC.c
./src/H5Fistore.c
Turned off printing of statistics on H5close() if H5AC_DEBUG
isn't defined.
