cache entries. Added test code to test the new feature. Also some
minor cleanum
h5committested -- passed on copper and sol. Failed on osage with
a configuration error that appears unrelated to my changes. Serial
test (debug mode) passes on Phoenix (Linux x86 2.6 kernel), so I
went ahead with the checkin.
Tested platform:
Kagiso only since it is only a comment block change. If it works in one
machine, it should work in all, I hope. Still need to check the parallel
build on copper.
handle flush callbacks which can dirty other entries, and
resize and/or rename the target entry.
This feature is needed by the fractal heap code.
Also added associated test code.
H5Commit tested. Test failed on heping, but the
error appears to be a syntax error in an un-related
file.
Tests on copper & sol passed, along with tests on
phoenix.
flag, and pin an entry as it is inserted.
The objective is to avoid some function call overhead in fheap.
Also added matching test code in test/cache.c & test/cache_common.c
(also testpar/t_cache.c checked in separately by accident)
h5commit tested
1) Added trace file support to the metadata cache. This allows capture
of all metadata cache calls in trace files for purposes of optimization
and debuging.
2) Added an expunge entry function. This allows an entry to be deleteded
from the cache without writing it to disk even if it is dirty.
3) Added a function call to resize pinned entries.
4) Added code to deal with entries that are dirty on load. This is
needed in support of a bug fix which can alter object headers on
load to repair files.
5) Added progress reporting code to the "MDC API smoke check" test in
cache_api.c. To enable the progress reporting, set report_progress
to TRUE in mdc_api_call_smoke_check().
Tested with h5committest, and a parallel test on phoenix (dual athelon
linux box).
Code cleanup
Description:
Trim trailing whitespace in Makefile.am and C/C++ source files to make
diffing changes easier.
Platforms tested:
None necessary, whitespace only change
1) Check in potential fix to unreproduceable bug in t_cache observed on
Cobalt
2) Check in code supporting added pinned entry features in the metadata
cache.
Description:
1) Elena encountered a bug in t_cache when running on Cobalt. On
investigation I found a potential race condition in t_cache that
could explain her observation.
2) Quincey requested additions to the metadata cache allowing renaming
of pinned entryies, and marking of pinned or protected entries as
dirty.
Solution:
1) Modified t_cache.c to prevent the potential race condition. Elena
was unable to reproduce the bug on Cobalt, so we don't know if my
fix actually addressed the problem.
2) Added code supporting the requested functions. The changes were
relatively trivial, but required substantial new test code.
Platforms tested:
h5committest, serial test on heping, parallel test on phoenix.
Misc. update:
Bug fix
Description:
Fix two test bugs:
1) Unitialized variable bug in test of H5C_get_entry_status()
(in test/cache.c)
2) Incorrect initialization of constant in t_cache when running
with MPE. (in testpar/t_cache.c)
Solution:
1) Don't test uninitialize variable.
2) Correct initialzation of constant in MPE case.
Platforms tested:
Tested on Heping and Copper.
Misc. update:
Code cleanup/feature twist
Description:
Adjust recent H5AC routines to take H5F_t pointers instead of H5C_t
pointers, to match the rest of the H5AC routines.
This change propagated into a few of the tests, which also had some
compiler warnings cleaned up...
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Linux 2.4/64 (mir) w/C++ & FORTRAN
Add pinned entry capability to cache.
Description:
For frequently accessed cache entries, the protect/unprotect overhead
is sometimes a bottleneck.
Solution:
Allow entries to be pinned in the cache. Pinned entries can't be
evicted, but can be flushed or modified.
Platforms tested:
h5committested -- minus one small typo in test/cache.c whose fix was
tested on copper and heping only.
Misc. update:
1) Add parallel test for metadata cache
2) Split serial test for metadata cache into two parts
3) Fix bug in which cache was flushed needlessly when the
cache wasn't full.
4) Performance improvements
5) Update API for parallel cache coherency bug fix.
Description:
See above.
Solution:
See above.
Platforms tested:
h5committest
Misc. update:
Code cleanup
Description:
Check in some of the code cleanups from working on the external link
support. (This doesn't include any of the external link features)
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OSX.4 (amazon)
Linux 2.4
Repair synchronization bug in the metadata cache in PHDF5
Also repair numerous other bugs that surfaced in testing the
bug fix.
Description:
While operations modifying metadata must be collective, we allow
independant reads. This allows metadata caches on different processes
to adjust to different sizes, and to place the entries on their dirty
lists in different orders. Since only process 0 actually writes
metadata to disk (all other processes thought they did, but the writes
were discarded on the theory that they had to be collective), this made
it possible for another process to modify metadata, flush it, and then
read it back in in its original form (pre-modification) form. The
possibilities for file corruption should be obvious.
Solution:
Make the policy that only process 0 can write to file explicit, and
visible to the metadata caches. Thus only process 0 may flush dirty
entries -- all other caches must retain dirty entries until they are
informed by process 0 that the entries are clean.
Synchronization is handled by counting the bytes of dirty cache entries
created, and then synching up between the caches whenever the sum
exceeds an (eventually user specified) limit. Dirty metadata creation
is consistent across all processes because all operations modifying
metadata must be collective.
This change uncovered may bugs which are repaired in this checkin.
It also required modification of H5HL and H5O to allocate file space
on insertion rather than on flush from cache.
Platforms tested:
H5committest, heping(parallel & serial)
Misc. update:
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Code cleanup
Description:
Refactor metadata cache to merge "dirtied" flag in with other flags for
H5AC_unprotect and H5C_unprotect.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
Interim checkin of code changes moving management of the is_dirty flag
into the cache code.
Description:
Prior to this checkin, management of the is_dirty flag was handled
above the level of the metadata cache. This can no longer be allowed,
as it introduces a race condition in the proposed fix for a cache
coherency bug in PHDF5.
Solution:
Move management fo the is_dirty flag to the cache code proper.
Entries are now marked as dirty via a flag on the unprotect call.
Platforms tested:
h5committested
Misc. update:
Fix a compile issue with the SX6 port.
Description:
In cache.c, invalid_configs array was declared as const, which caused
problems when elements of the array were passed to H5Pset_mdc_config()
Solution:
Remove the const modifier from the declaration of invalid_configs.
Platforms tested:
Heping (serial)
Misc. update:
Remove C99 types from new metadata cache related API calls
(take 2 -- missed one in the previous check in)
Description:
Windows (and perhaps others) don't like int32_t and int64_t. While
we have dealt with the issue internally, it is more of a problem
in API calls.
Solution:
Convert int32_t to int and int64_t to long int in the new metadata
cache related API calls.
Platforms tested:
heping
Misc. update:
Remove C99 types from new metadata cache related API calls
Description:
Windows (and perhaps others) don't like int32_t and int64_t. While
we have dealt with the issue internally, it is more of a problem
in API calls.
Solution:
Convert int32_t to int and int64_t to long int in the
H5AC_cache_config_t structure used by the new metadata cache
related API calls. Added explicit type casts to convert
between internal and external representations.
Platforms tested:
h5committested
Misc. update:
Add API calls allowing user control of the metadata cache.
Description:
Prior to this update, the metadata cache was not configurable
from outside the library.
Solution:
Add API calls allowing the user to configure the metadata cache
either at file open time, or for any open file. Also added calls
permitting the user to monitor cache size and hit rate. These
latter facilities are needed for "manual" cache size control
Platforms tested:
h5committested
Misc. update:
Reduce the run time of the cache tests.
Description:
Some of the cache tests run for quite a while, which has been
causing problems with our daily build and test. In my last
modification to cache.c, I added code to skip these stress
tests unless we were building in production mode. Since
things go faster in production mode, the extra tests are not
a major problem here.
However, this means that the stress tests are only run once
or twice a week.
Solution:
To try to deal with this, I modified the four longest tests
to throttle depending on whether NDEBUG is defined. When
NDEBUG is not defined, runtime for the cache tests should
be about 1/5th its regular run time. We will see if this
is enough of a reduction to avoid problems.
There is some doubt in my mind as to just how much good the
throttled tests do, but they should be better than nothing.
As mentioned before, the correct solution is to build some
proper, random test code.
Platforms tested:
h5committested
production and debug builds on heping.
Misc. update:
1) Provide facilities in cache to allow us to avoid a potential cache
consistency bug in the parallel case.
2) Clean up a off by one sanity checking bug.
3) Turn off execution of long running tests in debug mode.
Description:
1) In the parallel case, all writes to metadata must be collective,
but reads may not be. In pricipal, this allows us to different
contents in different caches. This isn't a problem as long as the
correct data is always on disk, but unless we can force certain
writes immediately, that need not be the case.
2) & 3) should need no further explanation.
Solution:
1) Add code allowing us to mark cache entries, and then force
these entries to be flushed at a later time.
Note that to actually avoid the bug, we will have to modify
existing code to use these new features.
2) & 3) should need no further explanation.
Platforms tested:
heping (serial debug and production)
committest (copper, sol, and heping). test failed on heping in the
c++ portion of the build, but at Quincey's siggestion, I am proceeding
with the checkin.
Misc. update:
Reduce run time of daily tests.
Description:
cache, the test program for the metadata cache has been taking a while
to execute.
Solution:
As a short term "fix", I have commented out all but one of the long
running test functions. Of course that means that we aren't running
these tests at present. I'm not sure that this is a good idea.
Platforms tested:
Serial on Heping.
Misc. update:
Code cleanup
Description:
Convert chunk iteration code to use skip lists instead of threaded, balanced
binary trees.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & szip
Too minor to require h5committest
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
Modify the cache code (H5C) to support automatic cache resizing to
adapt to the work load at run time.
Description:
Different applications require different sized caches to maintain
an acceptable hit rate. This set of changes attempts to provide the
ability to adjust to circumstances automatically.
Solution:
Added highly configurable code to allow the user to either set a
fixed cache size, or allow the cache to grow and shrink according to
conditions.
If enabled, cache size increases are triggered when the hit rate
drops below a user specified threshold in a user specified interval.
Cache size reductions (if enabled) are triggered when either the
hit rate exceeds some user specified threshold over a user specified
interval, when the cache contains "enough" entries that haven't been
accessed for a user specified interval, or some mix of the above.
See the header comments on the H5C_auto_size_ctl_t structure in
H5Cprivate.h for further details.
At present, the cache resize configuration options are not
accessible via the user API. Must add this.
Platforms tested:
h5committested, heping (serial), and copper (parallel)
Misc. update:
Code cleanup
Description:
Clean up a bunch of warnings and bring new code better inline with current
library coding practice.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Too minor to require h5committest
Misc. update:
Code cleanup
Description:
Fix problems when compiling with C++ compiler.
Also clean up some warnings with gcc 3.4.x
Platforms tested:
FreeBSD 4.10 (sleipnir)
Too minor to require h5committest
Description:
Replaced the old metadata cache with a cache with a modified LRU
replacement policy. This should improve the hit rate.
Solution:
Since we want to flush cache entries in increasing address order, I
used the threaded binary B-tree code to store the cache entries.
There is a fair bit of overhead here, so we may want to consider
other options.
While the code is designed to allow the support of other replacement
algorithms, at present, only a modified version of LRU is supported.
The modified LRU algorithm requires that a user selectable portion
of the cache entries be clean. The clean entries are evicted first
when writes are not permitted. If the pool of clean entries is used
up, the cache grows beyond its user specified maximum size. The
cache can also exceed its maximum size if the combined size of the
protected (or locked) entries exceeds the maximum size of the cache.
Platforms tested:
eirene (serial, parallel, fp), h5committested
Misc. update:
