binutils-gdb/gdb/testsuite/gdb.perf

The GDB Performance Testsuite
=============================

This README contains notes on hacking on GDB's performance testsuite.
For notes on GDB's regular testsuite or how to run the performance testsuite,
see ../README.

Generated tests
***************

The testcase generator lets us easily test GDB on large programs.
The "monster" tests are mocks of real programs where GDB's
performance has been a problem.  Often it is difficult to build
these monster programs, but when measuring performance one doesn't
need the "real" program, all one needs is something that looks like
the real program along the axis one is measuring; for example, the
number of CUs (compilation units).

Structure of generated tests
****************************

Generated tests consist of a binary and potentially any number of
shared libraries.  One of these shared libraries, called "tail", is
special.  It is used to provide mocks of system provided code, and
contains no generated code.  Typically system-provided libraries
are searched last which can have significant performance consequences,
so we provide a means to exercise that.

The binary and the generated shared libraries can have a mix of
manually written and generated code.  Manually written code is
specified with the {binary,gen_shlib}_extra_sources config parameters,
which are lists of source files in testsuite/gdb.perf.  Generated
files are controlled with various configuration knobs.

Once a large test program is built, it makes sense to use it as much
as possible (i.e., with multiple tests).  Therefore perf data collection
for generated tests is split into two passes: the first pass builds
all the generated tests, and the second pass runs all the performance
tests.  The first pass is called "build-perf" and the second pass is
called "check-perf".  See ../README for instructions on running the tests.

Generated test directory layout
*******************************

All output lives under testsuite/gdb.perf in the build directory.

Because some of the tests can get really large (and take potentially
minutes to compile), parallelism is built into their compilation.
Note however that we don't run the tests in parallel as it can skew
the results.

To keep things simple and stay consistent, we use the same
mechanism used by "make check-parallel".  There is one catch: we need
one .exp for each "worker" but the .exp file must come from the source
tree.  To avoid generating .exp files for each worker we invoke
lib/build-piece.exp for each worker with different arguments.
The file build.piece.exp lives in "lib" to prevent dejagnu from finding
it when it goes to look for .exp scripts to run.

Another catch is that each parallel build worker needs its own directory
so that their gdb.{log,sum} files don't collide.  On the other hand
its easier if their output (all the object files and shared libraries)
are in the same directory.

The above considerations yield the resulting layout:

$objdir/testsuite/gdb.perf/

	gdb.log, gdb.sum: result of doing final link and running tests

	workers/

		gdb.log, gdb.sum: result of gen-workers step

		$program_name/

			${program_name}-0.worker
			...
			${program_name}-N.worker: input to build-pieces step

	outputs/

		${program_name}/

			${program_name}-0/
			...
			${program_name}-N/

				gdb.log, gdb.sum: for each build-piece worker

			pieces/

				generated sources, object files, shlibs

			${run_name_1}: binary for test config #1
			...
			${run_name_N}: binary for test config #N

Generated test configuration knobs
**********************************

The monster program generator provides various knobs for building various
kinds of monster programs.  For a list of the knobs see function
GenPerfTest::init_testcase in testsuite/lib/perftest.exp.
Most knobs are self-explanatory.
Here is a description of the less obvious ones.

binary_extra_sources

	This is the list of non-machine generated sources that go
	into the test binary.  There must be at least one: the one
	with main.

class_specs

	List of pairs of keys and values.
	Supported keys are:
	count: number of classes
	  Default: 1
	name: list of namespaces and class name prefix
	  E.g., { ns0 ns1 foo } -> ns0::ns1::foo_<cu#>_{0,1,...}
	  There is no default, this value must be specified.
	nr_members: number of members
	  Default: 0
	nr_static_members: number of static members
	  Default: 0
	nr_methods: number of methods
	  Default: 0
	nr_inline_methods: number of inline methods
	  Default: 0
	nr_static_methods: number of static methods
	  Default: 0
	nr_static_inline_methods: number of static inline methods
	  Default: 0

	E.g.,
	class foo {};
	namespace ns1 { class bar {}; }
	would be represented as:
	{
	  { count 1 name { foo } }
	  { count 1 name { ns1 bar } }
	}

	The naming of each class is "class_<cu_nr>_<class_nr>",
	where <cu_nr> is the number of the compilation unit the
	class is defined in.

	There's currently no support for nesting classes in classes,
	or for specifying baseclasses or templates.

Misc. configuration knobs
*************************

These knobs control building or running of the test and are specified
like any global Tcl variable.

CAT_PROGRAM

	Default is /bin/cat, you shouldn't need to change this.

SHA1SUM_PROGRAM

	Default is /usr/bin/sha1sum.

PERF_TEST_COMPILE_PARALLELISM

	An integer, specifies the amount of parallelism in the builds.
	Akin to make's -j flag.  The default is 10.

Writing a generated test program
********************************

The best way to write a generated test program is to take an existing
one as boilerplate.  Two good examples are gmonster1.exp and gmonster2.exp.
gmonster1.exp builds a big binary with various custom manually written
code, and gmonster2 is (essentially) the equivalent binary split up over
several shared libraries.

Writing a performance test that uses a generated program
********************************************************

The best way to write a test is to take an existing one as boilerplate.
Good examples are gmonster1-*.exp and gmonster2-*.exp.

The naming used thus far is that "foo.exp" builds the test program
and there is one "foo-bar.exp" file for each performance test
that uses test program "foo".

In addition to writing the test driver .exp script, one must also
write a python script that is used to run the test.
This contents of this script is defined by the performance testsuite
harness.  It defines a class, which is a subclass of one of the
classes in gdb.perf/lib/perftest/perftest.py.
See gmonster-null-lookup.py for an example.

Note: Since gmonster1 and gmonster2 are treated as being variations of
the same program, each test shares the same python script.
E.g., gmonster1-null-lookup.exp and gmonster2-null-lookup.exp
both use gmonster-null-lookup.py.

Running performance tests for generated programs
************************************************

There are two steps: build and run.

Example:

bash$ make -j10 build-perf RUNTESTFLAGS="gmonster1.exp"
bash$ make -j10 check-perf RUNTESTFLAGS="gmonster1-null-lookup.exp" \
    GDB_PERFTEST_MODE=run