openssl/Configurations/README-design.md

Design document for the unified scheme data

How are things connected?

The unified scheme takes all its data from the build.info files seen throughout the source tree. These files hold the minimum information needed to build end product files from diverse sources. See the section on build.info files below.

From the information in build.info files, Configure builds up an information database as a hash table called %unified_info, which is stored in configdata.pm, found at the top of the build tree (which may or may not be the same as the source tree).

Configurations/common.tmpl uses the data from %unified_info to generate the rules for building end product files as well as intermediary files with the help of a few functions found in the build-file templates. See the section on build-file templates further down for more information.

build.info files

As mentioned earlier, build.info files are meant to hold the minimum information needed to build output files, and therefore only (with a few possible exceptions [1]) have information about end products (such as scripts, library files and programs) and source files (such as C files, C header files, assembler files, etc). Intermediate files such as object files are rarely directly referred to in build.info files (and when they are, it's always with the file name extension .o), they are inferred by Configure. By the same rule of minimalism, end product file name extensions (such as .so, .a, .exe, etc) are never mentioned in build.info. Their file name extensions will be inferred by the build-file templates, adapted for the platform they are meant for (see sections on %unified_info and build-file templates further down).

The variables PROGRAMS, LIBS, MODULES and SCRIPTS are used to declare end products. There are variants for them with _NO_INST as suffix (PROGRAM_NO_INST etc) to specify end products that shouldn't get installed.

The variables SOURCE, DEPEND, INCLUDE and DEFINE are indexed by a produced file, and their values are the source used to produce that particular produced file, extra dependencies, include directories needed, or C macros to be defined.

All their values in all the build.info throughout the source tree are collected together and form a set of programs, libraries, modules and scripts to be produced, source files, dependencies, etc etc etc.

Let's have a pretend example, a very limited contraption of OpenSSL, composed of the program apps/openssl, the libraries libssl and libcrypto, an module engines/ossltest and their sources and dependencies.

# build.info
LIBS=libcrypto libssl
INCLUDE[libcrypto]=include
INCLUDE[libssl]=include
DEPEND[libssl]=libcrypto

This is the top directory build.info file, and it tells us that two libraries are to be built, the include directory include/ shall be used throughout when building anything that will end up in each library, and that the library libssl depend on the library libcrypto to function properly.

# apps/build.info
PROGRAMS=openssl
SOURCE[openssl]=openssl.c
INCLUDE[openssl]=.. ../include
DEPEND[openssl]=../libssl

This is the build.info file in apps/, one may notice that all file paths mentioned are relative to the directory the build.info file is located in. This one tells us that there's a program to be built called apps/openss (the file name extension will depend on the platform and is therefore not mentioned in the build.info file). It's built from one source file, apps/openssl.c, and building it requires the use of . and include/ include directories (both are declared from the point of view of the apps/ directory), and that the program depends on the library libssl to function properly.

# crypto/build.info
LIBS=../libcrypto
SOURCE[../libcrypto]=aes.c evp.c cversion.c
DEPEND[cversion.o]=buildinf.h

GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
DEPEND[buildinf.h]=../Makefile
DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm

This is the build.info file in crypto/, and it tells us a little more about what's needed to produce libcrypto. LIBS is used again to declare that libcrypto is to be produced. This declaration is really unnecessary as it's already mentioned in the top build.info file, but can make the info file easier to understand. This is to show that duplicate information isn't an issue.

This build.info file informs us that libcrypto is built from a few source files, crypto/aes.c, crypto/evp.c and crypto/cversion.c. It also shows us that building the object file inferred from crypto/cversion.c depends on crypto/buildinf.h. Finally, it also shows the possibility to declare how some files are generated using some script, in this case a perl script, and how such scripts can be declared to depend on other files, in this case a perl module.

Two things are worth an extra note:

DEPEND[cversion.o] mentions an object file. DEPEND indexes is the only location where it's valid to mention them

# ssl/build.info
LIBS=../libssl
SOURCE[../libssl]=tls.c

This is the build.info file in ssl/, and it tells us that the library libssl is built from the source file ssl/tls.c.

# engines/build.info
MODULES=dasync
SOURCE[dasync]=e_dasync.c
DEPEND[dasync]=../libcrypto
INCLUDE[dasync]=../include

MODULES_NO_INST=ossltest
SOURCE[ossltest]=e_ossltest.c
DEPEND[ossltest]=../libcrypto.a
INCLUDE[ossltest]=../include

This is the build.info file in engines/, telling us that two modules called engines/dasync and engines/ossltest shall be built, that dasync's source is engines/e_dasync.c and ossltest's source is engines/e_ossltest.c and that the include directory include/ may be used when building anything that will be part of these modules. Also, both modules depend on the library libcrypto to function properly. ossltest is explicitly linked with the static variant of the library libcrypto. Finally, only dasync is being installed, as ossltest is only for internal testing.

When Configure digests these build.info files, the accumulated information comes down to this:

LIBS=libcrypto libssl
SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
DEPEND[crypto/cversion.o]=crypto/buildinf.h
INCLUDE[libcrypto]=include
SOURCE[libssl]=ssl/tls.c
INCLUDE[libssl]=include
DEPEND[libssl]=libcrypto

PROGRAMS=apps/openssl
SOURCE[apps/openssl]=apps/openssl.c
INCLUDE[apps/openssl]=. include
DEPEND[apps/openssl]=libssl

MODULES=engines/dasync
SOURCE[engines/dasync]=engines/e_dasync.c
DEPEND[engines/dasync]=libcrypto
INCLUDE[engines/dasync]=include

MODULES_NO_INST=engines/ossltest
SOURCE[engines/ossltest]=engines/e_ossltest.c
DEPEND[engines/ossltest]=libcrypto.a
INCLUDE[engines/ossltest]=include

GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
DEPEND[crypto/buildinf.h]=Makefile
DEPEND[util/mkbuildinf.pl]=util/Foo.pm

A few notes worth mentioning:

LIBS may be used to declare routine libraries only.

PROGRAMS may be used to declare programs only.

MODULES may be used to declare modules only.

The indexes for SOURCE must only be end product files, such as libraries, programs or modules. The values of SOURCE variables must only be source files (possibly generated).

INCLUDE and DEPEND shows a relationship between different files (usually produced files) or between files and directories, such as a program depending on a library, or between an object file and some extra source file.

When Configure processes the build.info files, it will take it as truth without question, and will therefore perform very few checks. If the build tree is separate from the source tree, it will assume that all built files and up in the build directory and that all source files are to be found in the source tree, if they can be found there. Configure will assume that source files that can't be found in the source tree (such as crypto/bildinf.h in the example above) are generated and will be found in the build tree.

The %unified_info database

The information in all the build.info get digested by Configure and collected into the %unified_info database, divided into the following indexes:

depends   => a hash table containing 'file' => [ 'dependency' ... ]
             pairs.  These are directly inferred from the DEPEND
             variables in build.info files.

modules   => a list of modules.  These are directly inferred from
             the MODULES variable in build.info files.

generate  => a hash table containing 'file' => [ 'generator' ... ]
             pairs.  These are directly inferred from the GENERATE
             variables in build.info files.

includes  => a hash table containing 'file' => [ 'include' ... ]
             pairs.  These are directly inferred from the INCLUDE
             variables in build.info files.

install   => a hash table containing 'type' => [ 'file' ... ] pairs.
             The types are 'programs', 'libraries', 'modules' and
             'scripts', and the array of files list the files of
             that type that should be installed.

libraries => a list of libraries.  These are directly inferred from
             the LIBS variable in build.info files.

programs  => a list of programs.  These are directly inferred from
             the PROGRAMS variable in build.info files.

scripts   => a list of scripts.  There are directly inferred from
             the SCRIPTS variable in build.info files.

sources   => a hash table containing 'file' => [ 'sourcefile' ... ]
             pairs.  These are indirectly inferred from the SOURCE
             variables in build.info files.  Object files are
             mentioned in this hash table, with source files from
             SOURCE variables, and AS source files for programs and
             libraries.

shared_sources =>
             a hash table just like 'sources', but only as source
             files (object files) for building shared libraries.

As an example, here is how the build.info files example from the section above would be digested into a %unified_info table:

our %unified_info = (
    "depends" =>
        {
            "apps/openssl" =>
                [
                    "libssl",
                ],
            "crypto/buildinf.h" =>
                [
                    "Makefile",
                ],
            "crypto/cversion.o" =>
                [
                    "crypto/buildinf.h",
                ],
            "engines/dasync" =>
                [
                    "libcrypto",
                ],
            "engines/ossltest" =>
                [
                    "libcrypto.a",
                ],
            "libssl" =>
                [
                    "libcrypto",
                ],
            "util/mkbuildinf.pl" =>
                [
                    "util/Foo.pm",
                ],
        },
    "modules" =>
        [
            "engines/dasync",
            "engines/ossltest",
        ],
    "generate" =>
        {
            "crypto/buildinf.h" =>
                [
                    "util/mkbuildinf.pl",
                    "\"\$(CC)",
                    "\$(CFLAGS)\"",
                    "\"$(PLATFORM)\"",
                ],
        },
    "includes" =>
        {
            "apps/openssl" =>
                [
                    ".",
                    "include",
                ],
            "engines/ossltest" =>
                [
                    "include"
                ],
            "libcrypto" =>
                [
                    "include",
                ],
            "libssl" =>
                [
                    "include",
                ],
            "util/mkbuildinf.pl" =>
                [
                    "util",
                ],
        }
    "install" =>
        {
            "modules" =>
                [
                    "engines/dasync",
                ],
            "libraries" =>
                [
                    "libcrypto",
                    "libssl",
                ],
            "programs" =>
                [
                    "apps/openssl",
                ],
       },
    "libraries" =>
        [
            "libcrypto",
            "libssl",
        ],
    "programs" =>
        [
            "apps/openssl",
        ],
    "sources" =>
        {
            "apps/openssl" =>
                [
                    "apps/openssl.o",
                ],
            "apps/openssl.o" =>
                [
                    "apps/openssl.c",
                ],
            "crypto/aes.o" =>
                [
                    "crypto/aes.c",
                ],
            "crypto/cversion.o" =>
                [
                    "crypto/cversion.c",
                ],
            "crypto/evp.o" =>
                [
                    "crypto/evp.c",
                ],
            "engines/e_dasync.o" =>
                [
                    "engines/e_dasync.c",
                ],
            "engines/dasync" =>
                [
                    "engines/e_dasync.o",
                ],
            "engines/e_ossltest.o" =>
                [
                    "engines/e_ossltest.c",
                ],
            "engines/ossltest" =>
                [
                    "engines/e_ossltest.o",
                ],
            "libcrypto" =>
                [
                    "crypto/aes.c",
                    "crypto/cversion.c",
                    "crypto/evp.c",
                ],
            "libssl" =>
                [
                    "ssl/tls.c",
                ],
            "ssl/tls.o" =>
                [
                    "ssl/tls.c",
                ],
        },
);

As can be seen, everything in %unified_info is fairly simple suggest of information. Still, it tells us that to build all programs, we must build apps/openssl, and to build the latter, we will need to build all its sources (apps/openssl.o in this case) and all the other things it depends on (such as libssl). All those dependencies need to be built as well, using the same logic, so to build libssl, we need to build ssl/tls.o as well as libcrypto, and to build the latter...

Build-file templates

Build-file templates are essentially build-files (such as Makefile on Unix) with perl code fragments mixed in. Those perl code fragment will generate all the configuration dependent data, including all the rules needed to build end product files and intermediary files alike. At a minimum, there must be a perl code fragment that defines a set of functions that are used to generates specific build-file rules, to build static libraries from object files, to build shared libraries from static libraries, to programs from object files and libraries, etc.

generatesrc - function that produces build file lines to generate
              a source file from some input.

              It's called like this:

                    generatesrc(src => "PATH/TO/tobegenerated",
                                generator => [ "generatingfile", ... ]
                                generator_incs => [ "INCL/PATH", ... ]
                                generator_deps => [ "dep1", ... ]
                                incs => [ "INCL/PATH", ... ],
                                deps => [ "dep1", ... ],
                                intent => one of "libs", "dso", "bin" );

              'src' has the name of the file to be generated.
              'generator' is the command or part of command to
              generate the file, of which the first item is
              expected to be the file to generate from.
              generatesrc() is expected to analyse and figure out
              exactly how to apply that file and how to capture
              the result.  'generator_incs' and 'generator_deps'
              are include directories and files that the generator
              file itself depends on.  'incs' and 'deps' are
              include directories and files that are used if $(CC)
              is used as an intermediary step when generating the
              end product (the file indicated by 'src').  'intent'
              indicates what the generated file is going to be
              used for.

src2obj     - function that produces build file lines to build an
              object file from source files and associated data.

              It's called like this:

                    src2obj(obj => "PATH/TO/objectfile",
                            srcs => [ "PATH/TO/sourcefile", ... ],
                            deps => [ "dep1", ... ],
                            incs => [ "INCL/PATH", ... ]
                            intent => one of "lib", "dso", "bin" );

              'obj' has the intended object file with `.o`
              extension, src2obj() is expected to change it to
              something more suitable for the platform.
              'srcs' has the list of source files to build the
              object file, with the first item being the source
              file that directly corresponds to the object file.
              'deps' is a list of explicit dependencies.  'incs'
              is a list of include file directories.  Finally,
              'intent' indicates what this object file is going
              to be used for.

obj2lib     - function that produces build file lines to build a
              static library file ("libfoo.a" in Unix terms) from
              object files.

              called like this:

                    obj2lib(lib => "PATH/TO/libfile",
                            objs => [ "PATH/TO/objectfile", ... ]);

              'lib' has the intended library file name *without*
              extension, obj2lib is expected to add that.  'objs'
              has the list of object files to build this library.

libobj2shlib - backward compatibility function that's used the
              same way as obj2shlib (described next), and was
              expected to build the shared library from the
              corresponding static library when that was suitable.
              NOTE: building a shared library from a static
              library is now DEPRECATED, as they no longer share
              object files.  Attempting to do this will fail.

obj2shlib   - function that produces build file lines to build a
              shareable object library file ("libfoo.so" in Unix
              terms) from the corresponding object files.

              called like this:

                    obj2shlib(shlib => "PATH/TO/shlibfile",
                              lib => "PATH/TO/libfile",
                              objs => [ "PATH/TO/objectfile", ... ],
                              deps => [ "PATH/TO/otherlibfile", ... ]);

              'lib' has the base (static) library file name
              *without* extension.  This is useful in case
              supporting files are needed (such as import
              libraries on Windows).
              'shlib' has the corresponding shared library name
              *without* extension.  'deps' has the list of other
              libraries (also *without* extension) this library
              needs to be linked with.  'objs' has the list of
              object files to build this library.

obj2dso     - function that produces build file lines to build a
              dynamic shared object file from object files.

              called like this:

                    obj2dso(lib => "PATH/TO/libfile",
                            objs => [ "PATH/TO/objectfile", ... ],
                            deps => [ "PATH/TO/otherlibfile",
                            ... ]);

              This is almost the same as obj2shlib, but the
              intent is to build a shareable library that can be
              loaded in runtime (a "plugin"...).

obj2bin     - function that produces build file lines to build an
              executable file from object files.

              called like this:

                    obj2bin(bin => "PATH/TO/binfile",
                            objs => [ "PATH/TO/objectfile", ... ],
                            deps => [ "PATH/TO/libfile", ... ]);

              'bin' has the intended executable file name
              *without* extension, obj2bin is expected to add
              that.  'objs' has the list of object files to build
              this library.  'deps' has the list of library files
              (also *without* extension) that the programs needs
              to be linked with.

in2script   - function that produces build file lines to build a
              script file from some input.

              called like this:

                    in2script(script => "PATH/TO/scriptfile",
                              sources => [ "PATH/TO/infile", ... ]);

              'script' has the intended script file name.
              'sources' has the list of source files to build the
              resulting script from.

Along with the build-file templates is the driving template Configurations/common.tmpl, which looks through all the information in %unified_info and generates all the rulesets to build libraries, programs and all intermediate files, using the rule generating functions defined in the build-file template.

As an example with the smaller build.info set we've seen as an example, producing the rules to build libcrypto would result in the following calls:

# Note: obj2shlib will only be called if shared libraries are
# to be produced.
# Note 2: obj2shlib must convert the '.o' extension to whatever
# is suitable on the local platform.
obj2shlib(shlib => "libcrypto",
          objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ],
          deps => [  ]);

obj2lib(lib => "libcrypto"
        objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ]);

src2obj(obj => "crypto/aes.o"
        srcs => [ "crypto/aes.c" ],
        deps => [ ],
        incs => [ "include" ],
        intent => "lib");

src2obj(obj => "crypto/evp.o"
        srcs => [ "crypto/evp.c" ],
        deps => [ ],
        incs => [ "include" ],
        intent => "lib");

src2obj(obj => "crypto/cversion.o"
        srcs => [ "crypto/cversion.c" ],
        deps => [ "crypto/buildinf.h" ],
        incs => [ "include" ],
        intent => "lib");

generatesrc(src => "crypto/buildinf.h",
            generator => [ "util/mkbuildinf.pl", "\"$(CC)",
                           "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
            generator_incs => [ "util" ],
            generator_deps => [ "util/Foo.pm" ],
            incs => [ ],
            deps => [ ],
            intent => "lib");

The returned strings from all those calls are then concatenated together and written to the resulting build-file.