The memory footprint of how we produced the Makefile was quite...
important, because we have all the processing in one perl snippet, and
generate the details of the build file by appending to the "magic"
variable $OUT. The result is that this variable gets to hold the
majority of the build file text, and depending on memory reallocation
strategies for strings, the heap may hold multiple (possibly not just
a few) copies of this string, almost all of them "freed" but still
taking up space. This has resulted in memory exhaustion.
We therefore change strategy, and generate the build file in two
phases, where the first phase generates the full template using small
perl snippets for each detail, and the second phase processes this
template. This is much kinder to process memory.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15310)
quotify1() and quotify_l() were in OpenSSL::Template, but should be
more widely usable.
configdata.pm.in's out_item() is also more widely useful and is
therefore moved to OpenSSL::Util as well, and renamed to dump_data().
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15310)
Use it in the automated workflows.
Fixes: #15247
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15248)
Add also update-fips-checksums to update the checksums in the
$(SRCDIR) if the $(SRCDIR) and $(BLDDIR) is different.
The fips-checksums and generate_fips_sources targets are always
produced (regardless of enable-fips) as nothing else depends on them
and they are developer targets.
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15229)
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15167)
This adds the following scripts:
util/lang-compress.pl:
Compress source code, which language is determined by the first argument.
For the moment, we know 'perl' (perlasm source code), 'C' (C source code)
and 'S' (Assembler with C preprocessor directives).
This removes comments and empty lines, and compresses series of horizontal
spaces to one single space in the languages where that's appropriate.
util/fips-checksums.sh:
Takes source file names as arguments, pushes them through
util/lang-compress.pl and unifdef with FIPS_MODE defined, and calculates
the checksum on the result.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8871)
It might not be necessary with the most recent toolchain versions
but apparently many 32bit linux architectures and commonly used
toolchain versions require this.
It is also harmless to include even on architectures that do not
need it.
Fixes#14083
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15086)
Pull request #14320 introduced the ability to use compiled programs
as generators in GENERATE rules of build.info files. Those generator
calls were wrapped by the Perl wrapper (wrap.pl) in the Unix makefile
template, but not on Windows.
This commit adds the missing wrapper for Windows, because for the
`fipsmodule.cnf` target it is essential that the `openssl fipsinstall`
command does not load any preinstalled openssl configuration file.
Fixes#13680
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13684)
To follow the principle "what you configure is what you install",
the `make install` target now includes the installation of the
fips provider (`make install_fips`) if (and only if) OpenSSL was
configured with fips support (`enable-fips`).
The `make install_fips` target exists as well and can be used
to install just the fips provider. It requires `enable-fips`
and issues an error message if `no-fips` was configured.
The anologue holds for the 'uninstall_fips' target.
Fixes#13693
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13684)
There is already a `providers/fipsmodule.cnf` target which is required by
the tests. Instead of creating another fipsmodule.cnf, the `install_fips`
target simply copies that configuration file to its final destination.
This commit also restores the minimal dependencies to build the `install_fips`
target immediately after configuring, which was broken after the removal
of the `install_sw` dependency.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13684)
According to the OpenSSL 3.0 Wiki, the file should be located at
$(DESTDIR)$(OPENSSLDIR)/fipsmodule.cnf
next to the openssl.cnf file.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13684)
The `openssl` app was previously called without a path, which
would generally invoke the system's copy of the openssl application.
Currently, that's most likely an openssl version 1.1.1 application,
which does not recognize the `fipsinstall` command and terminates
with an error message.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13684)
The FIPS hmac key is provided as a hexadezimal string, which needs to
be be prefixed with `hexkey:`, not `key:`.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13684)
Previously, we had dependency making pretty much hard coded in the
build file templates, with a bit of an exception for Unix family
platforms, where we had different cases depending on what dependency
making program was found.
With the Embarcadero C++ builder, a separate scheme appeared, with a
different logic.
This change merges the two, and introduces two config target
attributes:
makedepcmd The program to use, where this is relevant.
This replaces the earlier configuration
attribute 'makedepprog'.
makedep_scheme This is a keyword that can be used by build
files templates to produce different sorts of
commands, but most importantly, to pass as
argument to util/add-depend.pl, which uses
this keyword as a "producer" for the
dependency lines.
If the config target doesn't define the 'makedep_scheme' attribute,
Configure tries to figure it out by looking for GCC compatible
compilers or for the 'makedepend' command.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/15006)
The modified way to generate .d files had an unfortunate side effect,
that it outputs the whole preprocessed file and not just the dependency
lines, at least with MSVC's cl. That gave util/add-depends.pl a whole
lot more to read through, which impacts greatly on the performance of
dependency treatment.
We modify the process by adding a config target attribute 'make_depend',
which can be any suitable command for generating such lines. All it
needs is to also accept C flags and macro definitions.
Fixes#14994
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/15006)
This file is outside the source tree, so we have no business removing
it. This is especially concerning if that was the tarball the user
had to create the source tree.
Fixes#14981
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14985)
We remove a TODO(3.0) from the unix Makefile template. The current
approach works. It can be improved later.
Fixes#14403
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14887)
`import32.lib` serves the purpose for most Windows API libraries, including this one. For example, with a GNU `grep` utility:
>tdump %BDS%\lib\win32c\release\import32.lib | grep -B 3 -A 1 CertOpenStore
171E32 COMENT Purge: Yes, List: Yes, Class: 160 (0A0h), SubClass: 1 (01h)
Dynamic link import (IMPDEF)
Imported by: name
Internal Name: CertOpenStore
Module Name: CRYPT32.dll
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
MSVC's `link.exe` automatically finds `__cdecl` C functions (which are
decorated with a leading underscore by the compiler) when they are
mentioned in a `.def` file without the leading underscore. This is an
[under-documented feature][1] of MSVC's `link.exe`. C++Builder's
`ilink32.exe` doesn't do this, and thus needs the name-translation in
the `.def` file. Then `implib.exe` needs to be told to re-add it.
(The Clang-based `bcc32c.exe` doesn't implement the [`-vu` or `-u-`][2]
options to skip adding the leading underscore to `__cdecl` C function
names, so this is the only way to have things work with non-underscored
export names in the DLLs.)
[1]: https://github.com/MicrosoftDocs/cpp-docs/issues/2653
[2]: http://docwiki.embarcadero.com/RADStudio/Sydney/en/Options_Not_Supported_by_Clang-enhanced_C%2B%2B_Compilers#BCC32_Options_that_Are_Not_Supported_by_Clang-enhanced_C.2B.2B_Compilers
Also silence linker warnings on duplicate symbols and ensure that error-
case cleanup in link rules work in C++Builder's `make.exe`.
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
Microsoft NMake runs the entire command block in a single `cmd.exe`
session, so environment variables exported with `set` are seen by the
following commands. C++Builder's `make.exe` doesn't use `cmd.exe` but
executes each command line separately. While it emulates only certain
shell features itself, `set` isn't one of them.
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
C++Builder's `make.exe` cannot handle a command output redirection to a
quoted filename, saying: "The system cannot find the file specified."
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
C++Builder's `make.exe` complains if a target has no dependencies (e.g.
after variable expansion) and no lines of commands. Ensure there is a
blank command line if the dependency list is entirely made of variables.
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
C++Builder's `make.exe` has a bug in finding the rule of a quoted
dependency that doesn't exist in the filesystem. So for example:
A: "src\B" "out\C"
touch $@
out\C:
mkdir out
touch $@
leads to:
Fatal: '"out\C"' does not exist - don't know how to make it
This happens even with the `-N` option, and is different behavior from
Microsoft NMake which documents the feature of [quoted filepaths][1].
Commit cb663908 quoted all dependency filepaths, in case they are used
in a out-of-source build. The quoting is not done for target names,
however, which implies that the build directory is still expected to not
have spaces. It follows that we only need to quote non-generated source
files in dependency lists, since generated source files will be created
in the build directory.
Change the logic accordingly as a workaround, so that it works at least
for in-source builds with C++Builder's `make.exe`.
[1]: https://docs.microsoft.com/cpp/build/reference/long-filenames-in-a-makefile?view=msvc-160
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
The space is problematic with C++Builder (`bcc32.exe`). MSVC (`cl.exe`)
doesn't care.
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
The Windows toolchain is composed of utilites from both the Windows SDK
and the Visual Studio Build Tools. Move the configuration of the latter
utilities into the `VC-common` template inside `10-main.conf`, while
leaving the former utilities in `BASE_Windows`. This allows for other
Windows compilers to be configured inheriting from `BASE_Windows`.
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/13540)
We now use the MANSUFFIX "ossl" by default.
Fixes#14318
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14847)
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14806)
The `make install_fips` target failed
msp@debian:~/src/openssl$ make install_fips
*** Installing FIPS module
install providers/fips.so -> /opt/openssl-dev/lib/ossl-modules/fips.so
*** Installing FIPS module configuration
fipsinstall /opt/openssl-dev/ssl/fipsmodule.cnf
FATAL: Startup failure (dev note: apps_startup()) for ./apps/openssl
... No such file or directory:crypto/conf/conf_def.c:771:calling stat(fipsmodule.cnf)
...
make: *** [Makefile:3341: install_fips] Error 1
because the `openssl fipsinstall` command was loading a previously installed
configuration file instead of the copy shipped with the source tree.
msp@debian:~/src/openssl$ strace -f make install_fips |& grep openssl.cnf
[pid 128683] openat(AT_FDCWD, "/opt/openssl-dev/ssl/openssl.cnf", O_RDONLY) = 3
This issue reveiled a more general problem, which applies to the tests as well:
unless openssl is installed, the openssl app must not use any preinstalled
configuration file. This holds in particular when the preinstalled configuration
file load providers, which caused the above failure.
The most consistent way to achieve this behaviour is to set the OPENSSL_CONF
environment variable to the correct location in the util/wrap.pl perl wrapper.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14136)
This change includes swapping the PUT and SPT configuration,
includes of sys/stat.h and sys/types.h in the correct scope
to be picked up by SPT definitions.
Fixes: #14698Fixes: #14734
CLA: The author has the permission to grant the OpenSSL Team the right to use this change.
Signed-off-by: Randall S. Becker <rsbecker@nexbridge.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14736)
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14723)