glibc/scripts/build-many-glibcs.py

1676 lines
74 KiB
Python
Raw Normal View History

Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
#!/usr/bin/python3
# Build many configurations of glibc.
# Copyright (C) 2016-2019 Free Software Foundation, Inc.
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
# This file is part of the GNU C Library.
#
# The GNU C Library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# The GNU C Library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with the GNU C Library; if not, see
Prefer https to http for gnu.org and fsf.org URLs Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 13:40:42 +08:00
# <https://www.gnu.org/licenses/>.
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""Build many configurations of glibc.
This script takes as arguments a directory name (containing a src
subdirectory with sources of the relevant toolchain components) and a
description of what to do: 'checkout', to check out sources into that
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
directory, 'bot-cycle', to run a series of checkout and build steps,
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
'bot', to run 'bot-cycle' repeatedly, 'host-libraries', to build
libraries required by the toolchain, 'compilers', to build
cross-compilers for various configurations, or 'glibcs', to build
glibc for various configurations and run the compilation parts of the
testsuite. Subsequent arguments name the versions of components to
check out (<component>-<version), for 'checkout', or, for actions
other than 'checkout' and 'bot-cycle', name configurations for which
compilers or glibc are to be built.
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""
import argparse
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
import datetime
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
import email.mime.text
import email.utils
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
import json
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
import os
import re
import shutil
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
import smtplib
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
import stat
import subprocess
import sys
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
import time
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
import urllib.request
try:
subprocess.run
except:
class _CompletedProcess:
def __init__(self, args, returncode, stdout=None, stderr=None):
self.args = args
self.returncode = returncode
self.stdout = stdout
self.stderr = stderr
def _run(*popenargs, input=None, timeout=None, check=False, **kwargs):
assert(timeout is None)
with subprocess.Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input)
except:
process.kill()
process.wait()
raise
returncode = process.poll()
if check and returncode:
raise subprocess.CalledProcessError(returncode, popenargs)
return _CompletedProcess(popenargs, returncode, stdout, stderr)
subprocess.run = _run
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
class Context(object):
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""The global state associated with builds in a given directory."""
def __init__(self, topdir, parallelism, keep, replace_sources, strip,
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
full_gcc, action):
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""Initialize the context."""
self.topdir = topdir
self.parallelism = parallelism
self.keep = keep
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
self.replace_sources = replace_sources
self.strip = strip
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
self.full_gcc = full_gcc
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.srcdir = os.path.join(topdir, 'src')
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
self.versions_json = os.path.join(self.srcdir, 'versions.json')
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
self.build_state_json = os.path.join(topdir, 'build-state.json')
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
self.bot_config_json = os.path.join(topdir, 'bot-config.json')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.installdir = os.path.join(topdir, 'install')
self.host_libraries_installdir = os.path.join(self.installdir,
'host-libraries')
self.builddir = os.path.join(topdir, 'build')
self.logsdir = os.path.join(topdir, 'logs')
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
self.logsdir_old = os.path.join(topdir, 'logs-old')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.makefile = os.path.join(self.builddir, 'Makefile')
self.wrapper = os.path.join(self.builddir, 'wrapper')
self.save_logs = os.path.join(self.builddir, 'save-logs')
self.script_text = self.get_script_text()
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if action != 'checkout':
self.build_triplet = self.get_build_triplet()
self.glibc_version = self.get_glibc_version()
self.configs = {}
self.glibc_configs = {}
self.makefile_pieces = ['.PHONY: all\n']
self.add_all_configs()
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
self.load_versions_json()
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
self.load_build_state_json()
self.status_log_list = []
self.email_warning = False
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def get_script_text(self):
"""Return the text of this script."""
with open(sys.argv[0], 'r') as f:
return f.read()
def exec_self(self):
"""Re-execute this script with the same arguments."""
sys.stdout.flush()
os.execv(sys.executable, [sys.executable] + sys.argv)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def get_build_triplet(self):
"""Determine the build triplet with config.guess."""
config_guess = os.path.join(self.component_srcdir('gcc'),
'config.guess')
cg_out = subprocess.run([config_guess], stdout=subprocess.PIPE,
check=True, universal_newlines=True).stdout
return cg_out.rstrip()
def get_glibc_version(self):
"""Determine the glibc version number (major.minor)."""
version_h = os.path.join(self.component_srcdir('glibc'), 'version.h')
with open(version_h, 'r') as f:
lines = f.readlines()
starttext = '#define VERSION "'
for l in lines:
if l.startswith(starttext):
l = l[len(starttext):]
l = l.rstrip('"\n')
m = re.fullmatch('([0-9]+)\.([0-9]+)[.0-9]*', l)
return '%s.%s' % m.group(1, 2)
print('error: could not determine glibc version')
exit(1)
def add_all_configs(self):
"""Add all known glibc build configurations."""
self.add_config(arch='aarch64',
os_name='linux-gnu',
extra_glibcs=[{'variant': 'disable-multi-arch',
'cfg': ['--disable-multi-arch']}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='aarch64_be',
os_name='linux-gnu')
self.add_config(arch='alpha',
os_name='linux-gnu')
self.add_config(arch='arm',
os_name='linux-gnueabi',
extra_glibcs=[{'variant': 'v4t',
'ccopts': '-march=armv4t'}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='armeb',
os_name='linux-gnueabi')
self.add_config(arch='armeb',
os_name='linux-gnueabi',
variant='be8',
gcc_cfg=['--with-arch=armv7-a'])
self.add_config(arch='arm',
os_name='linux-gnueabihf',
gcc_cfg=['--with-float=hard', '--with-cpu=arm926ej-s'],
extra_glibcs=[{'variant': 'v7a',
'ccopts': '-march=armv7-a -mfpu=vfpv3'},
{'variant': 'v7a-disable-multi-arch',
'ccopts': '-march=armv7-a -mfpu=vfpv3',
'cfg': ['--disable-multi-arch']}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='armeb',
os_name='linux-gnueabihf',
gcc_cfg=['--with-float=hard', '--with-cpu=arm926ej-s'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='armeb',
os_name='linux-gnueabihf',
variant='be8',
gcc_cfg=['--with-float=hard', '--with-arch=armv7-a',
'--with-fpu=vfpv3'])
Add C-SKY port This patch add two abi combinations support for C-SKY ABIV2: soft-float little endian, hard float little endian. C-SKY ABI manual and architecture user guide are available from: https://github.com/c-sky/csky-doc * config.h.in (CSKYABI, CSKY_HARD_FLOAT): New Define. * scripts/build-many-glibcs.py: Add C-SKY targets. * sysdeps/csky/Implies: New file. * sysdeps/csky/Makefile: Likewise. * sysdeps/csky/abiv2/__longjmp.S: Likewise. * sysdeps/csky/abiv2/csky-mcount.S: Likewise. * sysdeps/csky/abiv2/dl-trampoline.S: Likewise. * sysdeps/csky/abiv2/memcmp.S: Likewise. * sysdeps/csky/abiv2/memcpy.S: Likewise. * sysdeps/csky/abiv2/memmove.S: Likewise. * sysdeps/csky/abiv2/memset.S: Likewise. * sysdeps/csky/abiv2/setjmp.S: Likewise. * sysdeps/csky/abiv2/start.S: Likewise. * sysdeps/csky/abiv2/strcmp.S: Likewise. * sysdeps/csky/abiv2/strcpy.S: Likewise. * sysdeps/csky/abiv2/strlen.S: Likewise. * sysdeps/csky/abiv2/tls-macros.h: Likewise. * sysdeps/csky/abort-instr.h: Likewise. * sysdeps/csky/atomic-machine.h: Likewise. * sysdeps/csky/bits/endian.h: Likewise. * sysdeps/csky/bits/fenv.h: Likewise. * sysdeps/csky/bits/link.h: Likewise. * sysdeps/csky/bits/setjmp.h: Likewise. * sysdeps/csky/bsd-_setjmp.S: Likewise. * sysdeps/csky/bsd-setjmp.S: Likewise. * sysdeps/csky/configure: Likewise. * sysdeps/csky/configure.ac: Likewise. * sysdeps/csky/dl-machine.h: Likewise. * sysdeps/csky/dl-procinfo.c: Likewise. * sysdeps/csky/dl-procinfo.h: Likewise. * sysdeps/csky/dl-sysdep.h: Likewise. * sysdeps/csky/dl-tls.h: Likewise. * sysdeps/csky/fpu/fclrexcpt.c: Likewise. * sysdeps/csky/fpu/fedisblxcpt.c: Likewise. * sysdeps/csky/fpu/feenablxcpt.c: Likewise. * sysdeps/csky/fpu/fegetenv.c: Likewise. * sysdeps/csky/fpu/fegetexcept.c: Likewise. * sysdeps/csky/fpu/fegetmode.c: Likewise. * sysdeps/csky/fpu/fegetround.c: Likewise. * sysdeps/csky/fpu/feholdexcpt.c: Likewise. * sysdeps/csky/fpu/fenv_libc.h: Likewise. * sysdeps/csky/fpu/fenv_private.h: Likewise. * sysdeps/csky/fpu/fesetenv.c: Likewise. * sysdeps/csky/fpu/fesetexcept.c: Likewise. * sysdeps/csky/fpu/fesetmode.c: Likewise. * sysdeps/csky/fpu/fesetround.c: Likewise. * sysdeps/csky/fpu/feupdateenv.c: Likewise. * sysdeps/csky/fpu/fgetexcptflg.c: Likewise. * sysdeps/csky/fpu/fix-fp-int-convert-overflow.h: Likewise. * sysdeps/csky/fpu/fraiseexcpt.c: Likewise. * sysdeps/csky/fpu/fsetexcptflg.c: Likewise. * sysdeps/csky/fpu/ftestexcept.c: Likewise. * sysdeps/csky/fpu/libm-test-ulps: Likewise. * sysdeps/csky/fpu/libm-test-ulps-name: Likewise. * sysdeps/csky/fpu_control.h: Likewise. * sysdeps/csky/gccframe.h: Likewise. * sysdeps/csky/jmpbuf-unwind.h: Likewise. * sysdeps/csky/ldsodefs.h: Likewise. * sysdeps/csky/libc-tls.c: Likewise. * sysdeps/csky/linkmap.h: Likewise. * sysdeps/csky/machine-gmon.h: Likewise. * sysdeps/csky/memusage.h: Likewise. * sysdeps/csky/nofpu/Implies: Likewise. * sysdeps/csky/nofpu/libm-test-ulps: Likewise. * sysdeps/csky/nofpu/libm-test-ulps-name: Likewise. * sysdeps/csky/nptl/Makefile: Likewise. * sysdeps/csky/nptl/bits/pthreadtypes-arch.h: Likewise. * sysdeps/csky/nptl/bits/semaphore.h: Likewise. * sysdeps/csky/nptl/pthread-offsets.h: Likewise. * sysdeps/csky/nptl/pthreaddef.h: Likewise. * sysdeps/csky/nptl/tcb-offsets.sym: Likewise. * sysdeps/csky/nptl/tls.h: Likewise. * sysdeps/csky/preconfigure: Likewise. * sysdeps/csky/sfp-machine.h: Likewise. * sysdeps/csky/sotruss-lib.c: Likewise. * sysdeps/csky/stackinfo.h: Likewise. * sysdeps/csky/sysdep.h: Likewise. * sysdeps/csky/tininess.h: Likewise. * sysdeps/csky/tst-audit.h: Likewise. * sysdeps/unix/sysv/linux/csky/Implies: Likewise. * sysdeps/unix/sysv/linux/csky/Makefile: Likewise. * sysdeps/unix/sysv/linux/csky/Versions: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/____longjmp_chk.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/clone.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/getcontext.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/setcontext.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/swapcontext.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/syscall.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/sysdep.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/ucontext_i.sym: Likewise. * sysdeps/unix/sysv/linux/csky/bits/procfs.h: Likewise. * sysdeps/unix/sysv/linux/csky/bits/shmlba.h: Likewise. * sysdeps/unix/sysv/linux/csky/c++-types.data: Likewise. * sysdeps/unix/sysv/linux/csky/configure: Likewise. * sysdeps/unix/sysv/linux/csky/configure.ac: Likewise. * sysdeps/unix/sysv/linux/csky/ipc_priv.h: Likewise. * sysdeps/unix/sysv/linux/csky/jmp_buf-macros.h: Likewise. * sysdeps/unix/sysv/linux/csky/kernel-features.h: Likewise. * sysdeps/unix/sysv/linux/csky/ld.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/ldconfig.h: Likewise. * sysdeps/unix/sysv/linux/csky/libBrokenLocale.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libanl.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libc.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libcrypt.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libdl.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libpthread.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libresolv.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/librt.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libthread_db.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libutil.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/localplt.data: Likewise. * sysdeps/unix/sysv/linux/csky/makecontext.c: Likewise. * sysdeps/unix/sysv/linux/csky/profil-counter.h: Likewise. * sysdeps/unix/sysv/linux/csky/pt-vfork.S: Likewise. * sysdeps/unix/sysv/linux/csky/register-dump.h: Likewise. * sysdeps/unix/sysv/linux/csky/shlib-versions: Likewise. * sysdeps/unix/sysv/linux/csky/sigcontextinfo.h: Likewise. * sysdeps/unix/sysv/linux/csky/sys/cachectl.h: Likewise. * sysdeps/unix/sysv/linux/csky/sys/ucontext.h: Likewise. * sysdeps/unix/sysv/linux/csky/sys/user.h: Likewise. * sysdeps/unix/sysv/linux/csky/syscalls.list: Likewise. * sysdeps/unix/sysv/linux/csky/sysdep.h: Likewise.
2018-12-21 09:48:04 +08:00
self.add_config(arch='csky',
os_name='linux-gnuabiv2',
variant='soft',
gcc_cfg=['--disable-multilib'])
self.add_config(arch='csky',
os_name='linux-gnuabiv2',
gcc_cfg=['--with-float=hard', '--disable-multilib'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='hppa',
os_name='linux-gnu')
self.add_config(arch='i686',
os_name='gnu')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='ia64',
os_name='linux-gnu',
first_gcc_cfg=['--with-system-libunwind'])
self.add_config(arch='m68k',
os_name='linux-gnu',
gcc_cfg=['--disable-multilib'])
self.add_config(arch='m68k',
os_name='linux-gnu',
variant='coldfire',
gcc_cfg=['--with-arch=cf', '--disable-multilib'])
self.add_config(arch='m68k',
os_name='linux-gnu',
variant='coldfire-soft',
gcc_cfg=['--with-arch=cf', '--with-cpu=54455',
'--disable-multilib'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='microblaze',
os_name='linux-gnu',
gcc_cfg=['--disable-multilib'])
self.add_config(arch='microblazeel',
os_name='linux-gnu',
gcc_cfg=['--disable-multilib'])
self.add_config(arch='mips64',
os_name='linux-gnu',
gcc_cfg=['--with-mips-plt'],
glibcs=[{'variant': 'n32'},
{'arch': 'mips',
'ccopts': '-mabi=32'},
{'variant': 'n64',
'ccopts': '-mabi=64'}])
self.add_config(arch='mips64',
os_name='linux-gnu',
variant='soft',
gcc_cfg=['--with-mips-plt', '--with-float=soft'],
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
glibcs=[{'variant': 'n32-soft'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'soft',
'arch': 'mips',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=32'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'n64-soft',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=64'}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='mips64',
os_name='linux-gnu',
variant='nan2008',
gcc_cfg=['--with-mips-plt', '--with-nan=2008',
'--with-arch-64=mips64r2',
'--with-arch-32=mips32r2'],
glibcs=[{'variant': 'n32-nan2008'},
{'variant': 'nan2008',
'arch': 'mips',
'ccopts': '-mabi=32'},
{'variant': 'n64-nan2008',
'ccopts': '-mabi=64'}])
self.add_config(arch='mips64',
os_name='linux-gnu',
variant='nan2008-soft',
gcc_cfg=['--with-mips-plt', '--with-nan=2008',
'--with-arch-64=mips64r2',
'--with-arch-32=mips32r2',
'--with-float=soft'],
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
glibcs=[{'variant': 'n32-nan2008-soft'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'nan2008-soft',
'arch': 'mips',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=32'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'n64-nan2008-soft',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=64'}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='mips64el',
os_name='linux-gnu',
gcc_cfg=['--with-mips-plt'],
glibcs=[{'variant': 'n32'},
{'arch': 'mipsel',
'ccopts': '-mabi=32'},
{'variant': 'n64',
'ccopts': '-mabi=64'}])
self.add_config(arch='mips64el',
os_name='linux-gnu',
variant='soft',
gcc_cfg=['--with-mips-plt', '--with-float=soft'],
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
glibcs=[{'variant': 'n32-soft'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'soft',
'arch': 'mipsel',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=32'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'n64-soft',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=64'}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='mips64el',
os_name='linux-gnu',
variant='nan2008',
gcc_cfg=['--with-mips-plt', '--with-nan=2008',
'--with-arch-64=mips64r2',
'--with-arch-32=mips32r2'],
glibcs=[{'variant': 'n32-nan2008'},
{'variant': 'nan2008',
'arch': 'mipsel',
'ccopts': '-mabi=32'},
{'variant': 'n64-nan2008',
'ccopts': '-mabi=64'}])
self.add_config(arch='mips64el',
os_name='linux-gnu',
variant='nan2008-soft',
gcc_cfg=['--with-mips-plt', '--with-nan=2008',
'--with-arch-64=mips64r2',
'--with-arch-32=mips32r2',
'--with-float=soft'],
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
glibcs=[{'variant': 'n32-nan2008-soft'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'nan2008-soft',
'arch': 'mipsel',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=32'},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'n64-nan2008-soft',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'ccopts': '-mabi=64'}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='nios2',
os_name='linux-gnu')
self.add_config(arch='powerpc',
os_name='linux-gnu',
gcc_cfg=['--disable-multilib', '--enable-secureplt'],
extra_glibcs=[{'variant': 'power4',
'ccopts': '-mcpu=power4',
'cfg': ['--with-cpu=power4']}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='powerpc',
os_name='linux-gnu',
variant='soft',
gcc_cfg=['--disable-multilib', '--with-float=soft',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
'--enable-secureplt'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='powerpc64',
os_name='linux-gnu',
gcc_cfg=['--disable-multilib', '--enable-secureplt'])
self.add_config(arch='powerpc64le',
os_name='linux-gnu',
gcc_cfg=['--disable-multilib', '--enable-secureplt'])
self.add_config(arch='riscv64',
os_name='linux-gnu',
variant='rv64imac-lp64',
gcc_cfg=['--with-arch=rv64imac', '--with-abi=lp64',
'--disable-multilib'])
self.add_config(arch='riscv64',
os_name='linux-gnu',
variant='rv64imafdc-lp64',
gcc_cfg=['--with-arch=rv64imafdc', '--with-abi=lp64',
'--disable-multilib'])
self.add_config(arch='riscv64',
os_name='linux-gnu',
variant='rv64imafdc-lp64d',
gcc_cfg=['--with-arch=rv64imafdc', '--with-abi=lp64d',
'--disable-multilib'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='s390x',
os_name='linux-gnu',
glibcs=[{},
{'arch': 's390', 'ccopts': '-m31'}])
self.add_config(arch='sh3',
os_name='linux-gnu')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='sh3eb',
os_name='linux-gnu')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='sh4',
os_name='linux-gnu')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='sh4eb',
os_name='linux-gnu')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='sh4',
os_name='linux-gnu',
variant='soft',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
gcc_cfg=['--without-fp'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='sh4eb',
os_name='linux-gnu',
variant='soft',
Remove --with-fp / --without-fp. There is a configure option --without-fp that specifies that nofpu sysdeps directories should be used instead of fpu directories. For most glibc configurations, this option is of no use: either there is no valid nofpu variant of that configuration, or there are no fpu or nofpu sysdeps directories for that processor and so the option does nothing. For a few configurations, if you are using a soft-float compiler this option is required, and failing to use it generally results in compilation errors from inline asm using unavailable floating-point instructions. We're moving away from --with-cpu to configuring glibc based on how the compiler generates code, and it is natural to do so for --without-fp as well; in most cases the soft-float and hard-float ABIs are incompatible so you have no hope of building a working glibc with an inappropriately configured compiler or libgcc. This patch eliminates --without-fp, replacing it entirely by automatic configuration based on the compiler. Configurations for which this is relevant (coldfire / mips / powerpc32 / sh) define a variable with_fp_cond in their preconfigure fragments (under the same conditions under which those fragments do anything); this is a preprocessor conditional which the toplevel configure script then uses in a test to determine which sysdeps directories to use. The config.make with-fp variable remains. It's used only by powerpc (sysdeps/powerpc/powerpc32/Makefile) to add -mhard-float to various flags variables. For powerpc, -mcpu= options can imply use of soft-float. That could be an issue if you want to build for e.g. 476fp, but are using --with-cpu=476 because there isn't a 476fp sysdeps directory. If in future we eliminate --with-cpu and replace it entirely by testing the compiler, it would be natural at that point to eliminate that code as well (as the user should then just use a compiler defaulting to 476fp and the 476 sysdeps directory would be used automatically). Tested for x86_64, and tested with build-many-glibcs.py that installed shared libraries are unchanged by this patch. * configure.ac (--with-fp): Remove configure option. (with_fp_cond): New variable. (libc_cv_with_fp): New configure test. Use this variable instead of with_fp. * configure: Regenerated. * config.make.in (with-fp): Use @libc_cv_with_fp@. * manual/install.texi (Configuring and compiling): Remove --without-fp. * INSTALL: Regenerated. * sysdeps/m68k/preconfigure (with_fp_cond): Define for ColdFire. * sysdeps/mips/preconfigure (with_fp_cond): Define. * sysdeps/powerpc/preconfigure (with_fp_cond): Define for 32-bit. * sysdeps/sh/preconfigure (with_fp_cond): Define. * scripts/build-many-glibcs.py (Context.add_all_configs): Do not use --without-fp to configure glibc.
2017-12-12 21:56:47 +08:00
gcc_cfg=['--without-fp'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='sparc64',
os_name='linux-gnu',
glibcs=[{},
{'arch': 'sparcv8',
'ccopts': '-m32 -mlong-double-128 -mcpu=leon3'}],
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'arch': 'sparcv9',
'ccopts': '-m32 -mlong-double-128 -mcpu=v9'}],
extra_glibcs=[{'variant': 'disable-multi-arch',
'cfg': ['--disable-multi-arch']},
{'variant': 'disable-multi-arch',
'arch': 'sparcv9',
'ccopts': '-m32 -mlong-double-128',
'cfg': ['--disable-multi-arch']}])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.add_config(arch='x86_64',
os_name='linux-gnu',
gcc_cfg=['--with-multilib-list=m64,m32,mx32'],
glibcs=[{},
{'variant': 'x32', 'ccopts': '-mx32'},
{'arch': 'i686', 'ccopts': '-m32 -march=i686'}],
extra_glibcs=[{'variant': 'disable-multi-arch',
'cfg': ['--disable-multi-arch']},
{'variant': 'enable-obsolete',
'cfg': ['--enable-obsolete-rpc',
'--enable-obsolete-nsl']},
{'variant': 'static-pie',
'cfg': ['--enable-static-pie']},
{'variant': 'x32-static-pie',
'ccopts': '-mx32',
'cfg': ['--enable-static-pie']},
{'variant': 'static-pie',
'arch': 'i686',
'ccopts': '-m32 -march=i686',
'cfg': ['--enable-static-pie']},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'variant': 'disable-multi-arch',
'arch': 'i686',
'ccopts': '-m32 -march=i686',
'cfg': ['--disable-multi-arch']},
{'variant': 'enable-obsolete',
'arch': 'i686',
'ccopts': '-m32 -march=i686',
'cfg': ['--enable-obsolete-rpc',
'--enable-obsolete-nsl']},
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
{'arch': 'i486',
'ccopts': '-m32 -march=i486'},
{'arch': 'i586',
'ccopts': '-m32 -march=i586'}])
def add_config(self, **args):
"""Add an individual build configuration."""
cfg = Config(self, **args)
if cfg.name in self.configs:
print('error: duplicate config %s' % cfg.name)
exit(1)
self.configs[cfg.name] = cfg
for c in cfg.all_glibcs:
if c.name in self.glibc_configs:
print('error: duplicate glibc config %s' % c.name)
exit(1)
self.glibc_configs[c.name] = c
def component_srcdir(self, component):
"""Return the source directory for a given component, e.g. gcc."""
return os.path.join(self.srcdir, component)
def component_builddir(self, action, config, component, subconfig=None):
"""Return the directory to use for a build."""
if config is None:
# Host libraries.
assert subconfig is None
return os.path.join(self.builddir, action, component)
if subconfig is None:
return os.path.join(self.builddir, action, config, component)
else:
# glibc build as part of compiler build.
return os.path.join(self.builddir, action, config, component,
subconfig)
def compiler_installdir(self, config):
"""Return the directory in which to install a compiler."""
return os.path.join(self.installdir, 'compilers', config)
def compiler_bindir(self, config):
"""Return the directory in which to find compiler binaries."""
return os.path.join(self.compiler_installdir(config), 'bin')
def compiler_sysroot(self, config):
"""Return the sysroot directory for a compiler."""
return os.path.join(self.compiler_installdir(config), 'sysroot')
def glibc_installdir(self, config):
"""Return the directory in which to install glibc."""
return os.path.join(self.installdir, 'glibcs', config)
def run_builds(self, action, configs):
"""Run the requested builds."""
if action == 'checkout':
self.checkout(configs)
return
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
if action == 'bot-cycle':
if configs:
print('error: configurations specified for bot-cycle')
exit(1)
self.bot_cycle()
return
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
if action == 'bot':
if configs:
print('error: configurations specified for bot')
exit(1)
self.bot()
return
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
if action == 'host-libraries' and configs:
print('error: configurations specified for host-libraries')
exit(1)
self.clear_last_build_state(action)
build_time = datetime.datetime.utcnow()
if action == 'host-libraries':
build_components = ('gmp', 'mpfr', 'mpc')
old_components = ()
old_versions = {}
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.build_host_libraries()
elif action == 'compilers':
build_components = ('binutils', 'gcc', 'glibc', 'linux', 'mig',
'gnumach', 'hurd')
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
old_components = ('gmp', 'mpfr', 'mpc')
old_versions = self.build_state['host-libraries']['build-versions']
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.build_compilers(configs)
else:
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
build_components = ('glibc',)
old_components = ('gmp', 'mpfr', 'mpc', 'binutils', 'gcc', 'linux',
'mig', 'gnumach', 'hurd')
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
old_versions = self.build_state['compilers']['build-versions']
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.build_glibcs(configs)
self.write_files()
self.do_build()
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
if configs:
# Partial build, do not update stored state.
return
build_versions = {}
for k in build_components:
if k in self.versions:
build_versions[k] = {'version': self.versions[k]['version'],
'revision': self.versions[k]['revision']}
for k in old_components:
if k in old_versions:
build_versions[k] = {'version': old_versions[k]['version'],
'revision': old_versions[k]['revision']}
self.update_build_state(action, build_time, build_versions)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
@staticmethod
def remove_dirs(*args):
"""Remove directories and their contents if they exist."""
for dir in args:
shutil.rmtree(dir, ignore_errors=True)
@staticmethod
def remove_recreate_dirs(*args):
"""Remove directories if they exist, and create them as empty."""
Context.remove_dirs(*args)
for dir in args:
os.makedirs(dir, exist_ok=True)
def add_makefile_cmdlist(self, target, cmdlist, logsdir):
"""Add makefile text for a list of commands."""
commands = cmdlist.makefile_commands(self.wrapper, logsdir)
self.makefile_pieces.append('all: %s\n.PHONY: %s\n%s:\n%s\n' %
(target, target, target, commands))
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
self.status_log_list.extend(cmdlist.status_logs(logsdir))
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def write_files(self):
"""Write out the Makefile and wrapper script."""
mftext = ''.join(self.makefile_pieces)
with open(self.makefile, 'w') as f:
f.write(mftext)
wrapper_text = (
'#!/bin/sh\n'
'prev_base=$1\n'
'this_base=$2\n'
'desc=$3\n'
'dir=$4\n'
'path=$5\n'
'shift 5\n'
'prev_status=$prev_base-status.txt\n'
'this_status=$this_base-status.txt\n'
'this_log=$this_base-log.txt\n'
'date > "$this_log"\n'
'echo >> "$this_log"\n'
'echo "Description: $desc" >> "$this_log"\n'
'printf "%s" "Command:" >> "$this_log"\n'
'for word in "$@"; do\n'
' if expr "$word" : "[]+,./0-9@A-Z_a-z-]\\\\{1,\\\\}\\$" > /dev/null; then\n'
' printf " %s" "$word"\n'
' else\n'
' printf " \'"\n'
' printf "%s" "$word" | sed -e "s/\'/\'\\\\\\\\\'\'/"\n'
' printf "\'"\n'
' fi\n'
'done >> "$this_log"\n'
'echo >> "$this_log"\n'
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'echo "Directory: $dir" >> "$this_log"\n'
'echo "Path addition: $path" >> "$this_log"\n'
'echo >> "$this_log"\n'
'record_status ()\n'
'{\n'
' echo >> "$this_log"\n'
' echo "$1: $desc" > "$this_status"\n'
' echo "$1: $desc" >> "$this_log"\n'
' echo >> "$this_log"\n'
' date >> "$this_log"\n'
' echo "$1: $desc"\n'
' exit 0\n'
'}\n'
'check_error ()\n'
'{\n'
' if [ "$1" != "0" ]; then\n'
' record_status FAIL\n'
' fi\n'
'}\n'
'if [ "$prev_base" ] && ! grep -q "^PASS" "$prev_status"; then\n'
' record_status UNRESOLVED\n'
'fi\n'
'if [ "$dir" ]; then\n'
' cd "$dir"\n'
' check_error "$?"\n'
'fi\n'
'if [ "$path" ]; then\n'
' PATH=$path:$PATH\n'
'fi\n'
'"$@" < /dev/null >> "$this_log" 2>&1\n'
'check_error "$?"\n'
'record_status PASS\n')
with open(self.wrapper, 'w') as f:
f.write(wrapper_text)
# Mode 0o755.
mode_exec = (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
stat.S_IROTH|stat.S_IXOTH)
os.chmod(self.wrapper, mode_exec)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
save_logs_text = (
'#!/bin/sh\n'
'if ! [ -f tests.sum ]; then\n'
' echo "No test summary available."\n'
' exit 0\n'
'fi\n'
'save_file ()\n'
'{\n'
' echo "Contents of $1:"\n'
' echo\n'
' cat "$1"\n'
' echo\n'
' echo "End of contents of $1."\n'
' echo\n'
'}\n'
'save_file tests.sum\n'
'non_pass_tests=$(grep -v "^PASS: " tests.sum | sed -e "s/^PASS: //")\n'
'for t in $non_pass_tests; do\n'
' if [ -f "$t.out" ]; then\n'
' save_file "$t.out"\n'
' fi\n'
'done\n')
with open(self.save_logs, 'w') as f:
f.write(save_logs_text)
os.chmod(self.save_logs, mode_exec)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def do_build(self):
"""Do the actual build."""
cmd = ['make', '-j%d' % self.parallelism]
subprocess.run(cmd, cwd=self.builddir, check=True)
def build_host_libraries(self):
"""Build the host libraries."""
installdir = self.host_libraries_installdir
builddir = os.path.join(self.builddir, 'host-libraries')
logsdir = os.path.join(self.logsdir, 'host-libraries')
self.remove_recreate_dirs(installdir, builddir, logsdir)
cmdlist = CommandList('host-libraries', self.keep)
self.build_host_library(cmdlist, 'gmp')
self.build_host_library(cmdlist, 'mpfr',
['--with-gmp=%s' % installdir])
self.build_host_library(cmdlist, 'mpc',
['--with-gmp=%s' % installdir,
'--with-mpfr=%s' % installdir])
cmdlist.add_command('done', ['touch', os.path.join(installdir, 'ok')])
self.add_makefile_cmdlist('host-libraries', cmdlist, logsdir)
def build_host_library(self, cmdlist, lib, extra_opts=None):
"""Build one host library."""
srcdir = self.component_srcdir(lib)
builddir = self.component_builddir('host-libraries', None, lib)
installdir = self.host_libraries_installdir
cmdlist.push_subdesc(lib)
cmdlist.create_use_dir(builddir)
cfg_cmd = [os.path.join(srcdir, 'configure'),
'--prefix=%s' % installdir,
'--disable-shared']
if extra_opts:
cfg_cmd.extend (extra_opts)
cmdlist.add_command('configure', cfg_cmd)
cmdlist.add_command('build', ['make'])
cmdlist.add_command('check', ['make', 'check'])
cmdlist.add_command('install', ['make', 'install'])
cmdlist.cleanup_dir()
cmdlist.pop_subdesc()
def build_compilers(self, configs):
"""Build the compilers."""
if not configs:
self.remove_dirs(os.path.join(self.builddir, 'compilers'))
self.remove_dirs(os.path.join(self.installdir, 'compilers'))
self.remove_dirs(os.path.join(self.logsdir, 'compilers'))
configs = sorted(self.configs.keys())
for c in configs:
self.configs[c].build()
def build_glibcs(self, configs):
"""Build the glibcs."""
if not configs:
self.remove_dirs(os.path.join(self.builddir, 'glibcs'))
self.remove_dirs(os.path.join(self.installdir, 'glibcs'))
self.remove_dirs(os.path.join(self.logsdir, 'glibcs'))
configs = sorted(self.glibc_configs.keys())
for c in configs:
self.glibc_configs[c].build()
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
def load_versions_json(self):
"""Load information about source directory versions."""
if not os.access(self.versions_json, os.F_OK):
self.versions = {}
return
with open(self.versions_json, 'r') as f:
self.versions = json.load(f)
def store_json(self, data, filename):
"""Store information in a JSON file."""
filename_tmp = filename + '.tmp'
with open(filename_tmp, 'w') as f:
json.dump(data, f, indent=2, sort_keys=True)
os.rename(filename_tmp, filename)
def store_versions_json(self):
"""Store information about source directory versions."""
self.store_json(self.versions, self.versions_json)
def set_component_version(self, component, version, explicit, revision):
"""Set the version information for a component."""
self.versions[component] = {'version': version,
'explicit': explicit,
'revision': revision}
self.store_versions_json()
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def checkout(self, versions):
"""Check out the desired component versions."""
default_versions = {'binutils': 'vcs-2.33',
'gcc': 'vcs-9',
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'glibc': 'vcs-mainline',
'gmp': '6.1.2',
'linux': '5.4',
'mpc': '1.1.0',
'mpfr': '4.0.2',
'mig': 'vcs-mainline',
'gnumach': 'vcs-mainline',
'hurd': 'vcs-mainline'}
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
use_versions = {}
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
explicit_versions = {}
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
for v in versions:
found_v = False
for k in default_versions.keys():
kx = k + '-'
if v.startswith(kx):
vx = v[len(kx):]
if k in use_versions:
print('error: multiple versions for %s' % k)
exit(1)
use_versions[k] = vx
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
explicit_versions[k] = True
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
found_v = True
break
if not found_v:
print('error: unknown component in %s' % v)
exit(1)
for k in default_versions.keys():
if k not in use_versions:
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
if k in self.versions and self.versions[k]['explicit']:
use_versions[k] = self.versions[k]['version']
explicit_versions[k] = True
else:
use_versions[k] = default_versions[k]
explicit_versions[k] = False
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
os.makedirs(self.srcdir, exist_ok=True)
for k in sorted(default_versions.keys()):
update = os.access(self.component_srcdir(k), os.F_OK)
v = use_versions[k]
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
if (update and
k in self.versions and
v != self.versions[k]['version']):
if not self.replace_sources:
print('error: version of %s has changed from %s to %s, '
'use --replace-sources to check out again' %
(k, self.versions[k]['version'], v))
exit(1)
shutil.rmtree(self.component_srcdir(k))
update = False
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if v.startswith('vcs-'):
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
revision = self.checkout_vcs(k, v[4:], update)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
else:
self.checkout_tar(k, v, update)
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
revision = v
self.set_component_version(k, v, explicit_versions[k], revision)
if self.get_script_text() != self.script_text:
# Rerun the checkout process in case the updated script
# uses different default versions or new components.
self.exec_self()
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def checkout_vcs(self, component, version, update):
"""Check out the given version of the given component from version
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
control. Return a revision identifier."""
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if component == 'binutils':
git_url = 'git://sourceware.org/git/binutils-gdb.git'
if version == 'mainline':
git_branch = 'master'
else:
trans = str.maketrans({'.': '_'})
git_branch = 'binutils-%s-branch' % version.translate(trans)
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
return self.git_checkout(component, git_url, git_branch, update)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
elif component == 'gcc':
if version == 'mainline':
branch = 'trunk'
else:
trans = str.maketrans({'.': '_'})
branch = 'branches/gcc-%s-branch' % version.translate(trans)
svn_url = 'svn://gcc.gnu.org/svn/gcc/%s' % branch
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
return self.gcc_checkout(svn_url, update)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
elif component == 'glibc':
git_url = 'git://sourceware.org/git/glibc.git'
if version == 'mainline':
git_branch = 'master'
else:
git_branch = 'release/%s/master' % version
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
r = self.git_checkout(component, git_url, git_branch, update)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.fix_glibc_timestamps()
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
return r
elif component == 'gnumach':
git_url = 'git://git.savannah.gnu.org/hurd/gnumach.git'
git_branch = 'master'
r = self.git_checkout(component, git_url, git_branch, update)
subprocess.run(['autoreconf', '-i'],
cwd=self.component_srcdir(component), check=True)
return r
elif component == 'mig':
git_url = 'git://git.savannah.gnu.org/hurd/mig.git'
git_branch = 'master'
r = self.git_checkout(component, git_url, git_branch, update)
subprocess.run(['autoreconf', '-i'],
cwd=self.component_srcdir(component), check=True)
return r
elif component == 'hurd':
git_url = 'git://git.savannah.gnu.org/hurd/hurd.git'
git_branch = 'master'
r = self.git_checkout(component, git_url, git_branch, update)
subprocess.run(['autoconf'],
cwd=self.component_srcdir(component), check=True)
return r
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
else:
print('error: component %s coming from VCS' % component)
exit(1)
def git_checkout(self, component, git_url, git_branch, update):
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
"""Check out a component from git. Return a commit identifier."""
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if update:
subprocess.run(['git', 'remote', 'prune', 'origin'],
cwd=self.component_srcdir(component), check=True)
if self.replace_sources:
subprocess.run(['git', 'clean', '-dxfq'],
cwd=self.component_srcdir(component), check=True)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
subprocess.run(['git', 'pull', '-q'],
cwd=self.component_srcdir(component), check=True)
else:
subprocess.run(['git', 'clone', '-q', '-b', git_branch, git_url,
self.component_srcdir(component)], check=True)
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
r = subprocess.run(['git', 'rev-parse', 'HEAD'],
cwd=self.component_srcdir(component),
stdout=subprocess.PIPE,
check=True, universal_newlines=True).stdout
return r.rstrip()
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def fix_glibc_timestamps(self):
"""Fix timestamps in a glibc checkout."""
# Ensure that builds do not try to regenerate generated files
# in the source tree.
srcdir = self.component_srcdir('glibc')
Touch more glibc source files in build-many-glibcs.py. build-many-glibcs.py currently copies the source tree to avoid issues with parallel builds trying to write into it. This copying can result in occasional spurious build failures from bots, when a "git gc" is in progress that changes .git contents while copying is taking place, and it would also be desirable to avoid the need to copy to save on disk space, I/O and memory used in build-many-glibcs.py builds. In preparation for removing the copying, this patch arranges for build-many-glibcs.py to touch more files on checkout so their timestamps do not result in make attempting to rebuild them. Before actually removing the copying, I intend to do further tests to ensure I haven't missed any other such makefile dependencies. This is of course without prejudice to possibly moving more of these files to being generated in the build directory rather than being checked in at all, where that can be done using build tools already required for the build. For sysdeps files (installed and otherwise) it would be necessary to make sure this does not affect the search ordering, for headers used in the build it would be necessary to ensure they are generated early enough, and for errlist.c there may be dual licensing reasons for keeping it checked in. Tested that a checkout with build-many-glibcs.py does touch the expected files and that a glibcs build for aarch64-linux-gnu succeeds. * scripts/build-many-glibcs.py (Context.fix_glibc_timestamps): Touch additional files.
2018-11-27 07:52:43 +08:00
# These files have Makefile dependencies to regenerate them in
# the source tree that may be active during a normal build.
# Some other files have such dependencies but do not need to
# be touched because nothing in a build depends on the files
# in question.
for f in ('sysdeps/gnu/errlist.c',
'sysdeps/mach/hurd/bits/errno.h'):
Touch more glibc source files in build-many-glibcs.py. build-many-glibcs.py currently copies the source tree to avoid issues with parallel builds trying to write into it. This copying can result in occasional spurious build failures from bots, when a "git gc" is in progress that changes .git contents while copying is taking place, and it would also be desirable to avoid the need to copy to save on disk space, I/O and memory used in build-many-glibcs.py builds. In preparation for removing the copying, this patch arranges for build-many-glibcs.py to touch more files on checkout so their timestamps do not result in make attempting to rebuild them. Before actually removing the copying, I intend to do further tests to ensure I haven't missed any other such makefile dependencies. This is of course without prejudice to possibly moving more of these files to being generated in the build directory rather than being checked in at all, where that can be done using build tools already required for the build. For sysdeps files (installed and otherwise) it would be necessary to make sure this does not affect the search ordering, for headers used in the build it would be necessary to ensure they are generated early enough, and for errlist.c there may be dual licensing reasons for keeping it checked in. Tested that a checkout with build-many-glibcs.py does touch the expected files and that a glibcs build for aarch64-linux-gnu succeeds. * scripts/build-many-glibcs.py (Context.fix_glibc_timestamps): Touch additional files.
2018-11-27 07:52:43 +08:00
to_touch = os.path.join(srcdir, f)
subprocess.run(['touch', '-c', to_touch], check=True)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
for dirpath, dirnames, filenames in os.walk(srcdir):
for f in filenames:
if (f == 'configure' or
f == 'preconfigure' or
f.endswith('-kw.h')):
to_touch = os.path.join(dirpath, f)
subprocess.run(['touch', to_touch], check=True)
def gcc_checkout(self, svn_url, update):
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
"""Check out GCC from SVN. Return the revision number."""
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if not update:
subprocess.run(['svn', 'co', '-q', svn_url,
self.component_srcdir('gcc')], check=True)
subprocess.run(['contrib/gcc_update', '--silent'],
cwd=self.component_srcdir('gcc'), check=True)
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
r = subprocess.run(['svnversion', self.component_srcdir('gcc')],
stdout=subprocess.PIPE,
check=True, universal_newlines=True).stdout
return r.rstrip()
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def checkout_tar(self, component, version, update):
"""Check out the given version of the given component from a
tarball."""
if update:
return
url_map = {'binutils': 'https://ftp.gnu.org/gnu/binutils/binutils-%(version)s.tar.bz2',
'gcc': 'https://ftp.gnu.org/gnu/gcc/gcc-%(version)s/gcc-%(version)s.tar.gz',
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'gmp': 'https://ftp.gnu.org/gnu/gmp/gmp-%(version)s.tar.xz',
'linux': 'https://www.kernel.org/pub/linux/kernel/v%(major)s.x/linux-%(version)s.tar.xz',
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'mpc': 'https://ftp.gnu.org/gnu/mpc/mpc-%(version)s.tar.gz',
'mpfr': 'https://ftp.gnu.org/gnu/mpfr/mpfr-%(version)s.tar.xz',
'mig': 'https://ftp.gnu.org/gnu/mig/mig-%(version)s.tar.bz2',
'gnumach': 'https://ftp.gnu.org/gnu/gnumach/gnumach-%(version)s.tar.bz2',
'hurd': 'https://ftp.gnu.org/gnu/hurd/hurd-%(version)s.tar.bz2'}
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if component not in url_map:
print('error: component %s coming from tarball' % component)
exit(1)
version_major = version.split('.')[0]
url = url_map[component] % {'version': version, 'major': version_major}
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
filename = os.path.join(self.srcdir, url.split('/')[-1])
response = urllib.request.urlopen(url)
data = response.read()
with open(filename, 'wb') as f:
f.write(data)
subprocess.run(['tar', '-C', self.srcdir, '-x', '-f', filename],
check=True)
os.rename(os.path.join(self.srcdir, '%s-%s' % (component, version)),
self.component_srcdir(component))
os.remove(filename)
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
def load_build_state_json(self):
"""Load information about the state of previous builds."""
if os.access(self.build_state_json, os.F_OK):
with open(self.build_state_json, 'r') as f:
self.build_state = json.load(f)
else:
self.build_state = {}
for k in ('host-libraries', 'compilers', 'glibcs'):
if k not in self.build_state:
self.build_state[k] = {}
if 'build-time' not in self.build_state[k]:
self.build_state[k]['build-time'] = ''
if 'build-versions' not in self.build_state[k]:
self.build_state[k]['build-versions'] = {}
if 'build-results' not in self.build_state[k]:
self.build_state[k]['build-results'] = {}
if 'result-changes' not in self.build_state[k]:
self.build_state[k]['result-changes'] = {}
if 'ever-passed' not in self.build_state[k]:
self.build_state[k]['ever-passed'] = []
def store_build_state_json(self):
"""Store information about the state of previous builds."""
self.store_json(self.build_state, self.build_state_json)
def clear_last_build_state(self, action):
"""Clear information about the state of part of the build."""
# We clear the last build time and versions when starting a
# new build. The results of the last build are kept around,
# as comparison is still meaningful if this build is aborted
# and a new one started.
self.build_state[action]['build-time'] = ''
self.build_state[action]['build-versions'] = {}
self.store_build_state_json()
def update_build_state(self, action, build_time, build_versions):
"""Update the build state after a build."""
build_time = build_time.replace(microsecond=0)
self.build_state[action]['build-time'] = str(build_time)
self.build_state[action]['build-versions'] = build_versions
build_results = {}
for log in self.status_log_list:
with open(log, 'r') as f:
log_text = f.read()
log_text = log_text.rstrip()
m = re.fullmatch('([A-Z]+): (.*)', log_text)
result = m.group(1)
test_name = m.group(2)
assert test_name not in build_results
build_results[test_name] = result
old_build_results = self.build_state[action]['build-results']
self.build_state[action]['build-results'] = build_results
result_changes = {}
all_tests = set(old_build_results.keys()) | set(build_results.keys())
for t in all_tests:
if t in old_build_results:
old_res = old_build_results[t]
else:
old_res = '(New test)'
if t in build_results:
new_res = build_results[t]
else:
new_res = '(Test removed)'
if old_res != new_res:
result_changes[t] = '%s -> %s' % (old_res, new_res)
self.build_state[action]['result-changes'] = result_changes
old_ever_passed = {t for t in self.build_state[action]['ever-passed']
if t in build_results}
new_passes = {t for t in build_results if build_results[t] == 'PASS'}
self.build_state[action]['ever-passed'] = sorted(old_ever_passed |
new_passes)
self.store_build_state_json()
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
def load_bot_config_json(self):
"""Load bot configuration."""
with open(self.bot_config_json, 'r') as f:
self.bot_config = json.load(f)
def part_build_old(self, action, delay):
"""Return whether the last build for a given action was at least a
given number of seconds ago, or does not have a time recorded."""
old_time_str = self.build_state[action]['build-time']
if not old_time_str:
return True
old_time = datetime.datetime.strptime(old_time_str,
'%Y-%m-%d %H:%M:%S')
new_time = datetime.datetime.utcnow()
delta = new_time - old_time
return delta.total_seconds() >= delay
def bot_cycle(self):
"""Run a single round of checkout and builds."""
print('Bot cycle starting %s.' % str(datetime.datetime.utcnow()))
self.load_bot_config_json()
actions = ('host-libraries', 'compilers', 'glibcs')
self.bot_run_self(['--replace-sources'], 'checkout')
self.load_versions_json()
if self.get_script_text() != self.script_text:
print('Script changed, re-execing.')
# On script change, all parts of the build should be rerun.
for a in actions:
self.clear_last_build_state(a)
self.exec_self()
check_components = {'host-libraries': ('gmp', 'mpfr', 'mpc'),
'compilers': ('binutils', 'gcc', 'glibc', 'linux',
'mig', 'gnumach', 'hurd'),
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
'glibcs': ('glibc',)}
must_build = {}
for a in actions:
build_vers = self.build_state[a]['build-versions']
must_build[a] = False
if not self.build_state[a]['build-time']:
must_build[a] = True
old_vers = {}
new_vers = {}
for c in check_components[a]:
if c in build_vers:
old_vers[c] = build_vers[c]
new_vers[c] = {'version': self.versions[c]['version'],
'revision': self.versions[c]['revision']}
if new_vers == old_vers:
print('Versions for %s unchanged.' % a)
else:
print('Versions changed or rebuild forced for %s.' % a)
if a == 'compilers' and not self.part_build_old(
a, self.bot_config['compilers-rebuild-delay']):
print('Not requiring rebuild of compilers this soon.')
else:
must_build[a] = True
if must_build['host-libraries']:
must_build['compilers'] = True
if must_build['compilers']:
must_build['glibcs'] = True
for a in actions:
if must_build[a]:
print('Must rebuild %s.' % a)
self.clear_last_build_state(a)
else:
print('No need to rebuild %s.' % a)
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
if os.access(self.logsdir, os.F_OK):
shutil.rmtree(self.logsdir_old, ignore_errors=True)
shutil.copytree(self.logsdir, self.logsdir_old)
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
for a in actions:
if must_build[a]:
build_time = datetime.datetime.utcnow()
print('Rebuilding %s at %s.' % (a, str(build_time)))
self.bot_run_self([], a)
self.load_build_state_json()
self.bot_build_mail(a, build_time)
print('Bot cycle done at %s.' % str(datetime.datetime.utcnow()))
def bot_build_mail(self, action, build_time):
"""Send email with the results of a build."""
if not ('email-from' in self.bot_config and
'email-server' in self.bot_config and
'email-subject' in self.bot_config and
'email-to' in self.bot_config):
if not self.email_warning:
print("Email not configured, not sending.")
self.email_warning = True
return
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
build_time = build_time.replace(microsecond=0)
subject = (self.bot_config['email-subject'] %
{'action': action,
'build-time': str(build_time)})
results = self.build_state[action]['build-results']
changes = self.build_state[action]['result-changes']
ever_passed = set(self.build_state[action]['ever-passed'])
versions = self.build_state[action]['build-versions']
new_regressions = {k for k in changes if changes[k] == 'PASS -> FAIL'}
all_regressions = {k for k in ever_passed if results[k] == 'FAIL'}
all_fails = {k for k in results if results[k] == 'FAIL'}
if new_regressions:
new_reg_list = sorted(['FAIL: %s' % k for k in new_regressions])
new_reg_text = ('New regressions:\n\n%s\n\n' %
'\n'.join(new_reg_list))
else:
new_reg_text = ''
if all_regressions:
all_reg_list = sorted(['FAIL: %s' % k for k in all_regressions])
all_reg_text = ('All regressions:\n\n%s\n\n' %
'\n'.join(all_reg_list))
else:
all_reg_text = ''
if all_fails:
all_fail_list = sorted(['FAIL: %s' % k for k in all_fails])
all_fail_text = ('All failures:\n\n%s\n\n' %
'\n'.join(all_fail_list))
else:
all_fail_text = ''
if changes:
changes_list = sorted(changes.keys())
changes_list = ['%s: %s' % (changes[k], k) for k in changes_list]
changes_text = ('All changed results:\n\n%s\n\n' %
'\n'.join(changes_list))
else:
changes_text = ''
results_text = (new_reg_text + all_reg_text + all_fail_text +
changes_text)
if not results_text:
results_text = 'Clean build with unchanged results.\n\n'
versions_list = sorted(versions.keys())
versions_list = ['%s: %s (%s)' % (k, versions[k]['version'],
versions[k]['revision'])
for k in versions_list]
versions_text = ('Component versions for this build:\n\n%s\n' %
'\n'.join(versions_list))
body_text = results_text + versions_text
msg = email.mime.text.MIMEText(body_text)
msg['Subject'] = subject
msg['From'] = self.bot_config['email-from']
msg['To'] = self.bot_config['email-to']
msg['Message-ID'] = email.utils.make_msgid()
msg['Date'] = email.utils.format_datetime(datetime.datetime.utcnow())
with smtplib.SMTP(self.bot_config['email-server']) as s:
s.send_message(msg)
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
def bot_run_self(self, opts, action, check=True):
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
"""Run a copy of this script with given options."""
cmd = [sys.executable, sys.argv[0], '--keep=none',
'-j%d' % self.parallelism]
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
if self.full_gcc:
cmd.append('--full-gcc')
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
cmd.extend(opts)
cmd.extend([self.topdir, action])
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
sys.stdout.flush()
subprocess.run(cmd, check=check)
def bot(self):
"""Run repeated rounds of checkout and builds."""
while True:
self.load_bot_config_json()
if not self.bot_config['run']:
print('Bot exiting by request.')
exit(0)
self.bot_run_self([], 'bot-cycle', check=False)
self.load_bot_config_json()
if not self.bot_config['run']:
print('Bot exiting by request.')
exit(0)
time.sleep(self.bot_config['delay'])
if self.get_script_text() != self.script_text:
print('Script changed, bot re-execing.')
self.exec_self()
Add build-many-glibcs.py bot-cycle action. This patch continues the process of setting up build-many-glibcs.py to run as a bot monitoring for and reporting on build issues by adding a bot-cycle action to the script. When this action is used, it will run the checkout action (re-execing itself if it was changed by that action), then rebuild whichever of host-libraries, compilers, glibcs should be rebuilt based on changed versions, time elapsed and state of previous builds. Email is sent with the results of the build (for each build action done). The rebuild logic is: if previous build time or versions aren't recorded, rebuild that component. If the script has changed, rebuild everything. If any relevant component version has changed, rebuild, except for not rebuilding compilers if the time indicated in the bot configuration has not passed since the last build of the compilers. If one piece is rebuilt then rebuild subsequent pieces as well. Using bot-cycle requires a configuration file bot-config.json in the toplevel directory used by build-many-glibcs.py. It might contain e.g. { "compilers-rebuild-delay": 604800, "email-from": "Example Name <user@example.org>", "email-server": "localhost", "email-subject": "GCC 6 %(action)s %(build-time)s build results", "email-to": "libc-testresults@sourceware.org" } My next intended step is adding a further action "bot" which loops running bot-cycle then sleeping for an amount of time given in bot-config.json. Then I'll set up a bot using that action (building with GCC 6 branch; a bot using GCC mainline may wait until the SH out-of-memory issues <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78460> are fixed; I expect the bot to mail to me until it seems ready to switch to mailing to gcc-testresults). * scripts/build-many-glibcs.py: Add bot-cycle to usage message. Import email.mime.text, email.utils and smtplib modules. (Context.__init__): Initialize self.bot_config_json. (Context.run_builds): Handle bot-cycle action. (Context.load_bot_config_json): New function. (Context.part_build_old): Likewise. (Context.bot_cycle): Likewise. (Context.bot_build_mail): Likewise. (Context.bot_run_self): Likewise. (get_parser): Allow bot-cycle action.
2016-12-01 02:56:37 +08:00
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
class Config(object):
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""A configuration for building a compiler and associated libraries."""
def __init__(self, ctx, arch, os_name, variant=None, gcc_cfg=None,
first_gcc_cfg=None, glibcs=None, extra_glibcs=None):
"""Initialize a Config object."""
self.ctx = ctx
self.arch = arch
self.os = os_name
self.variant = variant
if variant is None:
self.name = '%s-%s' % (arch, os_name)
else:
self.name = '%s-%s-%s' % (arch, os_name, variant)
self.triplet = '%s-glibc-%s' % (arch, os_name)
if gcc_cfg is None:
self.gcc_cfg = []
else:
self.gcc_cfg = gcc_cfg
if first_gcc_cfg is None:
self.first_gcc_cfg = []
else:
self.first_gcc_cfg = first_gcc_cfg
if glibcs is None:
glibcs = [{'variant': variant}]
if extra_glibcs is None:
extra_glibcs = []
glibcs = [Glibc(self, **g) for g in glibcs]
extra_glibcs = [Glibc(self, **g) for g in extra_glibcs]
self.all_glibcs = glibcs + extra_glibcs
self.compiler_glibcs = glibcs
self.installdir = ctx.compiler_installdir(self.name)
self.bindir = ctx.compiler_bindir(self.name)
self.sysroot = ctx.compiler_sysroot(self.name)
self.builddir = os.path.join(ctx.builddir, 'compilers', self.name)
self.logsdir = os.path.join(ctx.logsdir, 'compilers', self.name)
def component_builddir(self, component):
"""Return the directory to use for a (non-glibc) build."""
return self.ctx.component_builddir('compilers', self.name, component)
def build(self):
"""Generate commands to build this compiler."""
self.ctx.remove_recreate_dirs(self.installdir, self.builddir,
self.logsdir)
cmdlist = CommandList('compilers-%s' % self.name, self.ctx.keep)
cmdlist.add_command('check-host-libraries',
['test', '-f',
os.path.join(self.ctx.host_libraries_installdir,
'ok')])
cmdlist.use_path(self.bindir)
self.build_cross_tool(cmdlist, 'binutils', 'binutils',
['--disable-gdb',
'--disable-libdecnumber',
'--disable-readline',
'--disable-sim'])
if self.os.startswith('linux'):
self.install_linux_headers(cmdlist)
self.build_gcc(cmdlist, True)
if self.os == 'gnu':
self.install_gnumach_headers(cmdlist)
self.build_cross_tool(cmdlist, 'mig', 'mig')
self.install_hurd_headers(cmdlist)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
for g in self.compiler_glibcs:
cmdlist.push_subdesc('glibc')
cmdlist.push_subdesc(g.name)
g.build_glibc(cmdlist, True)
cmdlist.pop_subdesc()
cmdlist.pop_subdesc()
self.build_gcc(cmdlist, False)
cmdlist.add_command('done', ['touch',
os.path.join(self.installdir, 'ok')])
self.ctx.add_makefile_cmdlist('compilers-%s' % self.name, cmdlist,
self.logsdir)
def build_cross_tool(self, cmdlist, tool_src, tool_build, extra_opts=None):
"""Build one cross tool."""
srcdir = self.ctx.component_srcdir(tool_src)
builddir = self.component_builddir(tool_build)
cmdlist.push_subdesc(tool_build)
cmdlist.create_use_dir(builddir)
cfg_cmd = [os.path.join(srcdir, 'configure'),
'--prefix=%s' % self.installdir,
'--build=%s' % self.ctx.build_triplet,
'--host=%s' % self.ctx.build_triplet,
'--target=%s' % self.triplet,
'--with-sysroot=%s' % self.sysroot]
if extra_opts:
cfg_cmd.extend(extra_opts)
cmdlist.add_command('configure', cfg_cmd)
cmdlist.add_command('build', ['make'])
# Parallel "make install" for GCC has race conditions that can
# cause it to fail; see
# <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=42980>. Such
# problems are not known for binutils, but doing the
# installation in parallel within a particular toolchain build
# (as opposed to installation of one toolchain from
# build-many-glibcs.py running in parallel to the installation
# of other toolchains being built) is not known to be
# significantly beneficial, so it is simplest just to disable
# parallel install for cross tools here.
cmdlist.add_command('install', ['make', '-j1', 'install'])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
cmdlist.cleanup_dir()
cmdlist.pop_subdesc()
def install_linux_headers(self, cmdlist):
"""Install Linux kernel headers."""
arch_map = {'aarch64': 'arm64',
'alpha': 'alpha',
'arm': 'arm',
Add C-SKY port This patch add two abi combinations support for C-SKY ABIV2: soft-float little endian, hard float little endian. C-SKY ABI manual and architecture user guide are available from: https://github.com/c-sky/csky-doc * config.h.in (CSKYABI, CSKY_HARD_FLOAT): New Define. * scripts/build-many-glibcs.py: Add C-SKY targets. * sysdeps/csky/Implies: New file. * sysdeps/csky/Makefile: Likewise. * sysdeps/csky/abiv2/__longjmp.S: Likewise. * sysdeps/csky/abiv2/csky-mcount.S: Likewise. * sysdeps/csky/abiv2/dl-trampoline.S: Likewise. * sysdeps/csky/abiv2/memcmp.S: Likewise. * sysdeps/csky/abiv2/memcpy.S: Likewise. * sysdeps/csky/abiv2/memmove.S: Likewise. * sysdeps/csky/abiv2/memset.S: Likewise. * sysdeps/csky/abiv2/setjmp.S: Likewise. * sysdeps/csky/abiv2/start.S: Likewise. * sysdeps/csky/abiv2/strcmp.S: Likewise. * sysdeps/csky/abiv2/strcpy.S: Likewise. * sysdeps/csky/abiv2/strlen.S: Likewise. * sysdeps/csky/abiv2/tls-macros.h: Likewise. * sysdeps/csky/abort-instr.h: Likewise. * sysdeps/csky/atomic-machine.h: Likewise. * sysdeps/csky/bits/endian.h: Likewise. * sysdeps/csky/bits/fenv.h: Likewise. * sysdeps/csky/bits/link.h: Likewise. * sysdeps/csky/bits/setjmp.h: Likewise. * sysdeps/csky/bsd-_setjmp.S: Likewise. * sysdeps/csky/bsd-setjmp.S: Likewise. * sysdeps/csky/configure: Likewise. * sysdeps/csky/configure.ac: Likewise. * sysdeps/csky/dl-machine.h: Likewise. * sysdeps/csky/dl-procinfo.c: Likewise. * sysdeps/csky/dl-procinfo.h: Likewise. * sysdeps/csky/dl-sysdep.h: Likewise. * sysdeps/csky/dl-tls.h: Likewise. * sysdeps/csky/fpu/fclrexcpt.c: Likewise. * sysdeps/csky/fpu/fedisblxcpt.c: Likewise. * sysdeps/csky/fpu/feenablxcpt.c: Likewise. * sysdeps/csky/fpu/fegetenv.c: Likewise. * sysdeps/csky/fpu/fegetexcept.c: Likewise. * sysdeps/csky/fpu/fegetmode.c: Likewise. * sysdeps/csky/fpu/fegetround.c: Likewise. * sysdeps/csky/fpu/feholdexcpt.c: Likewise. * sysdeps/csky/fpu/fenv_libc.h: Likewise. * sysdeps/csky/fpu/fenv_private.h: Likewise. * sysdeps/csky/fpu/fesetenv.c: Likewise. * sysdeps/csky/fpu/fesetexcept.c: Likewise. * sysdeps/csky/fpu/fesetmode.c: Likewise. * sysdeps/csky/fpu/fesetround.c: Likewise. * sysdeps/csky/fpu/feupdateenv.c: Likewise. * sysdeps/csky/fpu/fgetexcptflg.c: Likewise. * sysdeps/csky/fpu/fix-fp-int-convert-overflow.h: Likewise. * sysdeps/csky/fpu/fraiseexcpt.c: Likewise. * sysdeps/csky/fpu/fsetexcptflg.c: Likewise. * sysdeps/csky/fpu/ftestexcept.c: Likewise. * sysdeps/csky/fpu/libm-test-ulps: Likewise. * sysdeps/csky/fpu/libm-test-ulps-name: Likewise. * sysdeps/csky/fpu_control.h: Likewise. * sysdeps/csky/gccframe.h: Likewise. * sysdeps/csky/jmpbuf-unwind.h: Likewise. * sysdeps/csky/ldsodefs.h: Likewise. * sysdeps/csky/libc-tls.c: Likewise. * sysdeps/csky/linkmap.h: Likewise. * sysdeps/csky/machine-gmon.h: Likewise. * sysdeps/csky/memusage.h: Likewise. * sysdeps/csky/nofpu/Implies: Likewise. * sysdeps/csky/nofpu/libm-test-ulps: Likewise. * sysdeps/csky/nofpu/libm-test-ulps-name: Likewise. * sysdeps/csky/nptl/Makefile: Likewise. * sysdeps/csky/nptl/bits/pthreadtypes-arch.h: Likewise. * sysdeps/csky/nptl/bits/semaphore.h: Likewise. * sysdeps/csky/nptl/pthread-offsets.h: Likewise. * sysdeps/csky/nptl/pthreaddef.h: Likewise. * sysdeps/csky/nptl/tcb-offsets.sym: Likewise. * sysdeps/csky/nptl/tls.h: Likewise. * sysdeps/csky/preconfigure: Likewise. * sysdeps/csky/sfp-machine.h: Likewise. * sysdeps/csky/sotruss-lib.c: Likewise. * sysdeps/csky/stackinfo.h: Likewise. * sysdeps/csky/sysdep.h: Likewise. * sysdeps/csky/tininess.h: Likewise. * sysdeps/csky/tst-audit.h: Likewise. * sysdeps/unix/sysv/linux/csky/Implies: Likewise. * sysdeps/unix/sysv/linux/csky/Makefile: Likewise. * sysdeps/unix/sysv/linux/csky/Versions: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/____longjmp_chk.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/clone.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/getcontext.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/setcontext.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/swapcontext.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/syscall.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/sysdep.S: Likewise. * sysdeps/unix/sysv/linux/csky/abiv2/ucontext_i.sym: Likewise. * sysdeps/unix/sysv/linux/csky/bits/procfs.h: Likewise. * sysdeps/unix/sysv/linux/csky/bits/shmlba.h: Likewise. * sysdeps/unix/sysv/linux/csky/c++-types.data: Likewise. * sysdeps/unix/sysv/linux/csky/configure: Likewise. * sysdeps/unix/sysv/linux/csky/configure.ac: Likewise. * sysdeps/unix/sysv/linux/csky/ipc_priv.h: Likewise. * sysdeps/unix/sysv/linux/csky/jmp_buf-macros.h: Likewise. * sysdeps/unix/sysv/linux/csky/kernel-features.h: Likewise. * sysdeps/unix/sysv/linux/csky/ld.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/ldconfig.h: Likewise. * sysdeps/unix/sysv/linux/csky/libBrokenLocale.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libanl.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libc.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libcrypt.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libdl.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libm.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libpthread.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libresolv.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/librt.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libthread_db.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/libutil.abilist: Likewise. * sysdeps/unix/sysv/linux/csky/localplt.data: Likewise. * sysdeps/unix/sysv/linux/csky/makecontext.c: Likewise. * sysdeps/unix/sysv/linux/csky/profil-counter.h: Likewise. * sysdeps/unix/sysv/linux/csky/pt-vfork.S: Likewise. * sysdeps/unix/sysv/linux/csky/register-dump.h: Likewise. * sysdeps/unix/sysv/linux/csky/shlib-versions: Likewise. * sysdeps/unix/sysv/linux/csky/sigcontextinfo.h: Likewise. * sysdeps/unix/sysv/linux/csky/sys/cachectl.h: Likewise. * sysdeps/unix/sysv/linux/csky/sys/ucontext.h: Likewise. * sysdeps/unix/sysv/linux/csky/sys/user.h: Likewise. * sysdeps/unix/sysv/linux/csky/syscalls.list: Likewise. * sysdeps/unix/sysv/linux/csky/sysdep.h: Likewise.
2018-12-21 09:48:04 +08:00
'csky': 'csky',
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'hppa': 'parisc',
'i486': 'x86',
'i586': 'x86',
'i686': 'x86',
'i786': 'x86',
'ia64': 'ia64',
'm68k': 'm68k',
'microblaze': 'microblaze',
'mips': 'mips',
'nios2': 'nios2',
'powerpc': 'powerpc',
's390': 's390',
'riscv32': 'riscv',
'riscv64': 'riscv',
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'sh': 'sh',
'sparc': 'sparc',
'x86_64': 'x86'}
linux_arch = None
for k in arch_map:
if self.arch.startswith(k):
linux_arch = arch_map[k]
break
assert linux_arch is not None
srcdir = self.ctx.component_srcdir('linux')
builddir = self.component_builddir('linux')
headers_dir = os.path.join(self.sysroot, 'usr')
cmdlist.push_subdesc('linux')
cmdlist.create_use_dir(builddir)
cmdlist.add_command('install-headers',
['make', '-C', srcdir, 'O=%s' % builddir,
'ARCH=%s' % linux_arch,
'INSTALL_HDR_PATH=%s' % headers_dir,
'headers_install'])
cmdlist.cleanup_dir()
cmdlist.pop_subdesc()
def install_gnumach_headers(self, cmdlist):
"""Install GNU Mach headers."""
srcdir = self.ctx.component_srcdir('gnumach')
builddir = self.component_builddir('gnumach')
cmdlist.push_subdesc('gnumach')
cmdlist.create_use_dir(builddir)
cmdlist.add_command('configure',
[os.path.join(srcdir, 'configure'),
'--build=%s' % self.ctx.build_triplet,
'--host=%s' % self.triplet,
'--prefix=',
'CC=%s-gcc -nostdlib' % self.triplet])
cmdlist.add_command('install', ['make', 'DESTDIR=%s' % self.sysroot,
'install-data'])
cmdlist.cleanup_dir()
cmdlist.pop_subdesc()
def install_hurd_headers(self, cmdlist):
"""Install Hurd headers."""
srcdir = self.ctx.component_srcdir('hurd')
builddir = self.component_builddir('hurd')
cmdlist.push_subdesc('hurd')
cmdlist.create_use_dir(builddir)
cmdlist.add_command('configure',
[os.path.join(srcdir, 'configure'),
'--build=%s' % self.ctx.build_triplet,
'--host=%s' % self.triplet,
'--prefix=',
'--disable-profile', '--without-parted',
'CC=%s-gcc -nostdlib' % self.triplet])
cmdlist.add_command('install', ['make', 'prefix=%s' % self.sysroot,
'no_deps=t', 'install-headers'])
cmdlist.cleanup_dir()
cmdlist.pop_subdesc()
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def build_gcc(self, cmdlist, bootstrap):
"""Build GCC."""
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
# libssp is of little relevance with glibc's own stack
# checking support. libcilkrts does not support GNU/Hurd (and
# has been removed in GCC 8, so --disable-libcilkrts can be
# removed once glibc no longer supports building with older
# GCC versions).
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
cfg_opts = list(self.gcc_cfg)
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
cfg_opts += ['--disable-libssp', '--disable-libcilkrts']
host_libs = self.ctx.host_libraries_installdir
cfg_opts += ['--with-gmp=%s' % host_libs,
'--with-mpfr=%s' % host_libs,
'--with-mpc=%s' % host_libs]
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if bootstrap:
tool_build = 'gcc-first'
# Building a static-only, C-only compiler that is
# sufficient to build glibc. Various libraries and
# features that may require libc headers must be disabled.
# When configuring with a sysroot, --with-newlib is
# required to define inhibit_libc (to stop some parts of
# libgcc including libc headers); --without-headers is not
# sufficient.
cfg_opts += ['--enable-languages=c', '--disable-shared',
'--disable-threads',
'--disable-libatomic',
'--disable-decimal-float',
'--disable-libffi',
'--disable-libgomp',
'--disable-libitm',
'--disable-libmpx',
'--disable-libquadmath',
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
'--disable-libsanitizer',
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'--without-headers', '--with-newlib',
'--with-glibc-version=%s' % self.ctx.glibc_version
]
cfg_opts += self.first_gcc_cfg
else:
tool_build = 'gcc'
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
# libsanitizer commonly breaks because of glibc header
# changes, or on unusual targets.
if not self.ctx.full_gcc:
cfg_opts += ['--disable-libsanitizer']
langs = 'all' if self.ctx.full_gcc else 'c,c++'
cfg_opts += ['--enable-languages=%s' % langs,
'--enable-shared', '--enable-threads']
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
self.build_cross_tool(cmdlist, 'gcc', tool_build, cfg_opts)
class Glibc(object):
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""A configuration for building glibc."""
def __init__(self, compiler, arch=None, os_name=None, variant=None,
cfg=None, ccopts=None):
"""Initialize a Glibc object."""
self.ctx = compiler.ctx
self.compiler = compiler
if arch is None:
self.arch = compiler.arch
else:
self.arch = arch
if os_name is None:
self.os = compiler.os
else:
self.os = os_name
self.variant = variant
if variant is None:
self.name = '%s-%s' % (self.arch, self.os)
else:
self.name = '%s-%s-%s' % (self.arch, self.os, variant)
self.triplet = '%s-glibc-%s' % (self.arch, self.os)
if cfg is None:
self.cfg = []
else:
self.cfg = cfg
self.ccopts = ccopts
def tool_name(self, tool):
"""Return the name of a cross-compilation tool."""
ctool = '%s-%s' % (self.compiler.triplet, tool)
if self.ccopts and (tool == 'gcc' or tool == 'g++'):
ctool = '%s %s' % (ctool, self.ccopts)
return ctool
def build(self):
"""Generate commands to build this glibc."""
builddir = self.ctx.component_builddir('glibcs', self.name, 'glibc')
installdir = self.ctx.glibc_installdir(self.name)
logsdir = os.path.join(self.ctx.logsdir, 'glibcs', self.name)
self.ctx.remove_recreate_dirs(installdir, builddir, logsdir)
cmdlist = CommandList('glibcs-%s' % self.name, self.ctx.keep)
cmdlist.add_command('check-compilers',
['test', '-f',
os.path.join(self.compiler.installdir, 'ok')])
cmdlist.use_path(self.compiler.bindir)
self.build_glibc(cmdlist, False)
self.ctx.add_makefile_cmdlist('glibcs-%s' % self.name, cmdlist,
logsdir)
def build_glibc(self, cmdlist, for_compiler):
"""Generate commands to build this glibc, either as part of a compiler
build or with the bootstrapped compiler (and in the latter case, run
tests as well)."""
srcdir = self.ctx.component_srcdir('glibc')
if for_compiler:
builddir = self.ctx.component_builddir('compilers',
self.compiler.name, 'glibc',
self.name)
installdir = self.compiler.sysroot
else:
builddir = self.ctx.component_builddir('glibcs', self.name,
'glibc')
installdir = self.ctx.glibc_installdir(self.name)
cmdlist.create_use_dir(builddir)
use_usr = self.os != 'gnu'
prefix = '/usr' if use_usr else ''
cfg_cmd = [os.path.join(srcdir, 'configure'),
'--prefix=%s' % prefix,
'--enable-profile',
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
'--build=%s' % self.ctx.build_triplet,
'--host=%s' % self.triplet,
'CC=%s' % self.tool_name('gcc'),
'CXX=%s' % self.tool_name('g++'),
'AR=%s' % self.tool_name('ar'),
'AS=%s' % self.tool_name('as'),
'LD=%s' % self.tool_name('ld'),
'NM=%s' % self.tool_name('nm'),
'OBJCOPY=%s' % self.tool_name('objcopy'),
'OBJDUMP=%s' % self.tool_name('objdump'),
'RANLIB=%s' % self.tool_name('ranlib'),
'READELF=%s' % self.tool_name('readelf'),
'STRIP=%s' % self.tool_name('strip')]
if self.os == 'gnu':
cfg_cmd += ['MIG=%s' % self.tool_name('mig')]
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
cfg_cmd += self.cfg
cmdlist.add_command('configure', cfg_cmd)
cmdlist.add_command('build', ['make'])
cmdlist.add_command('install', ['make', 'install',
'install_root=%s' % installdir])
# GCC uses paths such as lib/../lib64, so make sure lib
# directories always exist.
mkdir_cmd = ['mkdir', '-p',
os.path.join(installdir, 'lib')]
if use_usr:
mkdir_cmd += [os.path.join(installdir, 'usr', 'lib')]
cmdlist.add_command('mkdir-lib', mkdir_cmd)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
if not for_compiler:
if self.ctx.strip:
cmdlist.add_command('strip',
['sh', '-c',
('%s $(find %s/lib* -name "*.so")' %
(self.tool_name('strip'), installdir))])
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
cmdlist.add_command('check', ['make', 'check'])
cmdlist.add_command('save-logs', [self.ctx.save_logs],
always_run=True)
cmdlist.cleanup_dir()
class Command(object):
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""A command run in the build process."""
def __init__(self, desc, num, dir, path, command, always_run=False):
"""Initialize a Command object."""
self.dir = dir
self.path = path
self.desc = desc
trans = str.maketrans({' ': '-'})
self.logbase = '%03d-%s' % (num, desc.translate(trans))
self.command = command
self.always_run = always_run
@staticmethod
def shell_make_quote_string(s):
"""Given a string not containing a newline, quote it for use by the
shell and make."""
assert '\n' not in s
if re.fullmatch('[]+,./0-9@A-Z_a-z-]+', s):
return s
strans = str.maketrans({"'": "'\\''"})
s = "'%s'" % s.translate(strans)
mtrans = str.maketrans({'$': '$$'})
return s.translate(mtrans)
@staticmethod
def shell_make_quote_list(l, translate_make):
"""Given a list of strings not containing newlines, quote them for use
by the shell and make, returning a single string. If translate_make
is true and the first string is 'make', change it to $(MAKE)."""
l = [Command.shell_make_quote_string(s) for s in l]
if translate_make and l[0] == 'make':
l[0] = '$(MAKE)'
return ' '.join(l)
def shell_make_quote(self):
"""Return this command quoted for the shell and make."""
return self.shell_make_quote_list(self.command, True)
class CommandList(object):
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
"""A list of commands run in the build process."""
def __init__(self, desc, keep):
"""Initialize a CommandList object."""
self.cmdlist = []
self.dir = None
self.path = None
self.desc = [desc]
self.keep = keep
def desc_txt(self, desc):
"""Return the description to use for a command."""
return '%s %s' % (' '.join(self.desc), desc)
def use_dir(self, dir):
"""Set the default directory for subsequent commands."""
self.dir = dir
def use_path(self, path):
"""Set a directory to be prepended to the PATH for subsequent
commands."""
self.path = path
def push_subdesc(self, subdesc):
"""Set the default subdescription for subsequent commands (e.g., the
name of a component being built, within the series of commands
building it)."""
self.desc.append(subdesc)
def pop_subdesc(self):
"""Pop a subdescription from the list of descriptions."""
self.desc.pop()
def create_use_dir(self, dir):
"""Remove and recreate a directory and use it for subsequent
commands."""
self.add_command_dir('rm', None, ['rm', '-rf', dir])
self.add_command_dir('mkdir', None, ['mkdir', '-p', dir])
self.use_dir(dir)
def add_command_dir(self, desc, dir, command, always_run=False):
"""Add a command to run in a given directory."""
cmd = Command(self.desc_txt(desc), len(self.cmdlist), dir, self.path,
command, always_run)
self.cmdlist.append(cmd)
def add_command(self, desc, command, always_run=False):
"""Add a command to run in the default directory."""
cmd = Command(self.desc_txt(desc), len(self.cmdlist), self.dir,
self.path, command, always_run)
self.cmdlist.append(cmd)
def cleanup_dir(self, desc='cleanup', dir=None):
"""Clean up a build directory. If no directory is specified, the
default directory is cleaned up and ceases to be the default
directory."""
if dir is None:
dir = self.dir
self.use_dir(None)
if self.keep != 'all':
self.add_command_dir(desc, None, ['rm', '-rf', dir],
always_run=(self.keep == 'none'))
def makefile_commands(self, wrapper, logsdir):
"""Return the sequence of commands in the form of text for a Makefile.
The given wrapper script takes arguments: base of logs for
previous command, or empty; base of logs for this command;
description; directory; PATH addition; the command itself."""
# prev_base is the base of the name for logs of the previous
# command that is not always-run (that is, a build command,
# whose failure should stop subsequent build commands from
# being run, as opposed to a cleanup command, which is run
# even if previous commands failed).
prev_base = ''
cmds = []
for c in self.cmdlist:
ctxt = c.shell_make_quote()
if prev_base and not c.always_run:
prev_log = os.path.join(logsdir, prev_base)
else:
prev_log = ''
this_log = os.path.join(logsdir, c.logbase)
if not c.always_run:
prev_base = c.logbase
if c.dir is None:
dir = ''
else:
dir = c.dir
if c.path is None:
path = ''
else:
path = c.path
prelims = [wrapper, prev_log, this_log, c.desc, dir, path]
prelim_txt = Command.shell_make_quote_list(prelims, False)
cmds.append('\t@%s %s' % (prelim_txt, ctxt))
return '\n'.join(cmds)
Make build-many-glibcs.py store more information about builds. This patch makes build-many-glibcs.py store information about builds in JSON format. This is part of preparing it for use in a bot checking for regressions. The information stored is: time of last build (of host-libraries, compilers or glibcs); versions of components used in the last build (for compilers, host library versions are properly copied from those used for the previous host-libraries build, and for glibcs, component versions other than that of glibc are similarly copied from the last compilers build); PASS/FAIL/UNRESOLVED results of the individual build steps; a list of changed results; a list of tests (that are still run at all) that have ever been recorded to PASS. The first pieces of information are intended to be used by a bot to decide whether a rebuild is appropriate (based on some combination of elapsed time and changes to versions; a bot might want to rebuild glibcs if there had been any change but only rebuild compilers after enough time had elapsed, for example). All the information is intended to be used in generating mails with results information. This state is specifically for full builds (no individual configs for building compilers or glibcs specified). If individual configs are specified, build-time and build-versions information is cleared (since it will no longer accurately reflect the install directory contents), while the other information is left unchanged. This reflects the motivation of providing information for a bot checking for regressions; the contents of build-state.json in a tree used for manual builds that may be only for some configurations are not particularly important. * scripts/build-many-glibcs.py: Import datetime module. (Context.__init__): Load JSON build state. Initialize list of status logs. (Context.run_builds): Update saved build state. (Context.add_makefile_cmdlist): Update list of status logs. (Context.load_build_state_json): New function. (Context.store_build_state_json): Likewise. (Context.clear_last_build_state): Likewise. (Context.update_build_state): Likewise. (CommandList.status_logs): Likewise.
2016-11-26 08:10:24 +08:00
def status_logs(self, logsdir):
"""Return the list of log files with command status."""
return [os.path.join(logsdir, '%s-status.txt' % c.logbase)
for c in self.cmdlist]
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
def get_parser():
"""Return an argument parser for this module."""
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('-j', dest='parallelism',
help='Run this number of jobs in parallel',
type=int, default=os.cpu_count())
parser.add_argument('--keep', dest='keep',
help='Whether to keep all build directories, '
'none or only those from failed builds',
default='none', choices=('none', 'all', 'failed'))
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
parser.add_argument('--replace-sources', action='store_true',
help='Remove and replace source directories '
'with the wrong version of a component')
parser.add_argument('--strip', action='store_true',
help='Strip installed glibc libraries')
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
parser.add_argument('--full-gcc', action='store_true',
help='Build GCC with all languages and libsanitizer')
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
parser.add_argument('topdir',
help='Toplevel working directory')
parser.add_argument('action',
help='What to do',
Make build-many-glibcs.py support running as a bot. This patch makes build-many-glibcs.py support a "bot" action, for repeatedly running a checkout and build cycle. Two new configuration variables are used in bot-config.json. "delay" indicates the time to sleep after each bot-cycle round (regardless of whether that round actually ran any builds); "run" is a boolean, which is false if the bot should just exit (the point of this is that you can edit bot-config.json to set this to false to cause a running bot to exit cleanly between builds) and true if the bot should run. The bot does not exit if the bot-cycle process exits with error status (that can occur when sourceware's load limiting means anonymous version control access fails, for example), just sleeps until it's time to try again. The script is changed to flush stdout before running a subprocess in bot-cycle, so that when output is redirected (as expected for a bot) the status messages from bot-cycle appear in their proper position in its redirected output relative to the output from the subprocesses run, and to copy the logs directory before running builds in bot-cycle so that the logs from at least one complete build are always available for looking at how something failed, even while the next build is running. * scripts/build-many-glibcs.py: Add bot to usage message. Import time module. (Context.__init__): Initialize self.logsdir_old. (Context.run_builds): Handle bot action. (Context.bot_cycle): Copy logs directory before running builds. (Context.bot_run_self): Take argument for whether to check subprocess result. Flush stdout before running subprocess. (Context.bot): New function. (get_parser): Allow bot action.
2016-12-01 08:09:25 +08:00
choices=('checkout', 'bot-cycle', 'bot',
'host-libraries', 'compilers', 'glibcs'))
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
parser.add_argument('configs',
help='Versions to check out or configurations to build',
nargs='*')
return parser
def main(argv):
"""The main entry point."""
parser = get_parser()
opts = parser.parse_args(argv)
topdir = os.path.abspath(opts.topdir)
Make build-many-glibcs.py track component versions requested and used. This patch makes build-many-glibcs.py record the component versions checked out, and whether those versions were explicitly requested or defaults, in a file versions.json in the source directory. The preferred version of a component is the first of: one explicitly specified on the command line; one explicitly specified on the command line in a previous run of build-many-glibcs.py; a default version for that component. Thus, once you've run build-many-glibcs.py checkout once with the updated script (making sure to specify the right versions of any components previously checked out with a non-default version), in future you can just run it without version specifiers and it will know when a default-version component has changed its default version and so should be checked out again. Because you might have local changes and not want a default-version component checkout replaced, you need to pass the --replace-sources option to allow the script to delete and replace a component source directory automatically; otherwise, it will give an error if a version has changed. The script does not try to change branches of git or SVN checkouts without checking out from scratch; if the version number requested has changed and --replace-sources is used, the relevant source directory will be removed completely and a new one checked out from scratch. Apart from allowing automatic updates of components with default versions, this also facilitates bots reporting on the versions used in a given build. versions.json contains not just information on the version number and whether that was requested explicitly, but also git or SVN revision information intended to be used in email reports from bots. * scripts/build-many-glibcs.py: Import json module. (Context.__init__): Take replace_sources argument. Load versions.json. (Context.load_versions_json): New function. (Context.store_json): Likewise. (Context.store_versions_json): Likewise. (Context.set_component_version): Likewise. (Context.checkout): Update versions.json. Check for and handle changes of version. Prefer previously explicitly specified version to default version. (Context.checkout_vcs): Return a revision identifier. (Context.git_checkout): Likewise. (Context.gcc_checkout): Likewise. (get_parser): Add --replace-sources option. (main): Pass replace_sources argument to Context call.
2016-11-25 06:25:58 +08:00
ctx = Context(topdir, opts.parallelism, opts.keep, opts.replace_sources,
Add build-many-glibcs.py support for building more GCC libraries. Every so often we get libsanitizer or libgo builds breaking with new glibc because of some change in the glibc headers. glibc's build-many-glibcs.py deliberately disables libsanitizer and GCC languages other than C and C++ because the point is to test glibc and find glibc problems (including problems shown up by new compiler warnings in new GCC), not to test libsanitizer or libgo; if the compiler build fails because of libsanitizer or libgo failing to build, that could hide the existence of new problems in glibc. However, it seems reasonable to have a non-default mode where build-many-glibcs.py does build those additional pieces, which this patch adds. Note that I do not intend to run a build-many-glibcs.py bot with this new option. If people concerned with libsanitizer, libgo or other potentially affected GCC libraries wish to find out about such problems more quickly, they may wish to run such a bot or bots (and to monitor the results and fix issues found - obviously there will be some overlap with issues found by my bots not using that option). Note also that building a non-native Ada compiler requires a sufficiently recent native (or build-x-host, in general) Ada compiler to be used, possibly more or less the same version as being built. That needs to be in the PATH when build-many-glibcs.py --full-gcc is run; the script does not deal with setting up such a compiler (or any of the other host tools needed for building GCC and glibc, beyond the GMP / MPFR / MPC libraries), but perhaps it should, to avoid the need to keep updating such a compiler manually when running a bot. Tested by running build-many-glibcs.py with the new option, with mainline GCC. There are build failures for various configurations, which may be of interest to Go / Ada people even if you're not interested in running such a bot: * mips64 / mips64el (all configuration): ICE building libstdc++, as seen without using the new option <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87156>. * aarch64_be: error building libgo (little-endian aarch64 works fine): version.go:67:13: error: expected ';' or ')' or newline 67 | BigEndian = | ^ version.go:67:3: error: reference to undefined name 'BigEndian' 67 | BigEndian = | ^ * arm (all configurations): error building libgo: /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:29:5: error: reference to undefined name 'randomTrap' 29 | if randomTrap == 0 { | ^ /scratch/jmyers/glibc/many9/src/gcc/libgo/go/internal/syscall/unix/getrandom_linux.go:38:34: error: reference to undefined name 'randomTrap' 38 | r1, _, errno := syscall.Syscall(randomTrap, | ^ What's happening there is, I think, that the arm*b*-*-* case in libgo/configure.ac is wrongly matching arm-glibc-linux-gnueabi with the 'b' in the vendor part, and then something else is failing to handle GOARCH=armbe. Given that you can have configurations with multilibs of both endiannesses, endianness should always be detected by configure.ac, for all architectures, using a compile test of whether __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__, not based on textual matches to the host (= target at top-level) triplet. * armeb (all configurations): error building libada (for some reason the Arm libada configuration seems to do different things for EH for big-endian, which makes no sense to me and doesn't actually work): a-exexpr.adb:87:06: "System.Exceptions.Machine" is not a predefined library unit a-exexpr.adb:87:06: "Ada.Exceptions (body)" depends on "Ada.Exceptions.Exception_Propagation (body)" a-exexpr.adb:87:06: "Ada.Exceptions.Exception_Propagation (body)" depends on "System.Exceptions.Machine (spec)" * hppa: error building libgo (same error as for aarch64_be). * ia64: ICE building libgo. I've filed <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87281> for this. * m68k: ICE in the Go front end building libgo <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84948>. * microblaze, microblazeel, nios2, sh3, sh3eb: build failure in libada for lack of a libada port to those systems (I'm not sure sh3 would actually need anything different from sh4): a-cbdlli.ads:38:14: violation of restriction "No_Finalization" at system.ads:47 * i686-gnu: build failure in libada, might be fixed by the patch attached to <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81103> (not tested): terminals.c:1115:13: fatal error: termio.h: No such file or directory * scripts/build-many-glibcs.py (Context.__init__): Add full_gcc argument. (Config.build_gcc): Use --disable-libsanitizer for first GCC build, but not for second build if --full-gcc. Use --enable-languages=all for second build if --full-gcc. (get_parser): Add --full-gcc option. (main): Update call to Context.
2018-09-11 20:02:28 +08:00
opts.strip, opts.full_gcc, opts.action)
Add script to build many glibc configurations. This patch adds a Python (3.5 or later) script to build many different configurations of glibc, including building the required cross compilers first. It's not intended to change any patch testing requirements, although some people may wish to use it for high-risk patches such as adding warning options (and it can also be used to test building, including compiling tests, for an individual configuration, if e.g. you wish to do such a compilation test of a patch for an architecture it touches). The configurations include all the GNU/Linux ABI variants in <https://sourceware.org/glibc/wiki/ABIList> (although some do not yet build cleanly) and it would be desirable to cover enough other variants e.g. for CPUs using different sysdeps directories to test building each piece of code in glibc at least once. It would also be desirable to extend it to cover Hurd and NaCl, which might best be done by people familiar with those configurations. You call the script as build-many-glibcs.py /some/where thing-to-do <other-arguments> where /some/where is a working directory for the script. It will create and use subdirectories build, install, logs therein. You can use it with thing-to-do being "checkout" to create a subdirectory src therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the sources of those components, or create those directories manually (all except glibc can be symlinks to sources elsewhere). In the checkout case, by default it checks out GCC 6 branch, binutils 2.27 branch, glibc mainline and releases of other components. You can specify <component>-<version> to choose a version to check out, where <version> is "vcs-mainline" or "vcs-<branch>" to check out from version control (only supported for gcc, binutils, glibc) and otherwise a release version number to download and use a tarball; components not specified on the command line have default versions checked out. If you rerun "checkout" (with the same version specifications) it will update checkouts from version control, but will not detect cases where the location something is expected to be checked out from has changed. Other than "checkout", thing-to-do is one of host-libraries, compilers, glibcs. So you run, in that order: build-many-glibcs.py /some/where host-libraries build-many-glibcs.py /some/where compilers build-many-glibcs.py /some/where glibcs host-libraries is run once and then those libraries are used for all the compilers. compilers can be run once and then used many times for testing different glibc versions (so a bot only needs to update glibc and rerun the glibcs task, if using stable GCC / binutils; if testing the latest versions of the whole toolchain together including mainline GCC, it would probably want to update everything and rerun both compilers and glibcs). You can also name particular variants after "compilers" or "glibcs" to build just those variants (the possible variants are hardcoded in the script). I may add support for allowing the set of configurations to depend on the GCC version (to get cleaner default results), and optionally looping over architecture-independent glibc variants of CFLAGS and configure options as well, for every glibc configuration listed (e.g. -Os). GCC versions before 4.9 are not expected to work (the code uses --with-glibc-version to get the bootstrap GCC appropriately configured). There are various problems for particular configurations as well. Command-line options to the script: -jN to run N jobs in parallel (default the number of CPU cores reported by the system); --keep=all or --keep=failed to control keeping around build directories (default --keep=none). * scripts/build-many-glibcs.py: New file.
2016-11-12 05:07:08 +08:00
ctx.run_builds(opts.action, opts.configs)
if __name__ == '__main__':
main(sys.argv[1:])