curl/appveyor.sh

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

159 lines
5.5 KiB
Bash
Raw Normal View History

#!/usr/bin/env bash
#***************************************************************************
# _ _ ____ _
# Project ___| | | | _ \| |
# / __| | | | |_) | |
# | (__| |_| | _ <| |___
# \___|\___/|_| \_\_____|
#
# Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
#
# This software is licensed as described in the file COPYING, which
# you should have received as part of this distribution. The terms
# are also available at https://curl.se/docs/copyright.html.
#
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
# copies of the Software, and permit persons to whom the Software is
# furnished to do so, under the terms of the COPYING file.
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
# KIND, either express or implied.
#
# SPDX-License-Identifier: curl
#
###########################################################################
# shellcheck disable=SC3040,SC2039
set -eux; [ -n "${BASH:-}${ZSH_NAME:-}" ] && set -o pipefail
# build
if [ "${APPVEYOR_BUILD_WORKER_IMAGE}" = 'Visual Studio 2022' ]; then
openssl_root_win='C:/OpenSSL-v32-Win64'
else
openssl_root_win='C:/OpenSSL-v111-Win64'
fi
GHA: add three old (gcc 6, 7, 9) mingw-w64 jobs Re-implement old mingw-w64 jobs in GHA. This allows to use the latest Windows runners, replacing Windows Server 2012 R2 (gcc 6) and Windows Server 2016 (gcc 7, 9) with Windows Server 2022. GHA runners are also significantly faster, and allow running tests in parallel (`-j14`). It also offloads 3 more long-running jobs from AppVeyor CI. These jobs download (then cache) the mingw-w64 packages from their original location, which allows flexibility in choosing which versions and flavours (win32/POSIX, SEH/DWARF, 64/32-bit) we want to test in CI. The new jobs use these distros: - https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/ (for gcc 7, same as on AppVeyor) - https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/ (for gcc 6, same as on AppVeyor) - https://winlibs.com/ (for gcc 9) I matched existing AppVeyor job configs, with these differences: - gcc 6.4.0 instead of 6.3.0. (same distro as on AppVeyor, but the latest bugfix release) - gcc 9.5.0 instead of 9.1.0 and a different (but compatible) binary distro. (in AppVeyor this relies on an old MSYS2 pre-installed on the runner) - using win32 builds instead of posix for gcc 6.4.0 and 7.3.0. - websockets enabled. - always build examples. - always build tests (this wasn't done for 6.4.0 with AppVeyor CI). I did not replicate existing test exclusions, and oddly enough the few failures (so far) were different from MSYS2 jobs and also from their AppVeyor CI counterparts. Also: - delete redundant (default) `-u` option from `cygpath` calls. - allow matrix options to override default ones in CMake. - detect and use Windows-supplied curl for `TFLAGS` `-ac` option. (it's available in modern runners.) - delete the 3 AppVeyor CI jobs now replicated in GHA. - appveyor: prefer `SYSTEMROOT` over `WINDIR`. - tidy-up quotes. Job performance: ``` AppVeyor GHA w/examples w/tests -------- ---------- CMake, mingw-w64, gcc 6, Debug, x86, Schannel, Static, no-unity 1m25s 8m50s CMake, mingw-w64, gcc 7, Debug, x64, Schannel, Static, Unicode 31m45s 9m39s CMake, mingw-w64, gcc 9, Debug, x64, Schannel, Static 28m25s 13m38s ``` Based on these runs: https://ci.appveyor.com/project/curlorg/curl/builds/49880799 https://github.com/curl/curl/actions/runs/9218292508 Notice that building examples and tests is time consuming. We can tweak any build parameter as necessary to make them more useful and/or without clogging the job queue or introducing flakiness. Closes #13759
2024-05-23 20:54:49 +08:00
openssl_root="$(cygpath "${openssl_root_win}")"
if [ "${BUILD_SYSTEM}" = 'CMake' ]; then
options=''
[[ "${TARGET:-}" = *'ARM64'* ]] && SKIP_RUN='ARM64 architecture'
[ "${OPENSSL}" = 'ON' ] && options+=" -DOPENSSL_ROOT_DIR=${openssl_root_win}"
[ -n "${CURLDEBUG:-}" ] && options+=" -DENABLE_CURLDEBUG=${CURLDEBUG}"
[ "${PRJ_CFG}" = 'Debug' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG='
[ "${PRJ_CFG}" = 'Release' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE='
[[ "${PRJ_GEN}" = *'Visual Studio'* ]] && options+=' -DCMAKE_VS_GLOBALS=TrackFileAccess=false'
if [ "${PRJ_GEN}" = 'Visual Studio 9 2008' ]; then
[ "${DEBUG}" = 'ON' ] && [ "${SHARED}" = 'ON' ] && SKIP_RUN='Crash on startup in ENABLE_DEBUG=ON shared builds'
# Fails to run without this due to missing MSVCR90.dll / MSVCR90D.dll
options+=' -DCURL_STATIC_CRT=ON'
fi
# shellcheck disable=SC2086
cmake -B _bld "-G${PRJ_GEN}" ${TARGET:-} ${options} \
"-DCURL_USE_OPENSSL=${OPENSSL}" \
"-DCURL_USE_SCHANNEL=${SCHANNEL}" \
"-DHTTP_ONLY=${HTTP_ONLY}" \
"-DBUILD_SHARED_LIBS=${SHARED}" \
"-DCMAKE_UNITY_BUILD=${UNITY}" \
tests: speed up builds with single-binary test bundles Add support for single-block binaries that contain all libtests and unit tests respectively. Enable with: - autotools: `--enable-test-bundles` - cmake: `-DCURL_TEST_BUNDLES=ON` (They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON` options, for further speed-up.) Makes libtests and unit tests build _fast_, needing little disk space even in static mode. Similar to CMake unity mode, but with a custom script, also supporting autotools builds. The price is having to deal with symbols/macros colliding between `lib*.c` and `unit*.c` sources. Maybe with naming conventions or other solutions this can be improved gradually and reduce the need for manual intervention by `mk-bundle.mk`. I've included a script that does the bulk of detecting name collisions. Also: - CI: enable test bundles. - CI: build tests in more jobs. - lib2305: fix FILE handle leak. - unit1661: fix memleak found by torture test by releasing the `bufref` structure in `unit_stop()` that was allocated in `unit_setup()`. ``` test 1661...[bufref unit tests] Leak detected: memory still allocated: 13 bytes allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70 1661: torture FAILED: function number 1 in test. ``` Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41 Similar test suite builds with autotools default and cmake+bundle+unity: - GHA/Linux: 33s vs 7s https://github.com/curl/curl/actions/runs/10705668823/job/29681617374 - GHA/macOS 34s vs 2s https://github.com/curl/curl/actions/runs/10705668813/job/29681632885 - GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu) https://github.com/curl/curl/actions/runs/10705668811/job/29681607915 - GHA/Cygwin: 9m52 vs 32s https://github.com/curl/curl/actions/runs/10705668809/job/29681609965 - GHA/MSYS2: 3m52 vs 14s https://github.com/curl/curl/actions/runs/10705668808/job/29681624295 - GHA/mingw-w64: 5m45 vs 30s https://github.com/curl/curl/actions/runs/10705668808/job/29681628787 Autotools test suite builds compared between master -> `--enable-test-bundles`: - GHA/Linux: 33s -> 9s (run tests: 22m23 -> 20m44) https://github.com/curl/curl/actions/runs/10710030193/job/29695932185 https://github.com/curl/curl/actions/runs/10967831456/job/30458220344 - GHA/macOS: 25s -> 4s (run tests: 2m58 -> 2m24) https://github.com/curl/curl/actions/runs/10710030195/job/29695938444 https://github.com/curl/curl/actions/runs/10967831452/job/30458225762 - GHA/non-native (FreeBSD): 4m8 -> 3m12 (full workflow time, ~qemu) https://github.com/curl/curl/actions/runs/10710030198/job/29695928401 https://github.com/curl/curl/actions/runs/10967831458/job/30458212692 - GHA/Cygwin: 9m25 -> 1m9 (run tests: 9m19 -> 3m28) https://github.com/curl/curl/actions/runs/10710030212/job/29695928213 https://github.com/curl/curl/actions/runs/10967831453/job/30458213268 - GHA/MSYS2: 3m54 -> 32s (run tests: 6m3 -> 3m59) https://github.com/curl/curl/actions/runs/10710030190/job/29704850591 https://github.com/curl/curl/actions/runs/10967831449/job/30459280005 - GHA/mingw-w64: 5m42 -> 1m5 (run tests: 7m41 -> 5m36) https://github.com/curl/curl/actions/runs/10710030190/job/29704852058 https://github.com/curl/curl/actions/runs/10967831449/job/30459280862 - Azure MSYS2 mingw64 openssl: 38m55 -> 11m58 https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e - Azure Ubuntu default: 2m15 -> 55s (all build) https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7 https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7 Cmake test suite builds compared between master -> `-DCURL_TEST_BUNDLES=ON` + unity: - GHA/Linux: 29s -> 7s (run tests: 4m50 -> 4m57, 20m43 -> 20m45) https://github.com/curl/curl/actions/runs/10710030193/job/29695941814 https://github.com/curl/curl/actions/runs/10705668823/job/29681622201 - GHA/Linux old: 44s -> 13s (bundle+no unity) (run tests: 5m5 -> 5m6) https://github.com/curl/curl/actions/runs/10718264094/job/29719794727 https://github.com/curl/curl/actions/runs/10718653175/job/29721009613 - GHA/macOS: 32s -> 2s (run tests: 2m43 -> 2m40) https://github.com/curl/curl/actions/runs/10710030195/job/29695931956 https://github.com/curl/curl/actions/runs/10705668813/job/29681638937 - GHA/non-native (*BSD): inconclusive (full workflow time, ~qemu) https://github.com/curl/curl/actions/runs/10710030198 https://github.com/curl/curl/actions/runs/10705668811 - GHA/Cygwin: 3m9 -> 32s https://github.com/curl/curl/actions/runs/10710030212/job/29695929075 https://github.com/curl/curl/actions/runs/10705668809/job/29681609965 - GHA/MSYS2: 2m24 -> 14s https://github.com/curl/curl/actions/runs/10710030190/job/29704850996 https://github.com/curl/curl/actions/runs/10705668808/job/29681624295 - GHA/mingw-w64: 3m56 -> 30s (run tests: 4m2 -> 3m52) https://github.com/curl/curl/actions/runs/10710030190/job/29704852219 https://github.com/curl/curl/actions/runs/10705668808/job/29681631393 - GHA/mingw-w64-old: 7m19 -> 1m44 (run tests: 3m30 -> 2m53) https://github.com/curl/curl/actions/runs/10710030190/job/29704849763 https://github.com/curl/curl/actions/runs/10705668808/job/29681622329 - GHA/MSVC: 3m22 -> 13s (run tests: 9m43 -> 4m22) https://github.com/curl/curl/actions/runs/10710030190/job/29704850411 https://github.com/curl/curl/actions/runs/10705668808/job/29681623313 - AppVeyor CI MSVC 2008: 4m3 -> 45s (full build) - AppVeyor CI MSVC 2010: 2m56 -> 1m8 (full build) - AppVeyor CI MSVC 2022: 10m19 -> 2m23 (full build) https://ci.appveyor.com/project/curlorg/curl/builds/50538455 https://ci.appveyor.com/project/curlorg/curl/builds/50536558 - AppVeyor CI total build time: 10m30 (master) -> 6m48 (unity) -> 4m5 (bundle) -> 3m24 (bundle+unity) -> 5m7 (bundle+unity+all jobs building tests) Closes #14772
2024-09-02 20:46:00 +08:00
'-DCURL_TEST_BUNDLES=ON' \
'-DCURL_WERROR=ON' \
"-DENABLE_DEBUG=${DEBUG}" \
"-DENABLE_UNICODE=${ENABLE_UNICODE}" \
'-DCMAKE_INSTALL_PREFIX=C:/curl' \
"-DCMAKE_BUILD_TYPE=${PRJ_CFG}" \
'-DCURL_USE_LIBPSL=OFF'
# shellcheck disable=SC2086
if ! cmake --build _bld --config "${PRJ_CFG}" --parallel 2 -- ${BUILD_OPT:-}; then
if [ "${PRJ_GEN}" = 'Visual Studio 9 2008' ]; then
find . -name BuildLog.htm -exec dos2unix '{}' +
find . -name BuildLog.htm -exec cat '{}' +
fi
false
fi
if [ "${SHARED}" = 'ON' ]; then
PATH="$PWD/_bld/lib:$PATH"
fi
if [ "${OPENSSL}" = 'ON' ]; then
PATH="$PWD/_bld/lib:${openssl_root}:$PATH"
fi
curl='_bld/src/curl.exe'
elif [ "${BUILD_SYSTEM}" = 'VisualStudioSolution' ]; then
(
cd projects
./generate.bat "${VC_VERSION}"
msbuild.exe -maxcpucount "-property:Configuration=${PRJ_CFG}" "Windows/${VC_VERSION}/curl-all.sln"
)
curl="build/Win32/${VC_VERSION}/${PRJ_CFG}/curld.exe"
elif [ "${BUILD_SYSTEM}" = 'winbuild_vs2015' ]; then
./buildconf.bat
(
cd winbuild
cat << EOF > _make.bat
call "C:/Program Files/Microsoft SDKs/Windows/v7.1/Bin/SetEnv.cmd" /x64
call "C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/vcvarsall.bat" x86_amd64
nmake -f Makefile.vc mode=dll VC=14 "SSL_PATH=${openssl_root_win}" WITH_SSL=dll MACHINE=x64 DEBUG=${DEBUG} ENABLE_UNICODE=${ENABLE_UNICODE}
EOF
./_make.bat
rm _make.bat
)
curl="builds/libcurl-vc14-x64-${PATHPART}-dll-ssl-dll-ipv6-sspi/bin/curl.exe"
elif [ "${BUILD_SYSTEM}" = 'winbuild_vs2017' ]; then
./buildconf.bat
(
cd winbuild
cat << EOF > _make.bat
call "C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Auxiliary/Build/vcvars64.bat"
nmake -f Makefile.vc mode=dll VC=14.10 "SSL_PATH=${openssl_root_win}" WITH_SSL=dll MACHINE=x64 DEBUG=${DEBUG} ENABLE_UNICODE=${ENABLE_UNICODE}
EOF
./_make.bat
rm _make.bat
)
curl="builds/libcurl-vc14.10-x64-${PATHPART}-dll-ssl-dll-ipv6-sspi/bin/curl.exe"
fi
find . -name '*.exe' -o -name '*.dll'
if [ -z "${SKIP_RUN:-}" ]; then
"${curl}" --disable --version
else
echo "Skip running curl.exe. Reason: ${SKIP_RUN}"
fi
if false; then
cat CMakeFiles/CMakeConfigureLog.yaml 2>/dev/null || true
fi
# build tests
if [[ "${TFLAGS}" != 'skipall' ]] && \
[ "${BUILD_SYSTEM}" = 'CMake' ]; then
cmake --build _bld --config "${PRJ_CFG}" --parallel 2 --target testdeps
fi
# run tests
if [[ "${TFLAGS}" != 'skipall' ]] && \
[[ "${TFLAGS}" != 'skiprun' ]]; then
GHA: add three old (gcc 6, 7, 9) mingw-w64 jobs Re-implement old mingw-w64 jobs in GHA. This allows to use the latest Windows runners, replacing Windows Server 2012 R2 (gcc 6) and Windows Server 2016 (gcc 7, 9) with Windows Server 2022. GHA runners are also significantly faster, and allow running tests in parallel (`-j14`). It also offloads 3 more long-running jobs from AppVeyor CI. These jobs download (then cache) the mingw-w64 packages from their original location, which allows flexibility in choosing which versions and flavours (win32/POSIX, SEH/DWARF, 64/32-bit) we want to test in CI. The new jobs use these distros: - https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/ (for gcc 7, same as on AppVeyor) - https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/ (for gcc 6, same as on AppVeyor) - https://winlibs.com/ (for gcc 9) I matched existing AppVeyor job configs, with these differences: - gcc 6.4.0 instead of 6.3.0. (same distro as on AppVeyor, but the latest bugfix release) - gcc 9.5.0 instead of 9.1.0 and a different (but compatible) binary distro. (in AppVeyor this relies on an old MSYS2 pre-installed on the runner) - using win32 builds instead of posix for gcc 6.4.0 and 7.3.0. - websockets enabled. - always build examples. - always build tests (this wasn't done for 6.4.0 with AppVeyor CI). I did not replicate existing test exclusions, and oddly enough the few failures (so far) were different from MSYS2 jobs and also from their AppVeyor CI counterparts. Also: - delete redundant (default) `-u` option from `cygpath` calls. - allow matrix options to override default ones in CMake. - detect and use Windows-supplied curl for `TFLAGS` `-ac` option. (it's available in modern runners.) - delete the 3 AppVeyor CI jobs now replicated in GHA. - appveyor: prefer `SYSTEMROOT` over `WINDIR`. - tidy-up quotes. Job performance: ``` AppVeyor GHA w/examples w/tests -------- ---------- CMake, mingw-w64, gcc 6, Debug, x86, Schannel, Static, no-unity 1m25s 8m50s CMake, mingw-w64, gcc 7, Debug, x64, Schannel, Static, Unicode 31m45s 9m39s CMake, mingw-w64, gcc 9, Debug, x64, Schannel, Static 28m25s 13m38s ``` Based on these runs: https://ci.appveyor.com/project/curlorg/curl/builds/49880799 https://github.com/curl/curl/actions/runs/9218292508 Notice that building examples and tests is time consuming. We can tweak any build parameter as necessary to make them more useful and/or without clogging the job queue or introducing flakiness. Closes #13759
2024-05-23 20:54:49 +08:00
if [ -x "$(cygpath "${SYSTEMROOT}/System32/curl.exe")" ]; then
TFLAGS+=" -ac $(cygpath "${SYSTEMROOT}/System32/curl.exe")"
elif [ -x "$(cygpath 'C:/msys64/usr/bin/curl.exe')" ]; then
TFLAGS+=" -ac $(cygpath 'C:/msys64/usr/bin/curl.exe')"
fi
TFLAGS+=' -j0'
if [ "${BUILD_SYSTEM}" = 'CMake' ]; then
cmake --build _bld --config "${PRJ_CFG}" --target test-ci
else
(
TFLAGS="-a -p !flaky -r -rm ${TFLAGS}"
cd _bld/tests
./runtests.pl
)
fi
fi
# build examples
if [[ "${EXAMPLES}" = 'ON' ]] && \
[ "${BUILD_SYSTEM}" = 'CMake' ]; then
cmake --build _bld --config "${PRJ_CFG}" --parallel 2 --target curl-examples
fi