* Configure changes to detect AWS-LC
* CMakeLists.txt changes to detect AWS-LC
* Compile-time branches needed to support AWS-LC
* Correctly set OSSL_VERSION and report AWS-LC release number
* GitHub Actions script to build with autoconf and cmake against AWS-LC
AWS-LC is a BoringSSL/OpenSSL derivative
For more information see https://github.com/awslabs/aws-lc/Closes#10320
When a commit only contains tests, documentation, or cmake files, skip
those builds that aren't affected by those.
The file filters available on the CI services don't seem to allow
skipping individual jobs, only the entire workflow, so we can't get any
more fine-grained than this.
- added to: ngtcp2-quictls, ngtcp2-gnutls and the linux varians
quiche, bearssl, libressl, mbedtls, openssl3, rustls
- added disabled in ngtcp2-wolfssl due to weird SSL_connect() errors
not reproducable locally
Improvements on pytest:
- handling of systems with nghttpx in $PATH
- configure will seach $PATH got nghttpx used in pytest
- pytest fixes for managing nghttpx without h3 support
- ngtcp2-wolfssl: use a fully enabled wolfssl build
- lower parallel count for http/1.1 tests, since we do not
want to test excessive connections.
- check built curl for HTTPS-proxy support in proxy tests
- bearssl does not like one of our critical cert extensions, making
it non-critical now
- bearssl is too slow for test_12, skipping
- making sure we do h3 tests only when curl and server support is there
Closes#10699
Also skip builds on non-Windows platforms when only Windows build files
have changed.
This should reduce the number of useless builds and the associated
waiting time and chance of spurious failures, freeing resources for
new PRs.
Closes#10742
- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more beefy environment
Closes#10654
This reverts commit e0db842b2a.
This tool seems very restricted in how often it might be used by a
project and thus very quickly start to report fails simply because it
refuses to run when "there are more runs than allowed".
Closes#10613
- adding test_02_20 for reproducing the situation
- using recently released mod_h2 Apache module
- skipping test if an older version is installed
- adding installation of current mod_h2 to github pytest workflow
This reproduces the error reliable (for me) on the lib/http2.c version
of curl 7.88.0. And passes with the recent curl master.
Closes#10534
A temporary error with a remote server shouldn't cause a CI run to fail.
Also, put a cap on the time to download to fail faster on a misbehaving
server or connection and use HTTP compression where possible to reduce
download times.
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Closes#10349
These owners do not have the bandwidth/energy to do the reviews which
makes PRs stall and this ownership claim flawed. We can bring people
back when the situation is different.
Follow-up to c04c78ac87Closes#10386
The labeler language is quite restrictive right now so labels are added
quite conservatively, meaning that many PRs won't get labels when it's
"obvious" they should. It will still save some manual work on those
that it can label.
- adding '--with-test-httpd=<path>' to configure non-standard apache2
install
- python env and base classes for running httpd
- basic tests for connectivity with h1/h2/h3
- adding test cases for truncated responses in http versions.
- adding goaway test for HTTP/3.
- adding "stuttering" tests with parallel downloads in chunks with
varying delays between chunks.
- adding a curltest module to the httpd server, adding GOAWAY test.
- mod_curltest now installs 2 handlers
- 'echo': writing as response body what came as request body
- 'tweak': with query parameters to tweak response behaviour
- marked known fails as skip for now
Closes#10175
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
Set top-level permissions to None on all workflows, setting per-job
permissions. This avoids that new jobs inherit unwanted permissions.
Discussion: https://curl.se/mail/lib-2022-11/0028.html
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com>
Closes#9928
- adding support for HTTP/3 test cases via a nghttpx server that is
build with ngtcp2 and nghttp3.
- test2500 is the first test case, performing a simple GET.
- nghttpx is checked for support and the 'feature' nghttpx-h3
is set accordingly. test2500 will only run, when supported.
- a specific nghttpx location can be given in the environment
variable NGHTTPX or via the configure option
--with-test-nghttpx=<path>
Extend NGHTTPX config to H2 tests as well
* use $ENV{NGHTTPX} and the configured default also in http2 server starts
* always provide the empty test/nghttpx.conf to nghttpx. as it defaults to
reading /etc/nghttpx/nghttpx.conf otherwise.
Added nghttpx to CI ngtcp2 jobs to run h3 tests.
Closes#9031
Update bare GNU Make `Makefile.m32` to:
- Move objects into a subdirectory.
- Add support for MS-DOS. Tested with DJGPP.
- Add support for Watt-32 (on MS-DOS).
- Add support for AmigaOS.
- Rename `Makefile.m32` to `Makefile.mk`
- Replace `ARCH` with `TRIPLET`.
- Build `tool_hugehelp.c` proper (when tools are available).
- Drop MS-DOS compatibility macro `USE_ZLIB` (replaced by `HAVE_LIBZ`)
- Add support for `ZLIB_LIBS` to override `-lz`.
- Omit object files when building examples.
- Default `CC` to `gcc` once again, for convenience. (Caveat: compiler
name `cc` cannot be set now.)
- Set `-DCURL_NO_OLDIES` for examples, like autotools does.
- Delete `makefile.dj` files. Notice the configuration details and
defaults are not retained with the new method.
- Delete `makefile.amiga` files. A successful build needs a few custom
options. We're also not retaining all build details from the existing
Amiga make files.
- Rename `Makefile.m32` to `Makefile.mk` to reflect that they are not
Windows/MinGW32-specific anymore.
- Add support for new `CFG` options: `-map`, `-debug`, `-trackmem`
- Set `-DNDEBUG` by default.
- Allow using `-DOS=...` in all `lib/config-*.h` headers, syncing this
with `config-win32.h`.
- Look for zlib parts in `ZLIB_PATH/include` and `ZLIB_PATH/lib`
instead of bare `ZLIB_PATH`.
Note that existing build configurations for MS-DOS and AmigaOS likely
become incompatible with this change.
Example AmigaOS configuration:
```
export CROSSPREFIX=/opt/amiga/bin/m68k-amigaos-
export CC=gcc
export CPPFLAGS='-DHAVE_PROTO_BSDSOCKET_H'
export CFLAGS='-mcrt=clib2'
export LDFLAGS="${CFLAGS}"
export LIBS='-lnet -lm'
make -C lib -f Makefile.mk
make -C src -f Makefile.mk
```
Example MS-DOS configuration:
```
export CROSSPREFIX=/opt/djgpp/bin/i586-pc-msdosdjgpp-
export WATT_PATH=/opt/djgpp/net/watt
export ZLIB_PATH=/opt/djgpp
export OPENSSL_PATH=/opt/djgpp
export OPENSSL_LIBS='-lssl -lcrypt'
export CFG=-zlib-ssl
make -C lib -f Makefile.mk
make -C src -f Makefile.mk
```
Closes#9764
Also give each job on AppVeyor CI a human-readable name.
This aims to make job and therefore build failures more visible.
Reviewed-by: Marcel Raad
Closes#9769
This spellchecker checks markdown files. For this reason this job
converts all man pages in the repository to markdown with pandoc before
the check runs.
The perl script 'cleanspell' filters out details from the man page in
the process, to avoid the spellchecker trying to spellcheck things it
can't. Like curl specific symbols and the SYNOPSIS and EXAMPLE sections
of libcurl man pages.
The spell checker does not check words in sections that are within pre,
strong and em tags.
'spellcheck.words' is a custom word list with additional accepted words.
Closes#9523
Avoid letting outdated CI runs continue if a PR receives
new changes. Outside a PR we let them continue running
by tying the concurrency to the commit hash instead.
Also only let one CodeQL or Hacktoberfest job run at a time.
Other CI platforms we use have this build in, but GitHub
unfortunately neither by default nor with a simple option.
This saves CI resources and therefore a little energy.
Approved-by: Daniel Stenberg
Approved-by: Max Dymond
Closes#9533
This is an undocumented option similar to the `-Spath' option introduced
in cmake 3.13.
Replace all instances of `-Hpath' with `-Spath' in macos workflow.
Replace `-H. -Bpath' with `mkdir path; cd ./path; cmake ..' in zuul
scripts since it runs an older version of cmake.
Fixes#9008Closes#9014
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
- bump actions/checkout from 2 to 3
- bump actions/upload-artifact from 1 to 3
- bump github/codeql-actions from 1 to 2
- use version tag for actions/checkout
Closes#8843
Unfortunately the GitHub API requires a token with write permission
for both issues and pull-requests to edit labels on even just PRs.
Follow up to #7897
Replace test-nonflaky with test-ci and enable verbose output
in all remaining CIs except Zuul which is customized a lot.
Reviewed-by: Daniel Stenberg
Reviewed-by: Jay Satiro
Follow up to #7785Closes#7832
Automatically add hacktoberfest-accepted label to PRs opened between
September 30th and November 1st once a commit with a close reference
to it is pushed onto the master branch.
With this workflow we can participate in Hacktoberfest while not
relying on GitHub to identify PRs as merged due to our rebasing.
Requires hacktoberfest-accepted labels to exist for PRs on the
participating repository. Also requires hacktoberfest topic on
the participating repository to avoid applying to forked repos.
Reviewed-by: Daniel Stenberg
Fixes#7865Closes#7897
Warning: this will make existing curl command lines that use metalink to
stop working.
Reasons for removal:
1. We've found several security problems and issues involving the
metalink support in curl. The issues are not detailed here. When
working on those, it become apparent to the team that several of the
problems are due to the system design, metalink library API and what
the metalink RFC says. They are very hard to fix on the curl side
only.
2. The metalink usage with curl was only very briefly documented and was
not following the "normal" curl usage pattern in several ways, making
it surprising and non-intuitive which could lead to further security
issues.
3. The metalink library was last updated 6 years ago and wasn't so
active the years before that either. An unmaintained library means
there's a security problem waiting to happen. This is probably reason
enough.
4. Metalink requires an XML parsing library, which is complex code (even
the smaller alternatives) and to this day often gets security
updates.
5. Metalink is not a widely used curl feature. In the 2020 curl user
survey, only 1.4% of the responders said that they'd are using it. In
2021 that number was 1.2%. Searching the web also show very few
traces of it being used, even with other tools.
6. The torrent format and associated technology clearly won for
downloading large files from multiple sources in parallel.
Cloes #7176
The CMake option is now called CMAKE_USE_SCHANNEL
The winbuild flag is USE_SCHANNEL
The CI jobs and build scripts only use the new names and the new name
options
Tests now require 'Schannel' (when necessary)
Closes#5795
Align CodeQL action with existing CI actions:
- Update branch filter to avoid duplicate CI runs.
- Shorten workflow name due to informative job name.
Reviewed-by: Daniel Stenberg
Closes#5660
Reduce workload on Azure Pipelines and Travis CI while
consolidating macOS jobs onto less utilized GitHub Actions.
Reviewed-by: Daniel Stenberg
Closes#5124
Introducing the curl bug bounty program on hackerone. We now recommend
filing security issues directly in the hackerone ticket system which
only is readable to curl security team members.
Assisted-by: Daniel Gustafsson
Closes#3488
Issues and PRs with no activity for 180 days will get marked as stale,
and if no further activity happens within 14 more days, the issue gets
closed.
This follows our established policy of not letting stalled bugs "get in
the way": https://curl.haxx.se/docs/bugs.html#Closing_off_stalled_bugsCloses#1398
We're mostly saying just "curl" in lower case these days so here's a big
cleanup to adapt to this reality. A few instances are left as the
project could still formally be considered called cURL.
