Normally curl uses cryptographically strong random provided by the
selected SSL backend. If compiled without SSL support, a naive built-in
function was used instead.
Generally this was okay, but it will result in some downsides for non-
SSL builds, such as predictable temporary file names.
This change ensures that arc4random will be used instead, if available.
Closes#10672
Because automake used to delete depdirs at once (.deps) and there was an issue
with portability, curl's XC_AMEND_DISTCLEAN greps the Makefiles in an attempt
to build a list of all depfiles and delete them individually instead.
Since commit 08849db866b44510f6b8fd49e313c91a43a3dfd3, automake switched from
deleting directories to individual files. curl's custom logic now finds a lot
more results with the grep (the filtering of these results isn't great), which
causes a massive bloating of the Makefile in the order of O(n^2).
Also remove now-unused XC_AMEND_DISTCLEAN macro group
References: https://github.com/curl/curl/issues/9843
References: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59288
Reported-by: Ilmari Lauhakangas
Fixes#9843Closes#10661
- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more beefy environment
Closes#10654
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Closes#10349
- adding '--with-test-httpd=<path>' to configure non-standard apache2
install
- python env and base classes for running httpd
- basic tests for connectivity with h1/h2/h3
- adding test cases for truncated responses in http versions.
- adding goaway test for HTTP/3.
- adding "stuttering" tests with parallel downloads in chunks with
varying delays between chunks.
- adding a curltest module to the httpd server, adding GOAWAY test.
- mod_curltest now installs 2 handlers
- 'echo': writing as response body what came as request body
- 'tweak': with query parameters to tweak response behaviour
- marked known fails as skip for now
Closes#10175
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
- keeping the "current" easy handle registered at SSL* is no longer
necessary, since the "calling" data object is already stored in the
cfilter's context (and used by other SSL backends from there).
- The "detach" of an easy handle that goes out of scope is then avoided.
- using SSL_set0_wbio for clear reference counting where available.
Closes#10151
- adding support for HTTP/3 test cases via a nghttpx server that is
build with ngtcp2 and nghttp3.
- test2500 is the first test case, performing a simple GET.
- nghttpx is checked for support and the 'feature' nghttpx-h3
is set accordingly. test2500 will only run, when supported.
- a specific nghttpx location can be given in the environment
variable NGHTTPX or via the configure option
--with-test-nghttpx=<path>
Extend NGHTTPX config to H2 tests as well
* use $ENV{NGHTTPX} and the configured default also in http2 server starts
* always provide the empty test/nghttpx.conf to nghttpx. as it defaults to
reading /etc/nghttpx/nghttpx.conf otherwise.
Added nghttpx to CI ngtcp2 jobs to run h3 tests.
Closes#9031
`windres` is not always auto-detected by autotools when building for
Windows. When this happened, the build failed with a confusing error due
to the empty `RC` command:
```
/bin/bash ../libtool --tag=RC --mode=compile -I../include -DCURL_EMBED_MANIFEST -i curl.rc -o curl.o
[...]
Usage: /sandbox/curl/libtool [OPTION]... [MODE-ARG]...
Try 'libtool --help' for more information.
libtool: error: unrecognised option: '-I../include'
```
Improve this by verifying if `RC` is set, and fail with a clear error
otherwise.
Follow-up to 6de7322c03
Ref: https://curl.se/mail/lib-2022-10/0049.html
Reported-by: Thomas Glanzmann
Closes#9781
`WANT_IDN_PROTOTYPES` was necessary to avoid using a header that came
via an optional package. MS stopped distributing this package some
years ago and the winidn definitions are part of standard headers (via
`windows.h`) since Vista.
Auto-detect Vista inside `lib/idn_win32.c` and enable the manual
definitions if building for an older Windows.
This allows to delete this manual knob from all build-systems.
Also drop the `_SAL_VERSION` sub-case:
Our manual definitions are now only enabled with old systems. We assume
that code analysis is not run on such systems, allowing us to delete the
SAL-friendly flavour of these.
Reviewed-by: Jay Satiro
Closes#9793
Fixes: 73a070d96f/curl-autotools.sh (L44-L47)
On Windows this feature is present, but not the header used in the
detection logic. It also requires an elaborate enabler logic
(as seen in `lib/curl_setup.h`). Let's always allow it and let the
lib code deal with the details.
Closes#9688
Detecting headers and lib separately makes sense when headers come in
variations or with extra ones, but this wasn't the case here. These were
duplicate/parallel macros that we had to keep in sync with each other
for a working build. This patch leaves a single macro for each of these
dependencies:
- Rely on `HAVE_LIBZ`, delete parallel `HAVE_ZLIB_H`.
Also delete CMake logic making sure these two were in sync, along with
a toggle to turn off that logic, called `CURL_SPECIAL_LIBZ`.
Also delete stray `HAVE_ZLIB` defines.
There is also a `USE_ZLIB` variant in `lib/config-dos.h`. This patch
retains it for compatibility and deprecates it.
- Rely on `USE_LIBSSH2`, delete parallel `HAVE_LIBSSH2_H`.
Also delete `LIBSSH2_WIN32`, `LIBSSH2_LIBRARY` from
`winbuild/MakefileBuild.vc`, these have a role when building libssh2
itself. And `CURL_USE_LIBSSH`, which had no use at all.
Also delete stray `HAVE_LIBSSH2` defines.
- Rely on `USE_LIBSSH`, delete parallel `HAVE_LIBSSH_LIBSSH_H`.
Also delete `LIBSSH_WIN32`, `LIBSSH_LIBRARY` and `HAVE_LIBSSH` from
`winbuild/MakefileBuild.vc`, these were the result of copy-pasting the
libssh2 line, and were not having any use.
- Delete unused `HAVE_LIBPSL_H` and `HAVE_LIBPSL`.
Reviewed-by: Daniel Stenberg
Closes#9652
If curl_off_t turns out to be smaller than 8 bytes,
--with-n64-deprecated needs to be used to allow the build to
continue. This is to highlight the fact that support for such builds is
going away next year.
Also mentioned in DEPRECATED.md
Closes#9605
Move `LT_LANG([Windows Resource])` after `XC_LIBTOOL`, fixing:
- Warnings when running `autoreconf -fi`.
- Warning when compiling .rc files:
libtool: compile: unable to infer tagged configuration
libtool: error: specify a tag with '--tag'
Follow up to 6de7322c03
Ref: https://github.com/curl/curl/pull/9521#issuecomment-1256291156
Suggested-by: Patrick Monnerat
Closes#9582
This is the single place in libcurl code where it uses the "native"
s(n)printf() function. Used for writing floats. The use has been
reviewed and vetted and uses a HUGE target buffer, but switching to
snprintf() still makes this safer and removes build-time warnings.
Reported-by: Philip Heiduck
Fixes#9569Closes#9570
After this update autotools builds will compile and link `.rc` resources
to Windows executables. Bringing this feature on par with CMake and
Makefile.m32 builds. And also making it unnecessary to improvise these
steps manually, while monkey patching build files, e.g. [0].
You can customize the resource compiler via the `RC` envvar, and its
options via `RCFLAGS`.
This harmless warning may appear throughout the build, even though the
autotools manual documents [1] `RC` as a valid tag, and it fails when
omitting one:
`libtool: error: ignoring unknown tag RC`
[0] 535f19060d/curl-autotools.sh (L376-L382)
[1] https://www.gnu.org/software/libtool/manual/html_node/Tags.htmlCloses#9521
The check first checks that grep -E works, and only as a fallback tries
to find and use egrep. egrep is deprecated.
This change only corrects the output wording, not the checks themselves.
Closes#9471
A side effect of a previous change to configure (576e507c78)
exposed a non-critical issue that can happen if configure is called with
both '--without-ssl' and some parameter setting the use of a ssl library
(e.g. --with-gnutls). The configure script would end up assuming this is
a MultiSSL build, due to the way the case statement is written.
I have changed the order of the variables in the string concatenation
for the case statement and also tweaked the options so that
--without-ssl never turns the build into a MultiSSL one and also clearly
stating that there are conflicting parameters if the user sets it like
described above.
Closes#9414
Previously the configure script would just warn about this fact and
continue with TLS disabled build which is not always helpful. TLS should
be explicitly disabled if that is what the user wants.
Closes#9367
Commit b589696f added lines to some shell within AC_ARG_WITH macros, but
inadvertently failed to move the final closing ).
Quote the script section using braces.
So, if these problems have been around for a while, how did I find them?
Only because I did a configure including these options:
$ ./configure --with-openssl --without-rustls
SSL: enabled (OpenSSL)
Closes#9344
- based on ngtcp2 PR https://github.com/ngtcp2/ngtcp2/pull/505
- configure adapted to build against ngtcp2 wolfssl crypto lib
- quic code added for creation of WOLFSSL* instances
Closes#9290
This is a rewrite of the previously used GPLv3+exception licensed
file. With this change, there is no more reference to GPL so we can
remove that from LICENSES/.
Ref: #9220Closes#9291
The code that detects bsdsocket.library for AmigaOS did not work
for AmigaOS 4.x. This has been fixed and also cleaned up a little
to reduce duplication. Wasn't technically necessary before, but is
required when building with AmiSSL instead of OpenSSL.
Closes#9268
AmiSSL v5 is the latest version, featuring a port of OpenSSL 3.0.
Support for previous OpenSSL 1.1.x versions has been dropped, so
makes sense to enforce v5 as the minimum requirement. This also
allows all the AmiSSL stub workarounds to be removed as they are
now provided in a link library in the AmiSSL SDK.
Closes#9267
- Use the Windows API to seed the fallback random generator.
This ensures to always have a random seed, even when libcurl is built
with a vtls backend lacking a random generator API, such as rustls
(experimental), GSKit and certain mbedTLS builds, or, when libcurl is
built without a TLS backend. We reuse the Windows-specific random
function from the Schannel backend.
- Implement support for `BCryptGenRandom()` [1] on Windows, as a
replacement for the deprecated `CryptGenRandom()` [2] function.
It is used as the secure random generator for Schannel, and also to
provide entropy for libcurl's fallback random generator. The new
function is supported on Vista and newer via its `bcrypt.dll`. It is
used automatically when building for supported versions. It also works
in UWP apps (the old function did not).
- Clear entropy buffer before calling the Windows random generator.
This avoids using arbitrary application memory as entropy (with
`CryptGenRandom()`) and makes sure to return in a predictable state
when an API call fails.
[1] https://docs.microsoft.com/windows/win32/api/bcrypt/nf-bcrypt-bcryptgenrandom
[2] https://docs.microsoft.com/windows/win32/api/wincrypt/nf-wincrypt-cryptgenrandomCloses#9027
Referring to Daniel's article [1], making the init function thread-safe
was the last bit to make libcurl thread-safe as a whole. So the name of
the feature may as well be the more concise 'threadsafe', also telling
the story that libcurl is now fully thread-safe, not just its init
function. Chances are high that libcurl wants to remain so in the
future, so there is little likelihood of ever needing any other distinct
`threadsafe-<name>` feature flags.
For consistency we also shorten `CURL_VERSION_THREADSAFE_INIT` to
`CURL_VERSION_THREADSAFE`, update its description and reference libcurl's
thread safety documentation.
[1]: https://daniel.haxx.se/blog/2022/06/08/making-libcurl-init-more-thread-safe/
Reviewed-by: Daniel Stenberg
Reviewed-by: Jay Satiro
Closes#8989
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
This flag can be used to make sure that curl_global_init() is
thread-safe.
This can be useful for libraries that can't control what other
dependencies are doing with Curl.
Closes#8680
Enable Linux GSO in ngtcp2 QUIC. In order to recover from the
EAGAIN/EWOULDBLOCK by sendmsg with multiple packets in one GSO write,
packet buffer is now held by struct quicsocket. GSO write might fail in
runtime depending on NIC. Disable GSO if sendmsg returns EIO.
Closes#8909
