Prior to this change a workaround for Windows to recv before every send
was enabled by default. The way it works is a recv is called before
every send and saves the received data, in case send fails because in
Windows apparently that can wipe out the socket's internal received
data buffer.
This feature has led to several bugs because the way libcurl operates
it waits on a socket to read or to write, and may not at all times
check for buffered receive data.
Two recent significant bugs this workaround caused:
- Broken Schannel TLS 1.3 connections (#9431)
- HTTP/2 arbitrary hangs (#10253)
The actual code remains though it is disabled by default. Though future
changes to connection filter buffering could improve the situation IMO
it's just not tenable to manage this workaround.
Ref: https://github.com/curl/curl/issues/657
Ref: https://github.com/curl/curl/pull/668
Ref: https://github.com/curl/curl/pull/720
Ref: https://github.com/curl/curl/issues/9431
Ref: https://github.com/curl/curl/issues/10253
Closes https://github.com/curl/curl/pull/10409
curl would erroneously report CURLE_OUT_OF_MEMORY in some cases instead
of CURLE_URL_MALFORMAT. In other cases, it would erroneously return
CURLE_URL_MALFORMAT instead of CURLE_OUT_OF_MEMORY. Add a test case to
test the former condition.
Fixes#10130Closes#10414
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Closes#10349
- add test2600 as a unit test that triggers various connect conditions
and monitors behaviour, available in a debug build only.
- this exposed edge cases in connect.c that have been fixed
Closes#10312
- adding '--with-test-httpd=<path>' to configure non-standard apache2
install
- python env and base classes for running httpd
- basic tests for connectivity with h1/h2/h3
- adding test cases for truncated responses in http versions.
- adding goaway test for HTTP/3.
- adding "stuttering" tests with parallel downloads in chunks with
varying delays between chunks.
- adding a curltest module to the httpd server, adding GOAWAY test.
- mod_curltest now installs 2 handlers
- 'echo': writing as response body what came as request body
- 'tweak': with query parameters to tweak response behaviour
- marked known fails as skip for now
Closes#10175
Built-in TLS backends are detected at test time by scanning for their
names in the version string line returned by the cli tool: as this line
may also list the libssh configuration that mentions its own backend,
the curl backend may be wrongly determined.
In example, if the version line contains "libssh/0.10.4/openssl/zlib",
OpenSSL is detected as a curl-configured backend even if not.
This fix requires the backend names to appear as full words preceded by
spacing in the version line to be recognized as curl TLS backends.
Closes#10236
- Set all scripts +x, ie 644 => 755.
Prior to this change some scripts were not executable and therefore
could not be called directly.
~~~
git ls-files -s \*.{sh,pl,py} | grep -v 100755
~~~
Closes https://github.com/curl/curl/pull/10219
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
The unencoding stack is added to as Transfer-Encoding and
Content-Encoding fields are encountered with no distinction between the
two, meaning the stack will be incorrect if, e.g., the message has both
fields and a non-chunked Transfer-Encoding comes first. This commit
fixes this by ordering the stack with transfer encodings first.
Reviewed-by: Patrick Monnerat
Closes#10187
As in all other lib tests. This avoids a macro redefinition warning for
`_FILE_OFFSET_BITS` visible in the autobuilds.
Closes https://github.com/curl/curl/pull/10182
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.
- eyeballing cfilter that uses sub-filters for performing parallel connects
- socket cfilter for all transport types, including QUIC
- QUIC implementations in cfilter, can now participate in eyeballing
- connection setup is more dynamic in order to adapt to what filter did
really connect. Relevant to see if a SSL filter needs to be added or
if SSL has already been provided
- HTTP/3 test cases similar to HTTP/2
- multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche
- Fix for data attach/detach in VTLS filters that could lead to crashes
during parallel transfers.
- Eliminating setup() methods in cfilters, no longer needed.
- Improving Curl_conn_is_alive() to replace Curl_connalive() and
integrated ssl alive checks into cfilter.
- Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
connection into and persist it at the easy handle.
- Several more cfilter related cleanups and moves:
- stream_weigth and dependency info is now wrapped in struct
Curl_data_priority
- Curl_data_priority members depend is available in HTTP2|HTTP3
- Curl_data_priority members depend on NGHTTP2 support
- handling init/reset/cleanup of priority part of url.c
- data->state.priority same struct, but shallow copy for compares only
- PROTOPT_STREAM has been removed
- Curl_conn_is_mulitplex() now available to check on capability
- Adding query method to connection filters.
- ngtcp2+quiche: implementing query for max concurrent transfers.
- Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
- setting keepalive timestamp on connect
- DATA_SETUP is called after the connection has been completely
setup (but may not connected yet) to allow filters to initialize
data members they use.
- there is no socket to be had with msh3, it is unclear how select
shall work
- manual test via "curl --http3 https://curl.se" fail with "empty
reply from server".
- Various socket/conn related cleanups:
- Curl_socket is now Curl_socket_open and in cf-socket.c
- Curl_closesocket is now Curl_socket_close and in cf-socket.c
- Curl_ssl_use has been replaced with Cur_conn_is_ssl
- Curl_conn_tcp_accepted_set has been split into
Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
with a clearer purpose
Closes#10141
- tests/ftpserver.pl blocks when expecting a DATA connection from the
client.
- the previous 10 seconds were encountered repeatedly in torture tests
and let to long waits.
- 2 seconds should still be sufficient for current hw, but CI will show.
Closes#10178
The SHA-1 algorithm is deprecated (particularly for security-sensitive
applications) in a variety of OS environments. This already affects
RHEL-9 and derivatives, which are not willing to use certificates using
that algorithm. The fix is to use sha256 instead, which is already used
for most of the other certificates in the test suite.
Fixes#10135
This gets rid of issues related to sha1 signatures.
Manual steps after "make clean-certs" and "make build-certs":
- Copy tests/certs/stunnel-sv.pem to tests/stunnel.pem
(make clean-certs does not remove the original tests/stunnel.pem)
- Copy tests/certs/Server-localhost-sv.pubkey-pinned into --pinnedpubkey
options of tests/data/test2041 and tests/data/test2087
Closes#10153
The cookiefile entries are set into the handle and should remain set for
the lifetime of the handle so that duplicating it also duplicates the
list. Therefore, the struct field is moved from 'state' to 'set'.
Fixes#10133Closes#10134
Follow-up to ac612dfeee
strtoul() accepts a leading minus so better make sure there is none
Extended test 356 somewhat to use a huge negative 64 bit number that
otherwise becomes a low positive number.
Closes#10095
This commit fixes a bug in the dump-header feature regarding the
determination of the second fopen(3) option.
Reported-by: u20221022 on github
See #4753
See #4762Fixes#10074Closes#10079
Also, move the output of the nghttpx_h3 info to the general "Env:" line
in the test output header.
Reported-by: Marcel Raad
Ref: ca15b7512e (commitcomment-92015094)Closes#10044
