... and make Curl_cookie_add() require 'data' being set proper with an
assert.
The function has not worked with a NULL data for quite some time so this
just corrects the code and comment.
This is a different take than the proposed fixed in #10927
Reported-by: Kvarec Lezki
Ref: #10929Closes#10930
- move all code handling HTTP/2 frames for a particular
stream into a separate function to keep from confusing
the call `data` with the stream `data`.
Closes#10924
A typical mistake would be to try to set "https://" - including the
separator - this is now rejected as that would then lead to
url_get(... URL...) would get an invalid URL extracted.
Extended test 1560 to verify.
Closes#10911
Curl_http2_strerror was renamed to http2_strerror in
05b100aee2 and then http2_strerror was removed in
5808a0d0f5
This also fixes the following compiler error
lib/http2.h:41:33: error: unknown type name 'uint32_t'
lib/http2.h:1:1: note: 'uint32_t' is defined in header '<stdint.h>'
Closes#10912
The only user of this define was 'chkdecimalpoint' - a special purpose
test tool that was built but not used anymore (since 17c18fbc3 - Apr
2020).
Closes#10908
- remove NGHTTP2 members of `struct HTTP`
- add `void *h2_ctx` to `struct HTTP`
- add `void *h3_ctx` to `struct HTTP`
- separate h2/h3 pointers are needed for eyeballing
- manage local stream_ctx in http implementations
Closes#10877
- currently only on debug build and when env variable
CURL_PROXY_TUNNEL_H2 is present.
- will ALPN negotiate with the proxy server and switch
tunnel filter based on the protocol negotiated.
- http/1.1 tunnel code moved into cf-h1-proxy.[ch]
- http/2 tunnel code implemented in cf-h2-proxy.[ch]
- tunnel start and ALPN set remains in http_proxy.c
- moving all haproxy related code into cf-haproxy.[ch]
VTLS changes
- SSL filters rely solely on the "alpn" specification they
are created with and no longer check conn->bits.tls_enable_alpn.
- checks on which ALPN specification to use (or none at all) are
done in vtls.c when creating the filter.
Testing
- added a nghttpx forward proxy to the pytest setup that
speaks HTTP/2 and forwards all requests to the Apache httpd
forward proxy server.
- extending test coverage in test_10 cases
- adding proxy tests for direct/tunnel h1/h2 use of basic auth.
- adding test for http/1.1 and h2 proxy tunneling to pytest
Closes#10780
- eliminate receive loop in vtls to fill buffer. This may
lead to partial reads of data which is counter productive
- let http2 instead loop smarter to process pending network
data without transfer switches
scorecard improvements
- do not start caddy when only httpd is requested
- allow curl -v to stderr file on --curl-verbose
Closes#10891
Using bad numbers in an IPv4 numerical address now returns
CURLUE_BAD_HOSTNAME.
I noticed while working on trurl and it was originally reported here:
https://github.com/curl/trurl/issues/78
Updated test 1560 accordingly.
Closes#10894
Meaning that it would wrongly still store the fragment using spaces
instead of %20 if allowing space while also asking for URL encoding.
Discovered when playing with trurl.
Added test to lib1560 to verify the fix.
Closes#10887
- when rustls is told to recieve more TLS data and its internal
plaintext buffers are full, it returns an IOERROR
- avoid receiving TLS data while plaintext is not read empty
pytest:
- increase curl run timeout when invoking pytest with higher verbosity
Closes#10876
- ngtcp2: using bufq for recv stream data
- internal stream_ctx instead of `struct HTTP` members
for quiche, ngtcp2 and msh3
- no more QUIC related members in `struct HTTP`
- experimental use of recvmmsg(), disabled by default
- testing on my old debian box shows no throughput improvements.
- leaving it in, but disabled, for future revisit
- vquic: common UDP receive code for ngtcp2 and quiche
- vquic: common UDP send code for ngtcp2 and quiche
- added pytest skips for known msh3 failures
- fix unit2601 to survive torture testing
- quiche: using latest `master` from quiche and enabling large download
tests, now that key change is supported
- fixing test_07_21 where retry handling of starting a stream
was faulty
- msh3: use bufq for recv buffering headers and data
- msh3: replace fprintf debug logging with LOG_CF where possible
- msh3: force QUIC expire timers on recv/send to have more than
1 request per second served
Closes#10772
- use bufq for send/receive of network data
- usd bufq for send/receive of stream data
- use HTTP/2 flow control with no-auto updates to control the
amount of data we are buffering for a stream
HTTP/2 stream window set to 128K after local tests, defined
code constant for now
- elminiating PAUSEing nghttp2 processing when receiving data
since a stream can now take in all DATA nghttp2 forwards
Improved scorecard and adjuste http2 stream window sizes
- scorecard improved output formatting and options default
- scorecard now also benchmarks small requests / second
Closes#10771
RFC 7686 states that:
> Applications that do not implement the Tor
> protocol SHOULD generate an error upon the use of .onion and
> SHOULD NOT perform a DNS lookup.
Let's do that.
https://www.rfc-editor.org/rfc/rfc7686#section-2
Add test 1471 and 1472 to verify
Fixes#543Closes#10705
* Configure changes to detect AWS-LC
* CMakeLists.txt changes to detect AWS-LC
* Compile-time branches needed to support AWS-LC
* Correctly set OSSL_VERSION and report AWS-LC release number
* GitHub Actions script to build with autoconf and cmake against AWS-LC
AWS-LC is a BoringSSL/OpenSSL derivative
For more information see https://github.com/awslabs/aws-lc/Closes#10320
SSL backends like OpenSSL/wolfSSL and other return the content of one
TLS record on read, but usually there are more available.
Change the vtls cfilter recv() function to fill the given buffer until a
read would block.
Closes#10736
Some IP cameras send malformed RTSP interleaved frames sometimes, which
can cause curl_easy_perform return 1 (CURLE_UNSUPPORTED_PROTOCOL). This
change attempts to skip clearly incorrect RTSP interleaving frame data.
Closes#10808
Adding `bufq`:
- at init() time configured to hold up to `n` chunks of `m` bytes each.
- various methods for reading from and writing to it.
- `peek` support to get access to buffered data without copy
- `pass` support to allow buffer flushing on write if it becomes full
- use case: IO buffers for dynamic reads and writes that do not blow up
- distinct from `dynbuf` in that:
- it maintains a read position
- writes on a full bufq return CURLE_AGAIN instead of nuking itself
- Init options:
- SOFT_LIMIT: allow writes into a full bufq
- NO_SPARES: free empty chunks right away
- a `bufc_pool` that can keep a number of spare chunks to
be shared between different `bufq` instances
Adding `dynhds`:
- a straightforward list of name+value pairs as used for HTTP headers
- headers can be appended dynamically
- headers can be removed again
- headers can be replaced
- headers can be looked up
- http/1.1 formatting into a `dynbuf`
- configured at init() with limits on header counts and total string
sizes
- use case: pass a HTTP request or response around without being version
specific
- express a HTTP request without a curl easy handle (used in h2 proxy
tunnels)
- future extension possibilities:
- conversions of `dynhds` to nghttp2/nghttp3 name+value arrays
Closes#10720
As dynbufs always have a fixed maximum size which they are not allowed
to grow larger than, making sure that it never allocates a larger buffer
makes sure the buffer does not allocate memory that will never be used.
Closes#10845
The public 'curl_fileinfo' struct contained three fields that are for
internal purposes only. This change makes them unused in the public
struct.
The new private struct fields are also renamed to make this separation
more obvious internally.
Closes#10844
As they are not driving transfers or any socket activity, the main loop
does not need to iterate over these handles. A performance improvement.
They are instead only held in their own separate lists.
'data->multi' is kept a pointer to the multi handle as long as the easy
handle is actually part of it even when the handle is moved to the
pending/msgsent lists. It needs to know which multi handle it belongs
to, if for example curl_easy_cleanup() is called before the handle is
removed from the multi handle.
Alll 'data->multi' pointers of handles still part of the multi handle
gets cleared by curl_multi_cleanup() which "orphans" all previously
attached easy handles.
This is take 2. The first version was reverted for the 8.0.1 release.
Assisted-by: Stefan Eissing
Closes#10801
- make configure show on HTTP3 feature that both ngtcp2 and nghttp3
are in play
- define ENABLE_QUIC only when USE_NGTCP2 and USE_NGHTTP3 are defined
- add USE_NGHTTP3 in the ngtcp2 implementation
Fixes#10793Closes#10821
For GOOD_EASY_HANDLE and GOOD_MULTI_HANDLE checks
- allow NULL pointers to "just" return an error as before
- fail hard on nun-NULL pointers that no longer show the MAGICs
Closes#10812
Various compile failures in gskit.c;
- pipe_ssloverssl() needs Curl_easy data parameter for
Curl_conn_cf_get_socket(cf, data)
- key_passwd is in ssl_config, not conn_config
- close_on() has 2 parameters, not 4
- getsockopt() needs to call Curl_conn_cf_get_socket(), not
cxn->sock[FIRSTSOCKET]
Fixes#10799Closes#10800
It turns out c-ares returns an error when asked to resolve a host name with
ares_getaddrinfo using port number 0.
Reported as a c-ares bug here: https://github.com/c-ares/c-ares/issues/517
The work-around is to simply use port 80 instead, as the number typically does
not make a difference and a non-zero number works for c-ares.
Fixes#10759
Reported-by: Matt Jolly
Closes#10789
As they are not driving transfers or any socket activity, the main loop
does not need to iterate over these handles. A performance improvement.
They are instead only held in their own separate lists.
Assisted-by: Stefan Eissing
Ref: #10743Closes#10762
Linked lists themselves do not carry any allocations, so for the lists
that do not have have a set destructor we can just skip the
Curl_llist_destroy() call and save CPU time.
Closes#10764
all s3 requests default to UNSIGNED-PAYLOAD and add the required
x-amz-content-sha256 header. this allows CURLAUTH_AWS_SIGV4 to correctly
sign s3 requests to amazon with no additional configuration
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Closes#9995
- add QUIC/ngtcp2 detection in CMake with wolfSSL.
Because wolfSSL uses zlib if available, move compression detection
before TLS detection. (OpenSSL might also need this in the future.)
- wolfSSL 5.5.0 started using C99 types in its `quic.h` header, but it
doesn't #include the necessary C99 header itself, breaking builds
(unless another dependency pulled it by chance.) Add local workaround
for it. For this to work with all build tools, we had to fix our
header detection first. Ref: #10745
Ref: 6ad5f6ecc1Closes#10739
- use the defined, but so far not used, KEEP_SEND_HOLD bit for flow
control based suspend of sending in transfers.
Prior to this change KEEP_SEND_PAUSE bit was used instead, but that can
interfere with pausing streams from the user side via curl_easy_pause.
Fixes https://github.com/curl/curl/issues/10751
Closes https://github.com/curl/curl/pull/10753
Fix `stdint.h` and `inttypes.h` detection with non-autotools builds on
Windows. (autotools already auto-detected them accurately.)
`lib/config-win32.h` builds (e.g. `Makefile.mk`):
- set `HAVE_STDINT_H` where supported.
- set `HAVE_INTTYPES_H` for MinGW.
CMake:
- auto-detect them on Windows. (They were both force-disabled.)
- delete unused `CURL_PULL_STDINT_H`.
- delete unused `CURL_PULL_INTTYPES_H`.
- stop detecting `HAVE_STDINT_H` twice.
Present since the initial CMake commit: 4c5307b456
curl doesn't use these C99 headers, we need them now to workaround
broken wolfSSL builds. Ref: #10739
Once that clears up, we can delete these detections and macros (unless
we want to keep them for future us.)
Reviewed-by: Daniel Stenberg
Closes#10745
This is already how curl is documented to behave in Everything curl, but
in actuality only short POSTs skip this. This should knock 30 seconds
off a full run of the test suite since the 100-continue timeout will no
longer be hit.
Closes#10740
RST and connection close were not handled correctly during parallel
transfers, leading to aborted response bodies being reported complete.
Closes#10715
brotli v1.0.0 throughout current latest v1.0.9 and latest master [1]
trigger this warning.
It happened with CMake and GNU Make. autotools builds avoid it with
the `convert -I options to -isystem` macro.
llvm/clang:
```
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:204:34: warning: variable length array used [-Wvla]
const uint8_t encoded_buffer[BROTLI_ARRAY_PARAM(encoded_size)],
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./brotli/x64-ucrt/usr/include/brotli/port.h:253:34: note: expanded from macro 'BROTLI_ARRAY_PARAM'
^~~~~~
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:206:48: warning: variable length array used [-Wvla]
uint8_t decoded_buffer[BROTLI_ARRAY_PARAM(*decoded_size)]);
~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
./brotli/x64-ucrt/usr/include/brotli/port.h:253:35: note: expanded from macro 'BROTLI_ARRAY_PARAM'
~^~~~~
```
gcc:
```
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:204:5: warning: ISO C90 forbids variable length array 'encoded_buffer' [-Wvla]
204 | const uint8_t encoded_buffer[BROTLI_ARRAY_PARAM(encoded_size)],
| ^~~~~
./brotli/x64-ucrt/usr/include/brotli/decode.h:206:5: warning: ISO C90 forbids variable length array 'decoded_buffer' [-Wvla]
206 | uint8_t decoded_buffer[BROTLI_ARRAY_PARAM(*decoded_size)]);
| ^~~~~~~
```
[1] ed1995b6bd
Reviewed-by: Daniel Stenberg
Reviewed-by: Marcel Raad
Closes#10738
In pytest'ing the situation occored that wolfSSL reported an
IO error when the underlying BIO operation was returning an
CURLE_AGAIN condition.
Readding the `io_result` filter context member to detect such
situations.
Also, making sure that the returned CURLcode is initialized
on all recv operations outcome.
Closes#10716
This makes us debug libssh2 less and libcurl more when for example
running torture tests that otherwise will spend a lot of time in libssh2
functions.
We leave libssh2 to test libssh2.
Closes#10721
By letting curl_easy_header() and curl_easy_nextheader() store the
header data in their own struct storage when they return a pointer to
it, it makes it possible for applications to use them both in a loop.
Like the curl tool does.
Reported-by: Boris Okunskiy
Fixes#10704Closes#10707
- since 7.87.0 we lost adding the SSL filter for an active
FTP connection that uses SSL. This leads to hangers and timeouts
as reported in #10666.
Reported-by: SandakovMM on github
Fixes#10666Closes#10669
- add parameter to `conn_is_alive()` cfilter method that returns
if there is input data waiting on the connection
- refrain from re-using connnection from the cache that have
input pending
- adapt http/2 and http/3 alive checks to digest pending input
to check the connection state
- remove check_cxn method from openssl as that was just doing
what the socket filter now does.
- add tests for connection reuse with special server configs
Closes#10690
- a reset transfer (HTTP/2 RST) did not always lead to the proper
error message on receiving its response, leading to wrong reports
of a successful transfer
- test_05_02 was able to trigger this condition with increased transfer
count. The simulated response errors did not carry a 'Content-Length'
so only proper RST handling could detect the abort
- When doing such transfers in parallel, a connection could enter the
state where
a) it had been closed (GOAWAY received)
b) the RST had not been "seen" for the transfer yet
or c) the GOAWAY announced an error and the last successful
stream id was not checked against ongoing transfers
Closes#10693
- time_connect was not updated when the overall connection failed,
e.g. when SSL verification was unsuccessful, refs #10670
- rework gather those values to interrogate involved filters,
also from all eyeballing attempts, to report the maximum of
those values.
- added 3 test cases in test_06 to check reported values on
successful, partially failed and totally failed connections.
Reported-by: Master Inspire
Fixes#10670Closes#10671
Normally curl uses cryptographically strong random provided by the
selected SSL backend. If compiled without SSL support, a naive built-in
function was used instead.
Generally this was okay, but it will result in some downsides for non-
SSL builds, such as predictable temporary file names.
This change ensures that arc4random will be used instead, if available.
Closes#10672
Before this patch, enabling LDAPS required a manual C flag:
c1cfc31cfc/curl-cmake.sh (L105)
Fix this and enable LDAPS automatically when using `wldap32` (and
when not explicitly disabled). This matches autotools and `Makefile.mk`
behavior. Also remove issue from KNOWN_BUGS.
Add workaround for MSVS 2010 warning triggered by LDAPS now enabled
in more CI tests:
`ldap.c(360): warning C4306: 'type cast' : conversion from 'int' to 'void *' of greater size`
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/46408284/job/v8mwl9yfbmoeqwlr#L312
Reported-by: JackBoosY on github
Reviewed-by: Jay Satiro
Reviewed-by: Marcel Raad
Fixes#6284Closes#10674
Since abebb2b893, we set this macro for
all Windows `wldap32` builds using `Makefile.mk`.
For OpenLDAP builds this macro is not enough to enable LDAPS, and
OpenLDAP is not an option in `Makefile.mk`. For Novell LDAP it might
have helped, but it's also not an option anymore in `Makefile.mk`.
The future for LDAPS is that we should enable it by default without
extra build knobs.
Reviewed-by: Marcel Raad
Closes#10681
The feature is rarely used so this frees up data for the vast majority
of easy handles that don't use it.
Rename "protdata" to "ftpwc" since it is always an FTP wildcard struct
pointer. Made the state struct field an unsigned char to save space.
Closes#10639
- refs #10646 where reuse was attempted on closed connections in the
cache, leading to an exhaustion of retries on a transfer
- the mistake was that poll events like POLLHUP, POLLERR, etc
were regarded as "not dead".
- change cf-socket filter check to regard such events as inidication
of corpsiness.
- vtls filter checks: fixed interpretation of backend check result
when inconclusive to interrogate status further down the filter
chain.
Reported-by: SendSonS on github
Fixes#10646Closes#10652
In 7.87.0, if callback method for CURLOPT_SOCKOPTFUNCTION returns
CURL_SOCKOPT_ALREADY_CONNECTED then curl library used to return
CURLE_OK. n 7.88.0, now even if callback returns
CURL_SOCKOPT_ALREADY_CONNECTED, curl library still tries to connect to
socket by invoking method do_connect().
This is regression caused by commit
https://github.com/curl/curl/commit/71b7e0161032927cdfb
Fix: Check if we are already connected and return CURLE_OK.
Fixes#10626Closes#10648
- Change readwrite_upload() to call win_update_buffer_size() no more
than once a second to update SO_SNDBUF (send buffer limit).
Prior to this change during an upload readwrite_upload() could call
win_update_buffer_size() anywhere from hundreds of times per second to
an extreme test case of 100k per second (which is likely due to a bug,
see #10618). In the latter case WPA profiler showed
win_update_buffer_size was the highest capture count in
readwrite_upload. In any case the calls were excessive and unnecessary.
Ref: https://github.com/curl/curl/pull/2762
Closes https://github.com/curl/curl/pull/10611
- refs #10634 where errors in the HTTP/2 framing layer are observed.
- the bug was that on connection reuse, the code attempted to switch
in yet another layer of HTTP/2 handling instead of detecting that
this was already in place.
- added pytest testcase reproducing the issue.
Reported-by: rwmjones on github
Fixes#10634Closes#10643
- do not try to determine the remote address of a listen socket. There
is none.
- Update remote address of an accepted socket by getpeername() if
available.
Reported-by: Harry Sintonen
Fixes#10622Closes#10642
- when h2/h3 eyeballing was involved, unix domain socket
configurations were not honoured
- configuring --unix-socket will disable HTTP/3 as candidate for eyeballing
- combinatino of --unix-socket and --http3-only will fail during initialisation
- adding pytest test_11 to reproduce
Reported-by: Jelle van der Waa
Fixes#10633Closes#10641
Some IDN sequences are converted into "" (nothing), which can make this
function end up with a zero length host name and we cannot consider that
a valid host to continue with.
Reported-by: Maciej Domanski
Closes#10617
POLLRDBAND does not seem to be an general error and on Windows the value
for POLLIN is 768 and the value for POLLRDBAND is 512.
Fixes#10501
Reported-by: opensslonzos-github on github
Closes#10592
Since cbf57176 the Cirrus CI 'macOS arm64 SecureTransport http2' has
been failing due to c89 warnings mixed code/declaration. That commit is
not the cause so I assume something has changed in the CI outside of our
configuration. Anyway, we don't mix code/declaration so this is the fix
for that.
Closes https://github.com/curl/curl/pull/10574
As tested in test_02_07, when firing off 200 urls with --parallel, 199
wait for the first connection to be established. if that is multiuse,
urls are added up to its capacity.
The first url over capacity opens another connection. But subsequent
urls found the same situation and open a connection too. They should
have waited for the second connection to actually connect and make its
capacity known.
This change fixes that by
- setting `connkeep()` early in the HTTP setup handler. as otherwise
a new connection is marked as closeit by default and not considered
for multiuse at all
- checking the "connected" status for a candidate always and continuing
to PIPEWAIT if no alternative is found.
pytest:
- removed "skip" from test_02_07
- added test_02_07b to check that http/1.1 continues to work as before
Closes#10456
- sscanf is slow and complex, avoid it
- give up already if the string is 12 bytes or longer as no valid string
can be that long
- this can now be done without copy
Closes#10547
- Define OPENSSL_SUPPRESS_DEPRECATED.
OpenSSL 3 has deprecated some of the functions libcurl uses such as
those with DES, MD5 and ENGINE prefix. We don't have replacements for
those functions so the warnings were disabled in autotools and cmake
builds, but still showed in other builds.
Closes https://github.com/curl/curl/pull/10543
- sscanf() is rather complex and slow, strchr() much simpler
- the port number function does not need to fully verify the IPv6 address
anyway as it is done later in the hostname_check() function and doing
it twice is unnecessary.
Closes#10541
Add const qualifier to 5th argument of Curl_os400_sendto()
Make OS400 wrapper for sendto match the normal prototype of sendto()
with a const qualifier.
Fixes#10539Closes#10548
For SOCKS modes where a local host resolve is done.
It was previously disabled in 12d655d456, but a few local tests seem to
indicate that it works fine. Works now because of the SOCKS refactor of
4a4b63daaa that made it non-blocking.
Reported-by: roughtex on github
Fixes#10537Closes#10540
- connect timeout was used at half the configured value, if the
destination had 1 ip version 4 and other version 6 addresses
(or the other way around)
- extended test2600 to reproduce these cases
Reported-by: Michael Kaufmann
Fixes#10514Closes#10517
* do not process pending input data when copying pausedata to the
caller
* return CURLE_AGAIN if the output buffer could not be completely
written out.
Ref: #10525Closes#10529
Add an intermediate cast to `void *`, as done everywhere else when
casting from `sockaddr *` to `sockaddr_in *`.
Closes https://github.com/curl/curl/pull/10528
Turns out "/usr/include/et/com_err.h" typedefs this type (without proper
variable scoping).
comerr is the "common error description library" that apparently might be used
by krb5 code, which then makes this header get used in a curl build.
Reported-by: Bruno Henrique Batista Cruz da Silva
Fixed#10502Closes#10500
- adding pytest test_10 cases for proxy httpd setup tests
- fixing openssl bug in https: proxy hostname verification that
used the hostname of the request and not the proxy name.
Closes#10498
The protocol needs to know the size ahead of time, this is now a known
restriction and not a bug.
Also output a clearer error if the URL path does not contain proper
share.
Ref: #7896Closes#10484
This addresses undefined behaviour found using clang's UBsan:
curl/lib/setopt.c:177:14: runtime error: applying non-zero offset 1 to null pointer
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior curl/lib/setopt.c:177:14 in
Closes#10472
Changes numeric error codes into verbose error codes in two places.
Adds a prefix indicating that the error came from rustls, and in some
places which function it came from.
Adds special handling for RUSTLS_RESULT_UNEXPECTED_EOF, since the
default message of "Unexpected EOF" is insufficiently explanatory.
Closes#10463
Follow-up to e8b00fcd6a
Due to the new 'if(!nonblocking)' check on the block a level above,
there is no need to check for it again within the same conditional.
Detected by Coverity
Closes#10473
vquic stabilization
- udp send code shared between ngtcp2 and quiche
- quiche handling of data and events improved
ngtcp2 and pytest improvements
- fixes handling of "drain" situations, discovered in scorecard
tests with the Caddy server.
- improvements in handling transfers that have already data or
are already closed to make an early return on recv
pytest
- adding caddy tests when available
scorecard improvemnts.
- using correct caddy port
- allowing tests for only httpd or caddy
Closes#10451
- as reported in #10433, HTTP/2 uploads may stall when a response is
received before the upload is done. This happens when the
data->state.drain is set for such a transfer, as the special handling
in transfer.c from then on only cared about downloads.
- add continuation of uploads, if applicable, in this case.
- add pytest case test_07_12_upload_seq_large to reproduce this scenario
(although, current nghttp2 implementation is using drain less often)
Reported-by: Lucas Pardue
Fixes#10433Closes#10443
If curl is built with openldap support (USE_OPENLDAP=1) but does not
have also some other protocol (IMAP/SMTP/POP3) enabled that brings
in Curl_sasl_* functions, then the build will fail with undefined
references to various symbols:
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_decode_mech'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_parse_url_auth_option'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_cleanup'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_can_authenticate'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_continue'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_start'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_init'
This was tracked down to these functions bein used in openldap.c but
defined in curl_sasl.c and then forward in two vauth/ files to have
a guard against a set of #define configurations that was now extended
to cover also this case.
Example configuration targeted that could reproduce the problem:
curl 7.87.1-DEV () libcurl/7.87.1-DEV .... OpenLDAP/2.6.3
Protocols: file ftp ftps http https ldap ldaps
Closes#10445
Prior to this change a workaround for Windows to recv before every send
was enabled by default. The way it works is a recv is called before
every send and saves the received data, in case send fails because in
Windows apparently that can wipe out the socket's internal received
data buffer.
This feature has led to several bugs because the way libcurl operates
it waits on a socket to read or to write, and may not at all times
check for buffered receive data.
Two recent significant bugs this workaround caused:
- Broken Schannel TLS 1.3 connections (#9431)
- HTTP/2 arbitrary hangs (#10253)
The actual code remains though it is disabled by default. Though future
changes to connection filter buffering could improve the situation IMO
it's just not tenable to manage this workaround.
Ref: https://github.com/curl/curl/issues/657
Ref: https://github.com/curl/curl/pull/668
Ref: https://github.com/curl/curl/pull/720
Ref: https://github.com/curl/curl/issues/9431
Ref: https://github.com/curl/curl/issues/10253
Closes https://github.com/curl/curl/pull/10409
add a small buffer to nghttp2 session sending in order to aggregate
small SETTINGS/PRIO/WIN_UPDATE frames that nghttp2 "writes" to the
callback individually.
Ref: #10389Closes#10432
move Curl_ssl_setup_x509_store() call after the first send (ClientHello)
this gives time to parse CA anchors while waiting on the server reply
Ref: #10389Closes#10432
I believe the code was secure before this, but limiting the accepted
name length to what is used in the structures should help Coverity's
analysis.
Closes#10431
[CWE-570] V560: A part of conditional expression is always false: conn->bits.authneg.
[CWE-570] V560: A part of conditional expression is always false: conn->handler->protocol & (0 | 0).
https://pvs-studio.com/en/docs/warnings/v560/Closes#10399
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Closes#10349
Inspired by the recent fixes to escape.c, we should avoid calling
Curl_dyn_addf() in loops, perhaps in particular when adding something so
simple as %HH codes - for performance reasons. This change makes the
same thing for the URL parser's two URL-encoding loops.
Closes#10384
As a miniscule optimization, treat a path of the length 1 as the same as
non-existing, as it can only be a single leading slash, and that's what
we do for no paths as well.
Closes#10385
1. Add `USE_WOLFSSL` to the TLS backend priority list in
`lib/curl_ntlm_core.c`.
2. Fix `lib/curl_ntlm_core.h` to respect TLS backend priority, bringing
it in sync with the above list and `lib/curl_ntlm_core.c` itself.
Reported-by: Mark Roszko
Ref: https://github.com/curl/curl/issues/10321
3. Allow enabling both wolfSSL and OpenSSL at the same time in
`lib/Makefile.mk` bringing this in line with cmake/autotools builds.
Update logic to select the crypto-specific lib for `ngtcp2`, which
supports a single TLS backend at the same time.
Closes#10322
- add test2600 as a unit test that triggers various connect conditions
and monitors behaviour, available in a debug build only.
- this exposed edge cases in connect.c that have been fixed
Closes#10312
- Curl_write_plain/Curl_read_plain have been eliminated. Last code use
now uses Curl_conn_send/recv so that requests use conn->send/revc
callbacks which defaults to cfilters use.
- Curl_recv_plain/Curl_send_plain have been internalized in cf-socket.c.
- USE_RECV_BEFORE_SEND_WORKAROUND (active on Windows) has been moved
into cf-socket.c. The pre_recv buffer is held at the socket filter
context. `postponed_data` structures have been removed from
`connectdata`.
- the hanger in HTTP/2 request handling was a result of read buffering
on all sends and the multi handling is not prepared for this. The
following happens:
- multi preforms on a HTTP/2 easy handle
- h2 reads and processes data
- this leads to a send of h2 data
- which receives and buffers before the send
- h2 returns
- multi selects on the socket, but no data arrives (its in the buffer already)
the workaround now receives data in a loop as long as there is something in
the buffer. The real fix would be for multi to change, so that `data_pending`
is evaluated before deciding to wait on the socket.
io_buffer, optional, in cf-socket.c, http/2 sets state.drain if lower
filter have pending data.
This io_buffer is only available/used when the
-DUSE_RECV_BEFORE_SEND_WORKAROUND is active, e.g. on Windows
configurations. It also maintains the original checks on protocol
handler being HTTP and conn->send/recv not being replaced.
The HTTP/2 (nghttp2) cfilter now sets data->state.drain when it finds
out that the "lower" filter chain has still pending data at the end of
its IO operation. This prevents the processing from becoming stalled.
Closes#10280
