- bring bearssl handshake times down from +200ms down to other TLS backends
- vtls: improve generic get_select_socks() implementation
- tests: provide Apache with a suitable ssl session cache
Closes#11675
- check in h2 filter recv that stream actually exists
and return error if not
- add test for parallel, extreme h2 upgrades that fail if
connections get reused before fully switched
- add h2 upgrade upload test just for completeness
Closes#11563
- adding tests using very large passwords in auth
- fixes general http sending to treat h3 like h2, and
not like http1.1
- eliminate H2_HEADER max definitions and use the commmon
DYN_HTTP_REQUEST everywhere, different limits do not help
- fix http2 handling of requests denied by nghttp2 on send
to immediately report the refused stream
Closes#11509
- fix HTTP/2 check to not declare a connection dead when
the read attempt results in EAGAIN
- add H2-PROXY alive check as for HTTP/2 that was missing
and is needed
- add attach/detach around Curl_conn_is_alive() and remove
these in filter methods
- add checks for number of connections used in some test_10
proxy tunneling tests
Closes#11368
- refs #11357, where it was reported that HTTP/1.1 downgrades
no longer works
- fixed with suggested change
- added test_05_03 and a new handler in the curltest module
to reproduce that downgrades work
Fixes#11357Closes#11362
Reported-by: Jay Satiro
- refs #11203 where hyper was reported as being slow
- fixes hyper_executor_poll to loop until it is out of
tasks as advised by @seanmonstar in https://github.com/hyperium/hyper/issues/3237
- added a fix in hyper io handling for detecting EAGAIN
- added some debug logs to see IO results
- pytest http/1.1 test cases pass
- pytest h2 test cases fail on connection reuse. HTTP/2
connection reuse does not seem to work. Hyper submits
a request on a reused connection, curl's IO works and
thereafter hyper declares `Hyper: [1] operation was canceled: connection closed`
on stderr without any error being logged before.
Fixes#11203
Reported-by: Gisle Vanem
Advised-by: Sean McArthur
Closes#11344
- leave transfer loop when --limit-rate is in effect and has
been received
- adjust stream window size to --limit-rate plus some slack
to make the server observe the pacing we want
- add test case to confirm behaviour
Closes#11115
- doing a POST with `--digest` does an override on the initial request
with `Content-Length: 0`, but the http2 filter was unaware of that
and expected the originally request body. It did therefore not
send a final DATA frame with EOF flag to the server.
- The fix overrides any initial notion of post size when the `done_send`
event is triggered by the transfer loop, leading to the EOF that
is necessary.
- refs #11194. The fault did not happen in testing, as Apache httpd
never tries to read the request body of the initial request,
sends the 401 reply and closes the stream. The server used in the
reported issue however tried to read the EOF and timed out on the
request.
Reported-by: Aleksander Mazur
Fixes#11194
Cloes #11200
- refs #11157 and #11175 where uploads get stuck or lead to RST streams
- fixes our h2 send behaviour to continue sending in the nghttp2 session
as long as it wants to. This will empty our send buffer as long as
the remote stream/connection window allows.
- in case the window is exhausted, the data remaining in the send buffer
will wait for a WINDOW_UPDATE from the server. Which is a socket event
that engages our transfer loop again
- the problem in the issue was that we did not exhaust the window, but
left data in the sendbuffer and no further socket events did happen.
The server was just waiting for us to send more.
- relatedly, there was an issue fixed that closing a stream with KEEP_HOLD
set kept the transfer from shutting down - as it should have - leading
to a timeout.
Closes#11176
Make send buffer smaller to have progress and "upload done" reporting
closer to reality. Fix handling of send "drain" condition to no longer
trigger once the transfer loop reports it is done sending. Also do not
trigger the send "drain" on RST streams.
Background:
- a upload stall was reported in #11157 that timed out
- test_07_33a reproduces a problem with such a stall if the
server 404s the request and RSTs the stream.
- test_07_33b verifies a successful PUT, using the parameters
from #11157 and checks success
Ref: #11157Closes#11165
- nghttp2 does not free connection level window flow for
aborted streams
- when closing transfers, make sure that any buffered
response data is "given back" to the flow control window
- add tests test_02_22 and test_02_23 to reproduce
Closes#11052
This is required for the macros there to take effect for system
libraries. Specifically, including the system libraries first led to
warnings about `_FILE_OFFSET_BITS` being redefined in curl_config.h on
the Solaris autobuilds for ws-data.c and ws-pingpong.c.
Also make the curl includes come first for the other source files here
for consistency.
Closes https://github.com/curl/curl/pull/11046
Add both lib/ directories (src & build) to the search path so
curl_setup.h and its dependencies can be found.
Followup-to acd82c8b
Ref: #11006Closes#11036
- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_02 for server PUSH_PROMISEs using clients/h2-serverpush
- added test_02_21 for lib based downloads and pausing/unpausing transfers
curl url parser:
- added internal method `curl_url_set_authority()` for setting the
authority part of a url (used for PUSH_PROMISE)
http2:
- made logging of PUSH_PROMISE handling nicer
Placing python test requirements in requirements.txt files
- separate files to base test suite and http tests since use
and module lists differ
- using the files in the gh workflows
websocket test cases, fixes for we and bufq
- bufq: account for spare chunks in space calculation
- bufq: reset chunks that are skipped empty
- ws: correctly encode frames with 126 bytes payload
- ws: update frame meta information on first call of collect
callback that fills user buffer
- test client ws-data: some test/reporting improvements
Closes#11006
- `drain` was used by http/2 and http/3 implementations to indicate
that the transfer requires send/recv independant from its socket
poll state. Intended as a counter, it was used as bool flag only.
- a similar mechanism exists on `connectdata->cselect_bits` where
specific protocols can indicate something similar, only for the
whole connection.
- `cselect_bits` are cleard in transfer.c on use and, importantly,
also set when the transfer loop expended its `maxloops` tries.
`drain` was not cleared by transfer and the http2/3 implementations
had to take care of that.
- `dselect_bits` is cleared *and* set by the transfer loop. http2/3
does no longer clear it, only set when new events happen.
This change unifies the handling of socket poll overrides, extending
`cselect_bits` by a easy handle specific value and a common treatment in
transfers.
Closes#11005
- with `--proxy-http2` allow h2 ALPN negotiation to
forward proxies
- applies to http: requests against a https: proxy only,
as https: requests will auto-tunnel
- adding a HTTP/1 request parser in http1.c
- removed h2h3.c
- using new request parser in nghttp2 and all h3 backends
- adding test 2603 for request parser
- adding h2 proxy test cases to test_10_*
scorecard.py: request scoring accidentally always run curl
with '-v'. Removed that, expect double numbers.
labeller: added http1.* and h2-proxy sources to detection
Closes#10967
- currently only on debug build and when env variable
CURL_PROXY_TUNNEL_H2 is present.
- will ALPN negotiate with the proxy server and switch
tunnel filter based on the protocol negotiated.
- http/1.1 tunnel code moved into cf-h1-proxy.[ch]
- http/2 tunnel code implemented in cf-h2-proxy.[ch]
- tunnel start and ALPN set remains in http_proxy.c
- moving all haproxy related code into cf-haproxy.[ch]
VTLS changes
- SSL filters rely solely on the "alpn" specification they
are created with and no longer check conn->bits.tls_enable_alpn.
- checks on which ALPN specification to use (or none at all) are
done in vtls.c when creating the filter.
Testing
- added a nghttpx forward proxy to the pytest setup that
speaks HTTP/2 and forwards all requests to the Apache httpd
forward proxy server.
- extending test coverage in test_10 cases
- adding proxy tests for direct/tunnel h1/h2 use of basic auth.
- adding test for http/1.1 and h2 proxy tunneling to pytest
Closes#10780
- eliminate receive loop in vtls to fill buffer. This may
lead to partial reads of data which is counter productive
- let http2 instead loop smarter to process pending network
data without transfer switches
scorecard improvements
- do not start caddy when only httpd is requested
- allow curl -v to stderr file on --curl-verbose
Closes#10891
- when rustls is told to recieve more TLS data and its internal
plaintext buffers are full, it returns an IOERROR
- avoid receiving TLS data while plaintext is not read empty
pytest:
- increase curl run timeout when invoking pytest with higher verbosity
Closes#10876
- ngtcp2: using bufq for recv stream data
- internal stream_ctx instead of `struct HTTP` members
for quiche, ngtcp2 and msh3
- no more QUIC related members in `struct HTTP`
- experimental use of recvmmsg(), disabled by default
- testing on my old debian box shows no throughput improvements.
- leaving it in, but disabled, for future revisit
- vquic: common UDP receive code for ngtcp2 and quiche
- vquic: common UDP send code for ngtcp2 and quiche
- added pytest skips for known msh3 failures
- fix unit2601 to survive torture testing
- quiche: using latest `master` from quiche and enabling large download
tests, now that key change is supported
- fixing test_07_21 where retry handling of starting a stream
was faulty
- msh3: use bufq for recv buffering headers and data
- msh3: replace fprintf debug logging with LOG_CF where possible
- msh3: force QUIC expire timers on recv/send to have more than
1 request per second served
Closes#10772
- use bufq for send/receive of network data
- usd bufq for send/receive of stream data
- use HTTP/2 flow control with no-auto updates to control the
amount of data we are buffering for a stream
HTTP/2 stream window set to 128K after local tests, defined
code constant for now
- elminiating PAUSEing nghttp2 processing when receiving data
since a stream can now take in all DATA nghttp2 forwards
Improved scorecard and adjuste http2 stream window sizes
- scorecard improved output formatting and options default
- scorecard now also benchmarks small requests / second
Closes#10771
- will check built curl for http and https support and
skip all tests if not there
- will dump stdout/stderr/trace output on errored responses
Closes#10829
- we had a CI case once where `curl` seemingly did not
return and it was hard to guess what happened.
- make curl execution in test cases time out after 60 seconds
Closes#10783
- added to: ngtcp2-quictls, ngtcp2-gnutls and the linux varians
quiche, bearssl, libressl, mbedtls, openssl3, rustls
- added disabled in ngtcp2-wolfssl due to weird SSL_connect() errors
not reproducable locally
Improvements on pytest:
- handling of systems with nghttpx in $PATH
- configure will seach $PATH got nghttpx used in pytest
- pytest fixes for managing nghttpx without h3 support
- ngtcp2-wolfssl: use a fully enabled wolfssl build
- lower parallel count for http/1.1 tests, since we do not
want to test excessive connections.
- check built curl for HTTPS-proxy support in proxy tests
- bearssl does not like one of our critical cert extensions, making
it non-critical now
- bearssl is too slow for test_12, skipping
- making sure we do h3 tests only when curl and server support is there
Closes#10699
- add parameter to `conn_is_alive()` cfilter method that returns
if there is input data waiting on the connection
- refrain from re-using connnection from the cache that have
input pending
- adapt http/2 and http/3 alive checks to digest pending input
to check the connection state
- remove check_cxn method from openssl as that was just doing
what the socket filter now does.
- add tests for connection reuse with special server configs
Closes#10690
- a reset transfer (HTTP/2 RST) did not always lead to the proper
error message on receiving its response, leading to wrong reports
of a successful transfer
- test_05_02 was able to trigger this condition with increased transfer
count. The simulated response errors did not carry a 'Content-Length'
so only proper RST handling could detect the abort
- When doing such transfers in parallel, a connection could enter the
state where
a) it had been closed (GOAWAY received)
b) the RST had not been "seen" for the transfer yet
or c) the GOAWAY announced an error and the last successful
stream id was not checked against ongoing transfers
Closes#10693
- necessary ports are bound at start of test suite and then
given to server fixtures for use.
- this make parallel use of pytest (in separate directories),
practically safe for use as OS tend to not reuse such port numbers
for a while
Closes#10692
- time_connect was not updated when the overall connection failed,
e.g. when SSL verification was unsuccessful, refs #10670
- rework gather those values to interrogate involved filters,
also from all eyeballing attempts, to report the maximum of
those values.
- added 3 test cases in test_06 to check reported values on
successful, partially failed and totally failed connections.
Reported-by: Master Inspire
Fixes#10670Closes#10671
- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more beefy environment
Closes#10654
