Commit Graph

36 Commits

Author SHA1 Message Date
Stefan Eissing
709a6a3965
cfilters: send flush
Since data can be held in connection filter buffers when sending gives
EAGAIN, add methods to query this and perform flushing of those buffers.

The transfer loop will continue sending until all upload data is
processed and the connection is flushed.

- add `CF_QUERY_SEND_PENDING` to query filters
- add `CF_CTRL_DATA_SEND_FLUSH` to flush filters
- change `Curl_req_want_send()` to query the connection
  if it needs flushing
- use `Curl_req_want_send()` to determine the POLLOUT
  in the PERFORMING multi state
- implement flush handling in the HTTP/2 connection filter

Closes #14271
2024-08-03 19:55:45 +02:00
Stefan Eissing
911c3166b6
lib: add eos flag to send methods
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.

Closes #14220
2024-08-03 19:53:54 +02:00
Daniel Stenberg
25321de30e
Revert "lib: send eos flag"
This reverts commit be93299f10.
2024-07-19 01:38:05 +02:00
Stefan Eissing
be93299f10
lib: send eos flag
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.

Closes #14220
2024-07-18 23:27:35 +02:00
Daniel Stenberg
c074ba64a8
code: language cleanup in comments
Based on the standards and guidelines we use for our documentation.

 - expand contractions (they're => they are etc)
 - host name = > hostname
 - file name => filename
 - user name = username
 - man page => manpage
 - run-time => runtime
 - set-up => setup
 - back-end => backend
 - a HTTP => an HTTP
 - Two spaces after a period => one space after period

Closes #14073
2024-07-01 22:58:55 +02:00
Stefan Eissing
c9b95c0bb3
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
  involved (each transfer might carry its own) into a temporary list.
  Let each connection cache on the list contribute sockets and
  POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
  them peform their maintenance. This will make another non-blocking
  attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
  their poll events via the callback. When `multi_socket()` is invoked
  for a socket not known by an active transfer, forward this to the
  multi's cache for processing. On closing a connection, remove its
  socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
  which happen without a graceful shutdown and should
  no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
  on FTP up- and downloads.

Closes #13976
2024-06-26 08:33:17 +02:00
Stefan Eissing
385c62aabc
lib: xfer_setup and non-blocking shutdown
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
  which socket they operate on. Add a shutdown flag for secondary
  sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
  sending a transfer

Closes #13913
2024-06-11 13:41:03 +02:00
Stefan Eissing
c31041b17e
connection: shutdown TLS (for FTP) better
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.

    - add cfilter `Curl_cft_shutdown` callback
    - keep a shutdown start timestamp and timeout at connectdata
    - provide shutdown timeout default and member in
      `data->set.shutdowntimeout`.
    - provide methods for starting, interrogating and clearing
      shutdown timers
    - provide `Curl_conn_shutdown_blocking()` to shutdown the
      `sockindex` filter chain in a blocking way. Use that in FTP.
    - add `Curl_conn_cf_poll()` to wait for socket events during
      shutdown of a connection filter chain.
      This gets the monitoring sockets and events via the filters
      "adjust_pollset()" methods. This gives correct behaviour when
      shutting down a TLS connection through a HTTP/2 proxy.
    - Implement shutdown for all socket filters
      - for HTTP/2 and h2 proxying to send GOAWAY
      - for TLS backends to the best of their capabilities
      - for tcp socket filter to make a final, nonblocking
        receive to avoid unwanted RST states
    - add shutdown forwarding to happy eyeballers and
      https connect ballers when applicable.

Closes #13904
2024-06-10 13:08:12 +02:00
Viktor Szakats
fc8e0dee30
build: untangle UNITTESTS and DEBUGBUILD macros
- fix `DEBUGBUILD` guards that should be `UNITTESTS`, in libcurl code
  used by unit tests.
- fix guards for libcurl functions used in unit tests only.
- sync `UNITTEST` attribute between declarations and definitions.
- drop `DEBUGBUILD` guard from test `unit2600`.
- fix guards for libcurl HSTS code used by both a unit test (`unit1660`)
  and `test0446`.
- update an existing AppVeyor CI job to test the issues fixed.

This fixes building tests with `CURLDEBUG` enabled but `DEBUGBUILD`
disabled. This can happen when building tests with CMake with
`ENABLE_DEBUG=ON` in Release config, or with `ENABLE_CURLDEBUG=ON`
and _without_ `ENABLE_DEBUG=ON`. Possibly also with autotools
when using `--enable-curldebug` without `--enable-debug`.

Test results:
- before:
  https://ci.appveyor.com/project/curlorg/curl/builds/49835609
  https://ci.appveyor.com/project/curlorg/curl/builds/49898529/job/k8qpbs8idby70smw
  https://github.com/curl/curl/actions/runs/9259078835/job/25470318167?pr=13798#step:13:821
- after: https://ci.appveyor.com/project/curlorg/curl/builds/49839255
  (the two failures are unrelated, subject to PR #13705)

Ref: #13592 (issue discovery)
Ref: #13689 (CI testing this PR with `DEBUGBUILD`/`CURLDEBUG` combinations)
Closes #13694
2024-05-27 21:15:50 +02:00
Stefan Eissing
c6655f7029
http/2, http/3: decouple stream state from easy handle
- add `Curl_hash_offt` as hashmap between a `curl_off_t` and
  an object. Use this in h2+h3 connection filters to associate
  `data->id` with the internal stream state.
- changed implementations of all affected connection filters
- removed `h2_ctx*` and `h3_ctx*` from `struct HTTP` and thus
  the easy handle
- solves the problem of attaching "foreign protocol" easy handles
  during connection shutdown

Test 1616 verifies the new hash functions.

Closes #13204
2024-04-17 14:24:09 +02:00
RainRat
1087937992
misc: fix typos
Closes #13344
2024-04-11 15:44:22 +02:00
Stefan Eissing
3210101088
tls: use shared init code for TCP+QUIC
Closes #13172
2024-04-09 09:08:05 +02:00
RainRat
2cd78f525c misc: Fix typos in docs and lib
This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.

Author: RainRat on Github
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Dan Fandrich <dan@coneharvesters.com>
Closes: #13019
2024-03-01 09:59:48 +01:00
Stefan Eissing
3755153571
lib: Curl_read/Curl_write clarifications
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
  clarify when and at what level they operate
- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer
  setup of `conn->sockfd` and `conn->writesockfd` on which
  connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index
  as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
  naming consistency
- clarify that the special CURLE_AGAIN hangling to return
  `CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
  and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
  when it arrived in more than a single chunk

The method for sending not just raw bytes, but bytes that are either
"headers" or "body". The send abstraction stack, to to bottom, now is:

* `Curl_req_send()`: has parameter to indicate amount of header bytes,
  buffers all data.
* `Curl_xfer_send()`: knows on which socket index to send, returns
  amount of bytes sent.
* `Curl_conn_send()`: called with socket index, returns amount of bytes
  sent.

In addition there is `Curl_req_flush()` for writing out all buffered
bytes.

`Curl_req_send()` is active for requests without body,
`Curl_buffer_send()` still being used for others. This is because the
special quirks need to be addressed in future parts:

* `expect-100` handling
* `Curl_fillreadbuffer()` needs to add directly to the new
  `data->req.sendbuf`
* special body handlings, like `chunked` encodings and line end
  conversions will be moved into something like a Client Reader.

In functions of the pattern `CURLcode xxx_send(..., ssize_t *written)`,
replace the `ssize_t` with a `size_t`. It makes no sense to allow for negative
values as the returned `CURLcode` already specifies error conditions. This
allows easier handling of lengths without casting.

Closes #12964
2024-02-27 14:13:56 +01:00
Stefan Eissing
5929822114
lib: send rework
Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
  when and at what level they operate

- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer setup of
  `conn->sockfd` and `conn->writesockfd` on which connection filter
  chain to operate.

- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index as
  parameter.

- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
  consistency

- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
  with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
  returned by all other send() variants.

SingleRequest reshuffling

- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data

Closes #12963
2024-02-27 08:58:10 +01:00
Stefan Eissing
a0f94800d5
transfer: adjust_pollset improvements
- let `multi_getsock()` initialize the pollset in what the
  transfer state requires in regards to SEND/RECV
- change connection filters `adjust_pollset()` implementation
  to react on the presence of POLLIN/-OUT in the pollset and
  no longer check CURL_WANT_SEND/CURL_WANT_RECV
- cf-socket will no longer add POLLIN on its own
- http2 and http/3 filters will only do adjustments if the
  passed pollset wants to POLLIN/OUT for the transfer on
  the socket. This is similar to the HTTP/2 proxy filter
  and works in stacked filters.

Closes #12640
2024-01-07 14:36:25 +01:00
Daniel Stenberg
0510e8b58c
lib: fix comment typos
Five separate ones, found by codespell

Closes #12390
2023-11-23 12:35:59 +01:00
Stefan Eissing
37b5cf4fa0
cfilter: provide call to tell connection to forget a socket
- fixed libssh.c workaround for a socket being closed by
  the library
- eliminate the terrible hack in cf-socket.c to guess when
  this happened and try not closing the socket again.
- fixes race in eyeballing when socket could have failed to
  be closed for a discarded connect attempt

Closes #12207
2023-10-27 16:59:13 +02:00
Stefan Eissing
47f5b1a37f
lib: introduce struct easy_poll_set for poll information
Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.

Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.

This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.

Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.

After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.

Example:

 * transfer wants to send, adds POLLOUT
 * http/2 filter has a flow control block, removes POLLOUT and adds
   POLLIN (it is waiting on a WINDOW_UPDATE from the server)
 * TLS filter is connected and changes nothing
 * h2-proxy filter also has a flow control block on its tunnel stream,
   removes POLLOUT and adds POLLIN also.
 * socket filter is connected and changes nothing
 * The resulting pollset is then mixed together with all other transfers
   and their pollsets, just as before.

Use of `SEND_HOLD` is no longer necessary in the filters.

All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.

The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.

No noticeable difference in performance was detected in local scorecard
runs.

Closes #11833
2023-10-25 09:34:32 +02:00
Futaura
7ad4c9066e
cfilters: rename close/connect functions to avoid clashes
Rename `close` and `connect` in `struct Curl_cftype` for
consistency and to avoid clashes with macros of the same name
(the standard AmigaOS networking connect() function is implemented
via a macro).

Closes #11491
2023-07-20 23:35:33 +02:00
Daniel Stenberg
8e85764b7b
lib: remove unused functions, make single-use static
Closes #11174
2023-05-23 11:08:51 +02:00
Stefan Eissing
4ae2d9f24d
proxy: http2 proxy tunnel implementation
- currently only on debug build and when env variable
  CURL_PROXY_TUNNEL_H2 is present.
- will ALPN negotiate with the proxy server and switch
  tunnel filter based on the protocol negotiated.
- http/1.1 tunnel code moved into cf-h1-proxy.[ch]
- http/2 tunnel code implemented in cf-h2-proxy.[ch]
- tunnel start and ALPN set remains in http_proxy.c
- moving all haproxy related code into cf-haproxy.[ch]

VTLS changes
- SSL filters rely solely on the "alpn" specification they
  are created with and no longer check conn->bits.tls_enable_alpn.
- checks on which ALPN specification to use (or none at all) are
  done in vtls.c when creating the filter.

Testing
- added a nghttpx forward proxy to the pytest setup that
  speaks HTTP/2 and forwards all requests to the Apache httpd
  forward proxy server.
- extending test coverage in test_10 cases
- adding proxy tests for direct/tunnel h1/h2 use of basic auth.
- adding test for http/1.1 and h2 proxy tunneling to pytest

Closes #10780
2023-04-06 13:04:46 +02:00
Stefan Eissing
7c5637b8b4
url: fix logic in connection reuse to deny reuse on "unclean" connections
- add parameter to `conn_is_alive()` cfilter method that returns
  if there is input data waiting on the connection
- refrain from re-using connnection from the cache that have
  input pending
- adapt http/2 and http/3 alive checks to digest pending input
  to check the connection state
- remove check_cxn method from openssl as that was just doing
  what the socket filter now does.
- add tests for connection reuse with special server configs

Closes #10690
2023-03-07 12:50:31 +01:00
Stefan Eissing
257416023d
connect: fix time_connect and time_appconnect timer statistics
- time_connect was not updated when the overall connection failed,
  e.g. when SSL verification was unsuccessful, refs #10670
- rework gather those values to interrogate involved filters,
  also from all eyeballing attempts, to report the maximum of
  those values.
- added 3 test cases in test_06 to check reported values on
  successful, partially failed and totally failed connections.

Reported-by: Master Inspire
Fixes #10670
Closes #10671
2023-03-06 20:16:43 +01:00
John Bampton
0065b146eb
misc: fix spelling
Closes #10437
2023-02-08 08:30:05 +01:00
Stefan Eissing
671158242d
connections: introduce http/3 happy eyeballs
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
  the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
  succeed after 100ms (subject to change), a parallel attempt
  is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
  as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
  expires, the HTTPS-CONNECT filter checks if the QUIC filter
  has received any data from the server. If not, it will start
  the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
  for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
  implementation into the udp socket cfilter. Because unconnected
  UDP sockets are weird. For example they error when adding to a
  pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
  and closes immediately (so the local conn goes into DRAIN phase), the
  connection is torn down and a another attempt is made after a short grace
  period.
  This is the behaviour observed with nghttpx when we tell it to  shut
  down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
  Invoke:
    python3 tests/tests-httpd/scorecard.py --help
  for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
  report connection statistics. This is triggered when the connection
  has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
  a timestamp of when it happend. This allows for updating timers
  "later", e.g. a connect statistic after full connectivity has been
  reached.
- in case of HTTP eyeballing, the previous changes will update
  statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
  by a filter chain.
  Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
  for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
  checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.

Closes #10349
2023-02-02 09:57:34 +01:00
Stefan Eissing
5651a36d1a
cf-socket: improvements in socket I/O handling
- Curl_write_plain/Curl_read_plain have been eliminated. Last code use
  now uses Curl_conn_send/recv so that requests use conn->send/revc
  callbacks which defaults to cfilters use.
- Curl_recv_plain/Curl_send_plain have been internalized in cf-socket.c.
- USE_RECV_BEFORE_SEND_WORKAROUND (active on Windows) has been moved
  into cf-socket.c. The pre_recv buffer is held at the socket filter
  context.  `postponed_data` structures have been removed from
  `connectdata`.
- the hanger in HTTP/2 request handling was a result of read buffering
  on all sends and the multi handling is not prepared for this. The
  following happens:

   - multi preforms on a HTTP/2 easy handle
   - h2 reads and processes data
   - this leads to a send of h2 data
   - which receives and buffers before the send
   - h2 returns
   - multi selects on the socket, but no data arrives (its in the buffer already)
   the workaround now receives data in a loop as long as there is something in
   the buffer. The real fix would be for multi to change, so that `data_pending`
   is evaluated before deciding to wait on the socket.

io_buffer, optional, in cf-socket.c, http/2 sets state.drain if lower
filter have pending data.

This io_buffer is only available/used when the
-DUSE_RECV_BEFORE_SEND_WORKAROUND is active, e.g. on Windows
configurations. It also maintains the original checks on protocol
handler being HTTP and conn->send/recv not being replaced.

The HTTP/2 (nghttp2) cfilter now sets data->state.drain when it finds
out that the "lower" filter chain has still pending data at the end of
its IO operation. This prevents the processing from becoming stalled.

Closes #10280
2023-01-31 11:23:59 +01:00
Stefan Eissing
9e93bd47c2 vtls: Manage current easy handle in nested cfilter calls
The previous implementation cleared `data` so the outer invocation lost
its data, which could lead to a crash.

Bug: https://github.com/curl/curl/issues/10336
Reported-by: Fujii Hironori

Closes https://github.com/curl/curl/pull/10340
2023-01-26 03:05:01 -05:00
Stefan Eissing
db91dbbf2c
curl_log: for failf/infof and debug logging implementations
- new functions and macros for cfilter debugging
 - set CURL_DEBUG with names of cfilters where debug logging should be
   enabled
 - use GNUC __attribute__ to enable printf format checks during compile

Closes #10271
2023-01-12 10:14:17 +01:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
Stefan Eissing
71b7e01610
lib: connect/h2/h3 refactor
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.

 - eyeballing cfilter that uses sub-filters for performing parallel connects
 - socket cfilter for all transport types, including QUIC
 - QUIC implementations in cfilter, can now participate in eyeballing
 - connection setup is more dynamic in order to adapt to what filter did
   really connect.  Relevant to see if a SSL filter needs to be added or
   if SSL has already been provided
 - HTTP/3 test cases similar to HTTP/2
 - multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche

 - Fix for data attach/detach in VTLS filters that could lead to crashes
   during parallel transfers.
 - Eliminating setup() methods in cfilters, no longer needed.
 - Improving Curl_conn_is_alive() to replace Curl_connalive() and
   integrated ssl alive checks into cfilter.
 - Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
   connection into and persist it at the easy handle.

 - Several more cfilter related cleanups and moves:
   - stream_weigth and dependency info is now wrapped in struct
     Curl_data_priority
   - Curl_data_priority members depend is available in HTTP2|HTTP3
   - Curl_data_priority members depend on NGHTTP2 support
   - handling init/reset/cleanup of priority part of url.c
   - data->state.priority same struct, but shallow copy for compares only

 - PROTOPT_STREAM has been removed
   - Curl_conn_is_mulitplex() now available to check on capability

 - Adding query method to connection filters.
   - ngtcp2+quiche: implementing query for max concurrent transfers.

 - Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
   - setting keepalive timestamp on connect
   - DATA_SETUP is called after the connection has been completely
     setup (but may not connected yet) to allow filters to initialize
     data members they use.

 - there is no socket to be had with msh3, it is unclear how select
   shall work

 - manual test via "curl --http3 https://curl.se" fail with "empty
   reply from server".

 - Various socket/conn related cleanups:
   - Curl_socket is now Curl_socket_open and in cf-socket.c
   - Curl_closesocket is now Curl_socket_close and in cf-socket.c
   - Curl_ssl_use has been replaced with Cur_conn_is_ssl
   - Curl_conn_tcp_accepted_set has been split into
     Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
     with a clearer purpose

Closes #10141
2022-12-30 16:43:19 +01:00
Stefan Eissing
55807e6c05
tls: backends use connection filters for IO, enabling HTTPS-proxy
- OpenSSL (and compatible)
 - BearSSL
 - gnutls
 - mbedtls
 - rustls
 - schannel
 - secure-transport
 - wolfSSL (v5.0.0 and newer)

 This leaves only the following without HTTPS-proxy support:
 - gskit
 - nss
 - wolfSSL (versions earlier than v5.0.0)

Closes #9962
2022-11-28 13:56:23 +01:00
Stefan Eissing
f18956d0ca
cfilter: re-add conn as parameter to cfilter setup methods
- `Curl_ssl_get_config()` now returns the first config if no SSL proxy
  filter is active

- socket filter starts connection only on first invocation of its
  connect method

Fixes #9982
Closes #9983
2022-11-26 16:56:46 +01:00
Stefan Eissing
af22c2a546
vtls: localization of state data in filters
- almost all backend calls pass the Curl_cfilter intance instead of
   connectdata+sockindex
 - ssl_connect_data is remove from struct connectdata and made internal
   to vtls
 - ssl_connect_data is allocated in the added filter, kept at cf->ctx

 - added function to let a ssl filter access its ssl_primary_config and
   ssl_config_data this selects the propert subfields in conn and data,
   for filters added as plain or proxy
 - adjusted all backends to use the changed api
 - adjusted all backends to access config data via the exposed
   functions, no longer using conn or data directly

cfilter renames for clear purpose:

 - methods `Curl_conn_*(data, conn, sockindex)` work on the complete
   filter chain at `sockindex` and connection `conn`.
 - methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter
   instance.
 - methods `Curl_conn_cf()` work on/with filter instances at a
   connection.
 - rebased and resolved some naming conflicts
 - hostname validation (und session lookup) on SECONDARY use the same
   name as on FIRST (again).

new debug macros and removing connectdata from function signatures where not
needed.

adapting schannel for new Curl_read_plain paramter.

Closes #9919
2022-11-22 14:25:50 +01:00
Stefan Eissing
4a8b4a1b66
cfiler: filter types have flags indicating what they do
- Adding Curl_conn_is_ip_connected() to check if network connectivity
  has been reached

- having ftp wait for network connectivity before proceeding with
  transfers.

Fixes test failures 1631 and 1632 with hyper.

Closes #9952
2022-11-22 08:20:38 +01:00
Stefan Eissing
dafdb20a26
lib: connection filters (cfilter) addition to curl:
- general construct/destroy in connectdata
 - default implementations of callback functions
 - connect: cfilters for connect and accept
 - socks: cfilter for socks proxying
 - http_proxy: cfilter for http proxy tunneling
 - vtls: cfilters for primary and proxy ssl
 - change in general handling of data/conn
 - Curl_cfilter_setup() sets up filter chain based on data settings,
   if none are installed by the protocol handler setup
 - Curl_cfilter_connect() boot straps filters into `connected` status,
   used by handlers and multi to reach further stages
 - Curl_cfilter_is_connected() to check if a conn is connected,
   e.g. all filters have done their work
 - Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE
   indicators for multi select to work
 - Curl_cfilter_data_pending() asks filters if the have incoming
   data pending for recv
 - Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks
   installed in conn->recv/conn->send for io handling
 - Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters
   and addition/removal of a `data` from their connection
 - adding vtl functions to prevent use of Curl_ssl globals directly
   in other parts of the code.

Reviewed-by: Daniel Stenberg
Closes #9855
2022-11-11 15:17:51 +01:00