13624 Commits

Author SHA1 Message Date
Daniel Stenberg
c28e9478cb
openssl: when verifystatus fails, remove session id from cache
To prevent that it gets used in a subsequent transfer that skips the
verifystatus check since that check can't be done when the session id is
reused.

Reported-by: Hiroki Kurosawa
Closes #12760
2024-01-23 08:26:36 +01:00
Viktor Szakats
02f91d5b64
libssh2: use libssh2_session_callback_set2() with v1.11.1
To avoid a local hack to pass function pointers and to avoid
deprecation warnings when building with libssh2 v1.11.1 or newer:
```
lib/vssh/libssh2.c:3324:5: warning: 'libssh2_session_callback_set' is deprecated: since libssh2 1.11.1. Use libssh2_session_callback_set2() [-Wdeprecated-declarations]
lib/vssh/libssh2.c:3326:5: warning: 'libssh2_session_callback_set' is deprecated: since libssh2 1.11.1. Use libssh2_session_callback_set2() [-Wdeprecated-declarations]
```
Ref: https://github.com/curl/curl-for-win/actions/runs/7609484879/job/20720821100#step:3:4982

Ref: https://github.com/libssh2/libssh2/pull/1285
Ref: c0f69548be
Reviewed-by: Daniel Stenberg
Closes #12754
2024-01-22 16:10:45 +00:00
Daniel Stenberg
cdd905a985
transfer: make the select_bits_paused condition check both directions
If there is activity in a direction that is not paused, return false.

Reported-by: Sergey Bronnikov
Bug: https://curl.se/mail/lib-2024-01/0049.html
Closes #12740
2024-01-22 16:22:19 +01:00
Stefan Eissing
0535f6ec71
http3: initial support for OpenSSL 3.2 QUIC stack
- HTTP/3 for curl using OpenSSL's own QUIC stack together
  with nghttp3
- configure with `--with-openssl-quic` to enable curl to
  build this. This requires the nghttp3 library
- implementation with the following restrictions:
  * macOS has to use an unconnected UDP socket due to an
    issue in OpenSSL's datagram implementation
    See https://github.com/openssl/openssl/issues/23251
    This makes connections to non-reponsive servers hang.
  * GET requests will send the indicator that they have
    no body in a separate QUIC packet. This may result
    in processing delays or Transfer-Encodings on proxied
    requests
  * uploads that encounter blocks will use 100% cpu as
    detection of these flow control issue is not working
    (we have not figured out to pry that from OpenSSL).

Closes #12734
2024-01-22 16:15:45 +01:00
Stefan Eissing
524253dc90
rtsp: Convert assertion into debug log
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65934

- write excess bytes to the client where the standard excess bytes
  checks will report any wrongness and fail the transfer

Fixes #12738
Closes #12739
2024-01-19 12:15:36 +01:00
Daniel Stenberg
c8cffcb8d4
headers: remove assert from Curl_headers_push
The fuzzer managed to reach the function without a terminating CR or LF
so let's handle it normally. While there, remove the goto.

Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65839

Closes #12721
2024-01-19 09:16:44 +01:00
Viktor Szakats
434db995a7
mbedtls: fix -Wnull-dereference and -Wredundant-decls
- Silence warning in mbedTLS v3.5.1 public headers:
  ```
  ./mbedtls/_x64-linux-musl/usr/include/psa/crypto_extra.h:489:14: warning: redundant redeclaration of 'psa_set_key_domain_parameters' [-Wredundant-decls]
  ./mbedtls/_x64-linux-musl/usr/include/psa/crypto_struct.h:354:14: note: previous declaration of 'psa_set_key_domain_parameters' was here
  ```
  Ref: ecec68a2c1
  Ref: https://github.com/libssh2/libssh2/pull/1226

- Fix compiler warnings seen with gcc 9.2.0 + cmake unity:
  ```
  ./curl/lib/vtls/mbedtls.c: In function 'mbedtls_bio_cf_read':
  ./curl/lib/vtls/mbedtls.c:189:11: warning: null pointer dereference [-Wnull-dereference]
    189 |   nread = Curl_conn_cf_recv(cf->next, data, (char *)buf, blen, &result);
        |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ./curl/lib/vtls/mbedtls.c: In function 'mbedtls_bio_cf_write':
  ./curl/lib/vtls/mbedtls.c:168:14: warning: null pointer dereference [-Wnull-dereference]
    168 |   nwritten = Curl_conn_cf_send(cf->next, data, (char *)buf, blen, &result);
        |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ```

- delete stray `#else`.

Closes #12720
2024-01-18 07:09:31 +00:00
Jay Satiro
7aa24f32db cf-socket: show errno in tcpkeepalive error messages
- If the socket keepalive options (TCP_KEEPIDLE, etc) cannot be set
  then show the errno in the verbose error messages.

Ref: https://github.com/curl/curl/discussions/12715#discussioncomment-8151652

Closes https://github.com/curl/curl/pull/12726
2024-01-17 03:05:45 -05:00
Stefan Eissing
3378d2bd09
websockets: refactor decode chain
- use client writer stack for decoding frames
- move websocket protocol handler to ws.c

Closes #12713
2024-01-16 16:43:24 +01:00
Stefan Eissing
49ca84144e
websockets: check for negative payload lengths
- in en- and decoding, check the websocket frame payload lengths for
  negative values (from curl_off_t) and error the operation in that case
- add test 2307 to verify

Closes #12707
2024-01-16 14:56:15 +01:00
Jay Wu
c5801a28c5
url: don't set default CA paths for Secure Transport backend
As the default for this backend is the native CA store.

Closes #12704
2024-01-16 10:48:36 +01:00
Lin Sun
4224d6e0f3
asyn-ares: with modern c-ares, use its default timeout
Closes #12703
2024-01-16 10:45:58 +01:00
Stefan Eissing
036eb150d1
rtsp: deal with borked server responses
- enforce a response body length of 0, if the
  response has no Content-lenght. This is according
  to the RTSP spec.
- excess bytes in a response body are forwarded to
  the client writers which will report and fail the
  transfer

Follow-up to d7b6ce6
Fixes #12701
Closes #12706
2024-01-15 14:13:58 +01:00
Daniel Stenberg
72bd88adde
version: show only the libpsl version, not its dependencies
The libpsl version output otherwise also includes version number for its
dependencies, like IDN lib, but since libcurl does not use libpsl's IDN
functionality those components are not important.

Ref: https://github.com/curl/curl-for-win/issues/63
Closes #12700
2024-01-14 23:33:22 +01:00
Daniel Stenberg
e186ca6534
cf-h1-proxy: no CURLOPT_USERAGENT in CONNECT with hyper
Follow-up to 693cd1679361828a which was incomplete

Ref #12680
Closes #12697
2024-01-14 18:11:15 +01:00
Daniel Stenberg
adfffc39a3
transfer: remove warning: Value stored to 'blen' is never read
Detected by scan-build

Follow-up from 1cd2f0072f

Closes #12693
2024-01-14 14:33:46 +01:00
Stefan Eissing
d7b6ce64ce
lib: replace readwrite with write_resp
This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.

The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.

Curl's protocol handler `readwrite()` method been changed:

```diff
-  CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
-                        const char *buf, size_t blen,
-                        size_t *pconsumed, bool *readmore);
+  CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+                         bool is_eos, bool *done);
```

The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:

* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
  response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
  finished

This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.

The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.

HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.

The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:

```
static CURLcode readwrite_data(data, ...)
{
  do {
    Curl_xfer_recv_resp(data, buf)
    ...
    Curl_xfer_write_resp(data, buf)
    ...
  } while(interested);
  ...
}
```

All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.

All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.

Closes #12480
2024-01-13 17:23:42 +01:00
Stefan Eissing
5d044ad948
vquic: extract TLS setup into own source
- separate ngtcp2 specific parts out
- provide callback during init to allow ngtcp2 to apply its defaults

Closes #12678
2024-01-11 10:43:00 +01:00
Sergey Markelov
98543fc2cf
multi: remove total timer reset in file_do() while fetching file://
The total timer is properly reset in MSTATE_INIT.  MSTATE_CONNECT starts
with resetting the timer that is a start point for further multi states.
If file://, MSTATE_DO calls file_do() that should not reset the total
timer.  Otherwise, the total time is always less than the pre-transfer
and the start transfer times.

Closes #12682
2024-01-11 08:51:29 +01:00
Daniel Stenberg
693cd16793
http_proxy: a blank CURLOPT_USERAGENT should not be used in CONNECT
Extended test 80 to verify this.

Reported-by: Stefan Eissing
Fixes #12680
Closes #12681
2024-01-11 08:49:21 +01:00
Daniel Stenberg
dd0f680fc0
sectransp: do verify_cert without memdup for blobs
Since the information is then already stored in memory, this can avoid
an extra set of malloc + free calls.

Closes #12679
2024-01-10 23:22:52 +01:00
Daniel Stenberg
24ae4a07f3
hsts: remove assert for zero length domain
A zero length domain can happen if the HSTS parser is given invalid
input data which is not unheard of and is done by the fuzzer.

Follow-up from cfe7902111ae547873

Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65661

Closes #12676
2024-01-10 13:58:14 +01:00
Daniel Stenberg
a9e128d569
headers: make sure the trailing newline is not stored
extended test1940 to verify blank header fields too

Bug: https://curl.se/mail/lib-2024-01/0019.html
Reported-by: Dmitry Karpov
Closes #12675
2024-01-10 13:57:08 +01:00
Stefan Eissing
48d86999af
multi: pollset adjust, init with FIRSTSOCKET during connect
- `conn->sockfd` is set by `Curl_setup_transfer()`, but that
  is called *after* the connection has been established
- use `conn->sock[FIRSTSOCKET]` instead

Follow-up to a0f94800d507de
Closes #12664
2024-01-09 17:41:13 +01:00
Daniel Stenberg
ac4dbc9cb5
setopt: use memdup0 when cloning COPYPOSTFIELDS
Closes #12651
2024-01-09 09:23:15 +01:00
Daniel Stenberg
2959f45b7d
telnet: use dynbuf instad of malloc for escape buffer
Previously, send_telnet_data() would malloc + free a buffer every time
for escaping IAC codes. Now, it reuses a dynbuf for this purpose.

Closes #12652
2024-01-09 09:21:01 +01:00
Daniel Stenberg
a3abc81a48
pop3: replace calloc + memcpy with memdup0
... and make sure to return error on out of memory.

Closes #12650
2024-01-08 22:55:39 +01:00
Daniel Stenberg
cfe7902111
lib: add debug log outputs for CURLE_BAD_FUNCTION_ARGUMENT
Closes #12658
2024-01-08 22:48:24 +01:00
Daniel Stenberg
8e0323b4b5
mime: use memdup0 instead of malloc + memcpy
Closes #12649
2024-01-08 22:40:43 +01:00
Stefan Eissing
d0cb2c7a1f
url: init conn->sockfd and writesockfd to CURL_SOCKET_BAD
Also add more tracing to test 19

Follow-up to a0f9480

Fixes #12657
Closes #12659
2024-01-08 16:51:13 +01:00
Daniel Stenberg
e556470c23
connect: remove margin from eyeballer alloc
Presumably leftovers from debugging

Closes #12647
2024-01-08 09:30:45 +01:00
Daniel Stenberg
1058483615
ftp: only consider entry path if it has a length
Follow-up from 8edcfedc1a144f438bd1cdf814a0016cb

Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65631

Avoids a NULL pointer deref.

Closes #12648
2024-01-07 16:34:55 +01:00
Stefan Eissing
a0f94800d5
transfer: adjust_pollset improvements
- let `multi_getsock()` initialize the pollset in what the
  transfer state requires in regards to SEND/RECV
- change connection filters `adjust_pollset()` implementation
  to react on the presence of POLLIN/-OUT in the pollset and
  no longer check CURL_WANT_SEND/CURL_WANT_RECV
- cf-socket will no longer add POLLIN on its own
- http2 and http/3 filters will only do adjustments if the
  passed pollset wants to POLLIN/OUT for the transfer on
  the socket. This is similar to the HTTP/2 proxy filter
  and works in stacked filters.

Closes #12640
2024-01-07 14:36:25 +01:00
Daniel Stenberg
8edcfedc1a
ftp: use memdup0 to store the OS from a SYST 215 response
avoid malloc + direct buffer fiddle

Closes #12639
2024-01-06 11:26:31 +01:00
Daniel Stenberg
f4beef524a
ftp: use dynbuf to store entrypath
avoid direct malloc

Closes #12638
2024-01-06 11:25:37 +01:00
Lealem Amedie
afdb6c2d34
wolfssl: load certificate *chain* for PEM client certs
Closes #12634
2024-01-06 11:24:11 +01:00
Stefan Eissing
8e2d7b9fa4
http: adjust_pollset fix
do not add a socket for POLLIN when the transfer does not want to send
(for example is paused).

Follow-up to 47f5b1a

Reported-by: bubbleguuum on github
Fixes #12632
Closes #12633
2024-01-04 15:45:22 +01:00
Stefan Eissing
f712af2891
http3/quiche: fix result code on a stream reset
- fixes pytest failures in test 07_22
- aligns CURLcode values on stream reset with ngtcp2

Closes #12629
2024-01-03 13:10:21 +01:00
Daniel Stenberg
eeda18b1d2
setopt: clear mimepost when formp is freed
A precaution to avoid a possibly dangling pointer left behind.

Reported-by: Thomas Ferguson
Fixes #12608
Closes #12621
2024-01-02 23:32:43 +01:00
Gisle Vanem
8558647613
content_encoding: change return code to typedef'ed enum
... to work around a clang ubsan warning.

Fixes #12618
Closes #12622
2024-01-02 23:28:17 +01:00
Viktor Szakats
aff26089e8
schannel: fix -Warith-conversion gcc 13 warning
```
lib/vtls/schannel.c:1201:22: warning: conversion to 'unsigned int' from 'int' may change the sign of the result [-Warith-conversion]
 1201 |     *extension_len = *list_len +
      |                      ^
```

Closes #12616
2024-01-02 07:50:47 +00:00
Viktor Szakats
f0ea11b50b
asyn-thread: silence -Wcast-align warning for Windows
Seen with llvm/clang 17:
```
lib/asyn-thread.c:310:5: warning: cast from 'PCHAR' (aka 'char *') to 'struct thread_sync_data *' increases required alignment from 1 to 8 [-Wcast-align]
  310 |     CONTAINING_RECORD(overlapped, struct thread_sync_data, w8.overlapped);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.../llvm-mingw/aarch64-w64-mingw32/include/winnt.h:717:48: note: expanded from macro 'CONTAINING_RECORD'
  717 | #define CONTAINING_RECORD(address,type,field) ((type *)((PCHAR)(address) - (ULONG_PTR)(&((type *)0)->field)))
      |                                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

Follow-up to a6bbc87f9e9ffb46a1801dfb983e7534825ed56b #12482

Ref: https://github.com/curl/curl/pull/12482#issuecomment-1873017261
Closes #12615
2024-01-02 07:50:46 +00:00
Jay Satiro
b83729a339 quiche: return CURLE_HTTP3 on send to invalid stream
Prior to this change if a send failed on a stream in an invalid state
(according to quiche) and not marked as closed (according to libcurl)
then the send function would return CURLE_SEND_ERROR.

We already have similar code for ngtcp2 to return CURLE_HTTP3 in this
case.

Caught by test test_07_upload.py: test_07_22_upload_parallel_fail.

Fixes https://github.com/curl/curl/issues/12590
Closes https://github.com/curl/curl/pull/12597
2024-01-02 00:54:15 -05:00
Patrick Monnerat
4b6d44f6e9
openldap: fix STARTTLS
It was not working anymore since introduction of connection filters.

Also do not attempt to recover from a failing TLS negotiation with
CURLUSESSL_TRY.

Closes #12610
2023-12-31 17:11:51 +01:00
Jay Satiro
26f002e02e system_win32: fix a function pointer assignment warning
- Use CURLX_FUNCTION_CAST to suppress a function pointer assignment
  warning.

a6bbc87f added lookups of some Windows API functions and then cast them
like `*(FARPROC*)&Curl_funcname = address`. Some versions of gcc warn
about that as breaking strict-aliasing rules so this PR changes those
assignments to use CURLX_FUNCTION_CAST.

Bug: https://github.com/curl/curl/pull/12581#issuecomment-1869804317
Reported-by: Marcel Raad

Closes https://github.com/curl/curl/pull/12602
2023-12-28 03:38:44 -05:00
Jay Satiro
e251e858b9 vtls: fix missing multissl version info
- Fix erroneous buffer copy logic from ff74cef5.

Prior to this change the MultiSSL version info returned to the user
was empty.

Closes https://github.com/curl/curl/pull/12599
2023-12-27 17:49:27 -05:00
Patrick Monnerat
fa6e123929
openldap: fix an LDAP crash
Reported-by: Ozan Cansel
Fixes #12593
Closes #12600
2023-12-27 14:42:32 +01:00
Daniel Stenberg
68f96fc9bf
getinfo: CURLINFO_QUEUE_TIME_T
Returns the time, in microseconds, during which this transfer was held
in a waiting queue before it started "for real". A transfer might be put
in a queue if after getting started, it cannot create a new connection
etc due to set conditions and limits imposed by the application.

Ref: #12293
Closes #12368
2023-12-27 09:28:48 +01:00
Stefan Eissing
e492c7c524
transfer: fix upload rate limiting, add test cases
- add test cases for rate limiting uploads for all
  http versions
- fix transfer loop handling of limits. Signal a re-receive
  attempt only on exhausting maxloops without an EAGAIN
- fix `data->state.selectbits` forcing re-receive to also
  set re-sending when transfer is doing this.

Reported-by: Karthikdasari0423 on github
Fixes #12559
Closes #12586
2023-12-22 16:52:02 +01:00
Daniel Stenberg
8b1d229835
mbedtls: free the entropy when threaded
The entropy_free was never done for threaded builds, causing a small
(fixed) memory leak.

Reported-by: RevaliQaQ on github
Fixes #12584
Closes #12585
2023-12-22 10:58:07 +01:00