Change multi's book keeping of transfers to no longer use lists, but a
special table and bitsets for unsigned int values.
`multi-xfers` is the `uint_tbl` where `multi_add_handle()` inserts a new
transfer which assigns it a unique identifier `mid`. Use bitsets to keep
track of transfers that are in state "process" or "pending" or
"msgsent".
Use sparse bitsets to replace `conn->easyq` and event handlings tracking
of transfers per socket. Instead of pointers, keep the mids involved.
Provide base data structures and document them in docs/internal:
* `uint_tbl`: a table of transfers with `mid` as lookup key,
handing out a mid for adds between 0 - capacity.
* `uint_bset`: a bitset keeping unsigned ints from 0 - capacity.
* `uint_spbset`: a sparse bitset for keeping a small number of
unsigned int values
* `uint_hash`: for associating `mid`s with a pointer.
This makes the `mid` the recommended way to refer to transfers inside
the same multi without risk of running into a UAF.
Modifying table and bitsets is safe while iterating over them. Overall
memory requirements are lower as with the double linked list apprach.
Closes#16761
Currently the DNS server only responds to A and AAAA queries. It always
responds with a fixed response: the localhost address. Three times.
It should work fine over either IPv4 or IPv6, but I don't think it
matters much for curl testing.
The idea is to allow curl tests to use "normal" DNS hostnames (using the
normal name resolving code paths) and still use the local test servers.
This setup currently only works if curl is built with c-ares because
redirecting DNS requests to our test server when using getaddrinfo() is
not easy.
This should be extended to respond to HTTPS queries as well to allow
more testing there, as c-ares is always used for that.
Test 2102 is the first test using this.
Closes#17015
Very similar to 9f8bdd0eae5c1d441d9d901a7cf917a8ee215c7f, but affects
e.g. netrc file parsing.
Suggested-by: Graham Christensen <graham@grahamc.com>
Add test 744 to verify
Closes#17036
- test2088 verifies that mutual tls works
This adds a new certificate to generate which has the clientAuth key
usage enabled, and uses it to connect to a https-mtls server.
Closes#16923
Made the HTTPS-RR parser a little stricter while at it.
Drop the ALPN escape handling, that was not needed.
Make the hode handle (and ignore) duplicate ALPN entries.
Closes#16972
- allow and ignore "identity" as an encoding
- fail if any other encoder than chunked follows after chunked
- fail on unsolicited encodings - when the server encodes but curl did
not ask for it
Add test 1493 to 1496 to verify.
Disable test 319 as that is now broken: issue #16974
Reported-by: Jonathan Rosa
Fixes#16956Closes#16959
When multiple headers share the same name, AWS SigV4 expects them to be
merged into a single header line, with values comma-delimited in the
order they appeared.
Add libtest 1978 to verify.
Closes#16743
- if there are pending internal handles left in the list, they are
leftovers (from for example Doh) and must be freed.
- unlink_all_msgsent_handles() did not properly move all msgsent
handles over to the process list as intended
Fixes a DoH memory leak found by oss-fuzz.
Add test 2101 that can reproduce and verify.
Closes#16674
For the case when the connection struct is all setup, the protocol
handler allocates data in its setup_connection function, but the
connection struct is discarded again before used further because a
connection reuse is prefered. Then the handler's disconnect function was
not previously called, which then would lead to a memory leak.
I added test case 698 that reproduces the leak and the fix.
Reported-by: Philippe Antoine
Closes#16604
With this change, the argument passed to the CURLOPT_FOLLOWLOCATION
option is now instead a "mode" instead of just a boolean. Documentation
is extended to describe the two new modes.
Test 1571 to 1581 verify.
Closes#16473
Verified in test 455 and 487.
If the provided string cannot be base64-decoded, it will instead use
"[64dec-fail]" (without the quotes).
Documented
Ref: #16288Closes#16330
Keeping the relevant 'ssl_scache' in 'data->state' leads to problems
when the owner of the cache is cleaned up and this reference is left
dangling.
Remove the ref entirely and always find the ssl_scache at the current
share or multi.
Folded in #16260 (test 3208) to verify this fixes the bug with a
dangling reference when an easy handle is used with easy_perform first
and in a multi_perform after.
Ref: #16236Closes#16261
Made the netrc parser return a more consistent set of error codes and
error messages, and also return error properly if the .netrc file is
missing.
Add test 697 to verify error on missing netrc file.
Fixes#16163
Reported-by: arlt on github
Closes#16165
For ASN.1 tags with indefinite length, curl's own parser for TLS
backends that do not support certificate inspection calls itself
recursively. A malicious server certificate can then lead to high
recursion level exhausting the stack space.
This PR limits the recursion level to 16 which should be safe on all
architectures.
Added unit test 1657 to verify behaviour.
Fixes#16135
Reported-by: z2_
Closes#16137
RFC 6455 Section 5.2 notes that for bits RSV1, RSV2, and RSV3 of the
framing header, a non-zero value that is not defined by a negotiated
extension MUST Fail the WebSocket connection.
Test 2310 verifies
Closes#16069
- Change the swsbounce keyword to override the part number on a
subsequent request to the previous part number + 1.
Note the previous part number in this case is the part number that
was returned as a response to the previous request and contained
the swsbounce keyword.
Prior to this change swsbounce incremented the part number of the
subsequent request instead of overriding it, and did so in a more
limited fashion that prevented chaining swsbounce in multiple responses.
For example, if the test makes a request that causes the sws server to
return `<data>` as a response and that response contains `swsbounce`
then for the next response the sws server returns `<data1>`. If
`<data1>` also contains `swsbounce` then for the next response the sws
server now returns `<data2>` instead of the requested part.
Fixes https://github.com/curl/curl/discussions/16074
Closes https://github.com/curl/curl/pull/16085
This typically happens if CURL_CONNECT_ONLY is used and a second
curl_easy_perform() is attempted.
A connection "taken over" with CURL_CONNECT_ONLY cannot be ended any
other way than a curl_easy_cleanup() on the easy handle that holds it.
Add test 696 to verify.
Closes#16003
- Change setopt and pretransfer to always reset URL related variables
for a CURLU handle set CURLOPT_CURLU.
This change is to ensure we are in compliance with the doc which says
CURLU handles must be able to override a URL set via CURLOPT_URL and
that if the contents of the CURLU handle changes between transfers then
the updated contents must be used.
Prior to this change, although subsequent transfers appear to be
performed correctly in those cases, the work URL `data->state.url` was
not updated. CURLINFO_EFFECTIVE_URL returns data->state.url to the user
so it would return the URL from the initial transfer which was the wrong
URL. It's likely there are other cases as well.
Ref: https://curl.se/libcurl/c/CURLOPT_CURLU.html
Reported-by: Nicolás San Martín
Fixes https://github.com/curl/curl/issues/15984
Closes https://github.com/curl/curl/pull/15985
Subparts may have been previously used as a top-level mime structure and
thus not rewound.
New test 695 checks the proper functioning in these particular conditions.
Reported-by: Qriist on github
Fixes#15842Closes#15911
Follow-up to 40c264db617d0 after discussions on IRC.
The new style is
name[0-99]=contents
and
name[0-99]@filename
A) This does not cause the same problems with old curl versions trying
the new syntax as this way will cause old curls just fail with syntax
error and not risk using the wrong file.
B) Adds the same byte range support for "normal" assigns, which the
previous syntax did not. Thus lets a user get a partial content of a
variable etc.
Added test 790 and 791 to verify non-file assigns with ranges.
Closes#15862
- Parse the input string without modifying it.
Prior to this change a segfault could occur if the input string was
const because the tokenizer modified the input string. For example if
the user set CURLOPT_COOKIELIST to a const string then libcurl would
likely cause a crash when modifying that string. Even if the string was
not const or a crash did not occur there was still the incorrect and
unexpected modification of the user's input string.
This issue was caused by 30da1f59 (precedes 8.11.0) which refactored
some options parsing and eliminated the copy of the input string. Also,
an earlier commit f88cc654 incorrectly cast the input pointer when
passing it to strtok.
Co-authored-by: Daniel Stenberg
Closes https://github.com/curl/curl/pull/15826
Allowing --variable read a portion of provided files, makes curl work on
partial files for any options that accepts strings. Like --data and others.
The byte offset is provided within brackets, with a semicolon separator
like: --variable name@file;[100-200]"
Inspired by #14479
Assisted-by: Manuel Einfalt
Test 784 - 789. Documentation update provided.
Closes#15739
When a specific hostname matched, and only a password is set before
another machine is specified in the netrc file, the parser would not be
happy and stop there and return the password-only state. It instead
continued and did not return a match.
Add test 2005 to verify this case
Regression from e9b9bba, shipped in 8.11.1.
Reported-by: Ben Zanin
Fixes#15767Closes#15768
Designed to aid converting off from sscanf parsers. sscanf is hard to
use right, easy to mess up and often makes for sloppy error checking.
The new parsers allow more exact and pedandic parsing.
This new set of functions should be possible to use (and extend) and
switch over other libcurl parser code to use going forward.
Adapts the following to use the new functions:
- altsvc.c
- hsts.c
- http_aws_sigv4.c
Bonus: fewer memory copies, fewer stack buffers.
Test: Unit test1664
Docs: docs/internals/STRPARSE.md
Closes#15692
The date parser function is very forgiving and skips most "irrelevant"
characters in its hunt for a date to figure out. Therefore it is
important to make sure the date string is properly null terminated so
that it does not accidentally parse a piece of whatever text follows
after the date.
Add test483: test (overly) long expire dates in cookies
Closes#15709
- make sure that a match that returns a username also returns a
password, that should be blank if no password is found
- fix handling of multiple logins for same host where the password/login
order might be reversed.
- reject credentials provided in the .netrc if they contain ASCII control
codes - if the used protocol does not support such (like HTTP and WS do)
Reported-by: Harry Sintonen
Add test 478, 479 and 480 to verify. Updated unit 1304.
Closes#15586
The netrc init was only done in the Curl_open, meaning that a duplicated
handle would not get inited properly.
Added test 2309 to verify. It does netrc auth with a duplicated handle.
Regression from 3b43a05e000aa8f65bda513f733a
Reported-by: tranzystorekk on github
Fixes#15496Closes#15503
By keeping the headers in memory until we know the target file name,
then output them all.
Previously this option combination would cause an error.
Add test 1310 and 1492 to verify. Adjusted test 1460 to work in the new
conditions.
Closes#15110
Windows sometimes has issues when opening the same file twice, so these
test two situations where that could potentially occur.
Reported-by: ralfjunker on github
Ref: #15043Closes#15045
Use the same fallback for content-disposition cases as for regular -O
Add test692: verify -JO with URL without a file name
Reported-by: Brian Inglis
Fixes#14939Closes#14940
The SSL_Session object is mutated during connection inside openssl,
and it might not be thread-safe. Besides, according to documentation
of openssl:
```
SSL_SESSION objects keep internal link information about the session
cache list, when being inserted into one SSL_CTX object's session
cache. One SSL_SESSION object, regardless of its reference count,
must therefore only be used with one SSL_CTX object (and the SSL
objects created from this SSL_CTX object).
```
If I understand correctly, it is not safe to share it even in a
single thread.
Instead, serialize the SSL_SESSION before adding it to the cache,
and deserialize it after retrieving it from the cache, so that no
concurrent write to the same object is infeasible.
Also
- add a ci test for thread sanitizer
- add a test for sharing ssl sessions concurrently
- avoid redefining memory functions when not building libcurl, but
including the soruce in libtest
- increase the concurrent connections limit in sws
Notice that there are fix for a global data race for openssl which
is not yet release. The fix is cherry pick for the ci test with
thread sanitizer.
d8def79838Closes#14751
Some POP3 commands are multi-line, e.g. have responses terminated by a
last line with '.', but some are not. Define the known command
properties and fix response handling.
Add test case for STAT.
Fixes#14677
Reported-by: ralfjunker on github
Closes#14707
