Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general events like a timer.
HTTP installs a `CURL_CR_PROTOCOL` reader when announcing `Expect:
100-continue`. That reader works as follows:
- on first invocation, records time, starts the `EXPIRE_100_TIMEOUT`
timer, disables `KEEP_SEND`, enables `KEEP_SEND_TIMER` and returns 0,
eos=FALSE like a paused upload.
- on subsequent invocation it checks if the timer has expired. If so, it
enables `KEEP_SEND` and switches to passing through reads to the
underlying readers.
Transfer handling's `readwrite()` will be invoked when a timer expires
(like `EXPIRE_100_TIMEOUT`) or when data from the server arrives. Seeing
`KEEP_SEND_TIMER`, it will try to upload more data, which triggers
reading from the client readers again. Which then may lead to a new
pausing or cause the upload to start.
Flags and timestamps connected to this have been moved from
`SingleRequest` into the reader's context.
Closes#13110
The option is really two enums ORed together, so it needs special
attention to make the code output nice.
Added test 1481 to verify. Both the server and the proxy versions.
Reported-by: Boris Verkhovskiy
Fixes#13127Closes#13129
Use imperative mood consistently for the first sentence describing an
option.
"Set this" instead "tell curl to set" or "this sets..."
Plus some extra cleanups and rephrasing.
Closes#13106
... correctly, even when they follow an existing one without a space in
between.
Verify with test 467
Follow-up to 07dd60c05b
Reported-by: Geeknik Labs
Fixes#13101Closes#13102
... in its new build path.
Also update the test scripts to be more precise in error messages to
help us understand CI errors better.
Follow-up to f03c85635f
Ref: #13029Closes#13083
Create ASCII version of manpage without nroff
- build src/tool_hugegelp.c from the ascii manpage
- move the the manpage and the ascii version build to docs/cmdline-opts
- remove all use of nroff from the build process
- should make the build entirely reproducible (by avoiding nroff)
- partly reverts 2620aa9 to build libcurl option man pages one by one
in cmake because the appveyor builds got all crazy until I did
The ASCII version of the manpage
- is built with gen.pl, just like the manpage is
- has a right-justified column making the appearance similar to the previous
version
- uses a 4-space indent per level (instead of the old version's 7)
- does not do hyphenation of words (which nroff does)
History
We first made the curl build use nroff for building the hugehelp file in
December 1998, for curl 5.2.
Closes#13047
If a response without a status line is received, and the connection is
known to use HTTP/1.x (not HTTP/0.9), report the error "Invalid status
line" instead of "Received HTTP/0.9 when not allowed".
Closes#13045
- pytest has changed the signature of the hook pytest_report_header()
for some obscure reason and that change landed in our CI now
- remove the changed param that we never used anyway
Closes#13037
This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.
Author: RainRat on Github
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Dan Fandrich <dan@coneharvesters.com>
Closes: #13019
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
clarify when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer
setup of `conn->sockfd` and `conn->writesockfd` on which
connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index
as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
naming consistency
- clarify that the special CURLE_AGAIN hangling to return
`CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
when it arrived in more than a single chunk (to be made
into a sperate PR, also)
Added as documented [in
CLIENT-READER.md](5b1f31dfba/docs/CLIENT-READERS.md).
- old `Curl_buffer_send()` completely replaced by new `Curl_req_send()`
- old `Curl_fillreadbuffer()` replaced with `Curl_client_read()`
- HTTP chunked uploads are now formatted in a client reader added when
needed.
- FTP line-end conversions are done in a client reader added when
needed.
- when sending requests headers, remaining buffer space is filled with
body data for sending in "one go". This is independent of the request
body size. Resolves#12938 as now small and large requests have the
same code path.
Changes done to test cases:
- test513: now fails before sending request headers as this initial
"client read" triggers the setup fault. Behaves now the same as in
hyper build
- test547, test555, test1620: fix the length check in the lib code to
only fail for reads *smaller* than expected. This was a bug in the
test code that never triggered in the old implementation.
Closes#12969
When disabling all protocols without enabling any, the resulting
set of allowed protocols remained the default set. Clearing the
allowed set before inspecting the passed value from --proto make
the set empty even in the errorpath of no protocols enabled.
Co-authored-by: Dan Fandrich <dan@telarity.com>
Reported-by: Dan Fandrich <dan@telarity.com>
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Closes: #13004
Curl_read/Curl_write clarifications
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer setup of
`conn->sockfd` and `conn->writesockfd` on which connection filter
chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index as
parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
consistency
- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
returned by all other send() variants.
SingleRequest reshuffling
- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data
Closes#12963
Returns 1 if the previous transfer used a proxy, otherwise 0. Useful to
for example determine if a `NOPROXY` pattern matched the hostname or
not.
Extended test 970 and 972
- when data arrived in several chunks, the collection into
the passed buffer always started at offset 0, overwriting
the data already there.
adding test_20_07 to verify fix
- debug environment var CURL_WS_CHUNK_SIZE can be used to
influence the buffer chunk size used for en-/decoding.
Closes#12945
Also fix the tests. New implementation tested with GNU libmicrohttpd.
The new numbers in tests are real SHA-512/256 numbers (not just some
random ;) numbers ).
The previous realloc code in this code could trigger a compiler warning,
but since that code path cannot happen in normal circumstances it now
instead exits with an error message there.
Ref: #12887Closes#12890
- add test_05_04 for requests using http/1.0, http/1.1 and h2 against an
Apache resource that does an unclean TLS shutdown.
- revert special workarund in openssl.c for suppressing shutdown errors
on multiplexed connections
- vlts.c restore to its state before 9a90c9dd64Fixes#12885Fixes#12844Closes#12848
- test450: remove --config from the keywords
- test2080: change return code
- test428: add --config as a keyword
- test428: disable on Windows due to CI problems
Like when trying to import an environment variable that does not exist.
Also fix a bug for reading env variables when there is a default value
set.
Bug: https://curl.se/mail/archive-2024-02/0008.html
Reported-by: Brett Buddin
Add test 462 to verify.
Closes#12862
Create the line in a dynbuf. Aborts the reading of the file on
errors. Avoids having to always allocate maximum amount from the
start. Avoids direct malloc.
Closes#12846
- improve info logging when peer verification fails to indicate
if DNS name or ip address has been tried to match
- add test case for contacting https proxy with ip address
- add pytest env check on loaded credentials and re-issue
when they are no longer valid
- disable proxy ip address test for bearssl, since not supported there
Ref: #12831Closes#12838
and mark the stream for close, but return OK since the response this far
was ok - if headers were received. Partly because this is what curl has
done traditionally.
Test 499 verifies. Updates test 689.
Reported-by: Sergey Bronnikov
Bug: https://curl.se/mail/lib-2024-02/0000.htmlCloses#12842
- delete redundant warning suppressions for `-Wformat-nonliteral`.
This now relies on `CURL_PRINTF()` and it's theoratically possible
that this macro isn't active but the warning is. We're ignoring this
as a corner-case here.
- replace two pragmas with code changes to avoid the warnings.
Follow-up to aee4ebe591#12803
Follow-up to 0923012758#12540
Follow-up to 3829759bd0#12489
Reviewed-by: Daniel Stenberg
Closes#12812
- Use http authentication mechanisms as a default, not a preset.
Consider http authentication options which are mapped to SASL options as
a default (overriding the hardcoded default mask for the protocol) that
is ignored if a login option string is given.
Prior to this change, if some HTTP auth options were given, sasl mapped
http authentication options to sasl ones but merged them with the login
options.
That caused problems with the cli tool that sets the http login option
CURLAUTH_BEARER as a side-effect of --oauth2-bearer, because this flag
maps to more than one sasl mechanisms and the latter cannot be cleared
individually by the login options string.
New test 992 checks this.
Fixes https://github.com/curl/curl/issues/10259
Closes https://github.com/curl/curl/pull/12790
When checking if the user wants to replace the header, the check should
be case insensitive.
Adding test 461 to verify
Found-by: Dan Fandrich
Ref: #12782Closes#12784
The pingpong logic now uses its own dynbuf for receiving command
response data.
When the "final" response header for a commanad has been received, that
final line is left first in the recvbuf for the protocols to parse at
will. If there is additional data behind the final response line, the
'overflow' counter is indicate how many bytes.
Closes#12757
- switch all invidual files documenting command line options into .md,
as the documentation is now markdown-looking.
- made the parser treat 4-space indents as quotes
- switch to building the curl.1 manpage using the "mainpage.idx" file,
which lists the files to include to generate it, instead of using the
previous page-footer/headers. Also, those files are now also .md
ones, using the same format. I gave them underscore prefixes to make
them sort separately:
_NAME.md, _SYNOPSIS.md, _DESCRIPTION.md, _URL.md, _GLOBBING.md,
_VARIABLES.md, _OUTPUT.md, _PROTOCOLS.md, _PROGRESS.md, _VERSION.md,
_OPTIONS.md, _FILES.md, _ENVIRONMENT.md, _PROXYPREFIX.md,
_EXITCODES.md, _BUGS.md, _AUTHORS.md, _WWW.md, _SEEALSO.md
- updated test cases accordingly
Closes#12751
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:
- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)
tools:
- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown
This setup generates .3 versions of all the curldown versions at build time.
CI:
Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...
Closes#12730
- HTTP/3 for curl using OpenSSL's own QUIC stack together
with nghttp3
- configure with `--with-openssl-quic` to enable curl to
build this. This requires the nghttp3 library
- implementation with the following restrictions:
* macOS has to use an unconnected UDP socket due to an
issue in OpenSSL's datagram implementation
See https://github.com/openssl/openssl/issues/23251
This makes connections to non-reponsive servers hang.
* GET requests will send the indicator that they have
no body in a separate QUIC packet. This may result
in processing delays or Transfer-Encodings on proxied
requests
* uploads that encounter blocks will use 100% cpu as
detection of these flow control issue is not working
(we have not figured out to pry that from OpenSSL).
Closes#12734
- in en- and decoding, check the websocket frame payload lengths for
negative values (from curl_off_t) and error the operation in that case
- add test 2307 to verify
Closes#12707
