Remove the PROTOCOLS section from the source files completely and
instead generate them based on the header data in the curldown files.
It also generates TLS backend information for options marked for TLS as
protocol.
Closes#13175
All man pages that are listed to be for TLS now must also specify
exactly what TLS backends the option works for, or use All if they all
work.
cd2nroff makes sure this is done and that the listed backends exist.
Closes#13168
The mandatory header now has a mandatory list of protocols for which the
manpage is relevant.
Most man pages already has a "PROTOCOLS" section, but this introduces a
stricter way to specify the relevant protocols.
cd2nroff verifies that at least one protocol is mentioned (which can be
`*`).
This information is not used just yet, but A) the PROTOCOLS section can
now instead get generated and get a unified wording across all manpages
and B) this allows us to more reliably filter/search for protocol
specific manpages/options.
Closes#13166
For options that are listed for specific protocols, the protocols (shown
first within parentheses) are now output without the leading space in the
manpage output.
Closes#13105
Just a tidy up to contain 'ifdef' pollution of common
code parts with implementation specifics.
- remove the ifdef hyper unpausing in easy.c
- add hyper client reader for CURL_CR_PROTOCOL phase
that implements the unpause method for calling
the hyper waker if it is set
Closes#13075
Create ASCII version of manpage without nroff
- build src/tool_hugegelp.c from the ascii manpage
- move the the manpage and the ascii version build to docs/cmdline-opts
- remove all use of nroff from the build process
- should make the build entirely reproducible (by avoiding nroff)
- partly reverts 2620aa9 to build libcurl option man pages one by one
in cmake because the appveyor builds got all crazy until I did
The ASCII version of the manpage
- is built with gen.pl, just like the manpage is
- has a right-justified column making the appearance similar to the previous
version
- uses a 4-space indent per level (instead of the old version's 7)
- does not do hyphenation of words (which nroff does)
History
We first made the curl build use nroff for building the hugehelp file in
December 1998, for curl 5.2.
Closes#13047
- update client reader documentation
- client reader, add rewind capabilities
- tell creader to rewind on next start
- Curl_client_reset() will keep reader for future rewind if requested
- add Curl_client_cleanup() for freeing all resources independent of
rewinds
- add Curl_client_start() to trigger rewinds
- move rewind code from multi.c to sendf.c and make part of
"cr-in"'s implementation
- http, move the "resume_from" handling into the client readers
- the setup of a HTTP request is reshuffled to follow:
* determine method, target, auth negotiation
* install the client reader(s) for the request, including crlf
conversions and "chunked" encoding
* apply ranges to client reader
* concat request headers, upgrades, cookies, etc.
* complete request by determining Content-Length of installed
readers in combination with method
* send
- add methods for client readers to
* return the overall length they will generate (or -1 when unknown)
* return the amount of data on the CLIENT level, so that
expect-100 can decide if it want to apply itself
* set a "resume_from" offset or fail if unsupported
- struct HTTP has become largely empty now
- rename `Client_reader_*` to `Curl_creader_*`
Closes#13026
With the recent changes to completion file building, the files were
built always and only installation was selectively disabled. Now, when
they are disabled they aren't even built, avoiding a build-time error in
environments where it's not possible to run the curl binary that was
just created (e.g. if library paths were not set up correctly).
Follow-up to 0f7aba83c
Reported-by: av223119 on github
Fixes#13027Closes#13030
The code that attempted to skip building the shell completions didn't
work properly and tried to build them even if perl wasn't available.
This step, as well as the install step, is now properly skipped without
perl.
Follow-up to 89733e2ddCloses#13022
This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.
Author: RainRat on Github
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Dan Fandrich <dan@coneharvesters.com>
Closes: #13019
The --with-fish-functions-dir and --with-zsh-functions-dir options
currently have no effect on a normal build because the scripts/ directory
where they're used is not built. Add scripts/ to a normal build and
change the completion options to default to off to preserve the existing
behaviour.
Closes: #12906
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
clarify when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer
setup of `conn->sockfd` and `conn->writesockfd` on which
connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index
as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
naming consistency
- clarify that the special CURLE_AGAIN hangling to return
`CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
when it arrived in more than a single chunk (to be made
into a sperate PR, also)
Added as documented [in
CLIENT-READER.md](5b1f31dfba/docs/CLIENT-READERS.md).
- old `Curl_buffer_send()` completely replaced by new `Curl_req_send()`
- old `Curl_fillreadbuffer()` replaced with `Curl_client_read()`
- HTTP chunked uploads are now formatted in a client reader added when
needed.
- FTP line-end conversions are done in a client reader added when
needed.
- when sending requests headers, remaining buffer space is filled with
body data for sending in "one go". This is independent of the request
body size. Resolves#12938 as now small and large requests have the
same code path.
Changes done to test cases:
- test513: now fails before sending request headers as this initial
"client read" triggers the setup fault. Behaves now the same as in
hyper build
- test547, test555, test1620: fix the length check in the lib code to
only fail for reads *smaller* than expected. This was a bug in the
test code that never triggered in the old implementation.
Closes#12969
- When parsing .checksrc chomp the (CR)LF line ending.
Prior to this change on Windows checksrc.pl would not process the
symbols in .checksrc properly, since many git repos in Windows use auto
crlf to check out files with CRLF line endings.
Closes https://github.com/curl/curl/pull/12924
Make sure we use \< and \> in markdown all over so that it renders
correctly, on GitHub and elsewhere. cd2nroff now outputs a warning if it
finds an unescaled angle bracket.
Ref: #12854Closes#12869
- Use strict and warnings pragmas.
- If open() fails then show the reason.
- Set STDIN io layer :crlf so that input is properly read on Windows.
- When STDIN is used as input, the filename $f is now set to "STDIN".
Various error messages in single() use $f for the filename and this way
it is not undefined when STDIN.
Closes https://github.com/curl/curl/pull/12819
- cmake: enable `BUILD_DOCS` by default (this controls converting and
installing `.3` files from `.md` sources)
- cmake: speed up generating `.3` files by using a single command per
directory, instead of a single command per file. This reduces external
commands by about a thousand. (There remains some CMake logic kicking
in resulting in 500 -one per file- external `-E touch_nocreate` calls.)
- cd2nroff: add ability to process multiple input files.
- cd2nroff: add `-k` option to use the source filename to form the
output filename. (instead of the default in-file `Title:` line.)
Follow-up to 3f08d80b22
Follow-up to ea0b575dab#12753
Follow-up to eefcc1bda4#12730Closes#12762
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:
- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)
tools:
- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown
This setup generates .3 versions of all the curldown versions at build time.
CI:
Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...
Closes#12730
This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.
The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.
Curl's protocol handler `readwrite()` method been changed:
```diff
- CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
- const char *buf, size_t blen,
- size_t *pconsumed, bool *readmore);
+ CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+ bool is_eos, bool *done);
```
The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:
* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
finished
This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.
The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.
HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.
The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:
```
static CURLcode readwrite_data(data, ...)
{
do {
Curl_xfer_recv_resp(data, buf)
...
Curl_xfer_write_resp(data, buf)
...
} while(interested);
...
}
```
All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.
All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.
Closes#12480
Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.
Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.
There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.
grepping for `WIN32` remains useful to discover Windows-specific code.
Also:
- extend `checksrc` to ensure we're not using `WIN32` anymore.
- apply minor formatting here and there.
- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.
Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg
Closes#12376
This tool generates a scheme-matching table.
It iterates over a number of different initial and shift values in order
to find the hash algorithm that needs the smallest possible table.
The generated hash function, table and table size then needs to be used
by the url.c:Curl_getn_scheme_handler() function.
We use `stdint.h` unconditionally in all places except one. These uses
are imposed by external dependencies / features. nghttp2, quic, wolfSSL
and `HAVE_MACH_ABSOLUTE_TIME` do require this C99 header. It means that
any of these features make curl require a C99 compiler. (In case of
MSVC, this means Visual Studio 2010 or newer.)
This patch changes the single use of `stdint.h` guarded by
`HAVE_STDINT_H` to use `stdint.h` unconditionally. Also stop using
`inttypes.h` as an alternative there. `HAVE_INTTYPES_H` wasn't used
anywhere else, allowing to delete this feature check as well.
Closes#12275
Delete leftovers of the `crypt-auth` `./configure` option and
add the new ones that replaced them.
Follow-up to e92edfbef6#11490
Reviewed-by: Daniel Stenberg
Closes#12194
Uses scripts/cmp-config.pl two compare two curl_config.h files,
presumbly generated with configure and cmake. It displays the
differences and filters out a lot of known lines we ignore.
The script also shows the matches that were *not* used. Possibly
subjects for removal.
Closes#11964
Earlier this year we changed our own stderr variable to use the standard
name `stderr` (to avoid bugs where someone is using `stderr` instead of
the curl-tool specific variable). This solution needed to override the
standard `stderr` symbol via the preprocessor. This in turn didn't play
well with unity builds and caused curl tool to crash or stay silent due
to an uninitialized stderr. This was a hard to find issue, fixed by
manually breaking out one file from the unity sources.
To avoid two these two tricks, this patch implements a different
solution: Restore using our own local variable for our stderr output and
leave `stderr` as-is. To avoid using `stderr` by mistake, add a
`checksrc` rule (based on logic we already used in lib for `strerror`)
that detects any `stderr` use in `src` and points to using our own
variable instead: `tool_stderr`.
Follow-up to 06133d3e9b
Follow-up to 2f17a9b654Closes#11958
This should reduce false-positive to almost zero. Checks for presence in
unit tests if --unit is specified, which is intended for debug builds
where unit testing is enabled.
Closes#11932
EGD is Entropy Gathering Daemon, a socket-based entropy source supported
by pre-OpenSSL v1.1 versions and now deprecated. curl also deprecated it
a while ago.
Its detection in CMake was broken all along because OpenSSL libs were
not linked at the point of feature check.
Delete detection from both cmake and autotools, along with the related
source snippet, and the `--with-egd-socket=` `./configure` option.
Closes#11556
Originally these scripts filtered out names that have no space so that
they better avoid nick names not intended for credits. Such names are
not too commonly used, plus we now give credit even to those.
Additionally: non-latin names, like Asian, don't have spaces at all so
they were also filtered out and had to be manually added which made it
an error-prone operation where Asian names eventually easily fell off by
mistake.
Closes#11206
Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
the line is the favoured style overall with 329 instances.
Out of the 86 labels not at the start of the line:
* 75 were indented with the same indentation level of the following line
* 8 were indented with exactly one space
* 2 were indented with one fewer indentation level then the following
line
* 1 was indented with the indentation level of the following line minus
three space (probably unintentional)
Co-Authored-By: Viktor Szakats
Closes#11134
If the previous line starts with if/while/for AND ends with a closed
parenthesis and there's an equal number of open and closed parentheses
on that line, verify that this line is indented $indent more steps, if
not a cpp line.
Also adjust the fall-out from this fix.
Closes#11054
The open paren check wants to warn for spaces before open parenthesis
for if/while/for but also for any function call. In order to avoid
catching function pointer declarations, the logic allows a space if the
first character after the open parenthesis is an asterisk.
I also spotted what we did not include "switch" in the check but we should.
This check is a little lame, but we reduce this problem by not allowing
that space for if/while/for/switch.
Reported-by: Emanuele Torre
Closes#11044
