Add `mime` client reader. Encapsulates reading from mime parts, getting
their length, rewinding and unpausing.
- remove special mime handling from sendf.c and easy.c
- add general "unpause" method to client readers
- use new reader in http/imap/smtp
- make some mime functions static that are now only used internally
In addition:
- remove flag 'forbidchunk' as no longer needed
Closes#13039
Curl_read/Curl_write clarifications
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer setup of
`conn->sockfd` and `conn->writesockfd` on which connection filter
chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index as
parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
consistency
- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
returned by all other send() variants.
SingleRequest reshuffling
- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data
Closes#12963
Remove curl_mimepart object from UserDefined structure when
CURL_DISABLE_MIME flag is active. Reduce size of UserDefined structure.
Also remove unreachable code: when CURL_DISABLE_MIME is set, httpreq can
never have HTTPREQ_POST_MIME value and the same goes for the
CURL_DISABLE_FORM_API flag and the HTTPREQ_POST_FORM value
Closes#12948
The pingpong logic now uses its own dynbuf for receiving command
response data.
When the "final" response header for a commanad has been received, that
final line is left first in the recvbuf for the protocols to parse at
will. If there is additional data behind the final response line, the
'overflow' counter is indicate how many bytes.
Closes#12757
This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.
The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.
Curl's protocol handler `readwrite()` method been changed:
```diff
- CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
- const char *buf, size_t blen,
- size_t *pconsumed, bool *readmore);
+ CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+ bool is_eos, bool *done);
```
The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:
* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
finished
This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.
The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.
HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.
The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:
```
static CURLcode readwrite_data(data, ...)
{
do {
Curl_xfer_recv_resp(data, buf)
...
Curl_xfer_write_resp(data, buf)
...
} while(interested);
...
}
```
All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.
All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.
Closes#12480
- memdebug: update to not trigger `-Wformat-nonliteral` warnings.
- imap: mark `imap_sendf()` with `CURL_PRINTF()`.
- tool_msgs: mark static function with `CURL_PRINTF()`.
Follow-up to 3829759bd0#12489Closes#12540
https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].
Enable new recommended warnings (except `-Wsign-conversion`):
- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
printf arguments with it. This is a copy of existing
`CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
with redefinting the `printf` symbol:
https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
Follow-up to d5c0351055#2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
checks. Previously it was always disabled due to the internal `printf`
macro.
Fix them:
- fix bug where an `set_ipv6_v6only()` call was missed in builds with
`--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.
[1] 56c0fde389/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.mdCloses#12489
- use `data->state.dselect_bits` everywhere instead
- remove `bool *comeback` parameter as non-zero
`data->state.dselect_bits` will indicate that IO is
incomplete.
Closes#12512
GCC 14 introduces a new -Walloc-size included in -Wextra which gives:
```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operate.c:213:5: warning: allocation of insufficient size ‘1’ for type ‘struct per_transfer’ with size ‘480’ [-Walloc-size]
213 | p = calloc(sizeof(struct per_transfer), 1);
| ^
src/var.c: In function ‘addvariable’:
src/var.c:361:5: warning: allocation of insufficient size ‘1’ for type ‘struct var’ with size ‘32’ [-Walloc-size]
361 | p = calloc(sizeof(struct var), 1);
| ^
```
The calloc prototype is:
```
void *calloc(size_t nmemb, size_t size);
```
So, just swap the number of members and size arguments to match the
prototype, as we're initialising 1 struct of size `sizeof(struct
...)`. GCC then sees we're not doing anything wrong.
Closes#12292
This PR has these changes:
Renaming of unencode_* to cwriter, e.g. client writers
- documentation of sendf.h functions
- move max decode stack checks back to content_encoding.c
- define writer phase which was used as order before
- introduce phases for monitoring inbetween decode phases
- offering default implementations for init/write/close
Add type paramter to client writer's do_write()
- always pass all writes through the writer stack
- writers who only care about BODY data will pass other writes unchanged
add RAW and PROTOCOL client writers
- RAW used for Curl_debug() logging of CURLINFO_DATA_IN
- PROTOCOL used for updates to data->req.bytecount, max_filesize checks and
Curl_pgrsSetDownloadCounter()
- remove all updates of data->req.bytecount and calls to
Curl_pgrsSetDownloadCounter() and Curl_debug() from other code
- adjust test457 expected output to no longer see the excess write
Closes#12184
- cache more Windows config results for faster initialization.
- delete unused config macros `HAVE_SYS_UTSNAME_H`, `HAVE_SSL_H`.
- delete dead references to `sys/utsname.h`.
Closes#11551
- Implement AUTH=+LOGIN for CURLOPT_LOGIN_OPTIONS to prefer plaintext
LOGIN over SASL auth.
Prior to this change there was no method to be able to fall back to
LOGIN if an IMAP server advertises SASL capabilities. However, this may
be desirable for e.g. a misconfigured server.
Per: https://www.ietf.org/rfc/rfc5092.html#section-3.2
";AUTH=<enc-auth-type>" looks to be the correct way to specify what
authenication method to use, regardless of SASL or not.
Closes https://github.com/curl/curl/pull/10041
- add an `id` long to Curl_easy, -1 on init
- once added to a multi (or its own multi), it gets
a non-negative number assigned by the connection cache
- `id` is unique among all transfers using the same
cache until reaching LONG_MAX where it will wrap
around. So, not unique eternally.
- CURLINFO_CONN_ID returns the connection id attached to
data or, if none present, data->state.lastconnect_id
- variables and type declared in tool for write out
Closes#11185
Aka "jumbo" or "amalgamation" builds. It means to compile all sources
per target as a single C source. This is experimental.
You can enable it by passing `-DCMAKE_UNITY_BUILD=ON` to cmake.
It requires CMake 3.16 or newer.
It makes builds (much) faster, allows for better optimizations and tends
to promote less ambiguous code.
Also add a new AppVeyor CI job and convert an existing one to use
"unity" mode (one MSVC, one MinGW), and enable it for one macOS CI job.
Fix related issues:
- add missing include guard to `easy_lock.h`.
- rename static variables and functions (and a macro) with names reused
across sources, or shadowed by local variables.
- add an `#undef` after use.
- add a missing `#undef` before use.
- move internal definitions from `ftp.h` to `ftp.c`.
- `curl_memory.h` fixes to make it work when included repeatedly.
- stop building/linking curlx bits twice for a static-mode curl tool.
These caused doubly defined symbols in unity builds.
- silence missing extern declarations compiler warning for ` _CRT_glob`.
- fix extern declarations for `tool_freq` and `tool_isVistaOrGreater`.
- fix colliding static symbols in debug mode: `debugtime()` and
`statename`.
- rename `ssl_backend_data` structure to unique names for each
TLS-backend, along with the `ssl_connect_data` struct member
referencing them. This required adding casts for each access.
- add workaround for missing `[P]UNICODE_STRING` types in certain Windows
builds when compiling `lib/ldap.c`. To support "unity" builds, we had
to enable `SCHANNEL_USE_BLACKLISTS` for Schannel (a Windows
`schannel.h` option) _globally_. This caused an indirect inclusion of
Windows `schannel.h` from `ldap.c` via `winldap.h` to have it enabled
as well. This requires `[P]UNICODE_STRING` types, which is apperantly
not defined automatically (as seen with both MSVS and mingw-w64).
This patch includes `<subauth.h>` to fix it.
Ref: https://github.com/curl/curl/runs/13987772013
Ref: https://dev.azure.com/daniel0244/curl/_build/results?buildId=15827&view=logs&jobId=2c9f582d-e278-56b6-4354-f38a4d851906&j=2c9f582d-e278-56b6-4354-f38a4d851906&t=90509b00-34fa-5a81-35d7-5ed9569d331c
- tweak unity builds to compile `lib/memdebug.c` separately in memory
trace builds to avoid PP confusion.
- force-disable unity for test programs.
- do not compile and link libcurl sources to libtests _twice_ when libcurl
is built in static mode.
KNOWN ISSUES:
- running tests with unity builds may fail in cases.
- some build configurations/env may not compile in unity mode. E.g.:
https://ci.appveyor.com/project/curlorg/curl/builds/47230972/job/51wfesgnfuauwl8q#L250
Ref: https://github.com/libssh2/libssh2/issues/1034
Ref: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html
Ref: https://en.wikipedia.org/wiki/Unity_buildCloses#11095
If the previous line starts with if/while/for AND ends with a closed
parenthesis and there's an equal number of open and closed parentheses
on that line, verify that this line is indented $indent more steps, if
not a cpp line.
Also adjust the fall-out from this fix.
Closes#11054
By making sure we set state.upload based on the set.method value and not
independently as set.upload, we reduce confusion and mixup risks, both
internally and externally.
Closes#11017
- by using BIT() instead of bool
- imap: shrink struct
- ftp: make state 'unsigned char'
- ftp: sort ftp_conn struct entries on size
- urldata: use smaller fields for SSL version info storage
- pop3: reduce the pop3_conn struct size
- smtp: reduce the size of the smtp structs
Closes#10186
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.
- eyeballing cfilter that uses sub-filters for performing parallel connects
- socket cfilter for all transport types, including QUIC
- QUIC implementations in cfilter, can now participate in eyeballing
- connection setup is more dynamic in order to adapt to what filter did
really connect. Relevant to see if a SSL filter needs to be added or
if SSL has already been provided
- HTTP/3 test cases similar to HTTP/2
- multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche
- Fix for data attach/detach in VTLS filters that could lead to crashes
during parallel transfers.
- Eliminating setup() methods in cfilters, no longer needed.
- Improving Curl_conn_is_alive() to replace Curl_connalive() and
integrated ssl alive checks into cfilter.
- Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
connection into and persist it at the easy handle.
- Several more cfilter related cleanups and moves:
- stream_weigth and dependency info is now wrapped in struct
Curl_data_priority
- Curl_data_priority members depend is available in HTTP2|HTTP3
- Curl_data_priority members depend on NGHTTP2 support
- handling init/reset/cleanup of priority part of url.c
- data->state.priority same struct, but shallow copy for compares only
- PROTOPT_STREAM has been removed
- Curl_conn_is_mulitplex() now available to check on capability
- Adding query method to connection filters.
- ngtcp2+quiche: implementing query for max concurrent transfers.
- Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
- setting keepalive timestamp on connect
- DATA_SETUP is called after the connection has been completely
setup (but may not connected yet) to allow filters to initialize
data members they use.
- there is no socket to be had with msh3, it is unclear how select
shall work
- manual test via "curl --http3 https://curl.se" fail with "empty
reply from server".
- Various socket/conn related cleanups:
- Curl_socket is now Curl_socket_open and in cf-socket.c
- Curl_closesocket is now Curl_socket_close and in cf-socket.c
- Curl_ssl_use has been replaced with Cur_conn_is_ssl
- Curl_conn_tcp_accepted_set has been split into
Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
with a clearer purpose
Closes#10141
- fixes `Curl_ssl_cf_get_ssl()` to detect also the first filter instance
as ssl (refs #10053)
- replaces `Curl_ssl_use()` with the correct `Curl_conn_is_ssl()`
Closes#10054Fixes#10053
Reported-by: Patrick Monnerat
Deprecation and removal of codeset conversion support from the library
have released the strict need for an early binding of mime structures to
an easy handle (https://github.com/curl/curl/commit/2610142).
This constraint currently forces to create the handle before the mime
structure and the latter cannot be attached to another handle once
created (see https://curl.se/mail/lib-2022-08/0027.html).
This commit removes the handle pointers from the mime structures
allowing more flexibility on their use.
When an easy handle is duplicated, bound mime structures must however
still be duplicated too as their components hold send-time dynamic
information.
Closes#9927
- `Curl_ssl_get_config()` now returns the first config if no SSL proxy
filter is active
- socket filter starts connection only on first invocation of its
connect method
Fixes#9982Closes#9983
Commit 3b16575ae9 removed support for
building on Novell Netware, but a few leftover traces remained. This
removes the last bits.
Closes: #9966
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
- almost all backend calls pass the Curl_cfilter intance instead of
connectdata+sockindex
- ssl_connect_data is remove from struct connectdata and made internal
to vtls
- ssl_connect_data is allocated in the added filter, kept at cf->ctx
- added function to let a ssl filter access its ssl_primary_config and
ssl_config_data this selects the propert subfields in conn and data,
for filters added as plain or proxy
- adjusted all backends to use the changed api
- adjusted all backends to access config data via the exposed
functions, no longer using conn or data directly
cfilter renames for clear purpose:
- methods `Curl_conn_*(data, conn, sockindex)` work on the complete
filter chain at `sockindex` and connection `conn`.
- methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter
instance.
- methods `Curl_conn_cf()` work on/with filter instances at a
connection.
- rebased and resolved some naming conflicts
- hostname validation (und session lookup) on SECONDARY use the same
name as on FIRST (again).
new debug macros and removing connectdata from function signatures where not
needed.
adapting schannel for new Curl_read_plain paramter.
Closes#9919
This struct field MUST remain what the application set it to, so that
handle reuse and handle duplication work.
Instead, the request state bit 'no_body' is introduced for code flows
that need to change this in run-time.
Closes#9888
- general construct/destroy in connectdata
- default implementations of callback functions
- connect: cfilters for connect and accept
- socks: cfilter for socks proxying
- http_proxy: cfilter for http proxy tunneling
- vtls: cfilters for primary and proxy ssl
- change in general handling of data/conn
- Curl_cfilter_setup() sets up filter chain based on data settings,
if none are installed by the protocol handler setup
- Curl_cfilter_connect() boot straps filters into `connected` status,
used by handlers and multi to reach further stages
- Curl_cfilter_is_connected() to check if a conn is connected,
e.g. all filters have done their work
- Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE
indicators for multi select to work
- Curl_cfilter_data_pending() asks filters if the have incoming
data pending for recv
- Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks
installed in conn->recv/conn->send for io handling
- Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters
and addition/removal of a `data` from their connection
- adding vtl functions to prevent use of Curl_ssl globals directly
in other parts of the code.
Reviewed-by: Daniel Stenberg
Closes#9855
This commit replaces a self-made character check for alphanumeric
characters within imap_is_bchar() with the ISALNUM() macro, as it is
reduces the size of the code and makes the performance better, due to
ASCII arithmetic.
Closes#9289
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
Capabilities of sasl module are extended to exchange messages in binary
as an alternative to base64.
If http authentication flags have been set, those are used as sasl
default preferred mechanisms.
Closes#6930
If a server pipelines future responses within the STARTTLS response, the
former are preserved in the pingpong cache across TLS negotiation and
used as responses to the encrypted commands.
This fix detects pipelined STARTTLS responses and rejects them with an
error.
CVE-2021-22947
Bug: https://curl.se/docs/CVE-2021-22947.html
In imap and pop3, check if TLS is required even when capabilities
request has failed.
In ftp, ignore preauthentication (230 status of server greeting) if TLS
is required.
Bug: https://curl.se/docs/CVE-2021-22946.html
CVE-2021-22946
- the data needs to be "line-based" anyway since it's also passed to the
debug callback/application
- it makes infof() work like failf() and consistency is good
- there's an assert that triggers on newlines in the format string
- Also removes a few instances of "..."
- Removes the code that would append "..." to the end of the data *iff*
it was truncated in infof()
Closes#7357
Introducing a 'isproxy' argument to the connect function so that it
knows wether to store the time stamp or not.
Reported-by: Yongkang Huang
Fixes#7274Closes#7274
The libssh2 backend has SSH session associated with the connection but
the callback context is the easy handle, so when a connection gets
attached to a transfer, the protocol handler now allows for a custom
function to get used to set things up correctly.
Reported-by: Michael O'Farrell
Fixes#6898Closes#7078