Commit Graph

339 Commits

Author SHA1 Message Date
Daniel Stenberg
95cbcec8f9
urldata: merge "struct DynamicStatic" into "struct UrlState"
Both were used for the same purposes and there was no logical separation
between them. Combined, this also saves 16 bytes in less holes in my
test build.

Closes #6798
2021-03-26 23:19:20 +01:00
Daniel Stenberg
94719e7285
multi: rename the multi transfer states
While working on documenting the states it dawned on me that step one is
to use more descriptive names on the states. This also changes prefix on
the states to make them shorter in the source.

State names NOT ending with *ing are transitional ones.

Closes #6612
2021-02-16 16:42:46 +01:00
Daniel Stenberg
cfff12a0b3
lib: remove 'conn->data' completely
The Curl_easy pointer struct entry in connectdata is now gone. Just
before commit 215db086e0 landed on January 8, 2021 there were 919
references to conn->data.

Closes #6608
2021-02-16 10:48:57 +01:00
Daniel Stenberg
8335c6417e
hostip: remove conn->data from resolver functions
This also moves the 'async' struct from the connectdata struct into the
Curl_easy struct, which seems like a better home for it.

Closes #6497
2021-01-22 08:25:09 +01:00
Daniel Stenberg
215db086e0
lib: pass in 'struct Curl_easy *' to most functions
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.

- We mostly operate on transfers and not connections.

- We need the transfer handle to log, store data and more. Everything in
  libcurl is driven by a transfer (the CURL * in the public API).

- This work clarifies and separates the transfers from the connections
  better.

- We should avoid "conn->data". Since individual connections can be used
  by many transfers when multiplexing, making sure that conn->data
  points to the current and correct transfer at all times is difficult
  and has been notoriously error-prone over the years. The goal is to
  ultimately remove the conn->data pointer for this reason.

Closes #6425
2021-01-17 23:56:09 +01:00
Daniel Stenberg
9d7281ff17
global_init: debug builds allocates a byte in init
... to make build tools/valgrind warn if no curl_global_cleanup is
called.

This is conditionally only done for debug builds with the env variable
CURL_GLOBAL_INIT set.

Closes #6410
2021-01-06 15:13:45 +01:00
Daniel Stenberg
ac9a7245fc
speedcheck: exclude paused transfers
Paused transfers should not be stopped due to slow speed even when
CURLOPT_LOW_SPEED_LIMIT is set. Additionally, the slow speed timer is
now reset when the transfer is unpaused - as otherwise it would easily
just trigger immediately after unpausing.

Reported-by: Harry Sintonen
Fixes #6358
Closes #6359
2020-12-22 13:51:07 +01:00
Daniel Stenberg
4d2f800677
curl.se: new home
Closes #6172
2020-11-04 23:59:47 +01:00
Daniel Stenberg
2cfc4ed983
hsts: add read/write callbacks
- read/write callback options
- man pages for the 4 new setopts
- test 1915 verifies the callbacks

Closes #5896
2020-11-03 16:08:48 +01:00
Daniel Stenberg
7385610d0c
hsts: add support for Strict-Transport-Security
- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle

Closes #5896
2020-11-03 16:08:42 +01:00
Daniel Stenberg
7772344e17
pause: only trigger a reread if the unpause sticks
As an unpause might itself get paused again and then triggering another
reread doesn't help.

Follow-up from e040146f22 (shipped since 7.69.1)

Bug: https://curl.haxx.se/mail/lib-2020-09/0081.html
Patch-by: Kunal Chandarana
Fixes #5988
Closes #6013
2020-09-25 15:18:10 +02:00
Daniel Stenberg
3d64031fa7
symbian: drop support
The OS is deprecated. I see no traces of anyone having actually built
curl for Symbian after 2012.

The public headers are unmodified.

Closes #5989
2020-09-22 15:14:12 +02:00
Quentin Balland
7921b41ce1
easy_reset: clear retry counter
Closes #5975
Fixes #5974
2020-09-18 07:59:24 +02:00
Daniel Stenberg
f93455eb04
altsvc: clone setting in curl_easy_duphandle
The cache content is not duplicated, like other caches, but the setting
and specified file name are.

Test 1908 is extended to verify this somewhat. Since the duplicated
handle gets the same file name, the test unfortunately overwrites the
same file twice (with different contents) which makes it hard to check
automatically.

Closes #5923
2020-09-06 09:54:54 +02:00
Daniel Stenberg
3c9e021f86
Curl_easy: remember last connection by id, not by pointer
CVE-2020-8231

Bug: https://curl.haxx.se/docs/CVE-2020-8231.html

Reported-by: Marc Aldorasi
Closes #5824
2020-08-17 14:33:09 +02:00
Marc Hoersken
3186f50054
timeouts: move ms timeouts to timediff_t from int and long
Now that all functions in select.[ch] take timediff_t instead
of the limited int or long, we can remove type conversions
and related preprocessor checks to silence compiler warnings.

Avoiding conversions from time_t was already done in 842f73de.

Based upon #5262
Supersedes #5214, #5220 and #5221
Follow up to #5343 and #5479
Closes #5490
2020-06-06 20:05:58 +02:00
Daniel Stenberg
c4e6968127
url: alloc the download buffer at transfer start
... and free it as soon as the transfer is done. It removes the extra
alloc when a new size is set with setopt() and reduces memory for unused
easy handles.

In addition: the closure_handle now doesn't use an allocated buffer at
all but the smallest supported size as a stack based one.

Closes #5472
2020-05-30 23:14:33 +02:00
Gilles Vollant
cac5374298
setopt: support certificate options in memory with struct curl_blob
This change introduces a generic way to provide binary data in setopt
options, called BLOBs.

This change introduces these new setopts:

CURLOPT_ISSUERCERT_BLOB, CURLOPT_PROXY_SSLCERT_BLOB,
CURLOPT_PROXY_SSLKEY_BLOB, CURLOPT_SSLCERT_BLOB and CURLOPT_SSLKEY_BLOB.

Reviewed-by: Daniel Stenberg
Closes #5357
2020-05-15 13:03:59 +02:00
denzor
a9021716cc easy: fix dangling pointer on easy_perform fail
Closes https://github.com/curl/curl/pull/5363
2020-05-12 03:00:27 -04:00
Daniel Stenberg
ed35d6590e
dynbuf: introduce internal generic dynamic buffer functions
A common set of functions instead of many separate implementations for
creating buffers that can grow when appending data to them. Existing
functionality has been ported over.

In my early basic testing, the total number of allocations seem at
roughly the same amount as before, possibly a few less.

See docs/DYNBUF.md for a description of the API.

Closes #5300
2020-05-04 10:40:39 +02:00
Jay Satiro
3bfda07004 easy: Fix curl_easy_duphandle for builds missing IPv6 that use c-ares
- Ignore CURLE_NOT_BUILT_IN errors returned by c-ares functions in
  curl_easy_duphandle.

Prior to this change if c-ares was used as the resolver backend and
either it was too old or libcurl was built without IPv6 support then
some of our resolver functions could return CURLE_NOT_BUILT_IN to
curl_easy_duphandle causing it to fail.

Caused by c8f086b which shipped in 7.69.1.

Reported-by: Karl Chen

Fixes https://github.com/curl/curl/issues/5097
Closes https://github.com/curl/curl/pull/5100
2020-03-14 19:07:05 -04:00
Patrick Monnerat
96972ec1c0
mime: latch last read callback status.
In case a read callback returns a status (pause, abort, eof,
error) instead of a byte count, drain the bytes read so far but
remember this status for further processing.
Takes care of not losing data when pausing, and properly resume a
paused mime structure when requested.
New tests 670-673 check unpausing cases, with easy or multi
interface and mime or form api.

Fixes #4813
Reported-by: MrdUkk on github
2020-03-07 23:26:00 +01:00
Ernst Sjöstrand
c8f086bcc3
ares: store dns parameters for duphandle
With c-ares the dns parameters lives in ares_channel. Store them in the
curl handle and set them again in easy_duphandle.

Regression introduced in #3228 (6765e6d), shipped in curl 7.63.0.

Fixes #4893
Closes #5020
Signed-off-by: Ernst Sjöstrand <ernst.sjostrand@verisure.com>
2020-03-07 13:50:23 +01:00
Daniel Stenberg
e364546fb3
version: make curl_version* thread-safe without using global context
Closes #5010
2020-03-07 12:10:11 +01:00
Daniel Stenberg
e040146f22
pause: force a connection (re-)check after unpausing
There might be data available that was already read off the socket, for
example in the TLS layer.

Reported-by: Anders Berg
Fixes #4966
Closes #5049
2020-03-07 10:49:51 +01:00
Daniel Stenberg
e54b1885d1
pause: bail out on bad input
A NULL easy handle or an easy handle without an associated connection
cannot be paused or unpaused.

Closes #5050
2020-03-06 17:31:01 +01:00
Daniel Stenberg
64258bd0aa
pause: return early for calls that don't change pause state
Reviewed-by: Patrick Monnerat
Ref: #4833
Closes #5026
2020-03-06 10:25:40 +01:00
Daniel Stenberg
8aa04e9a24
Revert "pause: force-drain the transfer on unpause"
This reverts commit fa0216b294 (from #5000)

Clearly that didn't solve the problem correctly.

Reported-by: Christopher Reid
Reopens #4966
Fixes #5044
2020-03-05 23:45:41 +01:00
Daniel Stenberg
8cd4e6d81f
Revert "mime: latch last read callback status."
This reverts commit 87869e38d7.

Fixes #5014
Closes #5015
Reopens #4833
2020-03-02 08:07:49 +01:00
Patrick Monnerat
87869e38d7
mime: latch last read callback status.
In case a read callback returns a status (pause, abort, eof,
error) instead of a byte count, drain the bytes read so far but
remember this status for further processing.
Takes care of not losing data when pausing, and properly resume a
paused mime structure when requested.
New tests 670-673 check unpausing cases, with easy or multi
interface and mime or form api.

Fixes #4813
Reported-by: MrdUkk on github
Closes #4833
2020-03-02 00:00:50 +01:00
Daniel Stenberg
fa0216b294
pause: force-drain the transfer on unpause
... since the socket might not actually be readable anymore when for
example the data is already buffered in the TLS layer.

Fixes #4966
Reported-by: Anders Berg
Closes #5000
2020-02-29 10:53:09 +01:00
Daniel Stenberg
15f51474c8
http2: make pausing/unpausing set/clear local stream window
This reduces the HTTP/2 window size to 32 MB since libcurl might have to
buffer up to this amount of data in memory and yet we don't want it set
lower to potentially impact tranfer performance on high speed networks.

Requires nghttp2 commit b3f85e2daa629
(https://github.com/nghttp2/nghttp2/pull/1444) to work properly, to end
up in the next release after 1.40.0.

Fixes #4939
Closes #4940
2020-02-27 10:35:51 +01:00
Jay Satiro
55bb83c92b easy: remove dead code
multi is already assigned to data->multi by curl_multi_add_handle.

Closes https://github.com/curl/curl/pull/4900
2020-02-12 16:54:51 -05:00
Daniel Stenberg
0b030a5b23
global_init: move the IPv6 works status bool to multi handle
Previously it was stored in a global state which contributed to
curl_global_init's thread unsafety. This boolean is now instead figured
out in curl_multi_init() and stored in the multi handle. Less effective,
but thread safe.

Closes #4851
2020-01-28 08:03:22 +01:00
Daniel Stenberg
1ad49feb71
global_init: assume the EINTR bit by default
- Removed from global_init since it isn't thread-safe. The symbol will
  still remain to not break compiles, it just won't have any effect going
  forward.

- make the internals NOT loop on EINTR (the opposite from previously).
  It only risks returning from the select/poll/wait functions early, and that
  should be risk-free.

Closes #4840
2020-01-26 18:27:18 +01:00
Daniel Stenberg
920af1a664
wolfssh: make it init properly via Curl_ssh_init()
Closes #4846
2020-01-24 10:27:31 +01:00
Daniel Stenberg
6773c7ca65
wolfSSH: new SSH backend
Adds support for SFTP (not SCP) using WolfSSH.

Closes #4231
2020-01-12 17:19:12 +01:00
Daniel Stenberg
bc5d22c3de
global_init: undo the "intialized" bump in case of failure
... so that failures in the global init function don't count as a
working init and it can then be called again.

Reported-by: Paul Groke
Fixes #4636
Closes #4653
2019-12-02 00:04:56 +01:00
Jay Satiro
1f6a18685e lib: Move lib/ssh.h -> lib/vssh/ssh.h
Follow-up to 5b2d703 which moved ssh source files to vssh.

Closes https://github.com/curl/curl/pull/4609
2019-11-17 23:27:39 -05:00
Daniel Stenberg
32747aafa0
pause: avoid updating socket if done was already called
... avoids unnecesary recursive risk when the transfer is already done.

Reported-by: Richard Bowker
Fixes #4563
Closes #4574
2019-11-10 16:48:47 +01:00
Daniel Stenberg
dcd7e37c3a
url: make Curl_close() NULLify the pointer too
This is the common pattern used in the code and by a unified approach we
avoid mistakes.

Closes #4534
2019-10-30 09:36:21 +01:00
Daniel Stenberg
2e68e5a023
easy: part of conditional expression is always true: !result
Fixes warning detected by PVS-Studio
Fixes #4374
2019-09-20 08:07:20 +02:00
Daniel Stenberg
5050edb124
cleanup: move functions out of url.c and make them static
Closes #4289
2019-09-03 13:11:12 +02:00
Daniel Stenberg
d6dea75af7
vssh: move ssh init/cleanup functions into backend code 2019-08-17 16:57:55 +02:00
Daniel Stenberg
02346abc32
curl_multi_poll: a sister to curl_multi_wait() that waits more
Repeatedly we see problems where using curl_multi_wait() is difficult or
just awkward because if it has no file descriptor to wait for
internally, it returns immediately and leaves it to the caller to wait
for a small amount of time in order to avoid occasional busy-looping.

This is often missed or misunderstood, leading to underperforming
applications.

This change introduces curl_multi_poll() as a replacement drop-in
function that accepts the exact same set of arguments. This function
works identically to curl_multi_wait() - EXCEPT - for the case when
there's nothing to wait for internally, as then this function will by
itself wait for a "suitable" short time before it returns. This
effectiely avoids all risks of busy-looping and should also make it less
likely that apps "over-wait".

This also changes the curl tool to use this funtion internally when
doing parallel transfers and changes curl_easy_perform() to use it
internally.

Closes #4163
2019-08-06 09:33:29 +02:00
Jay Satiro
78ed3abe11 easy: resize receive buffer on easy handle reset
- In curl_easy_reset attempt to resize the receive buffer to its default
  size. If realloc fails then continue using the previous size.

Prior to this change curl_easy_reset did not properly handle resetting
the receive buffer (data->state.buffer). It reset the variable holding
its size (data->set.buffer_size) to the default size (READBUFFER_SIZE)
but then did not actually resize the buffer. If a user resized the
buffer by using CURLOPT_BUFFERSIZE to set the size smaller than the
default, later called curl_easy_reset and attempted to reuse the handle
then a heap overflow would very likely occur during that handle's next
transfer.

Reported-by: Felix Hädicke

Fixes https://github.com/curl/curl/issues/4143
Closes https://github.com/curl/curl/pull/4145
2019-07-25 22:24:06 -04:00
Daniel Stenberg
680f141332
unpause: trigger a timeout for event-based transfers
... so that timeouts or other state machine actions get going again
after a changing pause state. For example, if the last delivery was
paused there's no pending socket activity.

Reported-by: sstruchtrup on github
Fixes #3994
Closes #4001
2019-06-09 18:33:59 +02:00
Daniel Stenberg
8ece8177f1
cleanup: remove FIXME and TODO comments
They serve very little purpose and mostly just add noise. Most of them
have been around for a very long time. I read them all before removing
or rephrasing them.

Ref: #3876
Closes #3883
2019-05-16 09:16:56 +02:00
Steve Holme
52dfab65d6
auth: Rename the various authentication clean up functions
For consistency and to a avoid confusion.

Closes #3869
2019-05-12 18:37:00 +01:00
Marcel Raad
15fd9abfea
easy: fix another "clarify calculation precedence" warning
I missed this one in commit 6b3dde7fe6.
2019-05-12 13:36:45 +02:00