mirror/curl
2
0
Fork 0
mirror of https://github.com/curl/curl.git synced 2024-12-09 06:30:06 +08:00
Code Issues Packages Projects Releases Wiki Activity

lib: enhance client reader resume + rewind

Browse Source 
- update client reader documentation
- client reader, add rewind capabilities
    - tell creader to rewind on next start
    - Curl_client_reset() will keep reader for future rewind if requested
    - add Curl_client_cleanup() for freeing all resources independent of
      rewinds
    - add Curl_client_start() to trigger rewinds
    - move rewind code from multi.c to sendf.c and make part of
      "cr-in"'s implementation
- http, move the "resume_from" handling into the client readers
    - the setup of a HTTP request is reshuffled to follow:
      * determine method, target, auth negotiation
      * install the client reader(s) for the request, including crlf
        conversions and "chunked" encoding
      * apply ranges to client reader
      * concat request headers, upgrades, cookies, etc.
      * complete request by determining Content-Length of installed
        readers in combination with method
      * send
    - add methods for client readers to
      * return the overall length they will generate (or -1 when unknown)
      * return the amount of data on the CLIENT level, so that
        expect-100 can decide if it want to apply itself
      * set a "resume_from" offset or fail if unsupported
    - struct HTTP has become largely empty now
- rename `Client_reader_*` to `Curl_creader_*`

Closes #13026
This commit is contained in:
Stefan Eissing 2024-02-29 10:12:39 +01:00 committed by Daniel Stenberg
parent 9c7768cd88
commit 14bcea074a
No known key found for this signature in database
GPG Key ID: 5CC908FDB71E12C2
15 changed files with 752 additions and 474 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
docs/CLIENT-READERS.md
View File

@ -31,6 +31,11 @@ struct Curl_crtype {
char *buf, size_t blen, size_t *nread, bool *eos);
void (*do_close)(struct Curl_easy *data, struct Curl_creader *reader);
bool (*needs_rewind)(struct Curl_easy *data, struct Curl_creader *reader);
curl_off_t (*total_length)(struct Curl_easy *data,
struct Curl_creader *reader);
CURLcode (*resume_from)(struct Curl_easy *data,
struct Curl_creader *reader, curl_off_t offset);
CURLcode (*rewind)(struct Curl_easy *data, struct Curl_creader *reader);
};
struct Curl_creader {
@ -80,6 +85,36 @@ Implemented in `sendf.c` for phase `CURL_CR_CLIENT`, this reader get a buffer po
Sometimes it is necessary to send a request with client data again. Transfer handling can inquire via `Curl_client_read_needs_rewind()` if a rewind (e.g. a reset of the client data) is necessary. This asks all installed readers if they need it and give `FALSE` of none does.
## Upload Size
Many protocols need to know the amount of bytes delivered by the client readers in advance. They may invoke `Curl_creader_total_length(data)` to retrieve that. However, not all reader chains know the exact value beforehand. In that case, the call returns `-1` for "unknown".
Even if the length of the "raw" data is known, the length that is send may not. Example: with option `--crlf` the uploaded content undergoes line-end conversion. The line converting reader does not know in advance how many newlines it may encounter. Therefore it must return `-1` for any positive raw content length.
In HTTP, once the correct client readers are installed, the protocol asks the readers for the total length. If that is known, it can set `Content-Length:` accordingly. If not, it may choose to add an HTTP "chunked" reader.
In addition, there is `Curl_creader_client_length(data)` which gives the total length as reported by the reader in phase `CURL_CR_CLIENT` without asking other readers that may transform the raw data. This is useful in estimating the size of an upload. The HTTP protocol uses this to determine if `Expect: 100-continue` shall be done.
## Resuming
Uploads can start at a specific offset, if so requested. The "resume from" that offset. This applies to the reader in phase `CURL_CR_CLIENT` that delivers the "raw" content. Resumption can fail if the installed reader does not support it or if the offset is too large.
The total length reported by the reader changes when resuming. Example: resuming an upload of 100 bytes by 25 reports a total length of 75 afterwards.
If `resume_from()` is invoked twice, it is additive. There is currently no way to undo a resume.
## Rewinding
When a request is retried, installed client readers are discarded and replaced by new ones. This works only if the new readers upload the same data. For many readers, this is not an issue. The "null" reader always does the same. Also the `buf` reader, initialized with the same buffer, does this.
Readers operating on callbacks to the application need to "rewind" the underlying content. For example, when reading from a `FILE*`, the reader needs to `fseek()` to the beginning. The following methods are used:
1. `Curl_creader_needs_rewind(data)`: tells if a rewind is necessary, given the current state of the reader chain. If nothing really has been read so far, this returns `FALSE`.
2. `Curl_creader_will_rewind(data)`: tells if the reader chain rewinds at the start of the next request.
3. `Curl_creader_set_rewind(data, TRUE)`: marks the reader chain for rewinding at the start of the next request.
4. `Curl_client_start(data)`: tells the readers that a new request starts and they need to rewind if requested.
## Summary and Outlook
By adding the client reader interface, any protocol can control how/if it wants the curl transfer to send bytes for a request. The transfer loop becomes then blissfully ignorant of the specifics.
@ -87,7 +122,5 @@ By adding the client reader interface, any protocol can control how/if it wants
The protocols on the other hand no longer have to care to package data most efficiently. At any time, should more data be needed, it can be read from the client. This is used when sending HTTP requests headers to add as much request body data to the initial sending as there is room for.
Future enhancements based on the client readers:
* delegate the actual "rewinding" to the readers. The should know how it is done, eliminating the `readrewind.c` protocol specifics in `multi.c`.
* `expect-100` handling: place that into a HTTP specific reader at `CURL_CR_PROTOCOL` and eliminate the checks in the generic transfer parts.
* `eos` detection: `upload_done` is partly triggered now by comparing the number of bytes sent to a known size. This is no longer necessary since the core readers obey length restrictions.
* `eos forwarding`: transfer should forward an `eos` flag to the connection filters. Filters like HTTP/2 and HTTP/3 can make use of that, terminating streams early. This would also eliminate length checks in stream handling.

14
lib/c-hyper.c
View File

@ -363,7 +363,7 @@ CURLcode Curl_hyper_stream(struct Curl_easy *data,
k->exp100 = EXP100_SEND_DATA;
k->keepon |= KEEP_SEND;
Curl_expire_done(data, EXPIRE_100_TIMEOUT);
infof(data, "Done waiting for 100-continue");
infof(data, "Done waiting for 100-continue after %ldms", (long)ms);
if(data->hyp.exp100_waker) {
hyper_waker_wake(data->hyp.exp100_waker);
data->hyp.exp100_waker = NULL;
@ -848,7 +848,9 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
may be parts of the request that is not yet sent, since we can deal with
the rest of the request in the PERFORM phase. */
*done = TRUE;
Curl_client_reset(data);
result = Curl_client_start(data);
if(result)
return result;
/* Add collecting of headers written to client. For a new connection,
* we might have done that already, but reuse
@ -883,9 +885,9 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
return result;
}
result = Curl_http_resume(data, conn, httpreq);
result = Curl_http_req_set_reader(data, httpreq, &te);
if(result)
return result;
goto error;
result = Curl_http_range(data, httpreq);
if(result)
@ -1006,10 +1008,6 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
goto error;
}
result = Curl_http_body(data, conn, httpreq, &te);
if(result)
goto error;
if(data->state.aptr.host) {
result = Curl_hyper_header(data, headers, data->state.aptr.host);
if(result)

501
lib/http.c
View File

@ -430,50 +430,23 @@ static CURLcode http_perhapsrewind(struct Curl_easy *data,
{
struct HTTP *http = data->req.p.http;
curl_off_t bytessent;
curl_off_t expectsend = -1; /* default is unknown */
curl_off_t expectsend = Curl_creader_total_length(data);
if(!http)
/* If this is still NULL, we have not reach very far and we can safely
skip this rewinding stuff */
return CURLE_OK;
switch(data->state.httpreq) {
case HTTPREQ_GET:
case HTTPREQ_HEAD:
if(!expectsend)
/* not sending any body */
return CURLE_OK;
default:
break;
}
bytessent = data->req.writebytecount;
if(data->req.authneg) {
/* This is a state where we are known to be negotiating and we don't send
any data then. */
expectsend = 0;
}
else if(!conn->bits.protoconnstart) {
if(!conn->bits.protoconnstart)
/* HTTP CONNECT in progress: there is no body */
expectsend = 0;
}
else {
/* figure out how much data we are expected to send */
switch(data->state.httpreq) {
case HTTPREQ_POST:
case HTTPREQ_PUT:
if(data->state.infilesize != -1)
expectsend = data->state.infilesize;
break;
case HTTPREQ_POST_FORM:
case HTTPREQ_POST_MIME:
expectsend = http->postsize;
break;
default:
break;
}
}
data->state.rewindbeforesend = FALSE; /* default */
bytessent = data->req.writebytecount;
Curl_creader_set_rewind(data, FALSE);
if((expectsend == -1) || (expectsend > bytessent)) {
#if defined(USE_NTLM)
@ -490,7 +463,7 @@ static CURLcode http_perhapsrewind(struct Curl_easy *data,
/* rewind data when completely done sending! */
if(!data->req.authneg && (conn->writesockfd != CURL_SOCKET_BAD)) {
data->state.rewindbeforesend = TRUE;
Curl_creader_set_rewind(data, TRUE);
infof(data, "Rewind stream before next send");
}
@ -518,7 +491,7 @@ static CURLcode http_perhapsrewind(struct Curl_easy *data,
/* rewind data when completely done sending! */
if(!data->req.authneg && (conn->writesockfd != CURL_SOCKET_BAD)) {
data->state.rewindbeforesend = TRUE;
Curl_creader_set_rewind(data, TRUE);
infof(data, "Rewind stream before next send");
}
@ -543,9 +516,9 @@ static CURLcode http_perhapsrewind(struct Curl_easy *data,
closure so we can safely do the rewind right now */
}
if(Curl_client_read_needs_rewind(data)) {
if(Curl_creader_needs_rewind(data)) {
/* mark for rewind since if we already sent something */
data->state.rewindbeforesend = TRUE;
Curl_creader_set_rewind(data, TRUE);
infof(data, "Please rewind output before next send");
}
@ -604,7 +577,7 @@ CURLcode Curl_http_auth_act(struct Curl_easy *data)
if(pickhost || pickproxy) {
if((data->state.httpreq != HTTPREQ_GET) &&
(data->state.httpreq != HTTPREQ_HEAD) &&
!data->state.rewindbeforesend) {
!Curl_creader_will_rewind(data)) {
result = http_perhapsrewind(data, conn);
if(result)
return result;
@ -2021,13 +1994,11 @@ CURLcode Curl_http_target(struct Curl_easy *data,
return result;
}
CURLcode Curl_http_body(struct Curl_easy *data, struct connectdata *conn,
Curl_HttpReq httpreq, const char **tep)
#if !defined(CURL_DISABLE_MIME) || !defined(CURL_DISABLE_FORM_API)
static CURLcode set_post_reader(struct Curl_easy *data, Curl_HttpReq httpreq)
{
CURLcode result = CURLE_OK;
const char *ptr;
struct HTTP *http = data->req.p.http;
http->postsize = 0;
curl_off_t postsize = 0;
CURLcode result;
switch(httpreq) {
#ifndef CURL_DISABLE_MIME
@ -2056,6 +2027,7 @@ CURLcode Curl_http_body(struct Curl_easy *data, struct connectdata *conn,
#endif
default:
data->state.mimepost = NULL;
break;
}
#ifndef CURL_DISABLE_MIME
@ -2081,10 +2053,137 @@ CURLcode Curl_http_body(struct Curl_easy *data, struct connectdata *conn,
result = Curl_mime_rewind(data->state.mimepost);
if(result)
return result;
http->postsize = Curl_mime_size(data->state.mimepost);
postsize = Curl_mime_size(data->state.mimepost);
}
#endif
switch(httpreq) {
case HTTPREQ_POST_FORM:
case HTTPREQ_POST_MIME:
/* This is form posting using mime data. */
data->state.infilesize = postsize;
if(!postsize)
result = Curl_creader_set_null(data);
else {
/* Read from mime structure. We could do a special client reader
* for this, but replacing the callback seems to work fine. */
data->state.fread_func = (curl_read_callback) Curl_mime_read;
data->state.in = (void *) data->state.mimepost;
result = Curl_creader_set_fread(data, postsize);
}
return result;
default:
if(!postsize)
result = Curl_creader_set_null(data);
else
result = Curl_creader_set_fread(data, postsize);
return result;
}
/* never reached */
}
#endif
static CURLcode set_reader(struct Curl_easy *data, Curl_HttpReq httpreq)
{
CURLcode result = CURLE_OK;
curl_off_t postsize = data->state.infilesize;
DEBUGASSERT(data->conn);
if(data->req.authneg) {
return Curl_creader_set_null(data);
}
switch(httpreq) {
case HTTPREQ_PUT: /* Let's PUT the data to the server! */
if(!postsize)
result = Curl_creader_set_null(data);
else
result = Curl_creader_set_fread(data, postsize);
return result;
#if !defined(CURL_DISABLE_MIME) || !defined(CURL_DISABLE_FORM_API)
case HTTPREQ_POST_FORM:
case HTTPREQ_POST_MIME:
return set_post_reader(data, httpreq);
#endif
case HTTPREQ_POST:
/* this is the simple POST, using x-www-form-urlencoded style */
/* the size of the post body */
if(!postsize) {
result = Curl_creader_set_null(data);
}
else if(data->set.postfields) {
if(postsize > 0)
result = Curl_creader_set_buf(data, data->set.postfields,
(size_t)postsize);
else
result = Curl_creader_set_null(data);
}
else { /* we read the bytes from the callback */
result = Curl_creader_set_fread(data, postsize);
}
return result;
default:
/* HTTP GET/HEAD download, has no body, needs no Content-Length */
data->state.infilesize = 0;
return Curl_creader_set_null(data);
}
/* not reached */
}
static CURLcode http_resume(struct Curl_easy *data, Curl_HttpReq httpreq)
{
if((HTTPREQ_POST == httpreq || HTTPREQ_PUT == httpreq) &&
data->state.resume_from) {
/**********************************************************************
* Resuming upload in HTTP means that we PUT or POST and that we have
* got a resume_from value set. The resume value has already created
* a Range: header that will be passed along. We need to "fast forward"
* the file the given number of bytes and decrease the assume upload
* file size before we continue this venture in the dark lands of HTTP.
* Resuming mime/form posting at an offset > 0 has no sense and is ignored.
*********************************************************************/
if(data->state.resume_from < 0) {
/*
* This is meant to get the size of the present remote-file by itself.
* We don't support this now. Bail out!
*/
data->state.resume_from = 0;
}
if(data->state.resume_from && !data->req.authneg) {
/* only act on the first request */
CURLcode result;
result = Curl_creader_resume_from(data, data->state.resume_from);
if(result) {
failf(data, "Unable to resume from offset %" CURL_FORMAT_CURL_OFF_T,
data->state.resume_from);
return result;
}
}
}
return CURLE_OK;
}
CURLcode Curl_http_req_set_reader(struct Curl_easy *data,
Curl_HttpReq httpreq,
const char **tep)
{
CURLcode result = CURLE_OK;
const char *ptr;
result = set_reader(data, httpreq);
if(result)
return result;
result = http_resume(data, httpreq);
if(result)
return result;
ptr = Curl_checkheaders(data, STRCONST("Transfer-Encoding"));
if(ptr) {
/* Some kind of TE is requested, check if 'chunked' is chosen */
@ -2093,18 +2192,14 @@ CURLcode Curl_http_body(struct Curl_easy *data, struct connectdata *conn,
STRCONST("Transfer-Encoding:"), STRCONST("chunked"));
}
else {
if((conn->handler->protocol & PROTO_FAMILY_HTTP) &&
(((httpreq == HTTPREQ_POST_MIME || httpreq == HTTPREQ_POST_FORM) &&
http->postsize < 0) ||
((data->state.upload || httpreq == HTTPREQ_POST) &&
data->state.infilesize == -1))) {
if(data->req.authneg)
/* don't enable chunked during auth neg */
;
else if(Curl_use_http_1_1plus(data, conn)) {
if(conn->httpversion < 20)
/* HTTP, upload, unknown file size and not HTTP 1.0 */
data->req.upload_chunky = TRUE;
curl_off_t req_clen = Curl_creader_total_length(data);
if(req_clen < 0) {
/* indeterminate request content length */
if(Curl_use_http_1_1plus(data, data->conn)) {
/* On HTTP/1.1, enable chunked, on HTTP/2 and later we do not
* need it */
data->req.upload_chunky = (data->conn->httpversion < 20);
}
else {
failf(data, "Chunky upload is not supported by HTTP 1.0");
@ -2127,7 +2222,6 @@ static CURLcode addexpect(struct Curl_easy *data, struct dynbuf *r)
data->state.expect100header = FALSE;
/* Avoid Expect: 100-continue if Upgrade: is used */
if(data->req.upgr101 == UPGR101_INIT) {
struct HTTP *http = data->req.p.http;
/* For really small puts we don't use Expect: headers at all, and for
the somewhat bigger ones we allow the app to disable it. Just make
sure that the expect100header is always set to the preferred value
@ -2138,8 +2232,11 @@ static CURLcode addexpect(struct Curl_easy *data, struct dynbuf *r)
Curl_compareheader(ptr, STRCONST("Expect:"),
STRCONST("100-continue"));
}
else if(http->postsize > EXPECT_100_THRESHOLD || http->postsize < 0)
return expect100(data, r);
else {
curl_off_t client_len = Curl_creader_client_length(data);
if(client_len > EXPECT_100_THRESHOLD || client_len < 0)
return expect100(data, r);
}
}
return CURLE_OK;
}
@ -2148,85 +2245,48 @@ CURLcode Curl_http_req_complete(struct Curl_easy *data,
struct dynbuf *r, Curl_HttpReq httpreq)
{
CURLcode result = CURLE_OK;
struct HTTP *http = data->req.p.http;
curl_off_t req_clen;
DEBUGASSERT(data->conn);
#ifndef USE_HYPER
if(data->req.upload_chunky) {
result = Curl_httpchunk_add_reader(data);
if(result)
return result;
}
#endif
DEBUGASSERT(data->conn);
/* Get the request body length that has been set up */
req_clen = Curl_creader_total_length(data);
switch(httpreq) {
case HTTPREQ_PUT: /* Let's PUT the data to the server! */
if(data->req.authneg)
http->postsize = 0;
else
http->postsize = data->state.infilesize;
if((http->postsize != -1) && !data->req.upload_chunky &&
(data->req.authneg ||
!Curl_checkheaders(data, STRCONST("Content-Length")))) {
/* only add Content-Length if not uploading chunked */
result = Curl_dyn_addf(r, "Content-Length: %" CURL_FORMAT_CURL_OFF_T
"\r\n", http->postsize);
if(result)
goto out;
}
result = addexpect(data, r);
if(result)
goto out;
/* end of headers */
result = Curl_dyn_addn(r, STRCONST("\r\n"));
if(result)
goto out;
/* set the upload size to the progress meter */
Curl_pgrsSetUploadSize(data, http->postsize);
if(!http->postsize)
result = Client_reader_set_null(data);
else
result = Client_reader_set_fread(data, data->state.infilesize);
break;
case HTTPREQ_PUT:
case HTTPREQ_POST:
#if !defined(CURL_DISABLE_MIME) || !defined(CURL_DISABLE_FORM_API)
case HTTPREQ_POST_FORM:
case HTTPREQ_POST_MIME:
/* This is form posting using mime data. */
if(data->req.authneg) {
/* nothing to post! */
result = Curl_dyn_addn(r, STRCONST("Content-Length: 0\r\n\r\n"));
if(!result)
result = Client_reader_set_null(data);
if(result)
return result;
/* setup variables for the upcoming transfer */
Curl_xfer_setup(data, FIRSTSOCKET, -1, TRUE, -1);
break;
}
data->state.infilesize = http->postsize;
#endif
/* We only set Content-Length and allow a custom Content-Length if
we don't upload data chunked, as RFC2616 forbids us to set both
kinds of headers (Transfer-Encoding: chunked and Content-Length) */
if(http->postsize != -1 && !data->req.upload_chunky &&
(!Curl_checkheaders(data, STRCONST("Content-Length")))) {
kinds of headers (Transfer-Encoding: chunked and Content-Length).
We do not override a custom "Content-Length" header, but during
authentication negotiation that header is suppressed.
*/
if(req_clen >= 0 && !data->req.upload_chunky &&
(data->req.authneg ||
!Curl_checkheaders(data, STRCONST("Content-Length")))) {
/* we allow replacing this header if not during auth negotiation,
although it isn't very wise to actually set your own */
result = Curl_dyn_addf(r,
"Content-Length: %" CURL_FORMAT_CURL_OFF_T
"\r\n", http->postsize);
if(result)
goto out;
"\r\n", req_clen);
}
if(result)
goto out;
#ifndef CURL_DISABLE_MIME
/* Output mime-generated headers. */
{
if(data->state.mimepost &&
((httpreq == HTTPREQ_POST_FORM) || (httpreq == HTTPREQ_POST_MIME))) {
struct curl_slist *hdr;
for(hdr = data->state.mimepost->curlheaders; hdr; hdr = hdr->next) {
@ -2236,93 +2296,26 @@ CURLcode Curl_http_req_complete(struct Curl_easy *data,
}
}
#endif
if(httpreq == HTTPREQ_POST) {
if(!Curl_checkheaders(data, STRCONST("Content-Type"))) {
result = Curl_dyn_addn(r, STRCONST("Content-Type: application/"
"x-www-form-urlencoded\r\n"));
if(result)
goto out;
}
}
result = addexpect(data, r);
if(result)
goto out;
/* make the request end in a true CRLF */
result = Curl_dyn_addn(r, STRCONST("\r\n"));
if(result)
goto out;
/* set the upload size to the progress meter */
Curl_pgrsSetUploadSize(data, http->postsize);
if(!http->postsize)
result = Client_reader_set_null(data);
else {
/* Read from mime structure. We could do a special client reader
* for this, but replacing the callback seems to work fine. */
data->state.fread_func = (curl_read_callback) Curl_mime_read;
data->state.in = (void *) data->state.mimepost;
result = Client_reader_set_fread(data, data->state.infilesize);
}
break;
#endif
case HTTPREQ_POST:
/* this is the simple POST, using x-www-form-urlencoded style */
if(data->req.authneg)
http->postsize = 0;
else
/* the size of the post body */
http->postsize = data->state.infilesize;
/* We only set Content-Length and allow a custom Content-Length if
we don't upload data chunked, as RFC2616 forbids us to set both
kinds of headers (Transfer-Encoding: chunked and Content-Length) */
if((http->postsize != -1) && !data->req.upload_chunky &&
(data->req.authneg ||
!Curl_checkheaders(data, STRCONST("Content-Length")))) {
/* we allow replacing this header if not during auth negotiation,
although it isn't very wise to actually set your own */
result = Curl_dyn_addf(r, "Content-Length: %" CURL_FORMAT_CURL_OFF_T
"\r\n", http->postsize);
if(result)
goto out;
}
if(!Curl_checkheaders(data, STRCONST("Content-Type"))) {
result = Curl_dyn_addn(r, STRCONST("Content-Type: application/"
"x-www-form-urlencoded\r\n"));
if(result)
goto out;
}
result = addexpect(data, r);
if(result)
goto out;
result = Curl_dyn_addn(r, STRCONST("\r\n"));
if(result)
goto out;
if(!http->postsize) {
Curl_pgrsSetUploadSize(data, 0);
result = Client_reader_set_null(data);
}
else if(data->set.postfields) {
Curl_pgrsSetUploadSize(data, http->postsize);
if(http->postsize > 0)
result = Client_reader_set_buf(data, data->set.postfields,
(size_t)http->postsize);
else
result = Client_reader_set_null(data);
}
else { /* we read the bytes from the callback */
Curl_pgrsSetUploadSize(data, http->postsize);
result = Client_reader_set_fread(data, http->postsize);
}
break;
default:
/* HTTP GET/HEAD download, has no body, needs no Content-Length */
result = Curl_dyn_addn(r, STRCONST("\r\n"));
if(!result)
result = Client_reader_set_null(data);
break;
}
/* end of headers */
result = Curl_dyn_addn(r, STRCONST("\r\n"));
Curl_pgrsSetUploadSize(data, req_clen);
out:
if(!result) {
/* setup variables for the upcoming transfer */
@ -2427,7 +2420,7 @@ CURLcode Curl_http_range(struct Curl_easy *data,
}
else if((httpreq == HTTPREQ_POST || httpreq == HTTPREQ_PUT) &&
!Curl_checkheaders(data, STRCONST("Content-Range"))) {
curl_off_t req_clen = Curl_creader_total_length(data);
/* if a line like this was already allocated, free the previous one */
free(data->state.aptr.rangeline);
@ -2438,25 +2431,28 @@ CURLcode Curl_http_range(struct Curl_easy *data,
data->state.aptr.rangeline =
aprintf("Content-Range: bytes 0-%" CURL_FORMAT_CURL_OFF_T
"/%" CURL_FORMAT_CURL_OFF_T "\r\n",
data->state.infilesize - 1, data->state.infilesize);
req_clen - 1, req_clen);
}
else if(data->state.resume_from) {
/* This is because "resume" was selected */
curl_off_t total_expected_size =
data->state.resume_from + data->state.infilesize;
/* TODO: not sure if we want to send this header during authentication
* negotiation, but test1084 checks for it. In which case we have a
* "null" client reader installed that gives an unexpected length. */
curl_off_t total_len = data->req.authneg?
data->state.infilesize :
(data->state.resume_from + req_clen);
data->state.aptr.rangeline =
aprintf("Content-Range: bytes %s%" CURL_FORMAT_CURL_OFF_T
"/%" CURL_FORMAT_CURL_OFF_T "\r\n",
data->state.range, total_expected_size-1,
total_expected_size);
data->state.range, total_len-1, total_len);
}
else {
/* Range was selected and then we just pass the incoming range and
append total size */
data->state.aptr.rangeline =
aprintf("Content-Range: bytes %s/%" CURL_FORMAT_CURL_OFF_T "\r\n",
data->state.range, data->state.infilesize);
data->state.range, req_clen);
}
if(!data->state.aptr.rangeline)
return CURLE_OUT_OF_MEMORY;
@ -2465,87 +2461,6 @@ CURLcode Curl_http_range(struct Curl_easy *data,
return CURLE_OK;
}
CURLcode Curl_http_resume(struct Curl_easy *data,
struct connectdata *conn,
Curl_HttpReq httpreq)
{
if((HTTPREQ_POST == httpreq || HTTPREQ_PUT == httpreq) &&
data->state.resume_from) {
/**********************************************************************
* Resuming upload in HTTP means that we PUT or POST and that we have
* got a resume_from value set. The resume value has already created
* a Range: header that will be passed along. We need to "fast forward"
* the file the given number of bytes and decrease the assume upload
* file size before we continue this venture in the dark lands of HTTP.
* Resuming mime/form posting at an offset > 0 has no sense and is ignored.
*********************************************************************/
if(data->state.resume_from < 0) {
/*
* This is meant to get the size of the present remote-file by itself.
* We don't support this now. Bail out!
*/
data->state.resume_from = 0;
}
if(data->state.resume_from && !data->state.followlocation) {
/* only act on the first request */
/* Now, let's read off the proper amount of bytes from the
input. */
int seekerr = CURL_SEEKFUNC_CANTSEEK;
if(conn->seek_func) {
Curl_set_in_callback(data, true);
seekerr = conn->seek_func(conn->seek_client, data->state.resume_from,
SEEK_SET);
Curl_set_in_callback(data, false);
}
if(seekerr != CURL_SEEKFUNC_OK) {
curl_off_t passed = 0;
if(seekerr != CURL_SEEKFUNC_CANTSEEK) {
failf(data, "Could not seek stream");
return CURLE_READ_ERROR;
}
/* when seekerr == CURL_SEEKFUNC_CANTSEEK (can't seek to offset) */
do {
char scratch[4*1024];
size_t readthisamountnow =
(data->state.resume_from - passed > (curl_off_t)sizeof(scratch)) ?
sizeof(scratch) :
curlx_sotouz(data->state.resume_from - passed);
size_t actuallyread =
data->state.fread_func(scratch, 1, readthisamountnow,
data->state.in);
passed += actuallyread;
if((actuallyread == 0) || (actuallyread > readthisamountnow)) {
/* this checks for greater-than only to make sure that the
CURL_READFUNC_ABORT return code still aborts */
failf(data, "Could only read %" CURL_FORMAT_CURL_OFF_T
" bytes from the input", passed);
return CURLE_READ_ERROR;
}
} while(passed < data->state.resume_from);
}
/* now, decrease the size of the read */
if(data->state.infilesize>0) {
data->state.infilesize -= data->state.resume_from;
if(data->state.infilesize <= 0) {
failf(data, "File already completely uploaded");
return CURLE_PARTIAL_FILE;
}
}
/* we've passed, proceed as normal */
}
}
return CURLE_OK;
}
CURLcode Curl_http_firstwrite(struct Curl_easy *data,
struct connectdata *conn,
bool *done)
@ -2754,17 +2669,13 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
goto fail;
#endif
result = Curl_http_body(data, conn, httpreq, &te);
result = Curl_http_req_set_reader(data, httpreq, &te);
if(result)
goto fail;
p_accept = Curl_checkheaders(data,
STRCONST("Accept"))?NULL:"Accept: */*\r\n";
result = Curl_http_resume(data, conn, httpreq);
if(result)
goto fail;
result = Curl_http_range(data, httpreq);
if(result)
goto fail;
@ -2882,14 +2793,8 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
result = Curl_add_custom_headers(data, FALSE, &req);
if(!result) {
if((httpreq == HTTPREQ_GET) ||
(httpreq == HTTPREQ_HEAD))
Curl_pgrsSetUploadSize(data, 0); /* nothing */
/* req_send takes ownership of the 'req' memory on success */
result = Curl_http_req_complete(data, &req, httpreq);
if(!result && data->req.upload_chunky)
result = Curl_httpchunk_add_reader(data);
if(!result)
result = Curl_req_send(data, &req);
}
@ -3736,7 +3641,7 @@ static CURLcode http_rw_headers(struct Curl_easy *data,
if(k->httpcode >= 300) {
if((!data->req.authneg) && !conn->bits.close &&
!data->state.rewindbeforesend) {
!Curl_creader_will_rewind(data)) {
/*
* General treatment of errors when about to send data. Including :
* "417 Expectation Failed", while waiting for 100-continue.
@ -3805,7 +3710,7 @@ static CURLcode http_rw_headers(struct Curl_easy *data,
}
}
if(data->state.rewindbeforesend && !Curl_req_done_sending(data)) {
if(Curl_creader_will_rewind(data) && !Curl_req_done_sending(data)) {
/* We rewind before next send, continue sending now */
infof(data, "Keep sending data to get tossed away");
k->keepon |= KEEP_SEND;

13
lib/http.h
View File

@ -105,9 +105,9 @@ CURLcode Curl_http_statusline(struct Curl_easy *data,
CURLcode Curl_http_header(struct Curl_easy *data, struct connectdata *conn,
char *headp);
CURLcode Curl_transferencode(struct Curl_easy *data);
CURLcode Curl_http_body(struct Curl_easy *data, struct connectdata *conn,
Curl_HttpReq httpreq,
const char **teep);
CURLcode Curl_http_req_set_reader(struct Curl_easy *data,
Curl_HttpReq httpreq,
const char **tep);
CURLcode Curl_http_req_complete(struct Curl_easy *data,
struct dynbuf *r, Curl_HttpReq httpreq);
bool Curl_use_http_1_1plus(const struct Curl_easy *data,
@ -119,9 +119,6 @@ CURLcode Curl_http_cookies(struct Curl_easy *data,
#else
#define Curl_http_cookies(a,b,c) CURLE_OK
#endif
CURLcode Curl_http_resume(struct Curl_easy *data,
struct connectdata *conn,
Curl_HttpReq httpreq);
CURLcode Curl_http_range(struct Curl_easy *data,
Curl_HttpReq httpreq);
CURLcode Curl_http_firstwrite(struct Curl_easy *data,
@ -188,11 +185,11 @@ CURLcode Curl_http_auth_act(struct Curl_easy *data);
* HTTP unique setup
***************************************************************************/
struct HTTP {
curl_off_t postsize; /* off_t to handle large file sizes */
#ifndef CURL_DISABLE_HTTP
void *h2_ctx; /* HTTP/2 implementation context */
void *h3_ctx; /* HTTP/3 implementation context */
#else
char unused;
#endif
};

12
lib/http_chunks.c
View File

@ -624,6 +624,15 @@ static CURLcode cr_chunked_read(struct Curl_easy *data,
return CURLE_OK;
}
static curl_off_t cr_chunked_total_length(struct Curl_easy *data,
struct Curl_creader *reader)
{
/* this reader changes length depending on input */
(void)data;
(void)reader;
return -1;
}
/* HTTP chunked Transfer-Encoding encoder */
const struct Curl_crtype Curl_httpchunk_encoder = {
"chunked",
@ -631,6 +640,9 @@ const struct Curl_crtype Curl_httpchunk_encoder = {
cr_chunked_read,
cr_chunked_close,
Curl_creader_def_needs_rewind,
cr_chunked_total_length,
Curl_creader_def_resume_from,
Curl_creader_def_rewind,
sizeof(struct chunked_reader)
};

90
lib/multi.c
View File

@ -1816,93 +1816,6 @@ static CURLcode protocol_connect(struct Curl_easy *data,
return result; /* pass back status */
}
/*
* readrewind() rewinds the read stream. This is typically used for HTTP
* POST/PUT with multi-pass authentication when a sending was denied and a
* resend is necessary.
*/
static CURLcode readrewind(struct Curl_easy *data)
{
#if !defined(CURL_DISABLE_MIME) || !defined(CURL_DISABLE_FORM_API)
curl_mimepart *mimepart = &data->set.mimepost;
#endif
DEBUGASSERT(data->conn);
data->state.rewindbeforesend = FALSE; /* we rewind now */
/* explicitly switch off sending data on this connection now since we are
about to restart a new transfer and thus we want to avoid inadvertently
sending more data on the existing connection until the next transfer
starts */
data->req.keepon &= ~KEEP_SEND;
/* We have sent away data. If not using CURLOPT_POSTFIELDS or
CURLOPT_HTTPPOST, call app to rewind
*/
#if !defined(CURL_DISABLE_HTTP) && !defined(CURL_DISABLE_MIME)
if(data->conn->handler->protocol & PROTO_FAMILY_HTTP) {
if(data->state.mimepost)
mimepart = data->state.mimepost;
}
#endif
if(data->set.postfields ||
(data->state.httpreq == HTTPREQ_GET) ||
(data->state.httpreq == HTTPREQ_HEAD))
; /* no need to rewind */
#if !defined(CURL_DISABLE_MIME) || !defined(CURL_DISABLE_FORM_API)
else if(data->state.httpreq == HTTPREQ_POST_MIME ||
data->state.httpreq == HTTPREQ_POST_FORM) {
CURLcode result = Curl_mime_rewind(mimepart);
if(result) {
failf(data, "Cannot rewind mime/post data");
return result;
}
}
#endif
else {
if(data->set.seek_func) {
int err;
Curl_set_in_callback(data, true);
err = (data->set.seek_func)(data->set.seek_client, 0, SEEK_SET);
Curl_set_in_callback(data, false);
if(err) {
failf(data, "seek callback returned error %d", (int)err);
return CURLE_SEND_FAIL_REWIND;
}
}
else if(data->set.ioctl_func) {
curlioerr err;
Curl_set_in_callback(data, true);
err = (data->set.ioctl_func)(data, CURLIOCMD_RESTARTREAD,
data->set.ioctl_client);
Curl_set_in_callback(data, false);
infof(data, "the ioctl callback returned %d", (int)err);
if(err) {
failf(data, "ioctl callback returned error %d", (int)err);
return CURLE_SEND_FAIL_REWIND;
}
}
else {
/* If no CURLOPT_READFUNCTION is used, we know that we operate on a
given FILE * stream and we can actually attempt to rewind that
ourselves with fseek() */
if(data->state.fread_func == (curl_read_callback)fread) {
if(-1 != fseek(data->state.in, 0, SEEK_SET))
/* successful rewind */
return CURLE_OK;
}
/* no callback set or failure above, makes us fail at once */
failf(data, "necessary data rewind wasn't possible");
return CURLE_SEND_FAIL_REWIND;
}
}
return CURLE_OK;
}
/*
* Curl_preconnect() is called immediately before a connect starts. When a
* redirect is followed, this is then called multiple times during a single
@ -2169,9 +2082,6 @@ static CURLMcode multi_runsingle(struct Curl_multi *multi,
break;
case MSTATE_PROTOCONNECT:
if(data->state.rewindbeforesend)
result = readrewind(data);
if(!result && data->conn->bits.reuse) {
/* ftp seems to hang when protoconnect on reused connection
* since we handle PROTOCONNECT in general inside the filers, it

64
lib/request.c
View File

@ -49,8 +49,13 @@ CURLcode Curl_req_init(struct SingleRequest *req)
CURLcode Curl_req_start(struct SingleRequest *req,
struct Curl_easy *data)
{
CURLcode result;
req->start = Curl_now();
Curl_client_reset(data);
result = Curl_client_start(data);
if(result)
return result;
if(!req->sendbuf_init) {
Curl_bufq_init2(&req->sendbuf, data->set.upload_buffer_size, 1,
BUFQ_OPT_SOFT_LIMIT);
@ -82,14 +87,15 @@ CURLcode Curl_req_done(struct SingleRequest *req,
void Curl_req_reset(struct SingleRequest *req, struct Curl_easy *data)
{
struct bufq savebuf;
bool save_init;
struct curltime t0 = {0, 0};
/* This is a bit ugly. `req->p` is a union and we assume we can
* free this safely without leaks. */
Curl_safefree(req->p.http);
Curl_safefree(req->newurl);
Curl_client_reset(data);
if(req->sendbuf_init)
Curl_bufq_reset(&req->sendbuf);
#ifndef CURL_DISABLE_DOH
if(req->doh) {
@ -97,17 +103,45 @@ void Curl_req_reset(struct SingleRequest *req, struct Curl_easy *data)
Curl_close(&req->doh->probe[1].easy);
}
#endif
savebuf = req->sendbuf;
save_init = req->sendbuf_init;
memset(req, 0, sizeof(*req));
data->req.size = data->req.maxdownload = -1;
data->req.no_body = data->set.opt_no_body;
if(save_init) {
req->sendbuf = savebuf;
req->sendbuf_init = save_init;
}
/* Can no longer memset() this struct as we need to keep some state */
req->size = -1;
req->maxdownload = -1;
req->bytecount = 0;
req->writebytecount = 0;
req->start = t0;
req->headerbytecount = 0;
req->allheadercount = 0;
req->deductheadercount = 0;
req->headerline = 0;
req->offset = 0;
req->httpcode = 0;
req->keepon = 0;
req->start100 = t0;
req->exp100 = EXP100_SEND_DATA;
req->upgr101 = UPGR101_INIT;
req->timeofdoc = 0;
req->bodywrites = 0;
req->location = NULL;
req->newurl = NULL;
#ifndef CURL_DISABLE_COOKIES
req->setcookies = 0;
#endif
req->header = FALSE;
req->content_range = FALSE;
req->download_done = FALSE;
req->eos_written = FALSE;
req->eos_read = FALSE;
req->upload_done = FALSE;
req->upload_aborted = FALSE;
req->ignorebody = FALSE;
req->http_bodyless = FALSE;
req->chunk = FALSE;
req->ignore_cl = FALSE;
req->upload_chunky = FALSE;
req->getheader = FALSE;
req->forbidchunk = FALSE;
req->no_body = data->set.opt_no_body;
req->authneg = FALSE;
}
void Curl_req_free(struct SingleRequest *req, struct Curl_easy *data)
@ -118,7 +152,7 @@ void Curl_req_free(struct SingleRequest *req, struct Curl_easy *data)
Curl_safefree(req->newurl);
if(req->sendbuf_init)
Curl_bufq_free(&req->sendbuf);
Curl_client_reset(data);
Curl_client_cleanup(data);
#ifndef CURL_DISABLE_DOH
if(req->doh) {

3
lib/request.h
View File

@ -119,8 +119,6 @@ struct SingleRequest {
#ifndef CURL_DISABLE_DOH
struct dohdata *doh; /* DoH specific data for this request */
#endif
char fread_eof[2]; /* the body read callback (index 0) returned EOF or
the trailer read callback (index 1) returned EOF */
#ifndef CURL_DISABLE_COOKIES
unsigned char setcookies;
#endif
@ -129,6 +127,7 @@ struct SingleRequest {
BIT(download_done); /* set to TRUE when download is complete */
BIT(eos_written); /* iff EOS has been written to client */
BIT(eos_read); /* iff EOS has been read from the client */
BIT(rewind_read); /* iff reader needs rewind at next start */
BIT(upload_done); /* set to TRUE when all request data has been sent */
BIT(upload_aborted); /* set to TRUE when upload was aborted. Will also
* show `upload_done` as TRUE. */

37
lib/rtsp.c
View File

@ -225,8 +225,6 @@ static CURLcode rtsp_do(struct Curl_easy *data, bool *done)
Curl_RtspReq rtspreq = data->set.rtspreq;
struct RTSP *rtsp = data->req.p.rtsp;
struct dynbuf req_buffer;
curl_off_t postsize = 0; /* for ANNOUNCE and SET_PARAMETER */
curl_off_t putsize = 0; /* for ANNOUNCE and SET_PARAMETER */
const char *p_request = NULL;
const char *p_session_id = NULL;
@ -499,38 +497,41 @@ static CURLcode rtsp_do(struct Curl_easy *data, bool *done)
if(rtspreq == RTSPREQ_ANNOUNCE ||
rtspreq == RTSPREQ_SET_PARAMETER ||
rtspreq == RTSPREQ_GET_PARAMETER) {
curl_off_t req_clen; /* request content length */
if(data->state.upload) {
putsize = data->state.infilesize;
req_clen = data->state.infilesize;
data->state.httpreq = HTTPREQ_PUT;
result = Client_reader_set_fread(data, putsize);
result = Curl_creader_set_fread(data, req_clen);
if(result)
goto out;
}
else {
postsize = (data->state.infilesize != -1)?
data->state.infilesize:
(data->set.postfields? (curl_off_t)strlen(data->set.postfields):0);
data->state.httpreq = HTTPREQ_POST;
if(postsize > 0 && data->set.postfields)
result = Client_reader_set_buf(data, data->set.postfields,
(size_t)postsize);
else if(!postsize)
result = Client_reader_set_null(data);
else
result = Client_reader_set_fread(data, postsize);
if(data->set.postfields) {
size_t plen = strlen(data->set.postfields);
req_clen = (curl_off_t)plen;
result = Curl_creader_set_buf(data, data->set.postfields, plen);
}
else if(data->state.infilesize >= 0) {
req_clen = data->state.infilesize;
result = Curl_creader_set_fread(data, req_clen);
}
else {
req_clen = 0;
result = Curl_creader_set_null(data);
}
if(result)
goto out;
}
if(putsize > 0 || postsize > 0) {
if(req_clen > 0) {
/* As stated in the http comments, it is probably not wise to
* actually set a custom Content-Length in the headers */
if(!Curl_checkheaders(data, STRCONST("Content-Length"))) {
result =
Curl_dyn_addf(&req_buffer,
"Content-Length: %" CURL_FORMAT_CURL_OFF_T"\r\n",
(data->state.upload ? putsize : postsize));
req_clen);
if(result)
goto out;
}
@ -565,7 +566,7 @@ static CURLcode rtsp_do(struct Curl_easy *data, bool *done)
}
}
else {
result = Client_reader_set_null(data);
result = Curl_creader_set_null(data);
if(result)
goto out;
}

348
lib/sendf.c
View File

@ -51,6 +51,7 @@
#include "strdup.h"
#include "http2.h"
#include "progress.h"
#include "warnless.h"
#include "ws.h"
/* The last 3 #include files should be in this order */
@ -112,9 +113,9 @@ static void cl_reset_reader(struct Curl_easy *data)
}
}
void Curl_client_reset(struct Curl_easy *data)
void Curl_client_cleanup(struct Curl_easy *data)
{
DEBUGF(infof(data, "Curl_client_reset()"));
DEBUGF(infof(data, "Curl_client_cleanup()"));
cl_reset_reader(data);
cl_reset_writer(data);
@ -122,6 +123,54 @@ void Curl_client_reset(struct Curl_easy *data)
data->req.headerline = 0;
}
void Curl_client_reset(struct Curl_easy *data)
{
if(data->req.rewind_read) {
/* already requested */
DEBUGF(infof(data, "Curl_client_reset(), will rewind_read"));
}
else {
DEBUGF(infof(data, "Curl_client_reset(), clear readers"));
cl_reset_reader(data);
}
cl_reset_writer(data);
data->req.bytecount = 0;
data->req.headerline = 0;
}
CURLcode Curl_client_start(struct Curl_easy *data)
{
if(data->req.rewind_read) {
struct Curl_creader *r = data->req.reader_stack;
CURLcode result = CURLE_OK;
DEBUGF(infof(data, "client start, rewind readers"));
while(r) {
result = r->crt->rewind(data, r);
if(result) {
failf(data, "rewind of client reader '%s' failed: %d",
r->crt->name, result);
return result;
}
r = r->next;
}
data->req.rewind_read = FALSE;
cl_reset_reader(data);
}
return CURLE_OK;
}
bool Curl_creader_will_rewind(struct Curl_easy *data)
{
return data->req.rewind_read;
}
void Curl_creader_set_rewind(struct Curl_easy *data, bool enable)
{
data->req.rewind_read = !!enable;
}
/* Write data using an unencoding writer stack. "nbytes" is not
allowed to be 0. */
CURLcode Curl_cwriter_write(struct Curl_easy *data,
@ -475,8 +524,35 @@ bool Curl_creader_def_needs_rewind(struct Curl_easy *data,
return FALSE;
}
curl_off_t Curl_creader_def_total_length(struct Curl_easy *data,
struct Curl_creader *reader)
{
return reader->next?
reader->next->crt->total_length(data, reader->next) : -1;
}
CURLcode Curl_creader_def_resume_from(struct Curl_easy *data,
struct Curl_creader *reader,
curl_off_t offset)
{
(void)data;
(void)reader;
(void)offset;
return CURLE_READ_ERROR;
}
CURLcode Curl_creader_def_rewind(struct Curl_easy *data,
struct Curl_creader *reader)
{
(void)data;
(void)reader;
return CURLE_OK;
}
struct cr_in_ctx {
struct Curl_creader super;
curl_read_callback read_cb;
void *cb_user_data;
curl_off_t total_len;
curl_off_t read_len;
CURLcode error_result;
@ -489,6 +565,8 @@ static CURLcode cr_in_init(struct Curl_easy *data, struct Curl_creader *reader)
{
struct cr_in_ctx *ctx = (struct cr_in_ctx *)reader;
(void)data;
ctx->read_cb = data->state.fread_func;
ctx->cb_user_data = data->state.in;
ctx->total_len = -1;
ctx->read_len = 0;
return CURLE_OK;
@ -523,9 +601,9 @@ static CURLcode cr_in_read(struct Curl_easy *data,
blen = (size_t)remain;
}
nread = 0;
if(data->state.fread_func && blen) {
if(ctx->read_cb && blen) {
Curl_set_in_callback(data, true);
nread = data->state.fread_func(buf, 1, blen, data->state.in);
nread = ctx->read_cb(buf, 1, blen, ctx->cb_user_data);
Curl_set_in_callback(data, false);
ctx->has_used_cb = TRUE;
}
@ -596,12 +674,167 @@ static bool cr_in_needs_rewind(struct Curl_easy *data,
return ctx->has_used_cb;
}
static curl_off_t cr_in_total_length(struct Curl_easy *data,
struct Curl_creader *reader)
{
struct cr_in_ctx *ctx = (struct cr_in_ctx *)reader;
(void)data;
return ctx->total_len;
}
static CURLcode cr_in_resume_from(struct Curl_easy *data,
struct Curl_creader *reader,
curl_off_t offset)
{
struct cr_in_ctx *ctx = (struct cr_in_ctx *)reader;
int seekerr = CURL_SEEKFUNC_CANTSEEK;
DEBUGASSERT(data->conn);
/* already started reading? */
if(ctx->read_len)
return CURLE_READ_ERROR;
if(data->conn->seek_func) {
Curl_set_in_callback(data, true);
seekerr = data->conn->seek_func(data->conn->seek_client, offset, SEEK_SET);
Curl_set_in_callback(data, false);
}
if(seekerr != CURL_SEEKFUNC_OK) {
curl_off_t passed = 0;
if(seekerr != CURL_SEEKFUNC_CANTSEEK) {
failf(data, "Could not seek stream");
return CURLE_READ_ERROR;
}
/* when seekerr == CURL_SEEKFUNC_CANTSEEK (can't seek to offset) */
do {
char scratch[4*1024];
size_t readthisamountnow =
(offset - passed > (curl_off_t)sizeof(scratch)) ?
sizeof(scratch) :
curlx_sotouz(offset - passed);
size_t actuallyread;
Curl_set_in_callback(data, true);
actuallyread = ctx->read_cb(scratch, 1, readthisamountnow,
ctx->cb_user_data);
Curl_set_in_callback(data, false);
passed += actuallyread;
if((actuallyread == 0) || (actuallyread > readthisamountnow)) {
/* this checks for greater-than only to make sure that the
CURL_READFUNC_ABORT return code still aborts */
failf(data, "Could only read %" CURL_FORMAT_CURL_OFF_T
" bytes from the input", passed);
return CURLE_READ_ERROR;
}
} while(passed < offset);
}
/* now, decrease the size of the read */
if(ctx->total_len > 0) {
ctx->total_len -= offset;
if(ctx->total_len <= 0) {
failf(data, "File already completely uploaded");
return CURLE_PARTIAL_FILE;
}
}
/* we've passed, proceed as normal */
return CURLE_OK;
}
static CURLcode cr_in_rewind(struct Curl_easy *data,
struct Curl_creader *reader)
{
struct cr_in_ctx *ctx = (struct cr_in_ctx *)reader;
/* TODO: I wonder if we should rather give mime its own client
* reader type. This is messy. */
#if !defined(CURL_DISABLE_MIME) || !defined(CURL_DISABLE_FORM_API)
curl_mimepart *mimepart = &data->set.mimepost;
#endif
/* If we never invoked the callback, there is noting to rewind */
if(!ctx->has_used_cb)
return CURLE_OK;
/* We have sent away data. If not using CURLOPT_POSTFIELDS or
CURLOPT_HTTPPOST, call app to rewind
*/
#if !defined(CURL_DISABLE_HTTP) && !defined(CURL_DISABLE_MIME)
if(data->conn->handler->protocol & PROTO_FAMILY_HTTP) {
if(data->state.mimepost)
mimepart = data->state.mimepost;
}
#endif
#if !defined(CURL_DISABLE_MIME) || !defined(CURL_DISABLE_FORM_API)
if(ctx->read_cb == (curl_read_callback)Curl_mime_read) {
CURLcode result = Curl_mime_rewind(mimepart);
DEBUGF(infof(data, "cr_in, rewind mime/post data -> %d", result));
if(result) {
failf(data, "Cannot rewind mime/post data");
}
return result;
}
#endif
/* With mime out of the way, handle "normal" fread callbacks */
if(data->set.seek_func) {
int err;
Curl_set_in_callback(data, true);
err = (data->set.seek_func)(data->set.seek_client, 0, SEEK_SET);
Curl_set_in_callback(data, false);
DEBUGF(infof(data, "cr_in, rewind via set.seek_func -> %d", err));
if(err) {
failf(data, "seek callback returned error %d", (int)err);
return CURLE_SEND_FAIL_REWIND;
}
}
else if(data->set.ioctl_func) {
curlioerr err;
Curl_set_in_callback(data, true);
err = (data->set.ioctl_func)(data, CURLIOCMD_RESTARTREAD,
data->set.ioctl_client);
Curl_set_in_callback(data, false);
DEBUGF(infof(data, "cr_in, rewind via set.ioctl_func -> %d", (int)err));
if(err) {
failf(data, "ioctl callback returned error %d", (int)err);
return CURLE_SEND_FAIL_REWIND;
}
}
else {
/* If no CURLOPT_READFUNCTION is used, we know that we operate on a
given FILE * stream and we can actually attempt to rewind that
ourselves with fseek() */
if(data->state.fread_func == (curl_read_callback)fread) {
int err = fseek(data->state.in, 0, SEEK_SET);
DEBUGF(infof(data, "cr_in, rewind via fseek -> %d(%d)",
(int)err, (int)errno));
if(-1 != err)
/* successful rewind */
return CURLE_OK;
}
/* no callback set or failure above, makes us fail at once */
failf(data, "necessary data rewind wasn't possible");
return CURLE_SEND_FAIL_REWIND;
}
return CURLE_OK;
}
static const struct Curl_crtype cr_in = {
"cr-in",
cr_in_init,
cr_in_read,
Curl_creader_def_close,
cr_in_needs_rewind,
cr_in_total_length,
cr_in_resume_from,
cr_in_rewind,
sizeof(struct cr_in_ctx)
};
@ -730,12 +963,24 @@ static CURLcode cr_lc_read(struct Curl_easy *data,
return result;
}
static curl_off_t cr_lc_total_length(struct Curl_easy *data,
struct Curl_creader *reader)
{
/* this reader changes length depending on input */
(void)data;
(void)reader;
return -1;
}
static const struct Curl_crtype cr_lc = {
"cr-lineconv",
cr_lc_init,
cr_lc_read,
cr_lc_close,
Curl_creader_def_needs_rewind,
cr_lc_total_length,
Curl_creader_def_resume_from,
Curl_creader_def_rewind,
sizeof(struct cr_lc_ctx)
};
@ -756,7 +1001,8 @@ static CURLcode cr_lc_add(struct Curl_easy *data)
static CURLcode do_init_reader_stack(struct Curl_easy *data,
const struct Curl_crtype *crt,
struct Curl_creader **preader)
struct Curl_creader **preader,
curl_off_t clen)
{
CURLcode result;
@ -766,11 +1012,13 @@ static CURLcode do_init_reader_stack(struct Curl_easy *data,
return result;
data->req.reader_stack = *preader;
if(data->set.crlf
/* if we do not have 0 length init, and crlf conversion is wanted,
* add the reader for it */
if(clen && (data->set.crlf
#ifdef CURL_DO_LINEEND_CONV
|| data->state.prefer_ascii
#endif
) {
)) {
result = cr_lc_add(data);
if(result)
return result;
@ -779,13 +1027,13 @@ static CURLcode do_init_reader_stack(struct Curl_easy *data,
return result;
}
CURLcode Client_reader_set_fread(struct Curl_easy *data, curl_off_t len)
CURLcode Curl_creader_set_fread(struct Curl_easy *data, curl_off_t len)
{
CURLcode result;
struct Curl_creader *r;
cl_reset_reader(data);
result = do_init_reader_stack(data, &cr_in, &r);
result = do_init_reader_stack(data, &cr_in, &r, len);
if(!result && r) {
struct cr_in_ctx *ctx = (struct cr_in_ctx *)r;
DEBUGASSERT(r->crt == &cr_in);
@ -801,7 +1049,7 @@ CURLcode Curl_creader_add(struct Curl_easy *data,
struct Curl_creader **anchor = &data->req.reader_stack;
if(!*anchor) {
result = Client_reader_set_fread(data, data->state.infilesize);
result = Curl_creader_set_fread(data, data->state.infilesize);
if(result)
return result;
}
@ -826,7 +1074,7 @@ CURLcode Curl_client_read(struct Curl_easy *data, char *buf, size_t blen,
DEBUGASSERT(eos);
if(!data->req.reader_stack) {
result = Client_reader_set_fread(data, data->state.infilesize);
result = Curl_creader_set_fread(data, data->state.infilesize);
if(result)
return result;
DEBUGASSERT(data->req.reader_stack);
@ -837,7 +1085,7 @@ CURLcode Curl_client_read(struct Curl_easy *data, char *buf, size_t blen,
return result;
}
bool Curl_client_read_needs_rewind(struct Curl_easy *data)
bool Curl_creader_needs_rewind(struct Curl_easy *data)
{
struct Curl_creader *reader = data->req.reader_stack;
while(reader) {
@ -862,21 +1110,33 @@ static CURLcode cr_null_read(struct Curl_easy *data,
return CURLE_OK;
}
static curl_off_t cr_null_total_length(struct Curl_easy *data,
struct Curl_creader *reader)
{
/* this reader changes length depending on input */
(void)data;
(void)reader;
return 0;
}
static const struct Curl_crtype cr_null = {
"cr-null",
Curl_creader_def_init,
cr_null_read,
Curl_creader_def_close,
Curl_creader_def_needs_rewind,
cr_null_total_length,
Curl_creader_def_resume_from,
Curl_creader_def_rewind,
sizeof(struct Curl_creader)
};
CURLcode Client_reader_set_null(struct Curl_easy *data)
CURLcode Curl_creader_set_null(struct Curl_easy *data)
{
struct Curl_creader *r;
cl_reset_reader(data);
return do_init_reader_stack(data, &cr_null, &r);
return do_init_reader_stack(data, &cr_null, &r, 0);
}
struct cr_buf_ctx {
@ -918,23 +1178,57 @@ static bool cr_buf_needs_rewind(struct Curl_easy *data,
return ctx->index > 0;
}
static curl_off_t cr_buf_total_length(struct Curl_easy *data,
struct Curl_creader *reader)
{
struct cr_buf_ctx *ctx = (struct cr_buf_ctx *)reader;
(void)data;
return (curl_off_t)ctx->blen;
}
static CURLcode cr_buf_resume_from(struct Curl_easy *data,
struct Curl_creader *reader,
curl_off_t offset)
{
struct cr_buf_ctx *ctx = (struct cr_buf_ctx *)reader;
size_t boffset;
(void)data;
DEBUGASSERT(data->conn);
/* already started reading? */
if(ctx->index)
return CURLE_READ_ERROR;
if(offset <= 0)
return CURLE_OK;
boffset = (size_t)offset;
if(boffset > ctx->blen)
return CURLE_READ_ERROR;
ctx->buf += boffset;
ctx->blen -= boffset;
return CURLE_OK;
}
static const struct Curl_crtype cr_buf = {
"cr-buf",
Curl_creader_def_init,
cr_buf_read,
Curl_creader_def_close,
cr_buf_needs_rewind,
cr_buf_total_length,
cr_buf_resume_from,
Curl_creader_def_rewind,
sizeof(struct cr_buf_ctx)
};
CURLcode Client_reader_set_buf(struct Curl_easy *data,
CURLcode Curl_creader_set_buf(struct Curl_easy *data,
const char *buf, size_t blen)
{
CURLcode result;
struct Curl_creader *r;
cl_reset_reader(data);
result = do_init_reader_stack(data, &cr_buf, &r);
result = do_init_reader_stack(data, &cr_buf, &r, blen);
if(!result && r) {
struct cr_buf_ctx *ctx = (struct cr_buf_ctx *)r;
DEBUGASSERT(r->crt == &cr_buf);
@ -944,3 +1238,25 @@ CURLcode Client_reader_set_buf(struct Curl_easy *data,
}
return result;
}
curl_off_t Curl_creader_total_length(struct Curl_easy *data)
{
struct Curl_creader *r = data->req.reader_stack;
return r? r->crt->total_length(data, r) : -1;
}
curl_off_t Curl_creader_client_length(struct Curl_easy *data)
{
struct Curl_creader *r = data->req.reader_stack;
while(r && r->phase != CURL_CR_CLIENT)
r = r->next;
return r? r->crt->total_length(data, r) : -1;
}
CURLcode Curl_creader_resume_from(struct Curl_easy *data, curl_off_t offset)
{
struct Curl_creader *r = data->req.reader_stack;
while(r && r->phase != CURL_CR_CLIENT)
r = r->next;
return r? r->crt->resume_from(data, r, offset) : CURLE_READ_ERROR;
}

78
lib/sendf.h
View File

@ -61,8 +61,20 @@ CURLcode Curl_client_write(struct Curl_easy *data, int type, const char *ptr,
/**
* Free all resources related to client writing.
*/
void Curl_client_cleanup(struct Curl_easy *data);
/**
* Reset readers and writer chains, keep rewind information
* when necessary.
*/
void Curl_client_reset(struct Curl_easy *data);
/**
* A new request is starting, perform any ops like rewinding
* previous readers when needed.
*/
CURLcode Curl_client_start(struct Curl_easy *data);
/**
* Client Writers - a chain passing transfer BODY data to the client.
* Main application: HTTP and related protocols
@ -178,11 +190,16 @@ void Curl_cwriter_def_close(struct Curl_easy *data,
/* Client Reader Type, provides the implementation */
struct Curl_crtype {
const char *name; /* writer name. */
CURLcode (*do_init)(struct Curl_easy *data, struct Curl_creader *writer);
CURLcode (*do_init)(struct Curl_easy *data, struct Curl_creader *reader);
CURLcode (*do_read)(struct Curl_easy *data, struct Curl_creader *reader,
char *buf, size_t blen, size_t *nread, bool *eos);
void (*do_close)(struct Curl_easy *data, struct Curl_creader *reader);
bool (*needs_rewind)(struct Curl_easy *data, struct Curl_creader *reader);
curl_off_t (*total_length)(struct Curl_easy *data,
struct Curl_creader *reader);
CURLcode (*resume_from)(struct Curl_easy *data,
struct Curl_creader *reader, curl_off_t offset);
CURLcode (*rewind)(struct Curl_easy *data, struct Curl_creader *reader);
size_t creader_size; /* sizeof() allocated struct Curl_creader */
};
@ -212,6 +229,13 @@ void Curl_creader_def_close(struct Curl_easy *data,
struct Curl_creader *reader);
bool Curl_creader_def_needs_rewind(struct Curl_easy *data,
struct Curl_creader *reader);
curl_off_t Curl_creader_def_total_length(struct Curl_easy *data,
struct Curl_creader *reader);
CURLcode Curl_creader_def_resume_from(struct Curl_easy *data,
struct Curl_creader *reader,
curl_off_t offset);
CURLcode Curl_creader_def_rewind(struct Curl_easy *data,
struct Curl_creader *reader);
/**
* Convenience method for calling `reader->do_read()` that
@ -260,22 +284,64 @@ CURLcode Curl_client_read(struct Curl_easy *data, char *buf, size_t blen,
* TRUE iff client reader needs rewing before it can be used for
* a retry request.
*/
bool Curl_client_read_needs_rewind(struct Curl_easy *data);
bool Curl_creader_needs_rewind(struct Curl_easy *data);
/**
* TRUE iff client reader will rewind at next start
*/
bool Curl_creader_will_rewind(struct Curl_easy *data);
/**
* En-/disable rewind of client reader at next start.
*/
void Curl_creader_set_rewind(struct Curl_easy *data, bool enable);
/**
* Get the total length of bytes provided by the installed readers.
* This is independent of the amount already delivered and is calculated
* by all readers in the stack. If a reader like "chunked" or
* "crlf conversion" is installed, the returned length will be -1.
* @return -1 if length is indeterminate
*/
curl_off_t Curl_creader_total_length(struct Curl_easy *data);
/**
* Get the total length of bytes provided by the reader at phase
* CURL_CR_CLIENT. This may not match the amount of bytes read
* for a request, depending if other, encoding readers are also installed.
* However it allows for rough estimation of the overall length.
* @return -1 if length is indeterminate
*/
curl_off_t Curl_creader_client_length(struct Curl_easy *data);
/**
* Ask the installed reader at phase CURL_CR_CLIENT to start
* reading from the given offset. On success, this will reduce
* the `total_length()` by the amount.
* @param date the transfer to read client bytes for
* param offset the offset where to start reads from, negative
* values will be ignored.
* @return CURLE_OK if offset could be set
* CURLE_READ_ERROR if not supported by reader or seek/read failed
* of offset larger then total length
* CURLE_PARTIAL_FILE if offset led to 0 total length
*/
CURLcode Curl_creader_resume_from(struct Curl_easy *data, curl_off_t offset);
/**
* Set the client reader to provide 0 bytes, immediate EOS.
*/
CURLcode Client_reader_set_null(struct Curl_easy *data);
CURLcode Curl_creader_set_null(struct Curl_easy *data);
/**
* Set the client reader the reads from fread callback.
*/
CURLcode Client_reader_set_fread(struct Curl_easy *data, curl_off_t len);
CURLcode Curl_creader_set_fread(struct Curl_easy *data, curl_off_t len);
/**
* Set the client reader the reads from the supplied buf (NOT COPIED).
*/
CURLcode Client_reader_set_buf(struct Curl_easy *data,
const char *buf, size_t blen);
CURLcode Curl_creader_set_buf(struct Curl_easy *data,
const char *buf, size_t blen);
#endif /* HEADER_CURL_SENDF_H */

14
lib/smtp.c
View File

@ -748,7 +748,7 @@ static CURLcode smtp_perform_mail(struct Curl_easy *data)
}
/* Setup client reader for size and EOB conversion */
result = Client_reader_set_fread(data, data->state.infilesize);
result = Curl_creader_set_fread(data, data->state.infilesize);
if(result)
goto out;
/* Add the client reader doing STMP EOB escaping */
@ -1909,12 +1909,24 @@ static CURLcode cr_eob_read(struct Curl_easy *data,
return CURLE_OK;
}
static curl_off_t cr_eob_total_length(struct Curl_easy *data,
struct Curl_creader *reader)
{
/* this reader changes length depending on input */
(void)data;
(void)reader;
return -1;
}
static const struct Curl_crtype cr_eob = {
"cr-smtp-eob",
cr_eob_init,
cr_eob_read,
cr_eob_close,
Curl_creader_def_needs_rewind,
cr_eob_total_length,
Curl_creader_def_resume_from,
Curl_creader_def_rewind,
sizeof(struct cr_eob_ctx)
};

9
lib/transfer.c
View File

@ -982,6 +982,7 @@ CURLcode Curl_follow(struct Curl_easy *data,
&& !(data->set.keep_post & CURL_REDIR_POST_301)) {
infof(data, "Switch from POST to GET");
data->state.httpreq = HTTPREQ_GET;
Curl_creader_set_rewind(data, FALSE);
}
break;
case 302: /* Found */
@ -1007,6 +1008,7 @@ CURLcode Curl_follow(struct Curl_easy *data,
&& !(data->set.keep_post & CURL_REDIR_POST_302)) {
infof(data, "Switch from POST to GET");
data->state.httpreq = HTTPREQ_GET;
Curl_creader_set_rewind(data, FALSE);
}
break;
@ -1109,12 +1111,7 @@ CURLcode Curl_retry_request(struct Curl_easy *data, char **url)
prevent i.e HTTP transfers to return
error just because nothing has been
transferred! */
if(Curl_client_read_needs_rewind(data)) {
data->state.rewindbeforesend = TRUE;
infof(data, "state.rewindbeforesend = TRUE");
}
Curl_creader_set_rewind(data, TRUE);
}
return CURLE_OK;
}

3
lib/urldata.h
View File

@ -1387,9 +1387,6 @@ struct UrlState {
BIT(url_alloc); /* URL string is malloc()'ed */
BIT(referer_alloc); /* referer string is malloc()ed */
BIT(wildcard_resolve); /* Set to true if any resolve change is a wildcard */
BIT(rewindbeforesend);/* TRUE when the sending couldn't be stopped even
though it will be discarded. We must call the data
rewind callback before trying to send again. */
BIT(upload); /* upload request */
BIT(internal); /* internal: true if this easy handle was created for
internal use and the user does not have ownership of the

3
scripts/singleuse.pl
View File

@ -46,7 +46,8 @@ my %wl = (
'Curl_xfer_write_resp' => 'internal api',
'Curl_creader_def_init' => 'internal api',
'Curl_creader_def_close' => 'internal api',
);
'Curl_creader_def_total_length' => 'internal api',
);
my %api = (
'curl_easy_cleanup' => 'API',