curl_ws_recv() now receives data to fill up the provided buffer, but can
return a partial fragment. The function now also get a pointer to a
curl_ws_frame struct with metadata that also mentions the offset and
total size of the fragment (of which you might be receiving a smaller
piece). This way, large incoming fragments will be "streamed" to the
application. When the curl_ws_frame struct field 'bytesleft' is 0, the
final fragment piece has been delivered.
curl_ws_recv() was also adjusted to work with a buffer size smaller than
the fragment size. (Possibly needless to say as the fragment size can
now be 63 bit large).
curl_ws_send() now supports sending a piece of a fragment, in a
streaming manner, in addition to sending the entire fragment in a single
call if it is small enough. To send a huge fragment, curl_ws_send() can
be used to send it in many small calls by first telling libcurl about
the total expected fragment size, and then send the payload in N number
of separate invokes and libcurl will stream those over the wire.
The struct curl_ws_meta() returns is now called 'curl_ws_frame' and it
has been extended with two new fields: *offset* and *bytesleft*. To help
describe the passed on data chunk when a fragment is delivered in many
smaller pieces.
The documentation has been updated accordingly.
Closes#9636
Example programs targeting a deprecated feature/option are commented with
a warning about it.
Other examples are adapted to not use deprecated options.
Closes#9661
Detecting headers and lib separately makes sense when headers come in
variations or with extra ones, but this wasn't the case here. These were
duplicate/parallel macros that we had to keep in sync with each other
for a working build. This patch leaves a single macro for each of these
dependencies:
- Rely on `HAVE_LIBZ`, delete parallel `HAVE_ZLIB_H`.
Also delete CMake logic making sure these two were in sync, along with
a toggle to turn off that logic, called `CURL_SPECIAL_LIBZ`.
Also delete stray `HAVE_ZLIB` defines.
There is also a `USE_ZLIB` variant in `lib/config-dos.h`. This patch
retains it for compatibility and deprecates it.
- Rely on `USE_LIBSSH2`, delete parallel `HAVE_LIBSSH2_H`.
Also delete `LIBSSH2_WIN32`, `LIBSSH2_LIBRARY` from
`winbuild/MakefileBuild.vc`, these have a role when building libssh2
itself. And `CURL_USE_LIBSSH`, which had no use at all.
Also delete stray `HAVE_LIBSSH2` defines.
- Rely on `USE_LIBSSH`, delete parallel `HAVE_LIBSSH_LIBSSH_H`.
Also delete `LIBSSH_WIN32`, `LIBSSH_LIBRARY` and `HAVE_LIBSSH` from
`winbuild/MakefileBuild.vc`, these were the result of copy-pasting the
libssh2 line, and were not having any use.
- Delete unused `HAVE_LIBPSL_H` and `HAVE_LIBPSL`.
Reviewed-by: Daniel Stenberg
Closes#9652
The former way that also suggested using a non-existing file to just
enable the cookie engine could lead to developers maybe a bit carelessly
guessing a file name that will not exist, and then in a future due to
circumstances, such a file could be made to exist and then accidentally
libcurl would read cookies not actually meant to.
Reported-by: Trail of bits
Closes#9654
The ci-test is the normal makefile target invoked in CI jobs. This has
been using the -r option to runtests.pl since a long time, but I find
that it mostly just adds many lines to the test output report without
anyone caring much about those stats.
Remove it.
Closes#9656
This is a bit shorter and a lot safer.
Substrings of unescaped characters are added by a single call to reduce
overhead.
Extend test 1465 to handle more kind of escapes.
Closes#9653
C string hexadecimal-escaped characters may have more than 2 digits.
This results in a wrong C compiler interpretation of a 2-digit escaped
character when followed by an hex digit character.
The solution retained here is to represent such characters as 3-digit
octal escapes.
Adjust and extend test 1465 for this case.
Closes#9643
Exclude linker flags specifying depedency libs and libpaths, when
building against `libcurl.dll`. In such case these options are not
necessary (but may cause errors if not/wrongly configured.)
Also move and reword a comment on `CPPFLAGS` to not apply to
`UNICODE` options. These are necessary for all build targets.
Closes#9651
- Don't show TESTFAIL message (ie tests failed which aren't ignored) if
only ignored tests failed.
Before:
IGNORED: failed tests: 571 612 1056
TESTDONE: 1214 tests out of 1217 reported OK: 99%
Use of uninitialized value $failed in concatenation (.) or string at
./runtests.pl line 6290.
TESTFAIL: These test cases failed:
After:
IGNORED: failed tests: 571 612 1056
TESTDONE: 1214 tests out of 1217 reported OK: 99%
Closes https://github.com/curl/curl/pull/9648
- Correct the use of -all-static for static Windows CI builds.
curl_LDFLAGS was removed from the makefile when metalink support was
removed. LDFLAGS=-all-static is passed to make only, because it is not a
valid option for configure compilation tests.
Closes https://github.com/curl/curl/pull/9633
In a recent commit I mistakenly deleted this logic, after seeing a
reference to a filename ending with `.cvs` and thinking it must have
been long gone. Turns out this is an existing file. Restore the rule
and the necessary `COPY` definitions with it.
The restored logic is required for a successful build on a bare source
tree (as opposed to a source release tarball).
Also shorten an existing condition similar to the one added in this
patch.
Regression since 07a0047882Closes#9645
After this patch, we reduce the three copies of most `Makefile.m32`
logic to one. This now resides in `lib/Makefile.m32`. It makes future
updates easier, the code shorter, with a small amount of added
complexity.
`Makefile.m32` reduction:
| | bytes | LOC total | blank | comment | code |
|-------------------|-------:|----------:|-------:|---------:|------:|
| 7.85.0 | 34772 | 1337 | 79 | 192 | 1066 |
| before this patch | 17601 | 625 | 62 | 106 | 457 |
| after this patch | 11680 | 392 | 52 | 104 | 236 |
Details:
- Change rules to create objects for the `v*` subdirs in the `lib` dir.
This allows to use a shared compile rule and assumes that filenames
are not (and will not be) colliding across these directories.
`Makefile.m32` now also stores a list of these subdirs. They are
changing rarely though.
- Sync as much as possible between the three `Makefile.m32` scripts'
rules and their source/target sections.
- After this patch `CPPFLAGS` are all applied to the `src` sources once
again. This matches the behaviour of cmake/autotools. Only zlib ones
are actually required there.
- Use `.rc` names from `Makefile.inc` instead of keeping a duplicate.
- Change examples to link `libcurl.dll` by default. This makes building
trivial, even as a cross-build:
`CC=x86_64-w64-mingw32-gcc make -f Makefile.m32`
To run them, you need to move/copy or add-to-path `libcurl.dll`.
You can select static mode via `CFG=-static`.
- List more of the `Makefile.m32` config variables.
- Drop `.rc` support from examples. It made it fragile without much
benefit.
- Include a necessary system lib for the `externalsocket.c` example.
- Exclude unnecessary systems libs when building in `-dyn` mode.
Closes#9642
- Use brackets for the IPv6 address shown in verbose message when the
format is address:port so that it is less confusing.
Before: Trying 2606:4700:4700::1111:443...
After: Trying [2606:4700:4700::1111]:443...
Bug: https://curl.se/mail/archive-2022-02/0041.html
Reported-by: David Hu
Closes#9635
This patch overhauls `Makefile.m32` scripts, fixing a list of quirks,
making its behaviour and customization envvars align better with other
build systems, aiming for less code, that is easier to read, use and
maintain.
Details:
- Rename customization envvars:
`CURL_CC` -> `CC`
`CURL_RC` -> `RC`
`CURL_AR` -> `AR`
`CURL_LDFLAG_EXTRAS_DLL` -> `CURL_LDFLAGS_LIB`
`CURL_LDFLAG_EXTRAS_EXE` -> `CURL_LDFLAGS_BIN`
- Drop `CURL_STRIP` and `CURL_RANLIB`. These tools are no longer used.
- Accept `CFLAGS`, `CPPFLAGS`, `RCFLAGS`, `LDFLAGS` and `LIBS` envvars.
- Drop `CURL_CFLAG_EXTRAS`, `CURL_LDFLAG_EXTRAS`, `CURL_RCFLAG_EXTRAS` in
favor of the above.
- Do not automatically enable `zlib` with `libssh2`. `zlib` is optional
with `libssh2`.
- Omit unnecessary `CPPFLAGS` options when building `curl.exe` and
examples.
- Drop support for deprecated `-winssl` `CFG` option. Use `-schannel`
instead.
- Avoid late evaluation where not necessary (`=` -> `:=`).
- Drop support for `CURL_DLL_A_SUFFIX` to override the implib suffix.
Instead, use the standard naming scheme by default: `libcurl.dll.a`.
The toolchain recognizes the name, and selects it automatically when
asking for a `-shared` vs. `-static` build.
- Stop applying `strip` to `libcurl.a`. Follow-up from
16a58e9f93. There was no debug info to
strip since then.
- Stop setting `-O3`, `-W`, `-Wall` options. You can add these to
`CFLAGS` as desired.
- Always enable `-DCURL_DISABLE_OPENSSL_AUTO_LOAD_CONFIG` with OpenSSL,
to avoid that vulnerability on Windows.
- Add `-lbrotlicommon` to `LIBS` when using `brotli`.
- Do not enable `-nghttp3` without `-ngtcp2`.
- `-ssh2` and `-rtmp` options no longer try to auto-select a TLS-backend.
You need to set the backend explicitly. This scales better and avoids
issues with certain combinations (e.g. `libssh2` + `wolfssl` with no
`schannel`).
- Default to OpenSSL TLS-backend with `ngtcp2`. Possible to override via
`NGTCP2_LIBS`.
- Old, alternate method of enabling components (e.g. `SSH2=1`) no longer
supported.
- Delete `SPNEGO` references. They were no-ops.
- Drop support for Win9x environments.
- Allow setting `OPENSSL_LIBS` independently from `OPENSSL_LIBPATH`.
- Support autotools/CMake `libssh2` builds by default.
- Respect `CURL_DLL_SUFFIX` in `-dyn` mode when building `curl.exe` and
examples.
- Assume standard directory layout with `LIBCARES_PATH`. (Instead of the
long gone embedded one.)
- Stop static linking with c-ares by default. Add
`CPPFLAGS=-DCARES_STATICLIB` to enable it.
- Reorganize internal layout to avoid redundancy and emit clean diffs
between src/lib and example make files.
- Delete unused variables.
- Code cleanups/rework.
- Comment and indentation fixes.
Closes#9632
- Drop auto-detection of OpenSSL 1.0.2 and earlier. Now always defaulting
to OpenSSL 1.1.0 and later, LibreSSL and BoringSSL.
- Drop `Invalid path to OpenSSL package` detection. OpenSSL has been
using a standard file layout since 1.1.0, so this seems unnecessary
now.
- Drop special logic to enable Novell LDAP SDK support.
- Drop special logic to enable OpenLDAP LDAP SDK support. This seems
to be distinct from native OpenLDAP, with support implemented inside
`lib/ldap.c` (vs. `lib/openldap.c`) back when the latter did not exist
yet in curl.
- Add `-lwldap32` only if there is no other LDAP library (either native
OpenLDAP, or SDKs above) present.
- Update `doc/INSTALL.md` accordingly.
After this patch, it's necessary to make configration changes when using
OpenSSL 1.0.2 or earlier, or the two LDAP SDKs.
OpenSSL 1.0.2 and earlier:
```
export OPENSSL_INCLUDE = <path-to-openssl>/outinc
export OPENSSL_LIBPATH = <path-to-openssl>/out
export OPENSSL_LIBS = -lssl32 -leay32 -lgdi32
```
Novell LDAP SDK, previously enabled via `USE_LDAP_NOVELL=1`:
```
export CURL_CFLAG_EXTRAS = -I<path-to-sdk>/inc -DCURL_HAS_NOVELL_LDAPSDK
export CURL_LDFLAG_EXTRAS = -L<path-to-sdk>/lib/mscvc -lldapsdk -lldapssl -lldapx
```
OpenLDAP LDAP SDK, previously enabled via `USE_LDAP_OPENLDAP=1`:
```
export CURL_CFLAG_EXTRAS = -I<path-to-sdk>/include -DCURL_HAS_OPENLDAP_LDAPSDK
export CURL_LDFLAG_EXTRAS = -L<path-to-sdk>/lib -lldap -llber
```
I haven't tested these scenarios, and in general we recommend using
a recent OpenSSL release. Also, WinLDAP (the Windows default) and
OpenLDAP (via `-DUSE_OPENLDAP`) are the LDAP options actively worked on
in curl.
Closes#9631
SourceForge projects can now choose between two hostnames, with .io and
.net ending. Both support HTTPS by default now. Opening the other variant
will perm-redirected to the one chosen by the project.
The .io -> .net redirection is done insecurely.
Let's update the URLs to point to the current canonical endpoints to
avoid any redirects.
Closes#9630
- Add `-lcrypt32` once, and add it always for simplicity.
- Delete broken link and reference to the pre-Vista WinIDN add-on.
MS no longer distribute it.
- Delete related `WINIDN_PATH` option. IDN is a system lib since Vista.
- Sync `LIBCARES_PATH` default with the rest of dependencies.
- Delete version numbers from dependency path defaults.
- `libgsasl` package is now called `gsasl`.
- Delete `libexpat` and `libxml2` references. No longer used by curl.
- Delete `Edit the path below...` comments. We recommend to predefine
those envvars instead.
- `libcares.a` is not an internal dependency anymore. Stop using it as
such.
- `windres` `--include-dir` -> `-I`, `-F` -> `--target=` for readability.
- Delete `STRIP`, `CURL_STRIP`, `AR` references from `src/Makefile.m32`.
They were never used.
- Stop to `clean` some objects twice in `src/Makefile.m32`.
- Delete cvs-specific leftovers.
- Finish resource support in examples make file.
- Delete `-I<root>/lib` from examples make file.
- Fix copyright start year in examples make file.
- Delete duplicate `ftpuploadresume` input in examples make file.
- Sync OpenSSL lib order, `SYNC` support, `PROOT` use, dependency path
defaults, variables names and other internal bits between the three
make files.
- `lib/Makefile.m32` accepted custom options via `DLL_LIBS` envvar. This
was lib-specific and possibly accidental. Use `CURL_LDFLAG_EXTRAS_DLL`
envvar for the same effect.
- Fix linking `curl.exe` and examples to wrong static libs with
auto-detected OpenSSL 1.0.2 or earlier.
- Add `-lgdi32` for OpenSSL 1.0.2 and earlier only.
- Add link to Novell LDAP SDK and use a relative default path. Latest
version is from 2016, linked to an outdated OpenSSL 1.0.1.
- Whitespace and comment cleanups.
TODO in a next commit:
Delete built-in detection/logic for OpenSSL 1.0.2 and earlier, the Novell
LDAP SDK and the other LDAP SDK (which is _not_ OpenLDAP). Write up the
necessary custom envvars to configure them.
Closes#9616
Added in 68b215157f, while adding openldap
support. This is also the single mention of this constant in the source
tree and also in that commit. Based on these, it seems like an accident.
Delete this reference.
Reviewed-by: Daniel Stenberg
Closes#9625
The introduction of CURL_DISABLE_MIME came with some additional bugs:
- Disabled MIME is compiled-in anyway if SMTP and/or IMAP is enabled.
- CURLOPT_MIMEPOST, CURLOPT_MIME_OPTIONS and CURLOPT_HTTPHEADER are
conditioned on HTTP, although also needed for SMTP and IMAP MIME mail
uploads.
In addition, the CURLOPT_HTTPHEADER and --header documentation does not
mention their use for MIME mail.
This commit fixes the problems above.
Closes#9610
The existing code tried but did not properly reject alternative services
using negative or too large port numbers.
With this fix, the logic now also flushes the old entries immediately
before adding a new one, making a following header with an illegal entry
not flush the already stored entry.
Report from the ongoing source code audit by Trail of Bits.
Adjusted test 356 to verify.
Closes#9607
This header is for providing the argument types for recv() and send()
when built to not use a dedicated config-[platfor].h file.
Remove the slow brute-force checks from configure and cmake.
This change also removes the use of the types for select, as they were
not used in code.
Closes#9592
If curl_off_t turns out to be smaller than 8 bytes,
--with-n64-deprecated needs to be used to allow the build to
continue. This is to highlight the fact that support for such builds is
going away next year.
Also mentioned in DEPRECATED.md
Closes#9605
This function is currently located in the lib/http.c module and is
therefore disabled by the CURL_DISABLE_HTTP conditional token.
As it may be called by TLS backends, disabling HTTP results in an
undefined reference error at link time.
Move this function to vauth/vauth.c to always provide it and rename it
as Curl_auth_allowed_to_host() to respect the vauth module naming
convention.
Closes#9600
Curl_ossl_certchain() is now an exported function in lib/vtls/openssl.c that
can also be used from quiche.c and ngtcp2.c to get the cert chain for QUIC
connections as well.
The *certchain function was moved to the top of the file for this reason.
Reported-by: Eloy Degen
Fixes#9584Closes#9597
