To enhance test coverage for AES-GCM mode, we provided longer additional
testing patterns for AES-GCM testing.
Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>
Signed-off-by: Jerry Shih <jerry.shih@sifive.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21923)
To accelerate the performance of the AES-GCM mode, in this patch, we
have the specialized multi-block implementations for AES-128-GCM,
AES-192-GCM and AES-256-GCM.
Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>
Signed-off-by: Jerry Shih <jerry.shih@sifive.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21923)
c8ddeb7e64/doc/vector/riscv-crypto-vector-zvkb.adoc
Create `RISCV_HAS_ZVKB()` macro.
Use zvkb for SM4 instead of zvbb.
Use zvkb for ghash instead of zvbb.
We could just use the zvbb's subset `zvkb` for flexibility.
Signed-off-by: Jerry Shih <jerry.shih@sifive.com>
Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21923)
Changed "mutiple" to "multiple" for improved clarity and correctness.
Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21923)
The upcoming RISC-V vector crypto extensions feature
a Zvkg extension, that provides a vghmac.vv instruction.
This patch provides an implementation that utilizes this
extension if available.
Tested on QEMU and no regressions observed.
Signed-off-by: Christoph Müllner <christoph.muellner@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21923)
The RISC-V vector crypto extensions features a Zvbc extension
that provides a carryless multiplication ('vclmul.vv') instruction.
This patch provides an implementation that utilizes this
extension if available.
Tested on QEMU and no regressions observed.
Signed-off-by: Christoph Müllner <christoph.muellner@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21923)
Replace the random suffix with a counter, to make the
build reproducible.
Fixes#20954
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/22415)
The original text for the Apache + BSD dual licensing for riscv GCM and AES
perlasm was taken from other openSSL users like crypto/crypto/LPdir_unix.c .
Though Eric pointed out that the dual-licensing text could be read in a
way negating the second license [0] and suggested to clarify the text
even more.
So do this here for all of the GCM, AES and shared riscv.pm .
We already had the agreement of all involved developers for the actual
dual licensing in [0] and [1], so this is only a better clarification
for this.
[0] https://github.com/openssl/openssl/pull/20649#issuecomment-1589558790
[1] https://github.com/openssl/openssl/pull/21018
Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21357)
The implementation is not usable there at all.
Fixes#21301
Reviewed-by: Hugo Landau <hlandau@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21312)
(cherry picked from commit b256d32915)
Typos in doc/man* will be fixed in a different commit.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20910)
To allow re-use of the already reviewed openSSL crypto code for RISC-V in
other projects - like the Linux kernel, add a second license (2-clause BSD)
to the recently added GCM ghash functions.
Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Todd Short <todd.short@me.com>
(Merged from https://github.com/openssl/openssl/pull/20649)
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20519)
(cherry picked from commit 110dac5783)
RISC-V currently only offers a GMULT() callback for accelerated
processing. Let's implement the missing piece to have GHASH()
available as well. Like GMULT(), we provide a variant for
systems with the Zbkb extension (including brev8).
The integration follows the existing pattern for GMULT()
in RISC-V. We keep the C implementation as we need to decide
if we can call an optimized routine at run-time.
The C implementation is the fall-back in case we don't have
any extensions available that can be used to accelerate
the calculation.
Tested with all combinations of possible extensions
on QEMU (limiting the available instructions accordingly).
No regressions observed.
Signed-off-by: Christoph Müllner <christoph.muellner@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20078)
The existing GCM calculation provides some potential
for further optimizations. Let's use the demo code
from the RISC-V cryptography extension groups
(https://github.com/riscv/riscv-crypto), which represents
the extension architect's intended use of the clmul instruction.
The GCM calculation depends on bit and byte reversal.
Therefore, we use the corresponding instructions to do that
(if available at run-time).
The resulting computation becomes quite compact and passes
all tests.
Note, that a side-effect of this change is a reduced register
usage in .gmult(), which opens the door for an efficient .ghash()
implementation.
Signed-off-by: Christoph Müllner <christoph.muellner@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20078)
A recent commit introduced a Perl module for common code.
This patch changes the GCM code to use this module, removes duplicated code,
and moves the instruction encoding functions into the module.
Signed-off-by: Christoph Müllner <christoph.muellner@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20078)
In RISC-V we have multiple extensions, that can be
used to accelerate processing.
The known extensions are defined in riscv_arch.def.
From that file test functions of the following
form are generated: RISCV_HAS_$ext().
In recent commits new ways to define the availability
of these test macros have been defined. E.g.:
#define RV32I_ZKND_ZKNE_CAPABLE \
(RISCV_HAS_ZKND() && RISCV_HAS_ZKNE())
[...]
#define RV64I_ZKND_ZKNE_CAPABLE \
(RISCV_HAS_ZKND() && RISCV_HAS_ZKNE())
This leaves us with two different APIs to test capabilities.
Further, creating the same macros for RV32 and RV64 results
in duplicated code (see example above).
This inconsistent situation makes it hard to integrate
further code. So let's clean this up with the following steps:
* Replace RV32I_* and RV64I_* macros by RICSV_HAS_* macros
* Move all test macros into riscv_arch.h
* Use "AND" and "OR" to combine tests with more than one extension
* Rename include files for accelerated processing (remove extension
postfix).
We end up with compile time tests for RV32/RV64 and run-time tests
for available extensions. Adding new routines (e.g. for vector crypto
instructions) should be straightforward.
Testing showed no regressions.
Signed-off-by: Christoph Müllner <christoph.muellner@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20078)
These aren't currently checked when they are called in cipher_aes_gcm_hw_armv8.inc,
but they are declared as returning as size_t the number of bytes they have processed,
and the aes_gcm_*_*_kernel (unroll by 4) versions of these do return the correct
values.
Change-Id: Ic3eaf139e36e29e8779b5bd8b867c08fde37a337
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20191)
Signed-off-by: Xu Yizhou <xuyizhou1@huawei.com>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19619)
Since OPENSSL_malloc() and friends report ERR_R_MALLOC_FAILURE, and
at least handle the file name and line number they are called from,
there's no need to report ERR_R_MALLOC_FAILURE where they are called
directly, or when SSLfatal() and RLAYERfatal() is used, the reason
`ERR_R_MALLOC_FAILURE` is changed to `ERR_R_CRYPTO_LIB`.
There were a number of places where `ERR_R_MALLOC_FAILURE` was reported
even though it was a function from a different sub-system that was
called. Those places are changed to report ERR_R_{lib}_LIB, where
{lib} is the name of that sub-system.
Some of them are tricky to get right, as we have a lot of functions
that belong in the ASN1 sub-system, and all the `sk_` calls or from
the CRYPTO sub-system.
Some extra adaptation was necessary where there were custom OPENSSL_malloc()
wrappers, and some bugs are fixed alongside these changes.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19301)
s390x GHASH assembler implementation assumed it was called from a
gcm128_context structure where the Xi paramter to the ghash function was
embedded in that structure. Since the structure layout resembles the paramter
block required for kimd-GHASH, the assembler code simply assumed the 128 bytes
after Xi are the hash subkey.
This assumption was broken with the introduction of AES-GCM-SIV which uses the
GHASH implementation without a gcm128_context structure. Furthermore, the
bytes following the Xi input parameter to the GHASH function do not contain
the hash subkey. To fix this, we remove the assumption about the calling
context and build the parameter block on the stack. This requires some
copying of data to and from the stack. While this introduces a performance
degradation, new systems anyway use kma for GHASH/AES-GCM.
Finally fixes#18693 for s390x.
Signed-off-by: Juergen Christ <jchrist@linux.ibm.com>
Reviewed-by: Todd Short <todd.short@me.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18939)
Fixes#16721
This uses AES-ECB to create a counter mode AES-CTR32 (32bit counter, I could
not get AES-CTR to work as-is), and GHASH to implement POLYVAL. Optimally,
there would be separate polyval assembly implementation(s), but the only one
I could find (and it was SSE2 x86_64 code) was not Apache 2.0 licensed.
This implementation lives only in the default provider; there is no legacy
implementation.
The code offered in #16721 is not used; that implementation sits on top of
OpenSSL, this one is embedded inside OpenSSL.
Full test vectors from RFC8452 are included, except the 0 length plaintext;
that is not supported; and I'm not sure it's worthwhile to do so.
Reviewed-by: Hugo Landau <hlandau@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18693)
Rework of GCM code did not include s390x causing NULL pointer dereferences on
GCM operations other than AES-GCM on platforms that support kma. Fix this by
a proper setup of the function pointers.
Fixes: 92c9086e5c ("Use separate function to get GCM functions")
Signed-off-by: Juergen Christ <jchrist@linux.ibm.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18862)
Fixes openssl#18073.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Todd Short <todd.short@me.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18327)
Rename x86-32 assembly files from .s to .S. While processing the .S file
gcc will use the pre-processor whic will evaluate macros and ifdef. This
is turn will be used to enable the endbr32 opcode based on the __CET__
define.
Signed-off-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18353)
ghash-riscv64.pl implements 128-bit galois field multiplication for
use in the GCM mode using RISC-V carryless multiplication primitives.
The clmul-accelerated routine can be selected by setting the Zbb and
Zbc bits of the OPENSSL_riscvcap environment variable at runtime.
Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
Signed-off-by: Henry Brausen <henry.brausen@vrull.eu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17640)
Decoration prefix for some assembler labels in aes-gcm-avx512.pl was
fixed for mingw64 build.
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17868)
The EOR3 instruction is implemented with .inst, and the code here is enabled
using run-time detection of the CPU capabilities, so no need to explicitly
ask for the sha3 extension.
Fixes#17773
Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17795)
Increase the block numbers to 8 for every iteration. Increase the hash
table capacity. Make use of EOR3 instruction to improve the performance.
This can improve performance 25-40% on out-of-order microarchitectures
with a large number of fast execution units, such as Neoverse V1. We also
see 20-30% performance improvements on other architectures such as the M1.
Assembly code reviewd by Tom Cosgrove (ARM).
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15916)
Assembly code reviewed by Shricharan Srivatsan <ssrivat@us.ibm.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16854)
