mirror/openssl
2
0
Fork 0
mirror of https://github.com/openssl/openssl.git synced 2024-12-09 05:51:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
25391acc14
openssl/crypto/bn/asm/armv4-mont.pl

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

767 lines
20 KiB
Perl
Raw Normal View History

Add OpenSSL copyright to .pl files Reviewed-by: Richard Levitte <levitte@openssl.org>
2016-05-21 20:23:39 +08:00
#! /usr/bin/env perl
Copyright year updates Reviewed-by: Richard Levitte <levitte@openssl.org> Release: yes
2023-09-07 16:59:15 +08:00
# Copyright 2007-2023 The OpenSSL Project Authors. All Rights Reserved.
Add OpenSSL copyright to .pl files Reviewed-by: Richard Levitte <levitte@openssl.org>
2016-05-21 20:23:39 +08:00
#
Following the license change, modify the boilerplates in crypto/bn/ [skip ci] Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/7777)
2018-12-06 20:22:12 +08:00
# Licensed under the Apache License 2.0 (the "License"). You may not use
Add OpenSSL copyright to .pl files Reviewed-by: Richard Levitte <levitte@openssl.org>
2016-05-21 20:23:39 +08:00
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
# ====================================================================
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
# project. The module is, however, dual licensed under OpenSSL and
# CRYPTOGAMS licenses depending on where you obtain it. For further
# details see http://www.openssl.org/~appro/cryptogams/.
# ====================================================================
# January 2007.
# Montgomery multiplication for ARMv4.
#
# Performance improvement naturally varies among CPU implementations
# and compilers. The code was observed to provide +65-35% improvement
# [depending on key length, less for longer keys] on ARM920T, and
# +115-80% on Intel IXP425. This is compared to pre-bn_mul_mont code
# base and compiler generated code with in-lined umull and even umlal
Remove trailing whitespace from some files. The prevailing style seems to not have trailing whitespace, but a few lines do. This is mostly in the perlasm files, but a few C files got them after the reformat. This is the result of: find . -name '*.pl' | xargs sed -E -i '' -e 's/( |'$'\t'')*$//' find . -name '*.c' | xargs sed -E -i '' -e 's/( |'$'\t'')*$//' find . -name '*.h' | xargs sed -E -i '' -e 's/( |'$'\t'')*$//' Then bn_prime.h was excluded since this is a generated file. Note mkerr.pl has some changes in a heredoc for some help output, but other lines there lack trailing whitespace too. Reviewed-by: Kurt Roeckx <kurt@openssl.org> Reviewed-by: Matt Caswell <matt@openssl.org>
2016-10-11 00:01:24 +08:00
# instructions. The latter means that this code didn't really have an
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
# "advantage" of utilizing some "secret" instruction.
#
# The code is interoperable with Thumb ISA and is rather compact, less
# than 1/2KB. Windows CE port would be trivial, as it's exclusively
# about decorations, ABI and instruction syntax are identical.
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
# November 2013
#
# Add NEON code path, which handles lengths divisible by 8. RSA/DSA
# performance improvement on Cortex-A8 is ~45-100% depending on key
# length, more for longer keys. On Cortex-A15 the span is ~10-105%.
# On Snapdragon S4 improvement was measured to vary from ~70% to
# incredible ~380%, yes, 4.8x faster, for RSA4096 sign. But this is
# rather because original integer-only code seems to perform
# suboptimally on S4. Situation on Cortex-A9 is unfortunately
# different. It's being looked into, but the trouble is that
# performance for vectors longer than 256 bits is actually couple
# of percent worse than for integer-only code. The code is chosen
# for execution on all NEON-capable processors, because gain on
# others outweighs the marginal loss on Cortex-A9.
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
# September 2015
#
# Align Cortex-A9 performance with November 2013 improvements, i.e.
# NEON code is now ~20-105% faster than integer-only one on this
# processor. But this optimization further improved performance even
# on other processors: NEON code path is ~45-180% faster than original
# integer-only on Cortex-A8, ~10-210% on Cortex-A15, ~70-450% on
# Snapdragon S4.
Unify all assembler file generators They now generally conform to the following argument sequence: script.pl "$(PERLASM_SCHEME)" [ C preprocessor arguments ... ] \ $(PROCESSOR) <output file> However, in the spirit of being able to use these scripts manually, they also allow for no argument, or for only the flavour, or for only the output file. This is done by only using the last argument as output file if it's a file (it has an extension), and only using the first argument as flavour if it isn't a file (it doesn't have an extension). While we're at it, we make all $xlate calls the same, i.e. the $output argument is always quoted, and we always die on error when trying to start $xlate. There's a perl lesson in this, regarding operator priority... This will always succeed, even when it fails: open FOO, "something" || die "ERR: $!"; The reason is that '||' has higher priority than list operators (a function is essentially a list operator and gobbles up everything following it that isn't lower priority), and since a non-empty string is always true, so that ends up being exactly the same as: open FOO, "something"; This, however, will fail if "something" can't be opened: open FOO, "something" or die "ERR: $!"; The reason is that 'or' has lower priority that list operators, i.e. it's performed after the 'open' call. Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/9884)
2019-09-13 06:06:46 +08:00
# $output is the last argument if it looks like a file (it has an extension)
# $flavour is the first argument if it doesn't look like a file
my $output = $#ARGV >= 0 && $ARGV[$#ARGV] =~ m|\.\w+$| ? pop : undef;
my $flavour = $#ARGV >= 0 && $ARGV[0] !~ m|\.| ? shift : undef;
Add assembly support for 32-bit iOS. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-04-02 16:17:42 +08:00
if ($flavour && $flavour ne "void") {
$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;
( $xlate="${dir}arm-xlate.pl" and -f $xlate ) or
( $xlate="${dir}../../perlasm/arm-xlate.pl" and -f $xlate) or
die "can't locate arm-xlate.pl";
Unify all assembler file generators They now generally conform to the following argument sequence: script.pl "$(PERLASM_SCHEME)" [ C preprocessor arguments ... ] \ $(PROCESSOR) <output file> However, in the spirit of being able to use these scripts manually, they also allow for no argument, or for only the flavour, or for only the output file. This is done by only using the last argument as output file if it's a file (it has an extension), and only using the first argument as flavour if it isn't a file (it doesn't have an extension). While we're at it, we make all $xlate calls the same, i.e. the $output argument is always quoted, and we always die on error when trying to start $xlate. There's a perl lesson in this, regarding operator priority... This will always succeed, even when it fails: open FOO, "something" || die "ERR: $!"; The reason is that '||' has higher priority than list operators (a function is essentially a list operator and gobbles up everything following it that isn't lower priority), and since a non-empty string is always true, so that ends up being exactly the same as: open FOO, "something"; This, however, will fail if "something" can't be opened: open FOO, "something" or die "ERR: $!"; The reason is that 'or' has lower priority that list operators, i.e. it's performed after the 'open' call. Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/9884)
2019-09-13 06:06:46 +08:00
open STDOUT,"| \"$^X\" $xlate $flavour \"$output\""
or die "can't call $xlate: $1";
Add assembly support for 32-bit iOS. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-04-02 16:17:42 +08:00
} else {
Unify all assembler file generators They now generally conform to the following argument sequence: script.pl "$(PERLASM_SCHEME)" [ C preprocessor arguments ... ] \ $(PROCESSOR) <output file> However, in the spirit of being able to use these scripts manually, they also allow for no argument, or for only the flavour, or for only the output file. This is done by only using the last argument as output file if it's a file (it has an extension), and only using the first argument as flavour if it isn't a file (it doesn't have an extension). While we're at it, we make all $xlate calls the same, i.e. the $output argument is always quoted, and we always die on error when trying to start $xlate. There's a perl lesson in this, regarding operator priority... This will always succeed, even when it fails: open FOO, "something" || die "ERR: $!"; The reason is that '||' has higher priority than list operators (a function is essentially a list operator and gobbles up everything following it that isn't lower priority), and since a non-empty string is always true, so that ends up being exactly the same as: open FOO, "something"; This, however, will fail if "something" can't be opened: open FOO, "something" or die "ERR: $!"; The reason is that 'or' has lower priority that list operators, i.e. it's performed after the 'open' call. Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/9884)
2019-09-13 06:06:46 +08:00
$output and open STDOUT,">$output";
Add assembly support for 32-bit iOS. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-04-02 16:17:42 +08:00
}
armv4-mont.pl: addenum to previous commit#19749.
2010-07-08 23:06:01 +08:00
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
$num="r0"; # starts as num argument, but holds &tp[num-1]
$ap="r1";
$bp="r2"; $bi="r2"; $rp="r2";
$np="r3";
$tp="r4";
$aj="r5";
$nj="r6";
$tj="r7";
$n0="r8";
########### # r9 is reserved by ELF as platform specific, e.g. TLS pointer
$alo="r10"; # sl, gcc uses it to keep @GOT
$ahi="r11"; # fp
$nlo="r12"; # ip
########### # r13 is stack pointer
$nhi="r14"; # lr
########### # r15 is program counter
#### argument block layout relative to &tp[num-1], a.k.a. $num
$_rp="$num,#12*4";
# ap permanently resides in r1
$_bp="$num,#13*4";
# np permanently resides in r3
$_n0="$num,#14*4";
$_num="$num,#15*4"; $_bpend=$_num;
$code=<<___;
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
#include "arm_arch.h"
ARMv4 assembly pack: allow Thumb2 even in iOS build, and engage it in most modules. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-12-07 05:02:39 +08:00
#if defined(__thumb2__)
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
.syntax unified
.thumb
#else
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.code 32
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
#endif
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
ARM assembly pack: make it Windows-friendly. "Windows friendliness" means a) flipping .thumb and .text directives, b) always generate Thumb-2 code when asked(*); c) Windows-specific references to external OPENSSL_armcap_P. (*) so far *some* modules were compiled as .code 32 even if Thumb-2 was targeted. It works at hardware level because processor can alternate between the modes with no overhead. But clang --target=arm-windows's builtin assembler just refuses to compile .code 32... Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/8252)
2019-02-15 16:44:39 +08:00
.text
Remove inconsistency in ARM support. This facilitates "universal" builds, ones that target multiple architectures, e.g. ARMv5 through ARMv7. See commentary in Configure for details. Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Reviewed-by: Matt Caswell <matt@openssl.org>
2014-11-08 05:48:22 +08:00
#if __ARM_MAX_ARCH__>=7
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.align 5
.LOPENSSL_armcap:
ARM assembly pack: make it Windows-friendly. "Windows friendliness" means a) flipping .thumb and .text directives, b) always generate Thumb-2 code when asked(*); c) Windows-specific references to external OPENSSL_armcap_P. (*) so far *some* modules were compiled as .code 32 even if Thumb-2 was targeted. It works at hardware level because processor can alternate between the modes with no overhead. But clang --target=arm-windows's builtin assembler just refuses to compile .code 32... Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/8252)
2019-02-15 16:44:39 +08:00
# ifdef _WIN32
.word OPENSSL_armcap_P
# else
Add assembly support for 32-bit iOS. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-04-02 16:17:42 +08:00
.word OPENSSL_armcap_P-.Lbn_mul_mont
ARM assembly pack: make it Windows-friendly. "Windows friendliness" means a) flipping .thumb and .text directives, b) always generate Thumb-2 code when asked(*); c) Windows-specific references to external OPENSSL_armcap_P. (*) so far *some* modules were compiled as .code 32 even if Thumb-2 was targeted. It works at hardware level because processor can alternate between the modes with no overhead. But clang --target=arm-windows's builtin assembler just refuses to compile .code 32... Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/8252)
2019-02-15 16:44:39 +08:00
# endif
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
#endif
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
.global bn_mul_mont
.type bn_mul_mont,%function
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.align 5
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
bn_mul_mont:
Add assembly support for 32-bit iOS. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-04-02 16:17:42 +08:00
.Lbn_mul_mont:
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
ldr ip,[sp,#4] @ load num
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
stmdb sp!,{r0,r2} @ sp points at argument block
Remove inconsistency in ARM support. This facilitates "universal" builds, ones that target multiple architectures, e.g. ARMv5 through ARMv7. See commentary in Configure for details. Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Reviewed-by: Matt Caswell <matt@openssl.org>
2014-11-08 05:48:22 +08:00
#if __ARM_MAX_ARCH__>=7
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
tst ip,#7
bne .Lialu
ARM assembly pack: make it Windows-friendly. "Windows friendliness" means a) flipping .thumb and .text directives, b) always generate Thumb-2 code when asked(*); c) Windows-specific references to external OPENSSL_armcap_P. (*) so far *some* modules were compiled as .code 32 even if Thumb-2 was targeted. It works at hardware level because processor can alternate between the modes with no overhead. But clang --target=arm-windows's builtin assembler just refuses to compile .code 32... Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/8252)
2019-02-15 16:44:39 +08:00
ldr r0,.LOPENSSL_armcap
#if !defined(_WIN32)
adr r2,.Lbn_mul_mont
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
ldr r0,[r0,r2]
ARM assembly pack: make it Windows-friendly. "Windows friendliness" means a) flipping .thumb and .text directives, b) always generate Thumb-2 code when asked(*); c) Windows-specific references to external OPENSSL_armcap_P. (*) so far *some* modules were compiled as .code 32 even if Thumb-2 was targeted. It works at hardware level because processor can alternate between the modes with no overhead. But clang --target=arm-windows's builtin assembler just refuses to compile .code 32... Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/8252)
2019-02-15 16:44:39 +08:00
# endif
# if defined(__APPLE__) || defined(_WIN32)
Add assembly support for 32-bit iOS. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-04-02 16:17:42 +08:00
ldr r0,[r0]
ARM assembly pack: make it Windows-friendly. "Windows friendliness" means a) flipping .thumb and .text directives, b) always generate Thumb-2 code when asked(*); c) Windows-specific references to external OPENSSL_armcap_P. (*) so far *some* modules were compiled as .code 32 even if Thumb-2 was targeted. It works at hardware level because processor can alternate between the modes with no overhead. But clang --target=arm-windows's builtin assembler just refuses to compile .code 32... Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/8252)
2019-02-15 16:44:39 +08:00
# endif
Consistently use arm_arch.h constants in armcap assembly code. Most of the assembly uses constants from arm_arch.h, but a few references to ARMV7_NEON don't. Consistently use the macros everywhere. Signed-off-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org>
2016-02-24 00:41:55 +08:00
tst r0,#ARMV7_NEON @ NEON available?
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
ldmia sp, {r0,r2}
beq .Lialu
add sp,sp,#8
b bn_mul8x_mont_neon
.align 4
.Lialu:
#endif
cmp ip,#2
mov $num,ip @ load num
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
#ifdef __thumb2__
ittt lt
#endif
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
movlt r0,#0
addlt sp,sp,#2*4
Eliminate conditional final subtraction in Montgomery assembler modules.
2007-06-18 01:10:03 +08:00
blt .Labrt
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
stmdb sp!,{r4-r12,lr} @ save 10 registers
mov $num,$num,lsl#2 @ rescale $num for byte count
sub sp,sp,$num @ alloca(4*num)
sub sp,sp,#4 @ +extra dword
sub $num,$num,#4 @ "num=num-1"
add $tp,$bp,$num @ &bp[num-1]
add $num,sp,$num @ $num to point at &tp[num-1]
ldr $n0,[$_n0] @ &n0
ldr $bi,[$bp] @ bp[0]
ldr $aj,[$ap],#4 @ ap[0],ap++
ldr $nj,[$np],#4 @ np[0],np++
ldr $n0,[$n0] @ *n0
str $tp,[$_bpend] @ save &bp[num]
umull $alo,$ahi,$aj,$bi @ ap[0]*bp[0]
str $n0,[$_n0] @ save n0 value
mul $n0,$alo,$n0 @ "tp[0]"*n0
mov $nlo,#0
umlal $alo,$nlo,$nj,$n0 @ np[0]*n0+"t[0]"
mov $tp,sp
.L1st:
ldr $aj,[$ap],#4 @ ap[j],ap++
mov $alo,$ahi
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $nj,[$np],#4 @ np[j],np++
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
mov $ahi,#0
umlal $alo,$ahi,$aj,$bi @ ap[j]*bp[0]
mov $nhi,#0
umlal $nlo,$nhi,$nj,$n0 @ np[j]*n0
adds $nlo,$nlo,$alo
str $nlo,[$tp],#4 @ tp[j-1]=,tp++
adc $nlo,$nhi,#0
cmp $tp,$num
bne .L1st
adds $nlo,$nlo,$ahi
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $tp,[$_bp] @ restore bp
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
mov $nhi,#0
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $n0,[$_n0] @ restore n0
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
adc $nhi,$nhi,#0
str $nlo,[$num] @ tp[num-1]=
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
mov $tj,sp
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
str $nhi,[$num,#4] @ tp[num]=
.Louter:
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
sub $tj,$num,$tj @ "original" $num-1 value
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
sub $ap,$ap,$tj @ "rewind" ap to &ap[1]
ldr $bi,[$tp,#4]! @ *(++bp)
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
sub $np,$np,$tj @ "rewind" np to &np[1]
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
ldr $aj,[$ap,#-4] @ ap[0]
ldr $alo,[sp] @ tp[0]
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $nj,[$np,#-4] @ np[0]
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
ldr $tj,[sp,#4] @ tp[1]
mov $ahi,#0
umlal $alo,$ahi,$aj,$bi @ ap[0]*bp[i]+tp[0]
str $tp,[$_bp] @ save bp
mul $n0,$alo,$n0
mov $nlo,#0
umlal $alo,$nlo,$nj,$n0 @ np[0]*n0+"tp[0]"
mov $tp,sp
.Linner:
ldr $aj,[$ap],#4 @ ap[j],ap++
adds $alo,$ahi,$tj @ +=tp[j]
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $nj,[$np],#4 @ np[j],np++
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
mov $ahi,#0
umlal $alo,$ahi,$aj,$bi @ ap[j]*bp[i]
mov $nhi,#0
umlal $nlo,$nhi,$nj,$n0 @ np[j]*n0
adc $ahi,$ahi,#0
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $tj,[$tp,#8] @ tp[j+1]
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
adds $nlo,$nlo,$alo
str $nlo,[$tp],#4 @ tp[j-1]=,tp++
adc $nlo,$nhi,#0
cmp $tp,$num
bne .Linner
adds $nlo,$nlo,$ahi
mov $nhi,#0
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $tp,[$_bp] @ restore bp
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
adc $nhi,$nhi,#0
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
ldr $n0,[$_n0] @ restore n0
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
adds $nlo,$nlo,$tj
ldr $tj,[$_bpend] @ restore &bp[num]
armv4-mont.pl: profiler-assisted optimization gives 8%-14% improvement (more for longer keys) on RSA/DSA.
2011-08-13 20:38:41 +08:00
adc $nhi,$nhi,#0
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
str $nlo,[$num] @ tp[num-1]=
str $nhi,[$num,#4] @ tp[num]=
cmp $tp,$tj
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
#ifdef __thumb2__
itt ne
#endif
movne $tj,sp
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
bne .Louter
ldr $rp,[$_rp] @ pull rp
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
mov $aj,sp
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
add $num,$num,#4 @ $num to point at &tp[num]
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
sub $aj,$num,$aj @ "original" num value
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
mov $tp,sp @ "rewind" $tp
Eliminate conditional final subtraction in Montgomery assembler modules.
2007-06-18 01:10:03 +08:00
mov $ap,$tp @ "borrow" $ap
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
sub $np,$np,$aj @ "rewind" $np to &np[0]
Eliminate conditional final subtraction in Montgomery assembler modules.
2007-06-18 01:10:03 +08:00
subs $tj,$tj,$tj @ "clear" carry flag
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
.Lsub: ldr $tj,[$tp],#4
ldr $nj,[$np],#4
sbcs $tj,$tj,$nj @ tp[j]-np[j]
str $tj,[$rp],#4 @ rp[j]=
teq $tp,$num @ preserve carry
bne .Lsub
sbcs $nhi,$nhi,#0 @ upmost carry
mov $tp,sp @ "rewind" $tp
sub $rp,$rp,$aj @ "rewind" $rp
bn/asm/*-mont.pl: harmonize with BN_from_montgomery_word. Montgomery multiplication post-conditions in some of code paths were formally non-constant time. Cache access pattern was result-neutral, but a little bit asymmetric, which might have produced a signal [if processor reordered load and stores at run-time]. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/6141)
2018-05-01 04:59:51 +08:00
.Lcopy: ldr $tj,[$tp] @ conditional copy
ldr $aj,[$rp]
Eliminate conditional final subtraction in Montgomery assembler modules.
2007-06-18 01:10:03 +08:00
str sp,[$tp],#4 @ zap tp
bn/asm/*-mont.pl: harmonize with BN_from_montgomery_word. Montgomery multiplication post-conditions in some of code paths were formally non-constant time. Cache access pattern was result-neutral, but a little bit asymmetric, which might have produced a signal [if processor reordered load and stores at run-time]. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/6141)
2018-05-01 04:59:51 +08:00
#ifdef __thumb2__
it cc
#endif
movcc $aj,$tj
str $aj,[$rp],#4
teq $tp,$num @ preserve carry
Eliminate conditional final subtraction in Montgomery assembler modules.
2007-06-18 01:10:03 +08:00
bne .Lcopy
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
mov sp,$num
add sp,sp,#4 @ skip over tp[num+1]
Eliminate conditional final subtraction in Montgomery assembler modules.
2007-06-18 01:10:03 +08:00
ldmia sp!,{r4-r12,lr} @ restore registers
add sp,sp,#2*4 @ skip over {r0,r2}
mov r0,#1
ARM assembly pack: get ARMv7 instruction endianness right. Pointer out and suggested by: Ard Biesheuvel.
2014-06-07 03:27:18 +08:00
.Labrt:
#if __ARM_ARCH__>=5
ret @ bx lr
#else
tst lr,#1
Eliminate conditional final subtraction in Montgomery assembler modules.
2007-06-18 01:10:03 +08:00
moveq pc,lr @ be binary compatible with V4, yet
bx lr @ interoperable with Thumb ISA:-)
ARM assembly pack: get ARMv7 instruction endianness right. Pointer out and suggested by: Ard Biesheuvel.
2014-06-07 03:27:18 +08:00
#endif
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
.size bn_mul_mont,.-bn_mul_mont
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
___
{
my ($A0,$A1,$A2,$A3)=map("d$_",(0..3));
my ($N0,$N1,$N2,$N3)=map("d$_",(4..7));
my ($Z,$Temp)=("q4","q5");
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
my @ACC=map("q$_",(6..13));
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
my ($Bi,$Ni,$M0)=map("d$_",(28..31));
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
my $zero="$Z#lo";
my $temp="$Temp#lo";
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
my ($rptr,$aptr,$bptr,$nptr,$n0,$num)=map("r$_",(0..5));
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
my ($tinptr,$toutptr,$inner,$outer,$bnptr)=map("r$_",(6..11));
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
$code.=<<___;
Remove inconsistency in ARM support. This facilitates "universal" builds, ones that target multiple architectures, e.g. ARMv5 through ARMv7. See commentary in Configure for details. Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Reviewed-by: Matt Caswell <matt@openssl.org>
2014-11-08 05:48:22 +08:00
#if __ARM_MAX_ARCH__>=7
.arch armv7-a
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.fpu neon
.type bn_mul8x_mont_neon,%function
.align 5
bn_mul8x_mont_neon:
mov ip,sp
stmdb sp!,{r4-r11}
vstmdb sp!,{d8-d15} @ ABI specification says so
ldmia ip,{r4-r5} @ load rest of parameter block
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
mov ip,sp
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
cmp $num,#8
bhi .LNEON_8n
@ special case for $num==8, everything is in register bank...
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vld1.32 {${Bi}[0]}, [$bptr,:32]!
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
veor $zero,$zero,$zero
sub $toutptr,sp,$num,lsl#4
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vld1.32 {$A0-$A3}, [$aptr]! @ can't specify :32 :-(
and $toutptr,$toutptr,#-64
vld1.32 {${M0}[0]}, [$n0,:32]
mov sp,$toutptr @ alloca
vzip.16 $Bi,$zero
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmull.u32 @ACC[0],$Bi,${A0}[0]
vmull.u32 @ACC[1],$Bi,${A0}[1]
vmull.u32 @ACC[2],$Bi,${A1}[0]
vshl.i64 $Ni,@ACC[0]#hi,#16
vmull.u32 @ACC[3],$Bi,${A1}[1]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vadd.u64 $Ni,$Ni,@ACC[0]#lo
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
veor $zero,$zero,$zero
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmul.u32 $Ni,$Ni,$M0
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmull.u32 @ACC[4],$Bi,${A2}[0]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vld1.32 {$N0-$N3}, [$nptr]!
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmull.u32 @ACC[5],$Bi,${A2}[1]
vmull.u32 @ACC[6],$Bi,${A3}[0]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vzip.16 $Ni,$zero
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmull.u32 @ACC[7],$Bi,${A3}[1]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmlal.u32 @ACC[0],$Ni,${N0}[0]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
sub $outer,$num,#1
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmlal.u32 @ACC[1],$Ni,${N0}[1]
vmlal.u32 @ACC[2],$Ni,${N1}[0]
vmlal.u32 @ACC[3],$Ni,${N1}[1]
vmlal.u32 @ACC[4],$Ni,${N2}[0]
vmov $Temp,@ACC[0]
vmlal.u32 @ACC[5],$Ni,${N2}[1]
vmov @ACC[0],@ACC[1]
vmlal.u32 @ACC[6],$Ni,${N3}[0]
vmov @ACC[1],@ACC[2]
vmlal.u32 @ACC[7],$Ni,${N3}[1]
vmov @ACC[2],@ACC[3]
vmov @ACC[3],@ACC[4]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vshr.u64 $temp,$temp,#16
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmov @ACC[4],@ACC[5]
vmov @ACC[5],@ACC[6]
vadd.u64 $temp,$temp,$Temp#hi
vmov @ACC[6],@ACC[7]
veor @ACC[7],@ACC[7]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vshr.u64 $temp,$temp,#16
b .LNEON_outer8
.align 4
.LNEON_outer8:
vld1.32 {${Bi}[0]}, [$bptr,:32]!
veor $zero,$zero,$zero
vzip.16 $Bi,$zero
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vadd.u64 @ACC[0]#lo,@ACC[0]#lo,$temp
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmlal.u32 @ACC[0],$Bi,${A0}[0]
vmlal.u32 @ACC[1],$Bi,${A0}[1]
vmlal.u32 @ACC[2],$Bi,${A1}[0]
vshl.i64 $Ni,@ACC[0]#hi,#16
vmlal.u32 @ACC[3],$Bi,${A1}[1]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vadd.u64 $Ni,$Ni,@ACC[0]#lo
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
veor $zero,$zero,$zero
subs $outer,$outer,#1
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmul.u32 $Ni,$Ni,$M0
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmlal.u32 @ACC[4],$Bi,${A2}[0]
vmlal.u32 @ACC[5],$Bi,${A2}[1]
vmlal.u32 @ACC[6],$Bi,${A3}[0]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vzip.16 $Ni,$zero
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmlal.u32 @ACC[7],$Bi,${A3}[1]
vmlal.u32 @ACC[0],$Ni,${N0}[0]
vmlal.u32 @ACC[1],$Ni,${N0}[1]
vmlal.u32 @ACC[2],$Ni,${N1}[0]
vmlal.u32 @ACC[3],$Ni,${N1}[1]
vmlal.u32 @ACC[4],$Ni,${N2}[0]
vmov $Temp,@ACC[0]
vmlal.u32 @ACC[5],$Ni,${N2}[1]
vmov @ACC[0],@ACC[1]
vmlal.u32 @ACC[6],$Ni,${N3}[0]
vmov @ACC[1],@ACC[2]
vmlal.u32 @ACC[7],$Ni,${N3}[1]
vmov @ACC[2],@ACC[3]
vmov @ACC[3],@ACC[4]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vshr.u64 $temp,$temp,#16
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vmov @ACC[4],@ACC[5]
vmov @ACC[5],@ACC[6]
vadd.u64 $temp,$temp,$Temp#hi
vmov @ACC[6],@ACC[7]
veor @ACC[7],@ACC[7]
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vshr.u64 $temp,$temp,#16
bne .LNEON_outer8
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vadd.u64 @ACC[0]#lo,@ACC[0]#lo,$temp
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
mov $toutptr,sp
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vshr.u64 $temp,@ACC[0]#lo,#16
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
mov $inner,$num
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vadd.u64 @ACC[0]#hi,@ACC[0]#hi,$temp
add $tinptr,sp,#96
vshr.u64 $temp,@ACC[0]#hi,#16
vzip.16 @ACC[0]#lo,@ACC[0]#hi
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
b .LNEON_tail_entry
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.align 4
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
.LNEON_8n:
veor @ACC[0],@ACC[0],@ACC[0]
sub $toutptr,sp,#128
veor @ACC[1],@ACC[1],@ACC[1]
sub $toutptr,$toutptr,$num,lsl#4
veor @ACC[2],@ACC[2],@ACC[2]
and $toutptr,$toutptr,#-64
veor @ACC[3],@ACC[3],@ACC[3]
mov sp,$toutptr @ alloca
veor @ACC[4],@ACC[4],@ACC[4]
add $toutptr,$toutptr,#256
veor @ACC[5],@ACC[5],@ACC[5]
sub $inner,$num,#8
veor @ACC[6],@ACC[6],@ACC[6]
veor @ACC[7],@ACC[7],@ACC[7]
.LNEON_8n_init:
vst1.64 {@ACC[0]-@ACC[1]},[$toutptr,:256]!
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
subs $inner,$inner,#8
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vst1.64 {@ACC[2]-@ACC[3]},[$toutptr,:256]!
vst1.64 {@ACC[4]-@ACC[5]},[$toutptr,:256]!
vst1.64 {@ACC[6]-@ACC[7]},[$toutptr,:256]!
bne .LNEON_8n_init
add $tinptr,sp,#256
vld1.32 {$A0-$A3},[$aptr]!
add $bnptr,sp,#8
vld1.32 {${M0}[0]},[$n0,:32]
mov $outer,$num
b .LNEON_8n_outer
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.align 4
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
.LNEON_8n_outer:
vld1.32 {${Bi}[0]},[$bptr,:32]! @ *b++
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
veor $zero,$zero,$zero
vzip.16 $Bi,$zero
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
add $toutptr,sp,#128
vld1.32 {$N0-$N3},[$nptr]!
vmlal.u32 @ACC[0],$Bi,${A0}[0]
vmlal.u32 @ACC[1],$Bi,${A0}[1]
veor $zero,$zero,$zero
vmlal.u32 @ACC[2],$Bi,${A1}[0]
vshl.i64 $Ni,@ACC[0]#hi,#16
vmlal.u32 @ACC[3],$Bi,${A1}[1]
vadd.u64 $Ni,$Ni,@ACC[0]#lo
vmlal.u32 @ACC[4],$Bi,${A2}[0]
vmul.u32 $Ni,$Ni,$M0
vmlal.u32 @ACC[5],$Bi,${A2}[1]
vst1.32 {$Bi},[sp,:64] @ put aside smashed b[8*i+0]
vmlal.u32 @ACC[6],$Bi,${A3}[0]
vzip.16 $Ni,$zero
vmlal.u32 @ACC[7],$Bi,${A3}[1]
___
for ($i=0; $i<7;) {
$code.=<<___;
vld1.32 {${Bi}[0]},[$bptr,:32]! @ *b++
vmlal.u32 @ACC[0],$Ni,${N0}[0]
veor $temp,$temp,$temp
vmlal.u32 @ACC[1],$Ni,${N0}[1]
vzip.16 $Bi,$temp
vmlal.u32 @ACC[2],$Ni,${N1}[0]
vshr.u64 @ACC[0]#lo,@ACC[0]#lo,#16
vmlal.u32 @ACC[3],$Ni,${N1}[1]
vmlal.u32 @ACC[4],$Ni,${N2}[0]
vadd.u64 @ACC[0]#lo,@ACC[0]#lo,@ACC[0]#hi
vmlal.u32 @ACC[5],$Ni,${N2}[1]
vshr.u64 @ACC[0]#lo,@ACC[0]#lo,#16
vmlal.u32 @ACC[6],$Ni,${N3}[0]
vmlal.u32 @ACC[7],$Ni,${N3}[1]
vadd.u64 @ACC[1]#lo,@ACC[1]#lo,@ACC[0]#lo
vst1.32 {$Ni},[$bnptr,:64]! @ put aside smashed m[8*i+$i]
___
push(@ACC,shift(@ACC)); $i++;
$code.=<<___;
vmlal.u32 @ACC[0],$Bi,${A0}[0]
vld1.64 {@ACC[7]},[$tinptr,:128]!
vmlal.u32 @ACC[1],$Bi,${A0}[1]
veor $zero,$zero,$zero
vmlal.u32 @ACC[2],$Bi,${A1}[0]
vshl.i64 $Ni,@ACC[0]#hi,#16
vmlal.u32 @ACC[3],$Bi,${A1}[1]
vadd.u64 $Ni,$Ni,@ACC[0]#lo
vmlal.u32 @ACC[4],$Bi,${A2}[0]
vmul.u32 $Ni,$Ni,$M0
vmlal.u32 @ACC[5],$Bi,${A2}[1]
vst1.32 {$Bi},[$bnptr,:64]! @ put aside smashed b[8*i+$i]
vmlal.u32 @ACC[6],$Bi,${A3}[0]
vzip.16 $Ni,$zero
vmlal.u32 @ACC[7],$Bi,${A3}[1]
___
}
$code.=<<___;
vld1.32 {$Bi},[sp,:64] @ pull smashed b[8*i+0]
vmlal.u32 @ACC[0],$Ni,${N0}[0]
vld1.32 {$A0-$A3},[$aptr]!
vmlal.u32 @ACC[1],$Ni,${N0}[1]
vmlal.u32 @ACC[2],$Ni,${N1}[0]
vshr.u64 @ACC[0]#lo,@ACC[0]#lo,#16
vmlal.u32 @ACC[3],$Ni,${N1}[1]
vmlal.u32 @ACC[4],$Ni,${N2}[0]
vadd.u64 @ACC[0]#lo,@ACC[0]#lo,@ACC[0]#hi
vmlal.u32 @ACC[5],$Ni,${N2}[1]
vshr.u64 @ACC[0]#lo,@ACC[0]#lo,#16
vmlal.u32 @ACC[6],$Ni,${N3}[0]
vmlal.u32 @ACC[7],$Ni,${N3}[1]
vadd.u64 @ACC[1]#lo,@ACC[1]#lo,@ACC[0]#lo
vst1.32 {$Ni},[$bnptr,:64] @ put aside smashed m[8*i+$i]
add $bnptr,sp,#8 @ rewind
___
push(@ACC,shift(@ACC));
$code.=<<___;
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
sub $inner,$num,#8
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
b .LNEON_8n_inner
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
.align 4
.LNEON_8n_inner:
subs $inner,$inner,#8
vmlal.u32 @ACC[0],$Bi,${A0}[0]
vld1.64 {@ACC[7]},[$tinptr,:128]
vmlal.u32 @ACC[1],$Bi,${A0}[1]
vld1.32 {$Ni},[$bnptr,:64]! @ pull smashed m[8*i+0]
vmlal.u32 @ACC[2],$Bi,${A1}[0]
vld1.32 {$N0-$N3},[$nptr]!
vmlal.u32 @ACC[3],$Bi,${A1}[1]
it ne
addne $tinptr,$tinptr,#16 @ don't advance in last iteration
vmlal.u32 @ACC[4],$Bi,${A2}[0]
vmlal.u32 @ACC[5],$Bi,${A2}[1]
vmlal.u32 @ACC[6],$Bi,${A3}[0]
vmlal.u32 @ACC[7],$Bi,${A3}[1]
___
for ($i=1; $i<8; $i++) {
$code.=<<___;
vld1.32 {$Bi},[$bnptr,:64]! @ pull smashed b[8*i+$i]
vmlal.u32 @ACC[0],$Ni,${N0}[0]
vmlal.u32 @ACC[1],$Ni,${N0}[1]
vmlal.u32 @ACC[2],$Ni,${N1}[0]
vmlal.u32 @ACC[3],$Ni,${N1}[1]
vmlal.u32 @ACC[4],$Ni,${N2}[0]
vmlal.u32 @ACC[5],$Ni,${N2}[1]
vmlal.u32 @ACC[6],$Ni,${N3}[0]
vmlal.u32 @ACC[7],$Ni,${N3}[1]
vst1.64 {@ACC[0]},[$toutptr,:128]!
___
push(@ACC,shift(@ACC));
$code.=<<___;
vmlal.u32 @ACC[0],$Bi,${A0}[0]
vld1.64 {@ACC[7]},[$tinptr,:128]
vmlal.u32 @ACC[1],$Bi,${A0}[1]
vld1.32 {$Ni},[$bnptr,:64]! @ pull smashed m[8*i+$i]
vmlal.u32 @ACC[2],$Bi,${A1}[0]
it ne
addne $tinptr,$tinptr,#16 @ don't advance in last iteration
vmlal.u32 @ACC[3],$Bi,${A1}[1]
vmlal.u32 @ACC[4],$Bi,${A2}[0]
vmlal.u32 @ACC[5],$Bi,${A2}[1]
vmlal.u32 @ACC[6],$Bi,${A3}[0]
vmlal.u32 @ACC[7],$Bi,${A3}[1]
___
}
$code.=<<___;
it eq
subeq $aptr,$aptr,$num,lsl#2 @ rewind
vmlal.u32 @ACC[0],$Ni,${N0}[0]
vld1.32 {$Bi},[sp,:64] @ pull smashed b[8*i+0]
vmlal.u32 @ACC[1],$Ni,${N0}[1]
vld1.32 {$A0-$A3},[$aptr]!
vmlal.u32 @ACC[2],$Ni,${N1}[0]
add $bnptr,sp,#8 @ rewind
vmlal.u32 @ACC[3],$Ni,${N1}[1]
vmlal.u32 @ACC[4],$Ni,${N2}[0]
vmlal.u32 @ACC[5],$Ni,${N2}[1]
vmlal.u32 @ACC[6],$Ni,${N3}[0]
vst1.64 {@ACC[0]},[$toutptr,:128]!
vmlal.u32 @ACC[7],$Ni,${N3}[1]
bne .LNEON_8n_inner
___
push(@ACC,shift(@ACC));
$code.=<<___;
add $tinptr,sp,#128
vst1.64 {@ACC[0]-@ACC[1]},[$toutptr,:256]!
veor q2,q2,q2 @ $N0-$N1
vst1.64 {@ACC[2]-@ACC[3]},[$toutptr,:256]!
veor q3,q3,q3 @ $N2-$N3
vst1.64 {@ACC[4]-@ACC[5]},[$toutptr,:256]!
vst1.64 {@ACC[6]},[$toutptr,:128]
subs $outer,$outer,#8
vld1.64 {@ACC[0]-@ACC[1]},[$tinptr,:256]!
vld1.64 {@ACC[2]-@ACC[3]},[$tinptr,:256]!
vld1.64 {@ACC[4]-@ACC[5]},[$tinptr,:256]!
vld1.64 {@ACC[6]-@ACC[7]},[$tinptr,:256]!
itt ne
subne $nptr,$nptr,$num,lsl#2 @ rewind
bne .LNEON_8n_outer
add $toutptr,sp,#128
vst1.64 {q2-q3}, [sp,:256]! @ start wiping stack frame
vshr.u64 $temp,@ACC[0]#lo,#16
vst1.64 {q2-q3},[sp,:256]!
vadd.u64 @ACC[0]#hi,@ACC[0]#hi,$temp
vst1.64 {q2-q3}, [sp,:256]!
vshr.u64 $temp,@ACC[0]#hi,#16
vst1.64 {q2-q3}, [sp,:256]!
vzip.16 @ACC[0]#lo,@ACC[0]#hi
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
mov $inner,$num
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
b .LNEON_tail_entry
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
.align 4
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.LNEON_tail:
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vadd.u64 @ACC[0]#lo,@ACC[0]#lo,$temp
vshr.u64 $temp,@ACC[0]#lo,#16
vld1.64 {@ACC[2]-@ACC[3]}, [$tinptr, :256]!
vadd.u64 @ACC[0]#hi,@ACC[0]#hi,$temp
vld1.64 {@ACC[4]-@ACC[5]}, [$tinptr, :256]!
vshr.u64 $temp,@ACC[0]#hi,#16
vld1.64 {@ACC[6]-@ACC[7]}, [$tinptr, :256]!
vzip.16 @ACC[0]#lo,@ACC[0]#hi
.LNEON_tail_entry:
___
for ($i=1; $i<8; $i++) {
$code.=<<___;
vadd.u64 @ACC[1]#lo,@ACC[1]#lo,$temp
vst1.32 {@ACC[0]#lo[0]}, [$toutptr, :32]!
vshr.u64 $temp,@ACC[1]#lo,#16
vadd.u64 @ACC[1]#hi,@ACC[1]#hi,$temp
vshr.u64 $temp,@ACC[1]#hi,#16
vzip.16 @ACC[1]#lo,@ACC[1]#hi
___
push(@ACC,shift(@ACC));
}
push(@ACC,shift(@ACC));
$code.=<<___;
vld1.64 {@ACC[0]-@ACC[1]}, [$tinptr, :256]!
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
subs $inner,$inner,#8
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
vst1.32 {@ACC[7]#lo[0]}, [$toutptr, :32]!
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
bne .LNEON_tail
vst1.32 {${temp}[0]}, [$toutptr, :32] @ top-most bit
sub $nptr,$nptr,$num,lsl#2 @ rewind $nptr
subs $aptr,sp,#0 @ clear carry flag
add $bptr,sp,$num,lsl#2
.LNEON_sub:
ldmia $aptr!, {r4-r7}
ldmia $nptr!, {r8-r11}
sbcs r8, r4,r8
sbcs r9, r5,r9
sbcs r10,r6,r10
sbcs r11,r7,r11
teq $aptr,$bptr @ preserves carry
stmia $rptr!, {r8-r11}
bne .LNEON_sub
ldr r10, [$aptr] @ load top-most bit
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
mov r11,sp
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
veor q0,q0,q0
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
sub r11,$bptr,r11 @ this is num*4
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
veor q1,q1,q1
mov $aptr,sp
sub $rptr,$rptr,r11 @ rewind $rptr
mov $nptr,$bptr @ second 3/4th of frame
sbcs r10,r10,#0 @ result is carry flag
.LNEON_copy_n_zap:
ldmia $aptr!, {r4-r7}
ldmia $rptr, {r8-r11}
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
it cc
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
movcc r8, r4
vst1.64 {q0-q1}, [$nptr,:256]! @ wipe
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
itt cc
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
movcc r9, r5
movcc r10,r6
vst1.64 {q0-q1}, [$nptr,:256]! @ wipe
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
it cc
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
movcc r11,r7
ldmia $aptr, {r4-r7}
stmia $rptr!, {r8-r11}
sub $aptr,$aptr,#16
ldmia $rptr, {r8-r11}
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
it cc
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
movcc r8, r4
vst1.64 {q0-q1}, [$aptr,:256]! @ wipe
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
itt cc
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
movcc r9, r5
movcc r10,r6
vst1.64 {q0-q1}, [$nptr,:256]! @ wipe
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
it cc
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
movcc r11,r7
teq $aptr,$bptr @ preserves carry
stmia $rptr!, {r8-r11}
bne .LNEON_copy_n_zap
ARMv4 assembly pack: implement support for Thumb2. As some of ARM processors, more specifically Cortex-Mx series, are Thumb2-only, we need to support Thumb2-only builds even in assembly. Reviewed-by: Tim Hudson <tjh@openssl.org>
2015-09-24 00:41:27 +08:00
mov sp,ip
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
vldmia sp!,{d8-d15}
ldmia sp!,{r4-r11}
ARM assembly pack: get ARMv7 instruction endianness right. Pointer out and suggested by: Ard Biesheuvel.
2014-06-07 03:27:18 +08:00
ret @ bx lr
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
.size bn_mul8x_mont_neon,.-bn_mul8x_mont_neon
#endif
___
}
$code.=<<___;
.asciz "Montgomery multiplication for ARMv4/NEON, CRYPTOGAMS by <appro\@openssl.org>"
ARMv4 assembler: fix compilation failure. Fix is actually unconfirmed, but I can't think of any other cause for failure
2010-03-29 17:55:19 +08:00
.align 2
Remove inconsistency in ARM support. This facilitates "universal" builds, ones that target multiple architectures, e.g. ARMv5 through ARMv7. See commentary in Configure for details. Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Reviewed-by: Matt Caswell <matt@openssl.org>
2014-11-08 05:48:22 +08:00
#if __ARM_MAX_ARCH__>=7
Ensure there's only one copy of OPENSSL_armcap_P in libcrypto.a Change-Id: Ia94e528a2d55934435de6a2949784c52eb38d82f Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Tomas Mraz <tomas@openssl.org> (Merged from https://github.com/openssl/openssl/pull/20621)
2023-03-28 05:05:13 +08:00
.extern OPENSSL_armcap_P
bn/asm/armv4-mont.pl: add NEON code path.
2013-12-05 05:37:49 +08:00
#endif
Montgomery multiplication for ARMv4.
2007-01-12 05:43:25 +08:00
___
bn/asm/armv4-mont.pl: boost NEON performance. Close difference gap on Cortex-A9, which resulted in further improvement even on other processors. Reviewed-by: Richard Levitte <levitte@openssl.org>
2015-09-25 19:43:00 +08:00
foreach (split("\n",$code)) {
s/\`([^\`]*)\`/eval $1/ge;
s/\bq([0-9]+)#(lo|hi)/sprintf "d%d",2*$1+($2 eq "hi")/ge or
s/\bret\b/bx lr/g or
s/\bbx\s+lr\b/.word\t0xe12fff1e/g; # make it possible to compile with -march=armv4
print $_,"\n";
}
Also check for errors in x86_64-xlate.pl. In https://github.com/openssl/openssl/pull/10883, I'd meant to exclude the perlasm drivers since they aren't opening pipes and do not particularly need it, but I only noticed x86_64-xlate.pl, so arm-xlate.pl and ppc-xlate.pl got the change. That seems to have been fine, so be consistent and also apply the change to x86_64-xlate.pl. Checking for errors is generally a good idea. Reviewed-by: Richard Levitte <levitte@openssl.org> Reviewed-by: David Benjamin <davidben@google.com> (Merged from https://github.com/openssl/openssl/pull/10930)
2020-02-17 10:17:53 +08:00
close STDOUT or die "error closing STDOUT: $!";
Reference in New Issue Copy Permalink