mirror/glibc
2
0
Fork 0
mirror of git://sourceware.org/git/glibc.git synced 2024-11-27 03:41:23 +08:00
Code Issues Packages Projects Releases Wiki Activity
a9b3b770f5
glibc/sysdeps/aarch64/strchr.S

116 lines
3.1 KiB
ArmAsm
Raw Normal View History

[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
/* strchr - find a character in a string
Update copyright dates with scripts/update-copyrights
2023-01-07 05:08:04 +08:00
Copyright (C) 2014-2023 Free Software Foundation, Inc.
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library. If not, see
Prefer https to http for gnu.org and fsf.org URLs Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 13:40:42 +08:00
<https://www.gnu.org/licenses/>. */
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
#include <sysdep.h>
/* Assumptions:
*
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
* ARMv8-a, AArch64, Advanced SIMD.
* MTE compatible.
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
*/
#define srcin x0
#define chrin w1
#define result x0
#define src x2
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
#define tmp1 x1
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
#define tmp2 x3
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
#define vrepchr v0
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
#define vdata v1
#define qdata q1
#define vhas_nul v2
#define vhas_chr v3
#define vrepmask v4
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
#define vend v5
#define dend d5
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
/* Core algorithm.
For each 16-byte chunk we calculate a 64-bit syndrome value with four bits
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
per byte. Bits 0-1 are set if the relevant byte matched the requested
character, bits 2-3 are set if the byte is NUL or matched. Count trailing
zeroes gives the position of the matching byte if it is a multiple of 4.
If it is not a multiple of 4, there was no match. */
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
ENTRY (strchr)
aarch64: use PTR_ARG and SIZE_ARG instead of DELOUSE DELOUSE was added to asm code to make them compatible with non-LP64 ABIs, but it is an unfortunate name and the code was not compatible with ABIs where pointer and size_t are different. Glibc currently only supports the LP64 ABI so these macros are not really needed or tested, but for now the name is changed to be more meaningful instead of removing them completely. Some DELOUSE macros were dropped: clone, strlen and strnlen used it unnecessarily. The out of tree ILP32 patches are currently not maintained and will likely need a rework to rebase them on top of the time64 changes.
2020-12-17 18:03:05 +08:00
PTR_ARG (0)
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
bic src, srcin, 15
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
dup vrepchr.16b, chrin
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
ld1 {vdata.16b}, [src]
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
movi vrepmask.16b, 0x33
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
cmeq vhas_nul.16b, vdata.16b, 0
cmeq vhas_chr.16b, vdata.16b, vrepchr.16b
bit vhas_nul.16b, vhas_chr.16b, vrepmask.16b
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
lsl tmp2, srcin, 2
shrn vend.8b, vhas_nul.8h, 4 /* 128->64 */
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
fmov tmp1, dend
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
lsr tmp1, tmp1, tmp2
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
cbz tmp1, L(loop)
rbit tmp1, tmp1
clz tmp1, tmp1
/* Tmp1 is an even multiple of 2 if the target character was
found first. Otherwise we've found the end of string. */
tst tmp1, 2
add result, srcin, tmp1, lsr 2
csel result, result, xzr, eq
ret
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
.p2align 4
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
L(loop):
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
ldr qdata, [src, 16]
cmeq vhas_chr.16b, vdata.16b, vrepchr.16b
cmhs vhas_nul.16b, vhas_chr.16b, vdata.16b
umaxp vend.16b, vhas_nul.16b, vhas_nul.16b
fmov tmp1, dend
cbnz tmp1, L(end)
ldr qdata, [src, 32]!
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
cmeq vhas_chr.16b, vdata.16b, vrepchr.16b
cmhs vhas_nul.16b, vhas_chr.16b, vdata.16b
umaxp vend.16b, vhas_nul.16b, vhas_nul.16b
fmov tmp1, dend
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
cbz tmp1, L(loop)
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
sub src, src, 16
L(end):
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
#ifdef __AARCH64EB__
bif vhas_nul.16b, vhas_chr.16b, vrepmask.16b
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
shrn vend.8b, vhas_nul.8h, 4 /* 128->64 */
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
fmov tmp1, dend
#else
bit vhas_nul.16b, vhas_chr.16b, vrepmask.16b
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
shrn vend.8b, vhas_nul.8h, 4 /* 128->64 */
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
fmov tmp1, dend
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
rbit tmp1, tmp1
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
#endif
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
add src, src, 16
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
clz tmp1, tmp1
AArch64: Optimize strchr Simplify calculation of the mask using shrn. Unroll the main loop. Small strings are 20% faster on modern CPUs. Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
2023-01-11 21:52:08 +08:00
/* Tmp1 is a multiple of 4 if the target character was found. */
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
tst tmp1, 2
add result, src, tmp1, lsr 2
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
csel result, result, xzr, eq
ret
aarch64: MTE compatible strchr Introduce an Arm MTE compatible strchr implementation. The existing implementation assumes that any access to the pages in which the string resides is safe. This assumption is not true when MTE is enabled. This patch updates the algorithm to ensure that accesses remain within the bounds of an MTE tag (16-byte chunks) and improves overall performance. Benchmarked on Cortex-A72, Cortex-A53, Neoverse N1. Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
2020-06-05 23:20:50 +08:00
[AArch64] Add optimized strchr. Implementation of strchr for AArch64. Speedups taken from micro-bench show the improvements relative to the standard C code. The use of LD1 means we have identical code for both big- and little-endian systems.
2014-06-19 17:40:49 +08:00
END (strchr)
libc_hidden_builtin_def (strchr)
weak_alias (strchr, index)
Reference in New Issue Copy Permalink