Ard Biesheuvel 27c539aeff crypto: algapi - annotate expected branch behavior in crypto_inc() ...

To prevent unnecessary branching, mark the exit condition of the
primary loop as likely(), given that a carry in a 32-bit counter
occurs very rarely.

On arm64, the resulting code is emitted by GCC as

     9a8:   cmp     w1, #0x3
     9ac:   add     x3, x0, w1, uxtw
     9b0:   b.ls    9e0 <crypto_inc+0x38>
     9b4:   ldr     w2, [x3,#-4]!
     9b8:   rev     w2, w2
     9bc:   add     w2, w2, #0x1
     9c0:   rev     w4, w2
     9c4:   str     w4, [x3]
     9c8:   cbz     w2, 9d0 <crypto_inc+0x28>
     9cc:   ret

where the two remaining branch conditions (one for size < 4 and one for
the carry) are statically predicted as non-taken, resulting in optimal
execution in the vast majority of cases.

Also, replace the open coded alignment test with IS_ALIGNED().

Cc: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

2017-03-09 18:34:17 +08:00

..

asymmetric_keys

Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

2016-12-15 11:41:37 -08:00

async_tx

async_pq_val: fix DMA memory leak

2016-10-05 06:18:09 +05:30

.gitignore

crypto: rsa - add .gitignore for crypto/*.-asn1.[ch] files

2015-06-25 23:29:24 +08:00

842.c

crypto: acomp - add support for 842 via scomp

2016-10-25 11:08:33 +08:00

ablk_helper.c

crypto: ablk_helper - Fix cryptd reordering

2016-06-23 18:29:53 +08:00

ablkcipher.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

acompress.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

aead.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

aes_generic.c

crypto: aes-generic - drop alignment requirement

2017-02-11 17:50:43 +08:00

aes_ti.c

crypto: aes - add generic time invariant AES cipher

2017-02-11 17:50:43 +08:00

af_alg.c

crypto: af_alg - Forbid bind(2) when nokey child sockets are present

2016-01-18 18:16:33 +08:00

ahash.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

akcipher.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

algapi.c

crypto: algapi - annotate expected branch behavior in crypto_inc()

2017-03-09 18:34:17 +08:00

algboss.c

sched/headers: Prepare to move signal wakeup & sigpending methods from <linux/sched.h> into <linux/sched/signal.h>

2017-03-02 08:42:32 +01:00

algif_aead.c

sched/headers: Prepare to move signal wakeup & sigpending methods from <linux/sched.h> into <linux/sched/signal.h>

2017-03-02 08:42:32 +01:00

algif_hash.c

crypto: algif_hash - avoid zero-sized array

2016-12-27 17:50:52 +08:00

algif_rng.c

crypto: algif_rng - Remove obsolete const-removal cast

2015-04-22 09:30:21 +08:00

algif_skcipher.c

sched/headers: Prepare to move signal wakeup & sigpending methods from <linux/sched.h> into <linux/sched/signal.h>

2017-03-02 08:42:32 +01:00

ansi_cprng.c

crypto: ansi_cprng - Convert to new rng interface

2015-04-22 09:30:18 +08:00

anubis.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

api.c

sched/headers: Prepare to move signal wakeup & sigpending methods from <linux/sched.h> into <linux/sched/signal.h>

2017-03-02 08:42:32 +01:00

arc4.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

authenc.c

crypto: skcipher - Get rid of crypto_spawn_skcipher2()

2016-11-01 08:37:17 +08:00

authencesn.c

crypto: skcipher - Get rid of crypto_spawn_skcipher2()

2016-11-01 08:37:17 +08:00

blkcipher.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

blowfish_common.c

crypto: blowfish - split generic and common c code

2011-09-22 21:25:25 +10:00

blowfish_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

camellia_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

cast5_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

cast6_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

cast_common.c

crypto: make tables used from assembler __visible

2013-08-14 20:42:03 +10:00

cbc.c

crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic

2017-02-11 17:52:28 +08:00

ccm.c

crypto: ccm - move cbcmac input off the stack

2017-02-28 17:29:17 +08:00

chacha20_generic.c

crypto: chacha20 - convert generic and x86 versions to skcipher

2016-12-27 17:47:31 +08:00

chacha20poly1305.c

crypto: skcipher - Get rid of crypto_spawn_skcipher2()

2016-11-01 08:37:17 +08:00

cipher.c

crypto: api - Remove no-op exit_ops code

2016-10-21 11:03:42 +08:00

cmac.c

crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic

2017-02-11 17:52:28 +08:00

compress.c

crypto: api - Remove no-op exit_ops code

2016-10-21 11:03:42 +08:00

crc32_generic.c

crypto: crc32 - Rename generic implementation

2016-01-30 22:11:22 +08:00

crc32c_generic.c

crypto: crc32c - Fix crc32c soft dependency

2016-01-19 15:52:10 +08:00

crct10dif_common.c

crypto: crct10dif - Add fallback for broken initrds

2013-09-12 15:31:34 +10:00

crct10dif_generic.c

crypto: squash lines for simple wrapper functions

2016-09-13 20:27:26 +08:00

cryptd.c

crypto: cryptd - Add support for skcipher

2016-11-28 21:23:18 +08:00

crypto_engine.c

sched/headers: Prepare for new header dependencies before moving code to <uapi/linux/sched/types.h>

2017-03-02 08:42:27 +01:00

crypto_null.c

crypto: null - Remove default null blkcipher

2016-07-18 17:35:44 +08:00

crypto_user.c

crypto: acomp - add asynchronous compression api

2016-10-25 11:08:30 +08:00

crypto_wq.c

crypto: crypto_wq - Fix late crypto work queue initialization

2014-03-21 21:54:28 +08:00

ctr.c

crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic

2017-02-11 17:52:28 +08:00

cts.c

crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic

2017-02-11 17:52:28 +08:00

deflate.c

crypto: acomp - add support for deflate via scomp

2016-10-25 11:08:36 +08:00

des_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

dh_helper.c

crypto: dh - Add DH software implementation

2016-06-23 18:29:56 +08:00

dh.c

crypto: dh - Consistenly return negative error codes

2016-11-13 17:45:04 +08:00

drbg.c

Merge git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

2016-11-30 19:53:12 +08:00

ecb.c

crypto: include crypto- module prefix in template

2014-11-26 20:06:30 +08:00

ecc_curve_defs.h

crypto: ecdh - Add ECDH software support

2016-06-23 18:29:57 +08:00

ecc.c

crypto: ecdh - make ecdh_shared_secret unique

2016-06-24 21:24:59 +08:00

ecc.h

crypto: ecdh - make ecdh_shared_secret unique

2016-06-24 21:24:59 +08:00

ecdh_helper.c

crypto: ecdh - Add ECDH software support

2016-06-23 18:29:57 +08:00

ecdh.c

crypto: ecdh - make ecdh_shared_secret unique

2016-06-24 21:24:59 +08:00

echainiv.c

crypto: echainiv - Replace chaining with multiplication

2016-09-13 18:44:57 +08:00

fcrypt.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

fips.c

crypto: fips - Move fips_enabled sysctl into fips.c

2015-04-23 14:18:09 +08:00

gcm.c

crypto: skcipher - Get rid of crypto_spawn_skcipher2()

2016-11-01 08:37:17 +08:00

gf128mul.c

crypto: gf128mul - constify 4k and 64k multiplication tables

2017-03-09 18:34:16 +08:00

ghash-generic.c

crypto: ghash-generic - move common definitions to a new header file

2016-10-02 22:26:40 +08:00

hash_info.c

keys, trusted: select hash algorithm for TPM2 chips

2015-12-20 15:27:12 +02:00

hmac.c

crypto: include crypto- module prefix in template

2014-11-26 20:06:30 +08:00

internal.h

crypto: api - Remove no-op exit_ops code

2016-10-21 11:03:42 +08:00

jitterentropy-kcapi.c

crypto: jitterentropy - drop duplicate header module.h

2016-11-17 23:34:52 +08:00

jitterentropy.c

crypto: jitterentropy - Delete unnecessary checks before the function call "kzfree"

2015-06-25 23:18:33 +08:00

Kconfig

crypto: xts - Add ECB dependency

2017-02-23 20:11:06 +08:00

keywrap.c

crypto: keywrap - memzero the correct memory

2016-02-01 22:27:05 +08:00

khazad.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

kpp.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

lrw.c

crypto: lrw - Convert to skcipher

2016-11-28 21:23:17 +08:00

lz4.c

crypto: change LZ4 modules to work with new LZ4 module version

2017-02-24 17:46:57 -08:00

lz4hc.c

crypto: change LZ4 modules to work with new LZ4 module version

2017-02-24 17:46:57 -08:00

lzo.c

crypto: acomp - add support for lzo via scomp

2016-10-25 11:08:31 +08:00

Makefile

crypto: improve gcc optimization flags for serpent and wp512

2017-02-11 17:52:26 +08:00

mcryptd.c

sched/headers: Prepare for new header dependencies before moving code to <linux/sched/stat.h>

2017-03-02 08:42:34 +01:00

md4.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

md5.c

crypto: hash - add zero length message hash for shax and md5

2015-12-22 20:43:35 +08:00

memneq.c

crypto: memneq - fix for archs without efficient unaligned access

2013-12-09 20:09:12 +08:00

michael_mic.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

pcbc.c

crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic

2017-02-11 17:52:28 +08:00

pcrypt.c

crypto: aead - Remove CRYPTO_ALG_AEAD_NEW flag

2015-08-17 16:53:53 +08:00

poly1305_generic.c

crypto: poly1305 - Use unaligned access where required

2016-11-13 17:45:03 +08:00

proc.c

crypto: fips - Move fips_enabled sysctl into fips.c

2015-04-23 14:18:09 +08:00

ripemd.h

[CRYPTO] ripemd: Put all common RIPEMD values in header file

2008-07-10 20:35:12 +08:00

rmd128.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

rmd160.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

rmd256.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

rmd320.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

rng.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

rsa_helper.c

crypto: rsa - allow keys >= 2048 bits in FIPS mode

2016-08-24 21:07:10 +08:00

rsa-pkcs1pad.c

crypto: rsa-pkcs1pad - Handle leading zero for decryption

2016-09-22 17:42:08 +08:00

rsa.c

crypto: rsa - Generate fixed-length output

2016-07-01 23:45:18 +08:00

rsaprivkey.asn1

crypto: rsa - Store rest of the private key components

2016-07-05 23:05:26 +08:00

rsapubkey.asn1

crypto: akcipher - Changes to asymmetric key API

2015-10-14 22:23:16 +08:00

salsa20_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

scatterwalk.c

crypto: scatterwalk - Remove unnecessary aliasing check in map_and_copy

2016-11-22 15:02:25 +08:00

scompress.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

seed.c

crypto: prefix module autoloading with "crypto-"

2014-11-24 22:43:57 +08:00

seqiv.c

crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic

2017-02-11 17:52:28 +08:00

serpent_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

sha1_generic.c

crypto: hash - add zero length message hash for shax and md5

2015-12-22 20:43:35 +08:00

sha3_generic.c

crypto: sha3 - Add missing ULL suffixes for 64-bit constants

2016-08-08 23:43:46 +08:00

sha256_generic.c

crypto: hash - add zero length message hash for shax and md5

2015-12-22 20:43:35 +08:00

sha512_generic.c

crypto: sha512-generic - move to generic glue implementation

2015-04-10 21:39:41 +08:00

shash.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

simd.c

crypto: simd - Add simd skcipher helper

2016-11-28 21:23:18 +08:00

skcipher.c

crypto: Replaced gcc specific attributes with macros from compiler.h

2017-01-13 00:24:39 +08:00

tcrypt.c

crypto: tcrypt - Add debug prints

2017-01-23 22:50:24 +08:00

tcrypt.h

crypto: tcrypt - Add ChaCha20/Poly1305 speed tests

2015-07-17 21:20:20 +08:00

tea.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

testmgr.c

crypto: testmgr - add test cases for cbcmac(aes)

2017-02-11 17:50:44 +08:00

testmgr.h

Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

2017-03-04 10:42:53 -08:00

tgr192.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

twofish_common.c

crypto: twofish-x86_64-3way - add lrw support

2011-11-09 11:53:32 +08:00

twofish_generic.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

vmac.c

crypto: include crypto- module prefix in template

2014-11-26 20:06:30 +08:00

wp512.c

crypto: add missing crypto module aliases

2015-01-13 22:29:11 +11:00

xcbc.c

crypto: include crypto- module prefix in template

2014-11-26 20:06:30 +08:00

xor.c

crypto: xor - Fix warning when XOR_SELECT_TEMPLATE is unset

2016-08-31 23:00:48 +08:00

xts.c

crypto: xts - Propagate NEED_FALLBACK bit

2017-02-27 18:09:41 +08:00