Fix EVP_Cipher() for provided cipher implementations

EVP_Cipher() would return whatever ctx->cipher->ccipher() returned
with no regard for historical semantics.

We change this to first look if there is a ctx->cipher->ccipher(), and
in that case we treat the implementation as one with a custom cipher,
and "translate" it's return value like this: 0 => -1, 1 => outl, where
|outl| is the output length.

If there is no ctx->cipher->ccipher, we treat the implementation as
one without a custom cipher, call ctx->cipher->cupdate or
ctx->cipher->cfinal depending on input, and return whatever they
return (0 or 1).

Furthermore, we add a small hack in EVP_CIPHER_flags() to check if the
cipher is a provided one, and add EVP_CIPH_FLAG_CUSTOM_CIPHER to the
flags to be returned if there is a cipher->ccipher.  That way,
provided implementations never have to set that flag themselves, all
they need to do is to include a OSSL_FUNC_CIPHER_CIPHER function.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10137)

crypto/evp/evp_lib.c

@@ -298,15 +298,31 @@ int EVP_Cipher(EVP_CIPHER_CTX *ctx, unsigned char *out,
                const unsigned char *in, unsigned int inl)
 {
     if (ctx->cipher->prov != NULL) {
-        size_t outl = 0;         /* ignored */
-        int blocksize = EVP_CIPHER_CTX_block_size(ctx);
+        /*
+         * If the provided implementation has a ccipher function, we use it,
+         * and translate its return value like this: 0 => -1, 1 => outlen
+         *
+         * Otherwise, we call the cupdate function if in != NULL, or cfinal
+         * if in == NULL.  Regardless of which, we return what we got.
+         */
+        int ret = -1;
+        size_t outl = 0;
+        size_t blocksize = EVP_CIPHER_CTX_block_size(ctx);
 
         if (ctx->cipher->ccipher != NULL)
-            return
-                ctx->cipher->ccipher(ctx->provctx, out, &outl,
-                                     inl + (blocksize == 1 ? 0 : blocksize),
-                                     in, (size_t)inl);
-        return 0;
+            ret =  ctx->cipher->ccipher(ctx->provctx, out, &outl,
+                                        inl + (blocksize == 1 ? 0 : blocksize),
+                                        in, (size_t)inl)
+                ? (int)outl : -1;
+        else if (in != NULL)
+            ret = ctx->cipher->cupdate(ctx->provctx, out, &outl,
+                                       inl + (blocksize == 1 ? 0 : blocksize),
+                                       in, (size_t)inl);
+        else
+            ret = ctx->cipher->cfinal(ctx->provctx, out, &outl,
+                                      blocksize == 1 ? 0 : blocksize);
+
+        return ret;
     }
 
     return ctx->cipher->do_cipher(ctx, out, in, inl);
@@ -331,6 +347,10 @@ unsigned long EVP_CIPHER_flags(const EVP_CIPHER *cipher)
     params[0] = OSSL_PARAM_construct_ulong(OSSL_CIPHER_PARAM_FLAGS, &v);
     ok = evp_do_ciph_getparams(cipher, params);
 
+    /* Provided implementations may have a custom cipher_cipher */
+    if (cipher->prov != NULL && cipher->ccipher != NULL)
+        v |= EVP_CIPH_FLAG_CUSTOM_CIPHER;
+
     return ok != 0 ? v : 0;
 }
 

doc/man3/EVP_EncryptInit.pod

@@ -25,6 +25,7 @@ EVP_DecryptInit,
 EVP_DecryptFinal,
 EVP_CipherInit,
 EVP_CipherFinal,
+EVP_Cipher,
 EVP_get_cipherbyname,
 EVP_get_cipherbynid,
 EVP_get_cipherbyobj,
@@ -107,6 +108,9 @@ EVP_CIPHER_do_all_ex
                     const unsigned char *key, const unsigned char *iv, int enc);
  int EVP_CipherFinal(EVP_CIPHER_CTX *ctx, unsigned char *outm, int *outl);
 
+ int EVP_Cipher(EVP_CIPHER_CTX *ctx, unsigned char *out,
+                const unsigned char *in, unsigned int inl);
+
  int EVP_CIPHER_CTX_set_padding(EVP_CIPHER_CTX *x, int padding);
  int EVP_CIPHER_CTX_set_key_length(EVP_CIPHER_CTX *x, int keylen);
  int EVP_CIPHER_CTX_ctrl(EVP_CIPHER_CTX *ctx, int type, int arg, void *ptr);
@@ -251,6 +255,15 @@ EVP_CipherFinal_ex(). In previous releases they also cleaned up
 the B<ctx>, but this is no longer done and EVP_CIPHER_CTX_clean()
 must be called to free any context resources.
 
+EVP_Cipher() encrypts or decrypts a maximum I<inl> amount of bytes from
+I<in> and leaves the result in I<out>.
+If the cipher doesn't have the flag B<EVP_CIPH_FLAG_CUSTOM_CIPHER> set,
+then I<inl> must be a multiple of EVP_CIPHER_block_size().  If it isn't,
+the result is undefined.  If the cipher has that flag set, then I<inl>
+can be any size.
+This function is historic and shouldn't be used in an application, please
+consider using EVP_CipherUpdate() and EVP_CipherFinal_ex instead.
+
 EVP_get_cipherbyname(), EVP_get_cipherbynid() and EVP_get_cipherbyobj()
 return an EVP_CIPHER structure when passed a cipher name, a NID or an
 ASN1_OBJECT structure.
@@ -388,6 +401,11 @@ EVP_DecryptFinal_ex() returns 0 if the decrypt failed or 1 for success.
 EVP_CipherInit_ex() and EVP_CipherUpdate() return 1 for success and 0 for failure.
 EVP_CipherFinal_ex() returns 0 for a decryption failure or 1 for success.
 
+EVP_Cipher() returns the amount of encrypted / decrypted bytes, or -1
+on failure, if the flag B<EVP_CIPH_FLAG_CUSTOM_CIPHER> is set for the
+cipher.  EVP_Cipher() returns 1 on success or 0 on failure, if the flag
+B<EVP_CIPH_FLAG_CUSTOM_CIPHER> is not set for the cipher.
+
 EVP_CIPHER_CTX_reset() returns 1 for success and 0 for failure.
 
 EVP_get_cipherbyname(), EVP_get_cipherbynid() and EVP_get_cipherbyobj()