Avoid overflow in EVP_EncodeUpdate

An overflow can occur in the EVP_EncodeUpdate function which is used for
Base64 encoding of binary data. If an attacker is able to supply very large
amounts of input data then a length check can overflow resulting in a heap
corruption. Due to the very large amounts of data involved this will most
likely result in a crash.

Internally to OpenSSL the EVP_EncodeUpdate function is primarly used by the
PEM_write_bio* family of functions. These are mainly used within the
OpenSSL command line applications, so any application which processes
data from an untrusted source and outputs it as a PEM file should be
considered vulnerable to this issue.

User applications that call these APIs directly with large amounts of
untrusted data may also be vulnerable.

Issue reported by Guido Vranken.

CVE-2016-2105

Reviewed-by: Richard Levitte <levitte@openssl.org>

crypto/evp/encode.c

@@ -171,7 +171,7 @@ void EVP_EncodeUpdate(EVP_ENCODE_CTX *ctx, unsigned char *out, int *outl,
     if (inl <= 0)
         return;
     OPENSSL_assert(ctx->length <= (int)sizeof(ctx->enc_data));
-    if ((ctx->num + inl) < ctx->length) {
+    if (ctx->length - ctx->num > inl) {
         memcpy(&(ctx->enc_data[ctx->num]), in, inl);
         ctx->num += inl;
         return;