gh-135532: optimize calls to PyMem_Malloc in SHAKE digest computation

[picnixz]

• Issue: Rewrite & cleanup HACL*-based extension modules #135532

[picnixz]

!buildbot FIPS only

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit de09602 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135744%2Fmerge

The command will test the builders whose names match following regular expression: FIPS only

The builders matched are:

• AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
• AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR

[picnixz]

!buildbot FIPS only

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 7852a46 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135744%2Fmerge

The command will test the builders whose names match following regular expression: FIPS only

The builders matched are:

• AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
• AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR

[picnixz]

I should have thought a bit more but I don't know why we have a hard limit at 2**29. It looks like we're reserving 4 bits for the output length, just in case, but we should be able to use a digest of arbitrary length. There was a possible of overflows in 3.6 but this was fixed, and since then, as we're using OpenSSL, we don't have that issue anymore. And HACL* only requires the length to be an uint32_t.

So, I think we should limit ourselves to uint32_t in both cases, even when using OpenSSL. It doesn't make sense to have a digest of size 2**32. SHAKE-128 is 128-bit secure with 32 bytes while SHAKE-256 is 256-bit secure with 64 bytes for the output length.

Now, for OpenSSL's default provider, we have:

int EVP_DigestFinalXOF(EVP_MD_CTX *ctx, unsigned char *md, size_t size)
{
    int ret = 0;
    OSSL_PARAM params[2];
    size_t i = 0;

    if (ctx->digest == NULL) {
        ERR_raise(ERR_LIB_EVP, EVP_R_INVALID_NULL_ALGORITHM);
        return 0;
    }

    if (ctx->digest->prov == NULL)
        goto legacy;

    if (ctx->digest->dfinal == NULL) {
        ERR_raise(ERR_LIB_EVP, EVP_R_FINAL_ERROR);
        return 0;
    }

    if ((ctx->flags & EVP_MD_CTX_FLAG_FINALISED) != 0) {
        ERR_raise(ERR_LIB_EVP, EVP_R_FINAL_ERROR);
        return 0;
    }

    /*
     * For backward compatibility we pass the XOFLEN via a param here so that
     * older providers can use the supplied value. Ideally we should have just
     * used the size passed into ctx->digest->dfinal().
     */
    params[i++] = OSSL_PARAM_construct_size_t(OSSL_DIGEST_PARAM_XOFLEN, &size);
    params[i++] = OSSL_PARAM_construct_end();

    if (EVP_MD_CTX_set_params(ctx, params) >= 0)
        ret = ctx->digest->dfinal(ctx->algctx, md, &size, size);

    ctx->flags |= EVP_MD_CTX_FLAG_FINALISED;

    return ret;

legacy:
    if (EVP_MD_xof(ctx->digest)
        && size <= INT_MAX
        && ctx->digest->md_ctrl(ctx, EVP_MD_CTRL_XOF_LEN, (int)size, NULL)) {
        ret = ctx->digest->final(ctx, md);
        if (ctx->digest->cleanup != NULL) {
            ctx->digest->cleanup(ctx);
            EVP_MD_CTX_set_flags(ctx, EVP_MD_CTX_FLAG_CLEANED);
        }
        OPENSSL_cleanse(ctx->md_data, ctx->digest->ctx_size);
    } else {
        ERR_raise(ERR_LIB_EVP, EVP_R_NOT_XOF_OR_INVALID_LENGTH);
    }

    return ret;
}

So with a NULL provider, we expect size <= INT_MAX, even if it fits on a SIZE_T. On 32-bit systems, it's (1<<31). So the limit of 2**29 is arbitrary IMO and should be replaced by UINT32_MAX.

• With openssl legacy provider 16-bit platform: not supported
• with openssl modern provider 32/64-bit platforms: size_t supported, should be at least a uint32_t for Python.
• with HACL*: only uint32_t is supported

Thus, we can just require the digest length to fit on an uint32_t.

[picnixz]

PR is conditioned to #135767.