AVX512-specific heap buffer overflow with 3.0.4 release

[xry111]

Build OpenSSL-3.0.4 on a CPU with AVX512 (my CPU is a Core i7-1065G7) with:

CFLAGS="-O3 -g -fsanitize=address" ./config
make

Run a test:

make V=1 TESTS=test_exp test

The sanitizer complains:

==481618==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60c000089400 at pc 0x7f01e32a9509 bp 0x7fff643ec100 sp 0x7fff643ec0f8
READ of size 8 at 0x60c000089400 thread T0
    #0 0x7f01e32a9508 in bn_select_words crypto/bn/rsaz_exp.h:64
    #1 0x7f01e32a9508 in bn_reduce_once_in_place crypto/bn/rsaz_exp.h:74
    #2 0x7f01e32a9508 in ossl_rsaz_mod_exp_avx512_x2 crypto/bn/rsaz_exp_x2.c:223
    #3 0x7f01e3287dc8 in BN_mod_exp_mont_consttime_x2 crypto/bn/bn_exp.c:1448
    #4 0x4042c3 in test_mod_exp_x2 test/exptest.c:260
    #5 0x40611a in run_tests test/testutil/driver.c:370
    #6 0x4039ba in main test/testutil/main.c:30
    #7 0x7f01e2c29319 in __libc_start_call_main (/usr/lib/libc.so.6+0x29319)
    #8 0x7f01e2c293e4 in __libc_start_main_impl (/usr/lib/libc.so.6+0x293e4)
    #9 0x403c40 in _start (/home/xry111/sources/lfs/openssl-3.0.4/test/exptest+0x403c40)

0x60c000089400 is located 0 bytes to the right of 128-byte region [0x60c000089380,0x60c000089400)
allocated by thread T0 here:
    #0 0x7f01e3ae5107 in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f01e34aa7a8 in CRYPTO_zalloc crypto/mem.c:197

SUMMARY: AddressSanitizer: heap-buffer-overflow crypto/bn/rsaz_exp.h:64 in bn_select_words
Shadow bytes around the buggy address:
  0x0c1880009230: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c1880009240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1880009250: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1880009260: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c1880009270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c1880009280:[fa]fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1880009290: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c18800092a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c18800092b0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c18800092c0: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c18800092d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==481618==ABORTING

[xry111]

Bisect shows 10d8a10 may be the culprit.

[xry111]

Also reproduciable with master branch. Reverting 6d702ce can "fix" the issue.

[xry111]

Note that this is not just a sanitizer-related issue: I'm pretty sure it will blow up on any CPU with AVX-512, with or without sanitizer. But w/o sanitizer the error message is just an unhelpful invalid free (caused by heap corruption) so I used sanitizer in the report.

[alex]

We (pyca/cryptography) have started receiving bug reports of heap-corruption with our latest release (which includes 3.0.4) that I assume have this as a root cause.