Wycheproof failures #745

New issue

Open

opened 2026-04-29 04:19:19 -05:00 by karel-m · 3 comments

karel-m commented

2026-04-29 04:19:19 -05:00

(Migrated from github.com)

In Project Wycheproof there are currently 340 JSON files with test vectors.

I have:

used current https://github.com/C2SP/wycheproof
executed 65883 vectors (with libtomcrypt develop branch)
skipped 78428 vectors (mostly the given algo not implemented in libtomcrypt)

From the 65883 executed tests, we have 308 unique failed tcIds (0.5%):

  JSON file                            Fail/Total
  ------------------------------------ ------------
  aes_cbc_pkcs5_test.json                 3/216
  camellia_cbc_pkcs5_test.json            3/216
  ec_prime_order_curves_test.json         1/26
  ecdh_brainpoolP224r1_test.json          5/787
  ecdh_brainpoolP256r1_test.json          5/804
  ecdh_brainpoolP320r1_test.json          3/771
  ecdh_brainpoolP384r1_test.json          5/936
  ecdh_brainpoolP512r1_test.json          5/794
  ecdh_secp224r1_pem_test.json            3/714
  ecdh_secp224r1_test.json                3/714
  ecdh_secp256k1_test.json                3/752
  ecdh_secp256r1_ecpoint_test.json        9/355
  ecdh_secp256r1_pem_test.json           12/612
  ecdh_secp256r1_test.json               12/612
  ecdh_secp256r1_webcrypto_test.json      9/353
  ecdh_secp384r1_ecpoint_test.json       13/790
  ecdh_secp384r1_pem_test.json           16/1047
  ecdh_secp384r1_test.json               16/1047
  ecdh_secp384r1_webcrypto_test.json     13/794
  ecdh_secp521r1_ecpoint_test.json       12/661
  ecdh_secp521r1_pem_test.json           15/916
  ecdh_secp521r1_test.json               15/916
  ecdh_secp521r1_webcrypto_test.json     12/655
  ed448_test.json                         3/86
  x448_asn_test.json                     28/529
  x448_jwk_test.json                     28/523
  x448_pem_test.json                     28/510
  x448_test.json                         28/510

Do not take the numbers as exact; they may vary slightly, as I may have done some simplifications.

For now just this information, I will try to submit separate PRs to fix some of them.

In Project Wycheproof there are currently 340 JSON files with test vectors. I have: * used current https://github.com/C2SP/wycheproof * executed 65883 vectors (with libtomcrypt develop branch) * skipped 78428 vectors (mostly the given algo not implemented in libtomcrypt) From the 65883 executed tests, we have 308 unique failed tcIds (0.5%): ``` JSON file Fail/Total ------------------------------------ ------------ aes_cbc_pkcs5_test.json 3/216 camellia_cbc_pkcs5_test.json 3/216 ec_prime_order_curves_test.json 1/26 ecdh_brainpoolP224r1_test.json 5/787 ecdh_brainpoolP256r1_test.json 5/804 ecdh_brainpoolP320r1_test.json 3/771 ecdh_brainpoolP384r1_test.json 5/936 ecdh_brainpoolP512r1_test.json 5/794 ecdh_secp224r1_pem_test.json 3/714 ecdh_secp224r1_test.json 3/714 ecdh_secp256k1_test.json 3/752 ecdh_secp256r1_ecpoint_test.json 9/355 ecdh_secp256r1_pem_test.json 12/612 ecdh_secp256r1_test.json 12/612 ecdh_secp256r1_webcrypto_test.json 9/353 ecdh_secp384r1_ecpoint_test.json 13/790 ecdh_secp384r1_pem_test.json 16/1047 ecdh_secp384r1_test.json 16/1047 ecdh_secp384r1_webcrypto_test.json 13/794 ecdh_secp521r1_ecpoint_test.json 12/661 ecdh_secp521r1_pem_test.json 15/916 ecdh_secp521r1_test.json 15/916 ecdh_secp521r1_webcrypto_test.json 12/655 ed448_test.json 3/86 x448_asn_test.json 28/529 x448_jwk_test.json 28/523 x448_pem_test.json 28/510 x448_test.json 28/510 ``` Do not take the numbers as exact; they may vary slightly, as I may have done some simplifications. For now just this information, I will try to submit separate PRs to fix some of them.

❤️ 1

sjaeckel commented

2026-04-29 08:14:51 -05:00

(Migrated from github.com)

Would it maybe make sense to create a single PR with all the fixes as separate commits and link it to this issue?

karel-m commented

2026-04-29 08:36:21 -05:00

(Migrated from github.com)

not sure, the needed fixes are very different

karel-m commented

2026-04-29 15:21:22 -05:00

(Migrated from github.com)

With these fixes:

I am able to run wycheproof tests with these results:

testcases total=144311
- unsupported=77789 (not run, skipped)
- run=66522
- failed = 0 (which is great)
- acceptable.rejected=3159 (this is good)
- acceptable.tolerated=2163 (this is not that good but tolerated)

Just FYI skipped tests are:

# wycheproof2 unsupported:    475  unsupported AEAD algorithm AEGIS128
# wycheproof2 unsupported:    479  unsupported AEAD algorithm AEGIS128L
# wycheproof2 unsupported:    472  unsupported AEAD algorithm AEGIS256
# wycheproof2 unsupported:    202  unsupported AEAD algorithm AES-GCM-SIV
# wycheproof2 unsupported:    552  unsupported AEAD algorithm ARIA-CCM
# wycheproof2 unsupported:    311  unsupported AEAD algorithm ARIA-GCM
# wycheproof2 unsupported:    192  unsupported AEAD algorithm ASCON128
# wycheproof2 unsupported:    192  unsupported AEAD algorithm ASCON128A
# wycheproof2 unsupported:    192  unsupported AEAD algorithm ASCON80PQ
# wycheproof2 unsupported:    916  unsupported AEAD algorithm MORUS1280
# wycheproof2 unsupported:    458  unsupported AEAD algorithm MORUS640
# wycheproof2 unsupported:    267  unsupported ECDH curve sect283k1
# wycheproof2 unsupported:    260  unsupported ECDH curve sect283r1
# wycheproof2 unsupported:    265  unsupported ECDH curve sect409k1
# wycheproof2 unsupported:    258  unsupported ECDH curve sect409r1
# wycheproof2 unsupported:    267  unsupported ECDH curve sect571k1
# wycheproof2 unsupported:    257  unsupported ECDH curve sect571r1
# wycheproof2 unsupported:   2189  unsupported ECDSA hash SHAKE128
# wycheproof2 unsupported:   2562  unsupported ECDSA hash SHAKE256
# wycheproof2 unsupported:    123  unsupported IND-CPA algorithm AES-XTS
# wycheproof2 unsupported:    216  unsupported IND-CPA algorithm ARIA-CBC-PKCS5
# wycheproof2 unsupported:    311  unsupported MAC algorithm ARIA-CMAC
# wycheproof2 unsupported:    174  unsupported MAC algorithm HMACSM3
# wycheproof2 unsupported:    174  unsupported MAC algorithm KMAC128
# wycheproof2 unsupported:    261  unsupported MAC algorithm KMAC256
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHash-1-3
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHash-2-4
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHash-4-8
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHashX-2-4
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHashX-4-8
# wycheproof2 unsupported:   1528  unsupported MAC-with-IV algorithm VMAC-AES
# wycheproof2 unsupported:    780  unsupported RSA PSS hash
# wycheproof2 unsupported:    110  unsupported RSA private key
# wycheproof2 unsupported:   1091  unsupported RSA public key
# wycheproof2 unsupported:     19  unsupported schema bls_aggregate_verify_schema.json
# wycheproof2 unsupported:     34  unsupported schema bls_hash_to_g2_schema.json
# wycheproof2 unsupported:    114  unsupported schema bls_sig_verify_schema.json
# wycheproof2 unsupported:  31567  unsupported schema fpe_list_test_schema.json
# wycheproof2 unsupported:  25679  unsupported schema fpe_str_test_schema.json
# wycheproof2 unsupported:     83  unsupported schema json_web_crypto_schema_v1.json
# wycheproof2 unsupported:    139  unsupported schema json_web_encryption_schema_v1.json
# wycheproof2 unsupported:    401  unsupported schema json_web_signature_schema_v1.json
# wycheproof2 unsupported:   1024  unsupported schema keywrap_test_schema_v1.json
# wycheproof2 unsupported:    217  unsupported schema mldsa_sign_noseed_schema.json
# wycheproof2 unsupported:    281  unsupported schema mldsa_sign_seed_schema.json
# wycheproof2 unsupported:    485  unsupported schema mldsa_verify_schema.json
# wycheproof2 unsupported:    795  unsupported schema mlkem_encaps_test_schema.json
# wycheproof2 unsupported:    300  unsupported schema mlkem_keygen_seed_test_schema.json
# wycheproof2 unsupported:     21  unsupported schema mlkem_semi_expanded_decaps_test_schema.json
# wycheproof2 unsupported:    579  unsupported schema mlkem_test_schema.json
# wycheproof2 unsupported:    317  unsupported schema primality_test_schema_v1.json

Some algorithms (SM3, ML-KEM and ML-DSA) are waiting as pull requests.

Missing algorithms are clear, but there are some cases where we are pretty close, especially RSA and ECC combined with SHAKE128/256. We do support SHAKE128/256, but not in the form of an ltc_hash_descriptor that can be passed to e.g. rsa_sign_hash_ex. Maybe two simple helpers sha3_shake128_init + sha3_shake256_init (usable for ltc_hash_descriptor) based on sha3_shake_init would do the trick. I am not sure whether it is worth it.

With these fixes: - #750 - #749 - #748 - #747 - #746 I am able to run wycheproof tests with these results: ``` testcases total=144311 - unsupported=77789 (not run, skipped) - run=66522 - failed = 0 (which is great) - acceptable.rejected=3159 (this is good) - acceptable.tolerated=2163 (this is not that good but tolerated) ``` Just FYI skipped tests are: ``` # wycheproof2 unsupported: 475 unsupported AEAD algorithm AEGIS128 # wycheproof2 unsupported: 479 unsupported AEAD algorithm AEGIS128L # wycheproof2 unsupported: 472 unsupported AEAD algorithm AEGIS256 # wycheproof2 unsupported: 202 unsupported AEAD algorithm AES-GCM-SIV # wycheproof2 unsupported: 552 unsupported AEAD algorithm ARIA-CCM # wycheproof2 unsupported: 311 unsupported AEAD algorithm ARIA-GCM # wycheproof2 unsupported: 192 unsupported AEAD algorithm ASCON128 # wycheproof2 unsupported: 192 unsupported AEAD algorithm ASCON128A # wycheproof2 unsupported: 192 unsupported AEAD algorithm ASCON80PQ # wycheproof2 unsupported: 916 unsupported AEAD algorithm MORUS1280 # wycheproof2 unsupported: 458 unsupported AEAD algorithm MORUS640 # wycheproof2 unsupported: 267 unsupported ECDH curve sect283k1 # wycheproof2 unsupported: 260 unsupported ECDH curve sect283r1 # wycheproof2 unsupported: 265 unsupported ECDH curve sect409k1 # wycheproof2 unsupported: 258 unsupported ECDH curve sect409r1 # wycheproof2 unsupported: 267 unsupported ECDH curve sect571k1 # wycheproof2 unsupported: 257 unsupported ECDH curve sect571r1 # wycheproof2 unsupported: 2189 unsupported ECDSA hash SHAKE128 # wycheproof2 unsupported: 2562 unsupported ECDSA hash SHAKE256 # wycheproof2 unsupported: 123 unsupported IND-CPA algorithm AES-XTS # wycheproof2 unsupported: 216 unsupported IND-CPA algorithm ARIA-CBC-PKCS5 # wycheproof2 unsupported: 311 unsupported MAC algorithm ARIA-CMAC # wycheproof2 unsupported: 174 unsupported MAC algorithm HMACSM3 # wycheproof2 unsupported: 174 unsupported MAC algorithm KMAC128 # wycheproof2 unsupported: 261 unsupported MAC algorithm KMAC256 # wycheproof2 unsupported: 40 unsupported MAC algorithm SipHash-1-3 # wycheproof2 unsupported: 40 unsupported MAC algorithm SipHash-2-4 # wycheproof2 unsupported: 40 unsupported MAC algorithm SipHash-4-8 # wycheproof2 unsupported: 40 unsupported MAC algorithm SipHashX-2-4 # wycheproof2 unsupported: 40 unsupported MAC algorithm SipHashX-4-8 # wycheproof2 unsupported: 1528 unsupported MAC-with-IV algorithm VMAC-AES # wycheproof2 unsupported: 780 unsupported RSA PSS hash # wycheproof2 unsupported: 110 unsupported RSA private key # wycheproof2 unsupported: 1091 unsupported RSA public key # wycheproof2 unsupported: 19 unsupported schema bls_aggregate_verify_schema.json # wycheproof2 unsupported: 34 unsupported schema bls_hash_to_g2_schema.json # wycheproof2 unsupported: 114 unsupported schema bls_sig_verify_schema.json # wycheproof2 unsupported: 31567 unsupported schema fpe_list_test_schema.json # wycheproof2 unsupported: 25679 unsupported schema fpe_str_test_schema.json # wycheproof2 unsupported: 83 unsupported schema json_web_crypto_schema_v1.json # wycheproof2 unsupported: 139 unsupported schema json_web_encryption_schema_v1.json # wycheproof2 unsupported: 401 unsupported schema json_web_signature_schema_v1.json # wycheproof2 unsupported: 1024 unsupported schema keywrap_test_schema_v1.json # wycheproof2 unsupported: 217 unsupported schema mldsa_sign_noseed_schema.json # wycheproof2 unsupported: 281 unsupported schema mldsa_sign_seed_schema.json # wycheproof2 unsupported: 485 unsupported schema mldsa_verify_schema.json # wycheproof2 unsupported: 795 unsupported schema mlkem_encaps_test_schema.json # wycheproof2 unsupported: 300 unsupported schema mlkem_keygen_seed_test_schema.json # wycheproof2 unsupported: 21 unsupported schema mlkem_semi_expanded_decaps_test_schema.json # wycheproof2 unsupported: 579 unsupported schema mlkem_test_schema.json # wycheproof2 unsupported: 317 unsupported schema primality_test_schema_v1.json ``` Some algorithms (SM3, ML-KEM and ML-DSA) are waiting as pull requests. Missing algorithms are clear, but there are some cases where we are pretty close, especially RSA and ECC combined with SHAKE128/256. We do support SHAKE128/256, but not in the form of an `ltc_hash_descriptor` that can be passed to e.g. `rsa_sign_hash_ex`. Maybe two simple helpers `sha3_shake128_init` + `sha3_shake256_init` (usable for `ltc_hash_descriptor`) based on `sha3_shake_init` would do the trick. I am not sure whether it is worth it.