Validate siglen against modulus_len in PSS decode #807

Open

afonsojanu wants to merge 1 commit from afonsojanu/fix/pss-decode-siglen-check into develop

afonsojanu commented

2026-09-04 16:17:41 -05:00

(Migrated from github.com)

Fixes #802.

ltc_pkcs_1_pss_decode_mgf1 checks saltlen and modulus_len but never checks that siglen actually matches modulus_len. Later on it does:

XMEMCPY(DB, sig + x, modulus_len - hLen - 1);
x += modulus_len - hLen - 1;
XMEMCPY(hash, sig + x, hLen);

which copies modulus_len - 1 bytes total out of sig, based purely on modulus_len. If siglen < modulus_len, this reads past the end of the caller's buffer.

rsa_verify_hash_v2 happens to avoid triggering this today, because it always passes a buffer of exactly modulus_len (or modulus_len - 1 when modulus_bitlen % 8 == 1, handled via its own x - 1 adjustment) into the decode call. But ltc_pkcs_1_pss_decode_mgf1 is also reachable directly through the public (deprecated) pkcs_1_pss_decode() API, which forwards whatever siglen the caller passes with no validation at all — so a caller of that API supplying a genuinely truncated signature buffer hits the out-of-bounds read.

Fix: add siglen != modulus_len to the existing size-check if, matching the suggested fix in the issue.

Added a regression test in tests/pkcs_1_pss_test.c that calls pkcs_1_pss_decode() directly with a signature one byte shorter than the modulus and asserts it's rejected with CRYPT_PK_INVALID_SIZE rather than being processed. Verified with git-stash: reverting only the fix in pkcs_1_pss_decode.c makes the new assertion fail (on the very first PSS test vector) while everything else still passes; with the fix restored, the full suite passes (SUCCESS: passed=30 failed=0 nop=1).

Built and tested locally against LibTomMath (USE_LTM/LTM_DESC) on macOS/arm64.

Fixes #802. `ltc_pkcs_1_pss_decode_mgf1` checks `saltlen` and `modulus_len` but never checks that `siglen` actually matches `modulus_len`. Later on it does: ```c XMEMCPY(DB, sig + x, modulus_len - hLen - 1); x += modulus_len - hLen - 1; XMEMCPY(hash, sig + x, hLen); ``` which copies `modulus_len - 1` bytes total out of `sig`, based purely on `modulus_len`. If `siglen < modulus_len`, this reads past the end of the caller's buffer. `rsa_verify_hash_v2` happens to avoid triggering this today, because it always passes a buffer of exactly `modulus_len` (or `modulus_len - 1` when `modulus_bitlen % 8 == 1`, handled via its own `x - 1` adjustment) into the decode call. But `ltc_pkcs_1_pss_decode_mgf1` is also reachable directly through the public (deprecated) `pkcs_1_pss_decode()` API, which forwards whatever `siglen` the caller passes with no validation at all — so a caller of that API supplying a genuinely truncated signature buffer hits the out-of-bounds read. Fix: add `siglen != modulus_len` to the existing size-check `if`, matching the suggested fix in the issue. Added a regression test in `tests/pkcs_1_pss_test.c` that calls `pkcs_1_pss_decode()` directly with a signature one byte shorter than the modulus and asserts it's rejected with `CRYPT_PK_INVALID_SIZE` rather than being processed. Verified with git-stash: reverting only the fix in `pkcs_1_pss_decode.c` makes the new assertion fail (on the very first PSS test vector) while everything else still passes; with the fix restored, the full suite passes (`SUCCESS: passed=30 failed=0 nop=1`). Built and tested locally against LibTomMath (`USE_LTM`/`LTM_DESC`) on macOS/arm64.

This pull request is broken due to missing fork information.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin 'afonsojanu/fix/pss-decode-siglen-check:afonsojanu/fix/pss-decode-siglen-check'

git switch 'afonsojanu/fix/pss-decode-siglen-check'

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 'develop'

git merge --no-ff 'afonsojanu/fix/pss-decode-siglen-check'

git switch 'afonsojanu/fix/pss-decode-siglen-check'

git rebase 'develop'

git switch 'develop'

git merge --ff-only 'afonsojanu/fix/pss-decode-siglen-check'

git switch 'afonsojanu/fix/pss-decode-siglen-check'

git rebase 'develop'

git switch 'develop'

git merge --no-ff 'afonsojanu/fix/pss-decode-siglen-check'

git switch 'develop'

git merge --squash 'afonsojanu/fix/pss-decode-siglen-check'