Because `keylen` is unsigned the subtraction results in an integer
underflow if the key length is < 11 bytes.
This is only a problem when verifying signatures with a public key (for
private keys the plugin enforces a minimum modulus length) and to do so
we usually only use trusted keys. However, the x509 plugin actually
calls issued_by() on a parsed certificate to check if it is self-signed,
which is the reason this issue was found by OSS-Fuzz in the first place.
So, unfortunately, this can be triggered by sending an invalid client
cert to a peer.
Fixes: 5955db5b12 ("gmp: Don't parse PKCS1 v1.5 RSA signatures to verify them")
Fixes: CVE-2018-17540
Tobias Brunner
129ab919a8