lib: rsa: fix allocated size for rr and rrtmp in

A patch from »lib: rsa: distinguish between tpl and spl for« in state Mainline for u-boot

From: Heiko Stuebner <heiko.stuebner@...> Date: Wed, 20 May 2020 20:09:58 +0200

Commit-Message

When calculating rrtmp/rr rsa_gen_key_prop() tries to make (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] with rlen being num_bits * 2 On a 4096bit key this comes down to to 257 uint32_t elements in rr and 256 elements in rrtmp but with the current allocation rr and rrtmp only have 129 uint32_t elements. On 2048bit keys this works by chance as the defined max_rsa_size=4096 allocates a suitable number of elements, but with an actual 4096bit key this results in other memory parts getting overwritten. So as suggested by Heinrich Schuchardt just use the actual bis-size of the key as base for the size calculation, in turn making the code compatible to any future keysizes. Suggested-by: Heinrich Schuchardt <xypron.debian@...> Signed-off-by: Heiko Stuebner <heiko.stuebner@...>

Patch-Comment

changes in v6: - drop max_rsa_size and use the keysize as base changes in v4: - new patch lib/rsa/rsa-keyprop.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)

Statistics

  • 9 lines added
  • 5 lines removed

Changes

---------------------------- lib/rsa/rsa-keyprop.c -----------------------------
index 4b54db44c4..83b942615f 100644
@@ -654,14 +654,10 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop)
{
struct rsa_key rsa_key;
uint32_t *n = NULL, *rr = NULL, *rrtmp = NULL;
- const int max_rsa_size = 4096;
int rlen, i, ret;
*prop = calloc(sizeof(**prop), 1);
- n = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5));
- rr = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5));
- rrtmp = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5));
- if (!(*prop) || !n || !rr || !rrtmp) {
+ if (!(*prop)) {
ret = -ENOMEM;
goto err;
}
@@ -682,6 +678,14 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop)
}
memcpy((void *)(*prop)->modulus, &rsa_key.n[i], rsa_key.n_sz - i);
+ n = calloc(sizeof(uint32_t), 1 + ((*prop)->num_bits >> 5));
+ rr = calloc(sizeof(uint32_t), 1 + (((*prop)->num_bits * 2) >> 5));
+ rrtmp = calloc(sizeof(uint32_t), 1 + (((*prop)->num_bits * 2) >> 5));
+ if (!n || !rr || !rrtmp) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
/* exponent */
(*prop)->public_exponent = calloc(1, sizeof(uint64_t));
if (!(*prop)->public_exponent) {
 
 

Recent Patches

About Us

Sed lacus. Donec lectus. Nullam pretium nibh ut turpis. Nam bibendum. In nulla tortor, elementum vel, tempor at, varius non, purus. Mauris vitae nisl nec metus placerat consectetuer.

Read More...