Extend (& deprecate) lib_random #4

ghost · 2017-11-23T11:31:09Z

Work to date - please review it is what we want.
(history should be squashed)

…moment

…simple implementation that will produce random if the attacker does not have physical access to the chip

…uld probably be removed and replaced with ordinary EXPLORER-200.

deprecated random_create_generator_from_hw_seed() making it's build conditional (add 'RANDOM_ENABLE_HW_SEED=1' to the Makefile)

Make random_prng_server(...,client interface random_pool ?rpi,...) optinal

samchesney

Hi @robert-lytton, apologies for the delay in giving feedback on this pull request. I like the way the implementation is layered from bit to pool to prng, and have added a few questions and comments against particular parts of the source.

We have a few concrete use cases for this library as both a RNG and an entropy source now, and it looks like a common feature of them both is to have access to C functions with a prototype similar to the standard C int rand(void) function. I think that we should be able to provide that via a simple wrapper to the pool and prng APIs of this library, do you see any problem with that?
I will need to confirm that we will not need a CSPRNG for this, but I think not. Can the the values from the bit and pool APIs be considered cryptographically secure?
Did you have an implementation in mind for a CSPRNG? Would it have a very similar API to the PRNG added here?

Are the functions in random.h provided an a very low compute/fast PRNG alternative to random_prng.h?

Cheers,
Sam

samchesney · 2018-03-02T20:22:20Z

lib_random/api/random_bit.h

+#include <stdint.h>
+#include <stddef.h>
+
+#ifndef REFERENCE_PARAM


Is there a reason for defining REFERENCE_PARAM here (and in random.h) rather than using the macro from xccompat.h?

no,
xccompat.h should be included

random.h has now been removed - use clib alternatives.

samchesney · 2018-03-02T20:43:18Z

lib_random/src/random_bit.xc

+  timer tmr;
+  tmr :> last_time;
+  STORE32(last_time, &per_tile_last_time);
+  setps(0x60B, 2);


We could replace the magic numbers in the calls to getps() and setps() with the defines in xs2a_registers.h if this library is to support only XS2. Unfortunately the XS1 ring oscillator defines in xs1l_registers.h have a slightly different prefix, otherwise we could just include xs1_registers.h and have it work for both architectures...

the decision/roadmap/work to fix target headers has been on hold since Mr Fyles blocked it.
The single character fix is somewhere in bugzilla...

samchesney · 2018-03-02T20:47:38Z

lib_random/src/random_bit.xc

+  tmr :> time;
+  // If the timer wraps, we will miss the opportunity - tough!
+  // N.B. unsigned wrapping has defined behaviour.
+  if (time - last_time > TIME_FOR_ONE_BIT) {


Why do we delay reading the ring oscillator after it is stopped?

Ask henk,
I assume if we don't, the probability field is skewed.

comment added

ghost · 2018-08-13T10:35:08Z

I think that we should be able to provide that via a simple wrapper to the pool and prng APIs of this library, do you see any problem with that?

No, apart from extra code.
It may be possible to reduce the footprint for distinct use cases.

ghost · 2018-08-13T10:44:31Z

I will need to confirm that we will not need a CSPRNG for this

The PRNG is revealing 32bits out of either 57, 88 or 113 bits, the other bits remain hidden (unless the user connects a debugger).

It is up to the user to decide:
how many hidden bits are required;
how much entropy the seeds must start with;
how often entropy needs to be injected into the state;

I believe this is best left as an explicit implementation detail.

ghost · 2018-08-13T10:47:59Z

Are the functions in random.h ...

This is deprecated backward comparability.
Initially it attempted as adding entropy to the initial seed.
It now offers little over the version in stdlib.h viz rand(), srand(), drand48() etc.

Ensure CI system treats it as an xCORE application, and attempts a build.

Add Jenkins pipeline

fix magic numbers

ghost · 2018-08-15T11:42:57Z

I think that we should be able to provide that via a simple wrapper to the pool and prng APIs of this library, do you see any problem with that?

Simple wrapper?
The problem is the creation of interfaces and placement of the interfaced tasks - along with the passing of these parameters.

It would be nice if the XC language allowed collections of select-case statements to be built up and dynamically dispatched opaquely.
However, in the current language the list is statically presented at compiler time (select statement) and dynamically dispatched via guard statement etc.

Until that day arrives, I am not sure of the best way to mix C and XC, along with adding the necessary top level knitting, especially in an easy to use library.
May be we should leave this as an implementors business until a clear idiom (or XC v3) becomes available.

samchesney · 2018-08-29T15:02:38Z

I think that we should be able to provide that via a simple wrapper to the pool and prng APIs of this library, do you see any problem with that?

Simple wrapper?
The problem is the creation of interfaces and placement of the interfaced tasks - along with the passing of these parameters.

Starting the lib_random server as a task in xC is not an issue. However, we'll need C compatible API to access the client side, which we can use to set defines when building 3rd party libraries. These 3rd party libraries are configured as follows.

Kernel config:

#define configRAND32()    ulRand()

IP stack config:

#define ipconfigRAND32()    ulRand()

PKCS11 config:

/* Random number generation */

/* C_SeedRandom mixes additional seed material into the token's
 * random number generator. */
CK_PKCS11_FUNCTION_INFO(C_SeedRandom)
#ifdef CK_NEED_ARG_LIST
(
  CK_SESSION_HANDLE hSession,  /* the session's handle */
  CK_BYTE_PTR       pSeed,     /* the seed material */
  CK_ULONG          ulSeedLen  /* length of seed material */
);
#endif


/* C_GenerateRandom generates random data. */
CK_PKCS11_FUNCTION_INFO(C_GenerateRandom)
#ifdef CK_NEED_ARG_LIST
(
  CK_SESSION_HANDLE hSession,    /* the session's handle */
  CK_BYTE_PTR       RandomData,  /* receives the random data */
  CK_ULONG          ulRandomLen  /* # of bytes to generate */
);
#endif

TLS library config:

#define MBEDTLS_ENTROPY_HARDWARE_ALT

int mbedtls_hardware_poll(void * data, unsigned char * output, size_t len, size_t * olen) {
...
}

The following mbedTLS documentation looks useful:

ghost · 2018-08-29T17:09:50Z

Erm....
I'll read the documentation and see if I can work out how it works.

ghost · 2018-08-30T07:03:06Z

However, we'll need C compatible API to access the client side, which we can use to set defines when building 3rd party libraries. These 3rd party libraries are configured as follows.

This looks like an adapter is needed into a particular 3rd party library.
Such a specific adapter is not appropriate for a library - although having a general purpose C API that can be used by the adapter and others may be useful... but this is speculation.

I'll write suitable code and you can decide if it lives in the libraries 'examples', the libraries API or alongside the client (mbed) code.

ghost · 2018-08-30T07:17:23Z

#define configRAND32() ulRand()
#define ipconfigRAND32() ulRand()

These could use the stdlib.h rand() function.
As they will probably expose information to the outside world, they should not use the same seed as the tls - even if they share the same library code viz via a C wrapped random_prng.h.

ghost · 2018-08-30T07:21:08Z

p.s. here is the 'good enough' prng behind rand().... good enough for Donald good enough for me:

__extension__ static __thread unsigned long long _rand_next = 1;
void srand(unsigned int seed) {
  _rand_next = seed;
}
int rand(void) {
  /* This multiplier was obtained from Knuth, D.E., "The Art of
     Computer Programming," Vol 2, Seminumerical Algorithms, Third
     Edition, Addison-Wesley, 1998, p. 106 (line 26) & p. 108 */
  _rand_next =_rand_next * __extension__ 6364136223846793005LL + 1;
  return (int)((_rand_next >> 32) & RAND_MAX);
}

ghost · 2018-08-30T07:36:46Z

I've taken a 'quick' look at the cryptsoft API:
https://www.cryptsoft.com/pkcs11doc/v220/group__SEC__11__15__RANDOM__NUMBER__GENERATION__FUNCTIONS.html

This is generating random numbers for you, I believe.
So I assume all that is needed is the seed:
pSeed, /* the seed material */
This could be got from the random_pool.h.
I have no idea how, when & how often C_SeedRandom() is called to add entropy...
p.s. It could call random_bit.h directly, rather than having random_pool.h running too)

ghost · 2018-08-30T07:46:23Z

The TLS library looks like it need hooks into the entropy.
https://tls.mbed.org/api/entropy__poll_8h.html#a45ce4792a68304d592fb711bd8f2fc86
This is probably best done by implementing specific code for the TLS that sits ontop of the randome_bit.h API (I assume it keeps it's own pool)

ghost · 2018-08-30T07:47:26Z

Please do comment if I am incorrect regarding how these components work, or if the application architecture needs additional components that require something else.

samchesney · 2018-08-30T14:10:25Z

#define configRAND32() ulRand()
#define ipconfigRAND32() ulRand()

These could use the stdlib.h rand() function.
As they will probably expose information to the outside world, they should not use the same seed as the tls - even if they share the same library code viz via a C wrapped random_prng.h.

Understood, thank you. I've created https://github.com/xmos/amazon_freertos_experiment/issues/29 to track this.

samchesney · 2018-08-30T16:20:06Z

I've taken a 'quick' look at the cryptsoft API:
https://www.cryptsoft.com/pkcs11doc/v220/group__SEC__11__15__RANDOM__NUMBER__GENERATION__FUNCTIONS.html

This is generating random numbers for you, I believe.
So I assume all that is needed is the seed:
pSeed, /* the seed material */
This could be got from the random_pool.h.
I have no idea how, when & how often C_SeedRandom() is called to add entropy...
p.s. It could call random_bit.h directly, rather than having random_pool.h running too)

If we use random_bit() for this would it mean we were always passing a seed with of 0b00000000 or 0b00000001 and setting ulSeedLen to 1?

ghost · 2018-08-30T16:30:36Z

If we use random_bit() for this would it mean we were always passing a seed with of 0b00000000 or 0b00000001 and setting ulSeedLen to 1?

I have not looked at the implementation, but yes, that looks like one way of doing it.
Another way would be to have a timer interrupt gathering bits at a rate of 1bit/20000ticks (see #define TIME_FOR_ONE_BIT 20000) and then when it had 32bits it could send them all.
I have not looked at the expectations of the implementation so would not know what is better/more efficient.

It should be noted that random_pool.h could be used to gather the bits, but it seems too heavy weight an API (and it is written in XC). Thus implementing a really simple pooling function seems sensible in this case.

ghost · 2018-08-30T16:40:56Z

N.B.
If the 'cryptsoft' and 'mbed tls' libraries are running on differnt cores, they can each have their own random_bit.h source of entropy.

If they are running on the same core, they will need to share the bits co-operatively to prevent starvation or a race condition - the library is not logical-core safe by design! The random_bit_claim() is there to make this sharing safe.

samchesney · 2018-08-31T11:03:02Z

I found this documentation on how to port mbed TLS. So I agree, we can start off using the random_bit API for this too (and hopefully find that it performs well enough polling for a byte of entropy at a time in mbedtls_hardware_poll().

samchesney · 2018-08-31T11:03:50Z

I think all of the abov means that in actual fact this pull request is almost ready to merge?

samchesney · 2018-08-31T16:25:44Z

I've taken a 'quick' look at the cryptsoft API:
https://www.cryptsoft.com/pkcs11doc/v220/group__SEC__11__15__RANDOM__NUMBER__GENERATION__FUNCTIONS.html

This is generating random numbers for you, I believe.

Looking into this further, I'm not sure if I'm missing something. I thought this was specifying an API, but not providing an implementation?
I think we need to provide a matching implementation for C_GenerateRandom().

samchesney · 2018-09-17T09:54:53Z

examples/app_dieharder/src/gen.xc

+      interface random_prng prngi;
+      par {
+        [[distribute]]
+        random_prng_server(prngi, null, prng57, null);


@robert-lytton should the third argument passed be prng rather than prng57, or have I misunderstood?

samchesney · 2018-09-17T09:59:53Z

lib_random/api/random_prng.h

+
+/** Method that returns a pseudo-random number.
+ *
+ *  ====== WARNING ====================================================


@robert-lytton does this warning still hold true?
I'm trying to ensure I have correctly understood which applications this PRNG can be safely used for, but I'm struggling to reconcile the warning in the source with the comments in this review:

I will need to confirm that we will not need a CSPRNG for this

The PRNG is revealing 32bits out of either 57, 88 or 113 bits, the other bits remain hidden (unless the user connects a debugger).

It is up to the user to decide:
how many hidden bits are required;
how much entropy the seeds must start with;
how often entropy needs to be injected into the state;

I believe this is best left as an explicit implementation detail.

henkmuller and others added 7 commits October 20, 2017 12:07

First effort at extracting true random bits - horribly broken at the …

0fa8410

…moment

First version of true random that does something that appears alrightish

1bab776

First version of true random that does something that appears alrightish

07c4ed0

TOok all the clever bits out as they reduced randomness. Left with a …

4307eef

…simple implementation that will produce random if the attacker does not have physical access to the chip

Added board file on which random number generator was developed - sho…

456de38

…uld probably be removed and replaced with ordinary EXPLORER-200.

Added random_bit, random_pool, random_prng interfaces

3a6a84d

deprecated random_create_generator_from_hw_seed() making it's build conditional (add 'RANDOM_ENABLE_HW_SEED=1' to the Makefile)

Add examples/dieharder for validating random_prng.h

f6237d0

Make random_prng_server(...,client interface random_pool ?rpi,...) optinal

samchesney self-assigned this Nov 24, 2017

samchesney reviewed Mar 3, 2018

View reviewed changes

samchesney and others added 4 commits August 15, 2018 11:35

Add Jenkins pipeline

3862bd6

Rename dieharder -> app_dieharder

7b5e08d

Ensure CI system treats it as an xCORE application, and attempts a build.

Merge pull request #1 from samchesney/jenkins

9bfb1bc

Add Jenkins pipeline

Remove examples/AN002xx_... and random.h api

7fedf48

fix magic numbers

samchesney assigned samchesney and ghost and unassigned samchesney Aug 15, 2018

samchesney reviewed Sep 17, 2018

View reviewed changes

samchesney removed their assignment Oct 11, 2020

mbanth added the type:enhancement New feature or request label Apr 20, 2021

mbanth modified the milestone: Publish Before Resolution Apr 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend (& deprecate) lib_random #4

Extend (& deprecate) lib_random #4

ghost commented Nov 23, 2017

samchesney left a comment

samchesney Mar 2, 2018

ghost Aug 13, 2018

ghost Aug 15, 2018

samchesney Mar 2, 2018

ghost Aug 13, 2018

ghost Aug 13, 2018

ghost Aug 15, 2018

samchesney Mar 2, 2018

ghost Aug 13, 2018

ghost Aug 15, 2018

ghost commented Aug 13, 2018 •

edited by samchesney

Loading

ghost commented Aug 13, 2018

ghost commented Aug 13, 2018

ghost commented Aug 15, 2018

samchesney commented Aug 29, 2018

ghost commented Aug 29, 2018

ghost commented Aug 30, 2018

ghost commented Aug 30, 2018 •

edited by samchesney

Loading

ghost commented Aug 30, 2018

ghost commented Aug 30, 2018 •

edited by ghost

Loading

ghost commented Aug 30, 2018

ghost commented Aug 30, 2018

samchesney commented Aug 30, 2018

samchesney commented Aug 30, 2018

ghost commented Aug 30, 2018 •

edited by ghost

Loading

ghost commented Aug 30, 2018

samchesney commented Aug 31, 2018

samchesney commented Aug 31, 2018

samchesney commented Aug 31, 2018

samchesney Sep 17, 2018

samchesney Sep 17, 2018

Extend (& deprecate) lib_random #4

Are you sure you want to change the base?

Extend (& deprecate) lib_random #4

Conversation

ghost commented Nov 23, 2017

samchesney left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Aug 13, 2018 • edited by samchesney Loading

ghost commented Aug 13, 2018

ghost commented Aug 13, 2018

ghost commented Aug 15, 2018

samchesney commented Aug 29, 2018

ghost commented Aug 29, 2018

ghost commented Aug 30, 2018

ghost commented Aug 30, 2018 • edited by samchesney Loading

ghost commented Aug 30, 2018

ghost commented Aug 30, 2018 • edited by ghost Loading

ghost commented Aug 30, 2018

ghost commented Aug 30, 2018

samchesney commented Aug 30, 2018

samchesney commented Aug 30, 2018

ghost commented Aug 30, 2018 • edited by ghost Loading

ghost commented Aug 30, 2018

samchesney commented Aug 31, 2018

samchesney commented Aug 31, 2018

samchesney commented Aug 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Aug 13, 2018 •

edited by samchesney

Loading

ghost commented Aug 30, 2018 •

edited by samchesney

Loading

ghost commented Aug 30, 2018 •

edited by ghost

Loading

ghost commented Aug 30, 2018 •

edited by ghost

Loading