runtime: getRandomData slows Go programs down on Plan 9

[0intro]

We experienced the Go programs running on a Plan 9
machine on GCE to be irregularly slow.

For example:

% go test -v -run TestGoNil
=== RUN TestGoNil
--- PASS: TestGoNil (0.31s)
PASS
ok   runtime 0.397s
% go test -v -run TestGoNil
=== RUN TestGoNil
--- PASS: TestGoNil (5.27s)
PASS
ok   runtime 7.004s

With Go 1.4, the whole all.rc took more than 90 minutes
to complete.

After profiling the Go programs, we noticed that
sometimes, they spent much more time reading.

Fast:

% iostats go test -v -run TestGoNil
=== RUN TestGoNil
--- PASS: TestGoNil (0.41s)
PASS
ok      runtime 0.501s

read      23176119 bytes, 154556.8 Kb/sec
write     7041578 bytes, 49864.14 Kb/sec
protocol  31405437 bytes, 28192.42 Kb/sec
rpc       24803 count

Slow:

% iostats go test -v -run TestGoNil
=== RUN TestGoNil
--- PASS: TestGoNil (2.25s)
PASS
ok      runtime 2.332s

read      23172023 bytes, 11346.04 Kb/sec
write     7041588 bytes, 52504.14 Kb/sec
protocol  31401489 bytes, 10594.57 Kb/sec
rpc       24806 count

We figured out that extra time was spent reading /dev/random.

The Go runtime use random data to initialize hash table
keys and reduce collision. Every Go process starts by
acquiring random data from the operating system.
On Plan 9, it reads the /dev/random file, which provides
a stream of random numbers generated by the kernel.

https://github.com/golang/go/blob/d5e4c4/src/runtime/alg.go#L320

The generation of random numbers in the Plan 9 kernel
is trivial. A kernel process is running a loop incrementing
a variable. After each clock tick, the value of the variable
is read and appended in the random circular buffer.

https://github.com/0intro/plan9/blob/11e8c8/sys/src/9/port/random.c#L61

Generating random bytes this way is slow, but for some
reason, it's much slower on GCE than on QEMU or
real hardware.

Since every Go process is starting by reading 128 bytes
from /dev/random, the buffer regularly runs out of random
data, and the processes are blocking on read. This is
amplified by the fact that now, the Go compiler itself is
also a Go program.

We should probably not use /dev/random to initialize the
Go hash table keys on Plan 9. It's obviously too slow for
this use.

As a workaround, I wrote a simple user-space file system,
sitting on top of /dev/random, but providing pseudo-random
numbers, initialized from the kernel random source. Thus,
it is able to generate much more numbers per second.

This way, on GCE, the Go programs now run 3x to 9x
faster than before.

Now, on GCE, all.rc takes 10 minutes to complete with Go 1.4
and 15 minutes with the current master branch.

[randall77]

We shouldn't ever block waiting for random data for map hash initialization. It's a nice-to-have, but not critical. See https://go-review.googlesource.com/#/c/2582/

All of our ports read from /dev/urandom or equivalent, except plan9. Does plan9 have a /dev/urandom equivalent? If not, you might just want to do what nacl does, which is just use the clock. see os1_nacl.go.

[0intro]

Unfortunately, there is no equivalent to /dev/urandom on Plan 9.
However, the user space /dev/random file system I wrote does a
similar job.

We might want to do the same thing as nacl.