Intermittent build failure: fatal error: concurrent map read and map write (Viper config map)

[peterbroadhurst]

[2021-10-20T14:22:37.089Z]  INFO Project Firefly pid=9892
[2021-10-20T14:22:37.089Z]  INFO © Copyright 2021 Kaleido, Inc. pid=9892
[2021-10-20T14:22:37.095Z]  INFO Shutting down due to interrupt pid=9892
[2021-10-20T14:22:37.095Z]  INFO api listening on HTTP [::]:42831 pid=9892
fatal error: concurrent map read and map write

goroutine 34 [running]:
runtime.throw(0xe67d73, 0x21)
	/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/panic.go:1117 +0x72 fp=0xc0000ada10 sp=0xc0000ad9e0 pc=0x4412f2
runtime.mapaccess2_faststr(0xd65d60, 0xc000501680, 0xc0001b8f60, 0x4, 0x14469e0, 0xc000501600)
	/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/map_faststr.go:116 +0x4a5 fp=0xc0000ada80 sp=0xc0000ada10 pc=0x41d2e5
github.com/spf13/viper.(*Viper).searchMap(0xc000539380, 0xc000501680, 0xc0004c39e0, 0x1, 0x3, 0x14469e0, 0x0)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/viper.go:568 +0x5f fp=0xc0000adac8 sp=0xc0000ada80 pc=0x939edf
github.com/spf13/viper.(*Viper).isPathShadowedInDeepMap(0xc000539380, 0xc0004c39e0, 0x3, 0x3, 0xc000501680, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/viper.go:704 +0x7f fp=0xc0000adb18 sp=0xc0000adac8 pc=0x93a85f
github.com/spf13/viper.(*Viper).find(0xc000539380, 0xc0001b8f60, 0xf, 0x1, 0xc0000adc88, 0x2)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/viper.go:1222 +0xafd fp=0xc0000adc28 sp=0xc0000adb18 pc=0x93c6dd
github.com/spf13/viper.(*Viper).Get(0xc000539380, 0xc0001b8f50, 0xf, 0x0, 0x1444aa0)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/viper.go:798 +0x85 fp=0xc0000adcc0 sp=0xc0000adc28 pc=0x93aca5
github.com/spf13/viper.(*Viper).GetString(0xc000539380, 0xc0001b8f50, 0xf, 0xc0001b8f50, 0xf)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/viper.go:865 +0x3f fp=0xc0000add00 sp=0xc0000adcc0 pc=0x93b4bf
github.com/spf13/viper.GetString(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/viper.go:862
github.com/hyperledger/firefly/internal/config.(*configPrefix).GetString(0xc0002c0ad0, 0xe5180a, 0xa, 0xc0000add00, 0x416a1b)
	/home/runner/work/firefly/firefly/internal/config/config.go:527 +0x65 fp=0xc0000add38 sp=0xc0000add00 pc=0x949125
github.com/hyperledger/firefly/internal/apiserver.(*httpServer).createServer(0xc000390300, 0xf72178, 0xc0003a50c0, 0xc0001ca540, 0xc0004c4168, 0x0, 0x0, 0xc0001b8430)
	/home/runner/work/firefly/firefly/internal/apiserver/http_server.go:128 +0x97 fp=0xc0000ade18 sp=0xc0000add38 pc=0xb54797
github.com/hyperledger/firefly/internal/apiserver.newHTTPServer(0xf72178, 0xc0003a50c0, 0xe48d6b, 0x3, 0xc0001ca540, 0xc000370e40, 0xf7d318, 0xc0002c0ad0, 0xc0003a50c0, 0xd6d900, ...)
	/home/runner/work/firefly/firefly/internal/apiserver/http_server.go:103 +0x28a fp=0xc0000adea0 sp=0xc0000ade18 pc=0xb5418a
github.com/hyperledger/firefly/internal/apiserver.(*apiServer).Serve(0xc0003d6660, 0xf72178, 0xc0003a50c0, 0xf80bb0, 0xc00037e910, 0x0, 0x1)
	/home/runner/work/firefly/firefly/internal/apiserver/server.go:87 +0x2a5 fp=0xc0000adf18 sp=0xc0000adea0 pc=0xb57aa5
github.com/hyperledger/firefly/cmd.startFirefly(0xf72178, 0xc0003a50c0, 0xc0003dc410, 0xf80bb0, 0xc00037e910, 0xf60de0, 0xc0003d6660, 0xc000370cc0)
	/home/runner/work/firefly/firefly/cmd/firefly.go:163 +0x176 fp=0xc0000adfa0 sp=0xc0000adf18 pc=0xbf53f6
runtime.goexit()
	/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000adfa8 sp=0xc0000adfa0 pc=0x47ad01
created by github.com/hyperledger/firefly/cmd.run
	/home/runner/work/firefly/firefly/cmd/firefly.go:118 +0x4c5

Seen in #268

[awrichar]

Possibly related discussion here: spf13/viper#268

Viper seems to have no locking around Get/Set calls, and it's quite possible we're calling them concurrently.