Intermittent Unit Test

[nguyer]

https://github.com/hyperledger/firefly/actions/runs/4723939292/jobs/8382686472?pr=1275

time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'identity'"
time="2023-04-17T19:13:40Z" level=info msg="Config file change detected. Event=WRITE Name=/tmp/TestConfigListenerE2E2574459936/001/firefly.core Size=0 Hash=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
time="2023-04-17T19:13:40Z" level=info msg="Detected configuration file reload"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'database'"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'identity'"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'blockchain'"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'sharedstorage'"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'dataexchange'"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'tokens'"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'auth'"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'predefined'"
time="2023-04-17T19:13:40Z" level=info msg="Requesting stop of namespace 'ns1'"
time="2023-04-17T19:13:40Z" level=info msg="Namespace 'ns1' stopped"
time="2023-04-17T19:13:40Z" level=info msg="Requesting stop of namespace 'ns2'"
time="2023-04-17T19:13:40Z" level=info msg="Namespace 'ns2' stopped"
time="2023-04-17T19:13:40Z" level=info msg="Initialized namespace 'ns1' multiparty=true version=0"
time="2023-04-17T19:13:40Z" level=info msg="Namespace ns1 started"
time="2023-04-17T19:13:40Z" level=info msg="WAITER: Namespace started 'ns1'"
time="2023-04-17T19:13:40Z" level=info msg="WAITER: Namespace started 'ns1'"
time="2023-04-17T19:13:40Z" level=info msg="WAITER: Namespace started 'ns1'"
time="2023-04-17T19:13:40Z" level=info msg="Initialized namespace 'ns2' multiparty=false version=n/a"
time="2023-04-17T19:13:40Z" level=info msg="Namespace ns2 started"
time="2023-04-17T19:13:40Z" level=info msg="WAITER: Namespace started 'ns2'"
time="2023-04-17T19:13:40Z" level=info msg="WAITER: Namespace started 'ns2'"
time="2023-04-17T19:13:40Z" level=info msg="WAITER: Namespace started 'ns2'"
time="2023-04-17T19:13:40Z" level=info msg="Config file change detected. Event=WRITE Name=/tmp/TestConfigListenerE2E2574459936/001/firefly.core Size=3150 Hash=a8dbdf99e09c0ce9449269c8b0d5ee33801fb6d2591e1f6c95459ca346b0970b"
time="2023-04-17T19:13:40Z" level=info msg="Detected configuration file reload"
time="2023-04-17T19:13:40Z" level=error msg="Invalid object value '<nil>' for key 'identity'"
time="2023-04-17T19:13:40Z" level=info msg="Initialized namespace 'ns2' multiparty=false version=n/a"
time="2023-04-17T19:13:40Z" level=info msg="Namespace ns2 started"
time="2023-04-17T19:13:40Z" level=info msg="Initialized namespace 'ns3' multiparty=false version=n/a"
time="2023-04-17T19:13:40Z" level=info msg="Namespace ns3 started"
time="2023-04-17T19:13:40Z" level=info msg="Initialized namespace 'ns1' multiparty=true version=2"
time="2023-04-17T19:13:40Z" level=info msg="Namespace ns1 started"
panic: 
assert: mock: The method has been called over 1 times.
	Either do one more Mock.On("NamespaceRestarted").Return(...), or remove extra call.
	This call was unexpected:
		NamespaceRestarted(string,time.Time)
		0: "ns2"
		1: time.Date(2023, time.April, 17, 19, 13, 40, 93781890, time.Local)
	at: [/home/runner/work/firefly/firefly/internal/namespace/plugin.go:89 /home/runner/work/firefly/firefly/internal/namespace/manager.go:314 /home/runner/work/firefly/firefly/internal/namespace/retry.go:56 /home/runner/work/firefly/firefly/internal/namespace/manager.go:301 /home/runner/work/firefly/firefly/internal/namespace/asm_amd64.s:1571]

goroutine 50 [running]:
github.com/stretchr/testify/mock.(*Mock).fail(0xc000138500, {0xf112dc?, 0x8?}, {0xc0003fab00?, 0x2?, 0x2?})
	/home/runner/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:322 +0x145
github.com/stretchr/testify/mock.(*Mock).MethodCalled(0xc000138500, {0x114bd12, 0x12}, {0xc000161340, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:466 +0x2a5
github.com/stretchr/testify/mock.(*Mock).Called(0xeac700?, {0xc000161340, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:450 +0x148
github.com/hyperledger/firefly/mocks/eventsmocks.(*Plugin).NamespaceRestarted(0xc000138500, {0xc0004ed5b4, 0x3}, {0x0?, 0xc000343e08?, 0x1584de0?})
	/home/runner/work/firefly/firefly/mocks/eventsmocks/plugin.go:89 +0xd2
github.com/hyperledger/firefly/internal/namespace.(*namespaceManager).namespaceStarter.func1(0x7fd99ebe6108?)
	/home/runner/work/firefly/firefly/internal/namespace/manager.go:314 +0x2dd
github.com/hyperledger/firefly-common/pkg/retry.(*Retry).Do(0xc0001aada0, {0x1027a50, 0xc00022bbc0}, {0xc0004ec000, 0xd}, 0xc000343f98)
	/home/runner/go/pkg/mod/github.com/hyperledger/[email protected]/pkg/retry/retry.go:56 +0xe4
github.com/hyperledger/firefly/internal/namespace.(*namespaceManager).namespaceStarter(0xc00031c1c0, 0xc0001ec9a0)
	/home/runner/work/firefly/firefly/internal/namespace/manager.go:301 +0xd0
created by github.com/hyperledger/firefly/internal/namespace.(*namespaceManager).startNamespacesAndPlugins
	/home/runner/work/firefly/firefly/internal/namespace/manager.go:406 +0x8e

[nguyer]

I had this same test fail twice in a row, then it passed the third time. I have been unable to reproduce the failure locally on my machine. I have only observed it failing in GitHub actions.