Problems are detected but the test does not terminate

[MichaelRoeder]

When a problem is encountered, the test simply goes on. The following log was created by a test run without Docker containers:

2021-07-08 18:10:32,699 DEBUG [org.apache.cxf.common.logging.LogUtils] - <Using org.apache.cxf.common.logging.Log4jLogger for logging.>
2021-07-08 18:10:33,997 DEBUG [RabbitMqDockerizer] - <Creating container (imageName=rabbitmq:latest)>
2021-07-08 18:10:33,999 DEBUG [RabbitMqDockerizer] - <Attaching /tmp/session_1625760632151 volume to /share>
2021-07-08 18:10:34,227 DEBUG [RabbitMqDockerizer] - <Starting container (imageName=rabbitmq:latest, containerId=d10b229a533c)>
2021-07-08 18:10:34,227 DEBUG [RabbitMqDockerizer] - <Connecting container to networks (imageName=rabbitmq:latest)>
2021-07-08 18:10:38,215 DEBUG [RabbitMqDockerizer] - <Container started (imageName=rabbitmq:latest, containerId=d10b229a533c)>
2021-07-08 18:10:38,215 DEBUG [RabbitMqDockerizer] - <Starting monitoring & logs reading for container (imageName=rabbitmq:latest)>
2021-07-08 18:10:41,222 DEBUG [org.hobbit.sdk.utils.ComponentsExecutor] - <Processing CommandQueueListener>
2021-07-08 18:10:41,223 DEBUG [org.hobbit.sdk.utils.CommandQueueListener] - <Initializing...>
2021-07-08 18:10:41,269 WARN [org.hobbit.core.components.AbstractComponent] - <Couldn't connect to RabbitMQ with try #0. Next try in 5000ms.>
2021-07-08 18:10:46,334 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Couldn't get the id of this Docker container. Won't be able to create containers.>
2021-07-08 18:10:46,334 DEBUG [org.hobbit.sdk.utils.CommandQueueListener] - <Initialized. SessionID: session_1625760632151>
2021-07-08 18:10:46,351 DEBUG [org.hobbit.sdk.utils.commandreactions.ContainerCommandsReaction] - <CONTAINER_START command received with imageName=git.project-hobbit.eu:4567/ldcbench/ldcbench/ldcbench.lemming>
2021-07-08 18:10:46,352 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <This component will handle received commands in a single thread.>
2021-07-08 18:10:46,353 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <This component will handle received commands in a single thread.>
2021-07-08 18:10:46,354 DEBUG [org.hobbit.sdk.utils.ComponentsExecutor] - <Processing FileBasedRDFGraphGenerator>
2021-07-08 18:10:46,383 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Couldn't get the id of this Docker container. Won't be able to create containers.>
2021-07-08 18:10:46,384 ERROR [org.hobbit.sdk.utils.ComponentsExecutor] - <Error while running component FileBasedRDFGraphGenerator:>
java.lang.IllegalStateException: Couldn't find the expected variable HOBBIT_GENERATOR_ID from environment. Aborting.
	at org.hobbit.utils.EnvVariables.getVariableValue(EnvVariables.java:79)
	at org.hobbit.utils.EnvVariables.getIntValue(EnvVariables.java:561)
	at org.hobbit.utils.EnvVariables.getInt(EnvVariables.java:438)
	at org.hobbit.core.components.AbstractDataGenerator.init(AbstractDataGenerator.java:45)
	at org.dice_research.ldcbench.benchmark.DataGenerator.init(DataGenerator.java:275)
	at org.hobbit.sdk.utils.ComponentsExecutor.lambda$submit$0(ComponentsExecutor.java:79)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
2021-07-08 18:10:46,387 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <This component will handle received commands in a single thread.>
2021-07-08 18:10:46,396 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Couldn't get the id of this Docker container. Won't be able to create containers.>
2021-07-08 18:10:46,402 WARN [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Received a message with null correlationId. This is an error unless the other component uses an older version of HOBBIT core library.>
2021-07-08 18:10:46,402 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Correlating with the eldest request as a workaround.>
2021-07-08 18:10:46,414 DEBUG [org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction] - <Waiting ready signals for benchmarkController, dataGenerator, systemAdapter>
2021-07-08 18:10:46,414 DEBUG [org.hobbit.sdk.utils.commandreactions.ContainerCommandsReaction] - <CONTAINER_START command received with imageName=git.project-hobbit.eu:4567/ldcbench/ldcbench/ldcbench.system>
2021-07-08 18:10:46,414 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <This component will handle received commands in a single thread.>
2021-07-08 18:10:46,417 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <This component will handle received commands in a single thread.>
2021-07-08 18:10:46,418 DEBUG [org.hobbit.sdk.utils.ComponentsExecutor] - <Processing SystemAdapter>
2021-07-08 18:10:46,458 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Couldn't get the id of this Docker container. Won't be able to create containers.>
2021-07-08 18:10:46,459 WARN [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Received a message with null correlationId. This is an error unless the other component uses an older version of HOBBIT core library.>
2021-07-08 18:10:46,460 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Correlating with the eldest request as a workaround.>
2021-07-08 18:10:46,468 DEBUG [org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction] - <Waiting ready signals for benchmarkController, dataGenerator, systemAdapter>
2021-07-08 18:10:46,469 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Couldn't get the id of this Docker container. Won't be able to create containers.>
2021-07-08 18:10:46,471 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Couldn't get the id of this Docker container. Won't be able to create containers.>
2021-07-08 18:10:46,475 DEBUG [org.hobbit.sdk.utils.commandreactions.ContainerCommandsReaction] - <DOCKER_CONTAINER_TERMINATED ldcbench-lemming_0 received>
2021-07-08 18:10:46,475 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <This component will handle received commands in a single thread.>
2021-07-08 18:10:46,475 DEBUG [org.hobbit.sdk.utils.commandreactions.ContainerCommandsReaction] - <Sending BENCHMARK_FINISHED_SIGNAL signal>
2021-07-08 18:10:46,513 INFO [org.hobbit.core.components.AbstractCommandReceivingComponent] - <Couldn't get the id of this Docker container. Won't be able to create containers.>
2021-07-08 18:10:46,523 DEBUG [org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction] - <Waiting ready signals for benchmarkController, dataGenerator, systemAdapter>
2021-07-08 18:10:46,524 DEBUG [org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction] - <BENCHMARK_FINISHED_SIGNAL received>
2021-07-08 18:10:46,527 DEBUG [org.dice_research.ldcbench.system.SystemAdapter] - <Init()>
Exception in thread "pool-10-thread-1" java.lang.AssertionError
	at org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction.handleCmd(BenchmarkSignalsReaction.java:83)
	at org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction.handleCmd(BenchmarkSignalsReaction.java:19)
	at org.hobbit.sdk.utils.CommandQueueListener.receiveCommand(CommandQueueListener.java:96)
	at org.hobbit.core.components.AbstractCommandReceivingComponent.handleCmd(AbstractCommandReceivingComponent.java:266)
	at org.hobbit.sdk.utils.CommandQueueListener.handleCmd(CommandQueueListener.java:70)
	at org.hobbit.core.components.AbstractCommandReceivingComponent.handleCmd(AbstractCommandReceivingComponent.java:242)
	at org.hobbit.core.components.AbstractCommandReceivingComponent$1$1.run(AbstractCommandReceivingComponent.java:146)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
2021-07-08 18:10:46,529 DEBUG [org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction] - <SYSTEM_READY_SIGNAL signal received>
2021-07-08 18:10:46,529 DEBUG [org.hobbit.sdk.utils.commandreactions.BenchmarkSignalsReaction] - <Waiting ready signals for benchmarkController, dataGenerator>

It can be seen, that the AssertionError is simply ignored instead of reporting it further and terminating the test.

Apart from that, it is better to "log" errors using the logging framework instead of simply printing them to System.out (see https://github.com/hobbit-project/java-sdk/blob/master/src/main/java/org/hobbit/sdk/utils/commandreactions/BenchmarkSignalsReaction.java#L84)