DoS if cluster-state-entity-count-query is sent too soon

[slfritchie]

Is this a bug, feature request, or feedback?

Bug

What is the current behavior?

This should never happen: failure in /home/vagrant/wallaroo/lib/wallaroo/core/initialization/local_topology.pony at line 1681

What is the expected behavior?

Query results (successful or unsuccessful) are sent to the client.

What OS and version of Wallaroo are you using?

Ubuntu Xenial/18.04 LTS with 2 vCPUs & 4GB RAM + aloc-tee-source branch @ commit de983d7

Steps to reproduce?

1. Start a TCP sink process, then start 2 worker cluster.
2. Immediately run external_sender/external_sender -e $WALLAROO_EXTERNAL -t cluster-state-entity-count-query.

Complete log from worker2:

1568045309.560555,****RESILIENCE MODE is active****
1568045309.560580,****CLUSTERING MODE is active****
1568045309.560585,****AUTOSCALE MODE is active****
1568045309.560588,****CHECKPOINT TRACE is active****
1568045309.560592,****TRACE is active****
1568045309.560595,****IDENTIFY ROUTING IDS is active****
1568045309.560599,****DEBUG is active****
1568045309.561435,||| Resilience directory: /tmp|||
1568045309.561448,||| Log-rotation: false|||
1568045309.561642,Set up external channel listener on 127.0.0.1:7103
1568045309.562149,Transition to _WaitingForCheckpointInitiationEventLogPhase: checkpoint_id 1
1568045309.562185,_create_control_connection: call from line 113
1568045309.562276,Saving CheckpointId 0 and RollbackId 0
1568045309.563380,AUTOSCALE: Waiting for new autoscale event.
1568045309.563536,worker2 external: couldn't listen
1568045309.563693,EventLog: Writing CheckpointId for CheckpointId 0
1568045309.563716,Transition to _WaitingForCheckpointInitiationEventLogPhase: checkpoint_id 1
1568045309.564501,worker2 control: listening on 127.0.0.1:7124
1568045309.564533,_create_control_connection: call from line 588
1568045309.564543,_create_control_connection: target_name initializer host 127.0.0.1 service 7101
1568045309.564852,ControlSenderConnectNotifier: connected to initializer at 127.0.0.1:7101.
1568045309.564908,Sent control message to initializer
1568045309.566959,ControlChannelConnectNotifier: server closed
1568045309.567001,ControlSenderConnectNotifier: connection to initializer at 127.0.0.1:7101  closed

Complete log from initializer:

1568045308.546546,****RESILIENCE MODE is active****
1568045308.546578,****CLUSTERING MODE is active****
1568045308.546582,****AUTOSCALE MODE is active****
1568045308.546586,****CHECKPOINT TRACE is active****
1568045308.546590,****TRACE is active****
1568045308.546593,****IDENTIFY ROUTING IDS is active****
1568045308.546596,****DEBUG is active****
1568045308.546999,||| Resilience directory: /tmp|||
1568045308.547011,||| Log-rotation: false|||
1568045308.547177,Set up external channel listener on 127.0.0.1:7103
1568045308.547184,Running as Initializer...
1568045308.547633,Transition to _WaitingForCheckpointInitiationEventLogPhase: checkpoint_id 1
1568045308.547786,Saving CheckpointId 0 and RollbackId 0
1568045308.548761,AUTOSCALE: Waiting for new autoscale event.
1568045308.548949,initializer external: listening on 127.0.0.1:7103
1568045308.549061,EventLog: Writing CheckpointId for CheckpointId 0
1568045308.549084,Transition to _WaitingForCheckpointInitiationEventLogPhase: checkpoint_id 1
1568045308.549792,initializer control: listening on 127.0.0.1:7101
1568045309.566102,Received msg on Control Channel
1568045309.566126,Received IdentifyControlPortMsg on Control Channel
1568045309.566139,_create_control_connection: call from control_channel_tcp line 217
1568045309.566179,_create_control_connection: call from line 588
1568045309.566185,_create_control_connection: target_name worker2 host 127.0.0.1 service 7124
1568045309.566234,Worker worker2 control channel identified
1568045309.566240,All worker control channels identified
1568045309.566275,Received msg on External Channel
1568045309.566294,Received ExternalClusterStateEntityCountQueryMsg on External Channel
This should never happen: failure in /home/vagrant/wallaroo/lib/wallaroo/core/initialization/local_topology.pony at line 1681