DLM.DropChecker 'disappeared' condition causing possible graph execution race condition

[myxie]

Issue

Note: This is reproducible, but succeeds intermittently, making it difficult to determine what is the exact root cause across multiple sessions.

I was testing if I can get the a translated SubGraph running with the DIM/NM, prior to updating the node dropclass. To do so, I've used a basic SleepApp as the InputApplication for the SubGraph construct and a CopyApp as the OutputApplication.

In running the graph locally (which translates, partitions, and runs the 'main graph' fine), it has revealed a race condition where sometimes the middle File data drop (see below) is marked as disappeared, even though it has yet to be created by the SleepApp drop:

This leads to the drop being removed from the DLM, and any future attempt to interact with it causes the rest of the graph to fail due to a missing drop.uid key:

  File "/home/rwb/github/daliuge-icrar/daliuge-engine/dlg/lifecycle/dlm.py", line 475, in handleOpenedDrop
    drop = self._drops[uid]
KeyError: '2024-07-08T16:10:07_-5_0/0'

Outcome	Graph
Pass
Fail

Note: These graphs were run immediately after each other.

It appears the race condition is caused by the timeout of the DropChecker overlapping with the SleepApp, but it's unclear why this occurs for this SleepApp/File combination, and not others.

To confirm that this isn't a result of the subgraph-specific work, I've tested this with a similarly-structured 'normal' graph (attached), and replicated the failure.

Proposed solution

I don't know if this is actually a bug, or expected behaviour with something like the SleepApp, given that this code is from way back when the project originated. I suspect that the issue may be user error, and I've missed a configuration option. I have attached the alternative graph I made to replicate the bug.

TestSleepRaceNoSubGraph.txt

[myxie]

Thinking about this further, I suspect this may be a result of the SleepApp not actually writing anything, so the only time the FileDrop is created on disk is after it is setCompleted, as opposed to when the producer App is initialised and sets up it's outputs. I'm still not sure why this doesn't cause more problems, though.