graphdb batch writer resiliency

[Zenithar]

Context

The graphdb entity writers can be subject to unexpected behaviours from the JanusGraph backend (eg. backend not responding due to runtine errors).

The writers have been modified to add resilience to unexpected situation:

• The lock-free microbatcher pattern has been externalised to a dedicated instance to prevent design issues
• The batch processing are bound in time to react to the unresponsive backend issue (default: 60s)
• A retry pattern allows a batch to be retried with an capped limit. Batchs are lost if the retry max is reached. (default: 3)
• The writer supports parallel insertion workers. (default: 10)
• When a timeout error occurs, each batch size is divided by 2 to reduce the size of each attempt to prevent GraphBinary errors due to too large requests.

Sample warning triggering a retry with split batch after a unresponsive backend issue:

11:58:02 WARN Retrying write operation with smaller vertex batch (n:500 -> 250, r:0) app=kubehound dd.span_id=0 dd.trace_id=0

The batcher settings has been updated to reflect this new state:

• Insertion batch 500 -> 250
  • tunable with KH_BUILDER_VERTEX_BATCH_SIZE / KH_BUILDER_EDGE_BATCH_SIZE env. variables
  • or builder.egde/vertex.batch_size config key
• Concurrent insertion 1 -> 10
  • tunable with KH_JANUSGRAPH_WRITER_WORKER_COUNT
  • or janusgraph.writer_worker_count config key

[jt-dd]

Currently not working as the async synchro is not properly done. Also maybe adding a dedicated function to add element into the queue.

[jt-dd]

Nice. LTGM, can you add to the reference configuration file with the default value ?