non-blocking mode, better monitoring
- make blocking behavior configurable for kafkaMdm and grafanaNet routes. previously, grafanaNet was blocking, kafkaMdm was non-blocking. Now both default to non-blocking, but you can specify
blocking=true
.- nonblocking (default): when the route's buffer fills up, data will be discarded for that route, but everything else (e.g. other routes) will be unaffected. If you set your buffers large enough this won't be an issue.
rule of thumb: rate in metrics/s times how many seconds you want to be able to buffer in case of downstream issues. memory used will bebufSize * 100B
or usebufSize * 150B
to be extra safe. - blocking: when the route's buffer fills up, ingestion into the route will slow down/block, providing backpressure to the clients, and also blocking other routes from making progress. use this only if you know what you're doing and have smart clients that can gracefully handle the backpressure
- nonblocking (default): when the route's buffer fills up, data will be discarded for that route, but everything else (e.g. other routes) will be unaffected. If you set your buffers large enough this won't be an issue.
- monitor queue drops for non-blocking queues
- document route options better
- monitor queue size and ram used #218
- preliminary support for parsing out the new graphite tag format (kafkaMdm and grafanaNet route only)
the included, dashboard is updated accordingly. and also on https://grafana.com/dashboards/338