From a1261cef1435d12361dd79613f53a7f6c27090cd Mon Sep 17 00:00:00 2001 From: Sidhant Kohli Date: Wed, 23 Oct 2024 23:01:52 -0700 Subject: [PATCH 1/3] chore: add best practices in docs Signed-off-by: Sidhant Kohli --- docs/operations/best-practices.md | 25 +++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 26 insertions(+) create mode 100644 docs/operations/best-practices.md diff --git a/docs/operations/best-practices.md b/docs/operations/best-practices.md new file mode 100644 index 0000000000..1155d64883 --- /dev/null +++ b/docs/operations/best-practices.md @@ -0,0 +1,25 @@ +# Best Practices + +Few guidelines and tips to help with common use cases and FAQs + +## High latency operations in UDF + +If running a user code which has very high latency or high processing time + +1. Increase the `lookbackSeconds` for the autoscaling - This dictates how many seconds to look back for vertex average +processing rate (tps) and pending messages calculation, defaults to `120`. In case of tasks taking longer than +this duration, you need to increase`lookbackSeconds` so that the calculated average rate and pending messages won't +be 0 during the silent period. Refer [Autoscaling](https://numaflow.numaproj.io/user-guide/reference/autoscaling/) for more information +```yaml +apiVersion: numaflow.numaproj.io/v1alpha1 +kind: Pipeline +metadata: + name: my-pipeline +spec: + vertices: + - name: my-vertex + scale: + lookbackSeconds: 120 # Optional, defaults to 120. + .... +``` + diff --git a/mkdocs.yml b/mkdocs.yml index 1118c1ff43..53e43ec185 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -127,6 +127,7 @@ nav: - operations/metrics/metrics.md - operations/grafana.md - Security: operations/security.md + - Best Practices: operations/best-practices.md - Contributor Guide: - development/development.md - Specifications: From c1a202a50d6b31f3240438b200e22cfa2f3a8266 Mon Sep 17 00:00:00 2001 From: Sidhant Kohli Date: Fri, 1 Nov 2024 10:09:01 -0400 Subject: [PATCH 2/3] chore: folder Signed-off-by: Sidhant Kohli --- docs/operations/{best-practices.md => best-practices/udf.md} | 4 ++-- mkdocs.yml | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) rename docs/operations/{best-practices.md => best-practices/udf.md} (89%) diff --git a/docs/operations/best-practices.md b/docs/operations/best-practices/udf.md similarity index 89% rename from docs/operations/best-practices.md rename to docs/operations/best-practices/udf.md index 1155d64883..08202d5320 100644 --- a/docs/operations/best-practices.md +++ b/docs/operations/best-practices/udf.md @@ -1,6 +1,6 @@ -# Best Practices +# User Defined Functions -Few guidelines and tips to help with common use cases and FAQs +Few guidelines and tips to help with common use cases and FAQs for UDFs ## High latency operations in UDF diff --git a/mkdocs.yml b/mkdocs.yml index 69519abd47..5cb0599dc6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -128,7 +128,8 @@ nav: - operations/metrics/metrics.md - operations/grafana.md - Security: operations/security.md - - Best Practices: operations/best-practices.md + - Best Practices: + - UDF: "operations/best-practices/udf.md" - Contributor Guide: - development/development.md - Specifications: From 68d3305cc077eed27e45565dffe35cbb71717df7 Mon Sep 17 00:00:00 2001 From: Sidhant Kohli Date: Tue, 5 Nov 2024 13:34:34 -0800 Subject: [PATCH 3/3] chore: link Signed-off-by: Sidhant Kohli --- docs/operations/best-practices/udf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/best-practices/udf.md b/docs/operations/best-practices/udf.md index 08202d5320..22453ed9d5 100644 --- a/docs/operations/best-practices/udf.md +++ b/docs/operations/best-practices/udf.md @@ -9,7 +9,7 @@ If running a user code which has very high latency or high processing time 1. Increase the `lookbackSeconds` for the autoscaling - This dictates how many seconds to look back for vertex average processing rate (tps) and pending messages calculation, defaults to `120`. In case of tasks taking longer than this duration, you need to increase`lookbackSeconds` so that the calculated average rate and pending messages won't -be 0 during the silent period. Refer [Autoscaling](https://numaflow.numaproj.io/user-guide/reference/autoscaling/) for more information +be 0 during the silent period. Refer [Autoscaling](../../user-guide/reference/autoscaling.md) for more information ```yaml apiVersion: numaflow.numaproj.io/v1alpha1 kind: Pipeline