make k8s ingestion core

[georgew5656]

Makes kubernetes-overlord-extensions a core extension.

Description

I think it makes sense to make this extension (k8s based ingestion) a core extension.

• It has significant unit test coverage and I am aware of at least a few users using it in production.
• It also has features like a migration path to/from mm ingestion.

Release note

Make k8s ingestion a core extension

Key changed/added classes in this PR

• move kubernetes-overlord-extensions from extensions-contrib -> extensions-core

It looks like this made the standard build around 15MB bigger which seems okay to me (681->696MB).

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[vtlim]

For docs, can we add a redirect for other sources that link to the old location? For example:

druid/website/redirects.js

Lines 293 to 296 in 12e88b7

	{
	"from": "/docs/latest/development/extensions-contrib/google.html",
	"to": "/docs/latest/development/extensions-core/google"
	},

[FrankChen021]

@georgew5656 are there any docs that describe how to upgrade from a MM-based ingestion to k8s-based ingestion?
and under k8s-based ingestion, what's the suggested rolling update order?

[georgew5656]

@georgew5656 are there any docs that describe how to upgrade from a MM-based ingestion to k8s-based ingestion? and under k8s-based ingestion, what's the suggested rolling update order?

@FrankChen021 there's a section here describing the update plan: https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs#migrationkubernetes-and-worker-task-runner, basically you run the KubernetesAndWorkerRunner and progressively start assigning more tasks to Kubernetes instead of mm's, then you can tear down your middle managers

[FrankChen021]

Thanks for the reply.
I noticed that the k8sAndWorker was introduced in Druid 28, which means that there's no smooth migration path from MM to K8s before that. But this extension is introduced in Druid 25, can we add a note to the doc you mentioned that k8sAndWorker is supported since Druid 28?

With regard to the rolling update, we have a document about it. Under the k8s-based ingestion mode, what's suggested update order? should we update overlord after historical? I think we should add some text in this doc.

[georgew5656]

document

Thanks for the reply. I noticed that the k8sAndWorker was introduced in Druid 28, which means that there's no smooth migration path from MM to K8s before that. But this extension is introduced in Druid 25, can we add a note to the doc you mentioned that k8sAndWorker is supported since Druid 28?

With regard to the rolling update, we have a document about it. Under the k8s-based ingestion mode, what's suggested update order? should we update overlord after historical? I think we should add some text in this doc.

yep i can update those docs. in this case the update is basically several runtime props that apply to the overlord only (the other services don't change at all, the exception being the middle managers which get torn down after).

my recommendation for anyone considering switching is to first update to druid 28 normally (with middle managers), and then run the overlord-only update afterwards