diff --git a/docs/usage/install/ai/get-started-macvlan-zh_CN.md b/docs/usage/install/ai/get-started-macvlan-zh_CN.md index 6fd0586d3b..dd9a062c40 100644 --- a/docs/usage/install/ai/get-started-macvlan-zh_CN.md +++ b/docs/usage/install/ai/get-started-macvlan-zh_CN.md @@ -422,7 +422,7 @@ ## 基于 Webhook 自动注入 RDMA 网络资源 -在上述步骤中，我们展示了如何使用 SR-IOV 技术在 RoCE 和 Infiniband 网络环境中为容器提供 RDMA 通信能力。然而，当配置多网卡的 AI 应用时，过程会变得复杂。为简化这个过程，Spiderpool 通过 annotations(`cni.spidernet.io/rdma-resource-inject`) 支持对一组网卡配置进行分类。用户只需要为应用添加与网卡配置相同的注解，Spiderpool 就会通过 webhook 自动为应用注入所有具有相同注解的对应网卡和网络资源。 +在上述步骤中，我们展示了如何使用 SR-IOV 技术在 RoCE 和 Infiniband 网络环境中为容器提供 RDMA 通信能力。然而，当配置多网卡的 AI 应用时，过程会变得复杂。为简化这个过程，Spiderpool 通过 annotations(`cni.spidernet.io/rdma-resource-inject` 或 `cni.spidernet.io/network-resource-inject`) 支持对一组网卡配置进行分类。用户只需要为应用添加与网卡配置相同的注解，Spiderpool 就会通过 webhook 自动为应用注入所有具有相同注解的对应网卡和网络资源。`cni.spidernet.io/rdma-resource-inject` 只适用于 AI 场景，自动注入 RDMA 网卡及 RDMA Resources；`cni.spidernet.io/network-resource-inject` 不但可以用于 AI 场景，也支持 Underlay 场景。在未来我们希望都统一使用 `cni.spidernet.io/network-resource-inject` 支持这两种场景。 > 该功能仅支持 [ macvlan, ipvlan, sriov, ib-sriov, ipoib ] 这几种 cniType 的网卡配置。 @@ -440,7 +440,7 @@ > > 当前，完成配置变更后，您需要重启 spiderpool-controller 来使配置生效。 -2. 在创建 AI 算力网络的所有 SpiderMultusConfig 实例时，添加 key 为 "cni.spidernet.io/rdma-resource-inject" 的 annotation，value 可自定义任何值 +2. 在创建 AI 算力网络的所有 SpiderMultusConfig 实例时，添加 key 为 "cni.spidernet.io/rdma-resource-inject" 或 "cni.spidernet.io/network-resource-inject" 的 annotation，value 可自定义任何值 ```yaml apiVersion: spiderpool.spidernet.io/v2beta1 diff --git a/docs/usage/install/ai/get-started-macvlan.md b/docs/usage/install/ai/get-started-macvlan.md index 8ced42507f..06c43c7bec 100644 --- a/docs/usage/install/ai/get-started-macvlan.md +++ b/docs/usage/install/ai/get-started-macvlan.md @@ -420,7 +420,7 @@ The network planning for the cluster is as follows: ## Auto Inject RDMA Resources Based on Webhook -In the steps above, we demonstrated how to use SR-IOV technology to provide RDMA communication capabilities for containers in RoCE and Infiniband network environments. However, the process can become complex when configuring AI applications with multiple network cards. To simplify this process, Spiderpool supports classifying a set of network card configurations through annotations (`cni.spidernet.io/rdma-resource-inject`). Users only need to add the same annotation to the application, and Spiderpool will automatically inject all corresponding network cards and network resources with the same annotation into the application through a webhook. +In the steps above, we demonstrated how to use SR-IOV technology to provide RDMA communication capabilities for containers in RoCE and Infiniband network environments. However, the process can become complex when configuring AI applications with multiple network cards. To simplify this process, Spiderpool supports classifying a set of network card configurations through annotations (`cni.spidernet.io/rdma-resource-inject` or `cni.spidernet.io/network-resource-inject`). Users only need to add the same annotation to the application, and Spiderpool will automatically inject all corresponding network cards and network resources with the same annotation into the application through a webhook. `cni.spidernet.io/rdma-resource-inject` annotation is only applicable to AI scenarios, automatically injecting RDMA network cards and RDMA resources. `cni.spidernet.io/network-resource-inject` annotation can be used not only for AI scenarios but also supports underlay scenarios. In the future, we hope to uniformly use `cni.spidernet.io/network-resource-inject` to support both of these scenarios. > This feature only supports network card configurations with cniType of [macvlan, ipvlan, sriov, ib-sriov, ipoib]. @@ -438,7 +438,7 @@ In the steps above, we demonstrated how to use SR-IOV technology to provide RDMA > > Currently, after completing the configuration change, you need to restart the spiderpool-controller for the configuration to take effect. -2. When creating all SpiderMultusConfig instances for AI computing networks, add an annotation with the key "cni.spidernet.io/rdma-resource-inject" and a customizable value. +2. When creating all SpiderMultusConfig instances for AI computing networks, add an annotation with the key "cni.spidernet.io/rdma-resource-inject" (or "cni.spidernet.io/network-resource-inject") and a customizable value. ```yaml apiVersion: spiderpool.spidernet.io/v2beta1 diff --git a/docs/usage/install/ai/get-started-sriov-zh_CN.md b/docs/usage/install/ai/get-started-sriov-zh_CN.md index 78877db006..d6756b46d0 100644 --- a/docs/usage/install/ai/get-started-sriov-zh_CN.md +++ b/docs/usage/install/ai/get-started-sriov-zh_CN.md @@ -609,7 +609,7 @@ Spiderpool 使用了 [sriov-network-operator](https://github.com/k8snetworkplumb ## 基于 Webhook 自动注入 RDMA 网络资源 -在上述步骤中，我们展示了如何使用 SR-IOV 技术在 RoCE 和 Infiniband 网络环境中为容器提供 RDMA 通信能力。然而，当配置多网卡的 AI 应用时，过程会变得复杂。为简化这个过程，Spiderpool 通过 annotations(`cni.spidernet.io/rdma-resource-inject`) 支持对一组网卡配置进行分类。用户只需要为应用添加与网卡配置相同的注解，Spiderpool 就会通过 webhook 自动为应用注入所有具有相同注解的对应网卡和网络资源。 +在上述步骤中，我们展示了如何使用 SR-IOV 技术在 RoCE 和 Infiniband 网络环境中为容器提供 RDMA 通信能力。然而，当配置多网卡的 AI 应用时，过程会变得复杂。为简化这个过程，Spiderpool 通过 annotations(`cni.spidernet.io/rdma-resource-inject` 或 `cni.spidernet.io/network-resource-inject`) 支持对一组网卡配置进行分类。用户只需要为应用添加与网卡配置相同的注解，Spiderpool 就会通过 webhook 自动为应用注入所有具有相同注解的对应网卡和网络资源。`cni.spidernet.io/rdma-resource-inject` 只适用于 AI 场景，自动注入 RDMA 网卡及 RDMA Resources；`cni.spidernet.io/network-resource-inject` 不但可以用于 AI 场景，也支持 Underlay 场景。在未来我们希望都统一使用 `cni.spidernet.io/network-resource-inject` 支持这两种场景。 > 该功能仅支持 [ macvlan, ipvlan, sriov, ib-sriov, ipoib ] 这几种 cniType 的网卡配置。 @@ -627,7 +627,7 @@ Spiderpool 使用了 [sriov-network-operator](https://github.com/k8snetworkplumb > > 当前，完成配置变更后，您需要重启 spiderpool-controller 来使配置生效。 -2. 在创建 AI 算力网络的所有 SpiderMultusConfig 实例时，添加 key 为 "cni.spidernet.io/rdma-resource-inject" 的 annotation，value 可自定义任何值 +2. 在创建 AI 算力网络的所有 SpiderMultusConfig 实例时，添加 key 为 "cni.spidernet.io/rdma-resource-inject" 或 "cni.spidernet.io/network-resource-inject" 的 annotation，value 可自定义任何值 ```yaml apiVersion: spiderpool.spidernet.io/v2beta1 diff --git a/docs/usage/install/ai/get-started-sriov.md b/docs/usage/install/ai/get-started-sriov.md index 446c156eea..8f1ca7bc82 100644 --- a/docs/usage/install/ai/get-started-sriov.md +++ b/docs/usage/install/ai/get-started-sriov.md @@ -610,7 +610,7 @@ For clusters using Infiniband networks, if there is a [UFM management platform]( ## Auto Inject RDMA Resources Based on Webhook -In the steps above, we demonstrated how to use SR-IOV technology to provide RDMA communication capabilities for containers in RoCE and Infiniband network environments. However, the process can become complex when configuring AI applications with multiple network cards. To simplify this process, Spiderpool supports classifying a set of network card configurations through annotations (`cni.spidernet.io/rdma-resource-inject`). Users only need to add the same annotation to the application, and Spiderpool will automatically inject all corresponding network cards and network resources with the same annotation into the application through a webhook. +In the steps above, we demonstrated how to use SR-IOV technology to provide RDMA communication capabilities for containers in RoCE and Infiniband network environments. However, the process can become complex when configuring AI applications with multiple network cards. To simplify this process, Spiderpool supports classifying a set of network card configurations through annotations (`cni.spidernet.io/rdma-resource-inject` or `cni.spidernet.io/network-resource-inject`). Users only need to add the same annotation to the application, and Spiderpool will automatically inject all corresponding network cards and network resources with the same annotation into the application through a webhook. `cni.spidernet.io/rdma-resource-inject` annotation is only applicable to AI scenarios, automatically injecting RDMA network cards and RDMA resources. `cni.spidernet.io/network-resource-inject` annotation can be used not only for AI scenarios but also supports underlay scenarios. In the future, we hope to uniformly use `cni.spidernet.io/network-resource-inject` to support both of these scenarios. > This feature only supports network card configurations with cniType of [macvlan, ipvlan, sriov, ib-sriov, ipoib]. @@ -628,7 +628,7 @@ In the steps above, we demonstrated how to use SR-IOV technology to provide RDMA > > Currently, after completing the configuration change, you need to restart the spiderpool-controller for the configuration to take effect. -2. When creating all SpiderMultusConfig instances for AI computing networks, add an annotation with the key "cni.spidernet.io/rdma-resource-inject" and a customizable value. +2. When creating all SpiderMultusConfig instances for AI computing networks, add an annotation with the key "cni.spidernet.io/rdma-resource-inject" (or "cni.spidernet.io/network-resource-inject") and a customizable value. ```yaml apiVersion: spiderpool.spidernet.io/v2beta1 diff --git a/docs/usage/install/overlay/get-started-calico-zh_cn.md b/docs/usage/install/overlay/get-started-calico-zh_cn.md index b982774c14..36a4377c26 100644 --- a/docs/usage/install/overlay/get-started-calico-zh_cn.md +++ b/docs/usage/install/overlay/get-started-calico-zh_cn.md @@ -159,6 +159,7 @@ EOF ``` > `spec.macvlan.master` 设置为 `ens192`, `ens192`必须存在于主机上。并且 `spec.macvlan.spiderpoolConfigPools.IPv4IPPool`设置的子网和 `ens192`保持一致。 +> 如果需要为 Pod 添加多张 Underlay 网卡的可以参考 [**Multi-Underlay-NIC**](./multi-underlay-nic-zh_CN.md)。 创建成功后, 查看 Multus NAD 是否成功创建: diff --git a/docs/usage/install/overlay/get-started-calico.md b/docs/usage/install/overlay/get-started-calico.md index 1a12d4cb42..12911e44f0 100644 --- a/docs/usage/install/overlay/get-started-calico.md +++ b/docs/usage/install/overlay/get-started-calico.md @@ -130,6 +130,7 @@ EOF ``` > The subnet should be consistent with the subnet of `ens192` on the nodes, and ensure that the IP addresses do not conflict with any existing ones. +> If you need to add multiple underlay NICs to a Pod, you can refer to[**Multi-Underlay-NIC**](./multi-underlay-nic.md). ### Create SpiderMultusConfig diff --git a/docs/usage/install/overlay/get-started-cilium-zh_cn.md b/docs/usage/install/overlay/get-started-cilium-zh_cn.md index 0a099b4094..7f151fad7b 100644 --- a/docs/usage/install/overlay/get-started-cilium-zh_cn.md +++ b/docs/usage/install/overlay/get-started-cilium-zh_cn.md @@ -135,6 +135,7 @@ EOF Note: > subnet 应该与节点网卡 ens192 的子网保持一致，并且不与现有任何 IP 冲突。 +> 如果需要为 Pod 添加多张 Underlay 网卡的可以参考 [**Multi-Underlay-NIC**](./multi-underlay-nic-zh_CN.md)。 ### 创建 SpiderMultusConfig diff --git a/docs/usage/install/overlay/get-started-cilium.md b/docs/usage/install/overlay/get-started-cilium.md index 2985a5b385..d54781ac4f 100644 --- a/docs/usage/install/overlay/get-started-cilium.md +++ b/docs/usage/install/overlay/get-started-cilium.md @@ -133,6 +133,7 @@ EOF ``` > The subnet should be consistent with the subnet of `ens192` on the nodes, and ensure that the IP addresses do not conflict with any existing ones. +> If you need to add multiple underlay NICs to a Pod, you can refer to[**Multi-Underlay-NIC**](./multi-underlay-nic.md). ### Create SpiderMultusConfig diff --git a/docs/usage/install/overlay/multi-underlay-nic-zh_CN.md b/docs/usage/install/overlay/multi-underlay-nic-zh_CN.md new file mode 100644 index 0000000000..9b80531b9d --- /dev/null +++ b/docs/usage/install/overlay/multi-underlay-nic-zh_CN.md @@ -0,0 +1,96 @@ +# Calico with multi underlay NIC + +[**English**](./multi-underlay-nic.md) | **简体中文** + +## 基于 Webhook 自动为 Pod 附加多张 Underlay 网卡 + + 本文集群节点网卡: `ens192` 所在子网为 `10.6.0.0/16`，`ens193` 所在子网为 `10.7.0.0/16`，以此创建 SpiderIPPool: + + ```shell + $ cat < 注意：使用 webhook 自动注入网络资源功能时，不能为应用添加其他网络配置注解(如 `k8s.v1.cni.cncf.io/networks` 和 `ipam.spidernet.io ippools`等)，否则会影响资源自动注入功能。 + +2. 当 Pod 被创建后，可观测到 Pod 被自动注入了网卡 annotation + + ```yaml + ... + spec: + template: + metadata: + annotations: + k8s.v1.cni.cncf.io/networks: |- + [{"name":"macvlan-ens192","namespace":"spiderpool"}, + {"name":"macvlan-ens193","namespace":"spiderpool"}] + .... + ``` diff --git a/docs/usage/install/overlay/multi-underlay-nic.md b/docs/usage/install/overlay/multi-underlay-nic.md new file mode 100644 index 0000000000..c6880a376a --- /dev/null +++ b/docs/usage/install/overlay/multi-underlay-nic.md @@ -0,0 +1,96 @@ +# Calico with multi underlay NIC + +**English** | [**简体中文**](./multi-underlay-nic-zh_CN.md) + +## Auto attach multiple underlay NICs to Pod based on Webhook + + The subnet for the interface `ens192` on the cluster nodes here is `10.6.0.0/16`. The subnet for the interface `ens193` on the cluster nodes here is `10.7.0.0/16`. Create SpiderIPPools using these subnets: + + ```shell + $ cat < Note: When using the webhook automatic injection of network resources feature, do not add other network configuration annotations (such as `k8s.v1.cni.cncf.io/networks` and `ipam.spidernet.io/ippools`) to the application, as it will affect the automatic injection of resources. + +2. Once the Pod is created, you can observe that the Pod has been automatically injected with network card annotations. + + ```yaml + ... + spec: + template: + metadata: + annotations: + k8s.v1.cni.cncf.io/networks: |- + [{"name":"macvlan-ens192","namespace":"spiderpool"}, + {"name":"macvlan-ens193","namespace":"spiderpool"}] + .... + ``` diff --git a/pkg/constant/k8s.go b/pkg/constant/k8s.go index 9b88acde5e..4213d39582 100644 --- a/pkg/constant/k8s.go +++ b/pkg/constant/k8s.go @@ -105,8 +105,9 @@ const ( AnnoDraCdiVersion = AnnotationPre + "/cdi-version" // webhook - PodMutatingWebhookName = "pods.spiderpool.spidernet.io" - AnnoPodResourceInject = CNIAnnotationPre + "/rdma-resource-inject" + PodMutatingWebhookName = "pods.spiderpool.spidernet.io" + AnnoPodResourceInject = CNIAnnotationPre + "/rdma-resource-inject" + AnnoNetworkResourceInject = CNIAnnotationPre + "/network-resource-inject" ) const ( diff --git a/pkg/multuscniconfig/multusconfig_mutate.go b/pkg/multuscniconfig/multusconfig_mutate.go index 5967060773..9081b57cfb 100644 --- a/pkg/multuscniconfig/multusconfig_mutate.go +++ b/pkg/multuscniconfig/multusconfig_mutate.go @@ -53,15 +53,17 @@ func mutateSpiderMultusConfig(ctx context.Context, smc *spiderpoolv2beta1.Spider } // inject the labels - value, ok := smc.Annotations[constant.AnnoPodResourceInject] - if !ok { - return - } + for _, v := range []string{constant.AnnoPodResourceInject, constant.AnnoNetworkResourceInject} { + value, ok := smc.Annotations[v] + if !ok { + continue + } - if smc.Labels == nil { - smc.Labels = make(map[string]string) + if smc.Labels == nil { + smc.Labels = make(map[string]string) + } + smc.Labels[v] = value } - smc.Labels[constant.AnnoPodResourceInject] = value } func setMacvlanDefaultConfig(macvlanConfig *spiderpoolv2beta1.SpiderMacvlanCniConfig) { diff --git a/pkg/multuscniconfig/multusconfig_validate.go b/pkg/multuscniconfig/multusconfig_validate.go index d72a92e401..2582aa1817 100644 --- a/pkg/multuscniconfig/multusconfig_validate.go +++ b/pkg/multuscniconfig/multusconfig_validate.go @@ -89,6 +89,7 @@ func validateCNIConfig(multusConfig *spiderpoolv2beta1.SpiderMultusConfig) *fiel } _, injectRdmaResource := multusConfig.Annotations[constant.AnnoPodResourceInject] + _, injectNetworkResource := multusConfig.Annotations[constant.AnnoNetworkResourceInject] switch *multusConfig.Spec.CniType { case constant.MacvlanCNI: if multusConfig.Spec.MacvlanConfig == nil { @@ -115,6 +116,12 @@ func validateCNIConfig(multusConfig *spiderpoolv2beta1.SpiderMultusConfig) *fiel } } + if injectNetworkResource { + if err := ValidateNetworkResouce(multusConfig.Name, multusConfig.Namespace, multusConfig.Spec.MacvlanConfig.RdmaResourceName, multusConfig.Spec.MacvlanConfig.SpiderpoolConfigPools); err != nil { + return field.Invalid(macvlanConfigField, *multusConfig.Spec.MacvlanConfig, err.Error()) + } + } + case constant.IPVlanCNI: if multusConfig.Spec.IPVlanConfig == nil { return field.Required(ipvlanConfigField, fmt.Sprintf("no %s specified", ipvlanConfigField.String())) @@ -140,6 +147,12 @@ func validateCNIConfig(multusConfig *spiderpoolv2beta1.SpiderMultusConfig) *fiel } } + if injectNetworkResource { + if err := ValidateNetworkResouce(multusConfig.Name, multusConfig.Namespace, multusConfig.Spec.IPVlanConfig.RdmaResourceName, multusConfig.Spec.IPVlanConfig.SpiderpoolConfigPools); err != nil { + return field.Invalid(ipvlanConfigField, *multusConfig.Spec.IPVlanConfig, err.Error()) + } + } + case constant.SriovCNI: if multusConfig.Spec.SriovConfig == nil { return field.Required(sriovConfigField, fmt.Sprintf("no %s specified", sriovConfigField.String())) @@ -171,6 +184,12 @@ func validateCNIConfig(multusConfig *spiderpoolv2beta1.SpiderMultusConfig) *fiel } } + if injectNetworkResource { + if err := ValidateNetworkResouce(multusConfig.Name, multusConfig.Namespace, multusConfig.Spec.SriovConfig.ResourceName, multusConfig.Spec.SriovConfig.SpiderpoolConfigPools); err != nil { + return field.Invalid(sriovConfigField, *multusConfig.Spec.SriovConfig, err.Error()) + } + } + case constant.IBSriovCNI: if multusConfig.Spec.IbSriovConfig == nil { return field.Required(ibsriovConfigField, fmt.Sprintf("no %s specified", ibsriovConfigField.String())) @@ -190,6 +209,12 @@ func validateCNIConfig(multusConfig *spiderpoolv2beta1.SpiderMultusConfig) *fiel } } + if injectNetworkResource { + if err := ValidateNetworkResouce(multusConfig.Name, multusConfig.Namespace, multusConfig.Spec.IbSriovConfig.ResourceName, multusConfig.Spec.IbSriovConfig.SpiderpoolConfigPools); err != nil { + return field.Invalid(ibsriovConfigField, *multusConfig.Spec.IbSriovConfig, err.Error()) + } + } + case constant.IPoIBCNI: if multusConfig.Spec.IpoibConfig == nil { return field.Required(ipoibConfigField, fmt.Sprintf("no %s specified", ipoibConfigField.String())) @@ -209,9 +234,15 @@ func validateCNIConfig(multusConfig *spiderpoolv2beta1.SpiderMultusConfig) *fiel } } + if injectNetworkResource { + if err := ValidateNetworkResouce(multusConfig.Name, multusConfig.Namespace, multusConfig.Spec.IpoibConfig.Master, multusConfig.Spec.IpoibConfig.SpiderpoolConfigPools); err != nil { + return field.Invalid(ipoibConfigField, *multusConfig.Spec.IpoibConfig, err.Error()) + } + } + case constant.OvsCNI: - if injectRdmaResource { - return field.Forbidden(cniTypeField, fmt.Sprintf("the cniType %s does not support RDMA resource injected", *multusConfig.Spec.CniType)) + if injectRdmaResource || injectNetworkResource { + return field.Forbidden(cniTypeField, fmt.Sprintf("the cniType %s does not support RDMA resource or network resource injected", *multusConfig.Spec.CniType)) } if multusConfig.Spec.OvsConfig == nil { return field.Required(ovsConfigField, fmt.Sprintf("no %s specified", ovsConfigField.String())) @@ -250,8 +281,8 @@ func validateCNIConfig(multusConfig *spiderpoolv2beta1.SpiderMultusConfig) *fiel } case constant.CustomCNI: - if injectRdmaResource { - return field.Forbidden(cniTypeField, fmt.Sprintf("the cniType %s does not support RDMA resource injected", *multusConfig.Spec.CniType)) + if injectRdmaResource || injectNetworkResource { + return field.Forbidden(cniTypeField, fmt.Sprintf("the cniType %s does not support RDMA resource or network resource injected", *multusConfig.Spec.CniType)) } // multusConfig.Spec.CustomCNIConfig can be empty if checkExistedConfig(&(multusConfig.Spec), constant.CustomCNI) { diff --git a/pkg/multuscniconfig/utils.go b/pkg/multuscniconfig/utils.go index ee8fb0151d..5d68ee62ec 100644 --- a/pkg/multuscniconfig/utils.go +++ b/pkg/multuscniconfig/utils.go @@ -268,3 +268,19 @@ func ValidateRdmaResouce(enableRdma bool, name, namespace, rdmaResourceName stri return nil } + +func ValidateNetworkResouce(name, namespace, resourceName string, ippools *v2beta1.SpiderpoolPools) error { + if len(resourceName) == 0 { + return nil + } + + if ippools == nil { + return fmt.Errorf("No any ippools configured for spidermultusconfig %s/%s", namespace, name) + } + + if len(ippools.IPv4IPPool)+len(ippools.IPv6IPPool) == 0 { + return fmt.Errorf("No any ippools configured for spidermultusconfig %s/%s", namespace, name) + } + + return nil +} diff --git a/pkg/podmanager/pod_webhook.go b/pkg/podmanager/pod_webhook.go index 2cc086a8f2..7afa560969 100644 --- a/pkg/podmanager/pod_webhook.go +++ b/pkg/podmanager/pod_webhook.go @@ -74,12 +74,18 @@ func (pw *podWebhook) Default(ctx context.Context, obj runtime.Object) error { zap.String("Pod", pod.GenerateName)) mutateLogger.Sugar().Debugf("Request Pod: %+v", *pod) - _, ok := pod.Annotations[constant.AnnoPodResourceInject] - if !ok { + needInject := false + for _, anno := range []string{constant.AnnoPodResourceInject, constant.AnnoNetworkResourceInject} { + if _, ok := pod.Annotations[anno]; ok { + mutateLogger.Sugar().Debugf("Pod %s/%s is annotated with %s, start injecting network resources", pod.Namespace, pod.GenerateName, anno) + needInject = true + } + } + + if !needInject { return nil } - mutateLogger.Sugar().Debugf("Pod %s/%s is annotated with %s, start injecting network resources", pod.Namespace, pod.GenerateName, constant.AnnoPodResourceInject) err := podNetworkMutatingWebhook(pw.spiderClient, pod) if err != nil { mutateLogger.Sugar().Errorf("Failed to inject network resources for pod %s/%s: %v", pod.Namespace, pod.GenerateName, err) diff --git a/pkg/podmanager/utils.go b/pkg/podmanager/utils.go index 40b45c6677..c1e68dc37a 100644 --- a/pkg/podmanager/utils.go +++ b/pkg/podmanager/utils.go @@ -72,38 +72,41 @@ func IsStaticIPPod(enableStatefulSet, enableKubevirtStaticIP bool, pod *corev1.P // Returns: // - An error if any step in the process fails, nil otherwise func podNetworkMutatingWebhook(spiderClient crdclientset.Interface, pod *corev1.Pod) error { - multusLabelValue, ok := pod.Annotations[constant.AnnoPodResourceInject] - if !ok { - return nil - } - - labelSelector := metav1.LabelSelector{ - MatchExpressions: []metav1.LabelSelectorRequirement{ - { - Key: constant.AnnoPodResourceInject, - Operator: metav1.LabelSelectorOpIn, - Values: []string{multusLabelValue}, + for _, anno := range []string{constant.AnnoPodResourceInject, constant.AnnoNetworkResourceInject} { + multusLabelValue, ok := pod.Annotations[anno] + if !ok { + continue + } + labelSelector := metav1.LabelSelector{ + MatchExpressions: []metav1.LabelSelectorRequirement{ + { + Key: anno, + Operator: metav1.LabelSelectorOpIn, + Values: []string{multusLabelValue}, + }, }, - }, - } + } - selector, err := metav1.LabelSelectorAsSelector(&labelSelector) - if err != nil { - return fmt.Errorf("failed to create label selector: %v", err) - } + selector, err := metav1.LabelSelectorAsSelector(&labelSelector) + if err != nil { + return fmt.Errorf("failed to create label selector: %v", err) + } - multusConfigs, err := spiderClient.SpiderpoolV2beta1().SpiderMultusConfigs("").List(context.TODO(), metav1.ListOptions{ - LabelSelector: selector.String(), - }) - if err != nil { - return err - } + multusConfigs, err := spiderClient.SpiderpoolV2beta1().SpiderMultusConfigs("").List(context.TODO(), metav1.ListOptions{ + LabelSelector: selector.String(), + }) + if err != nil { + return err + } - if len(multusConfigs.Items) == 0 { - return fmt.Errorf("No spidermultusconfigs with annotation: %v:%v found", constant.AnnoPodResourceInject, multusLabelValue) + if len(multusConfigs.Items) == 0 { + return fmt.Errorf("No spidermultusconfigs with annotation: %v:%v found", anno, multusLabelValue) + } + + return InjectPodNetwork(pod, *multusConfigs) } - return InjectPodNetwork(pod, *multusConfigs) + return nil } // injectPodNetwork injects network configurations into the pod based on the provided SpiderMultusConfigs. diff --git a/test/doc/spidermultus.md b/test/doc/spidermultus.md index 03238f63d0..2da62e8c9c 100644 --- a/test/doc/spidermultus.md +++ b/test/doc/spidermultus.md @@ -32,3 +32,5 @@ | M00028 | return an err if rdma is not enabled when spidermutlus with annotation: cni.spidernet.io/rdma-resource-inject | p3 | | done | | | M00029 | return an err if no ippools config when spidermutlus with annotation: cni.spidernet.io/rdma-resource-inject | p3 | | done | | | M00030 | return an err if cniType is not in [macvlan,ipvlan,sriov,ib-sriov,ipoib] when spidermutlus with annotation: cni.spidernet.io/rdma-resource-inject | p3 | | done | | +| M00031 | resoucename and ippools config must be both set when spidermutlus with annotation: cni.spidernet.io/network-resource-inject | p3 | | done | | +| M00032 | return an err if resoucename is set without ippools config when spidermutlus with annotation: cni.spidernet.io/network-resource-inject | p3 | | done | | \ No newline at end of file diff --git a/test/e2e/spidermultus/spidermultus_test.go b/test/e2e/spidermultus/spidermultus_test.go index 07a2c0728f..d8536b74d1 100644 --- a/test/e2e/spidermultus/spidermultus_test.go +++ b/test/e2e/spidermultus/spidermultus_test.go @@ -881,6 +881,65 @@ var _ = Describe("test spidermultus", Label("SpiderMultusConfig"), func() { Expect(err).To(HaveOccurred()) }) + It("resoucename and ippools config must be both set when spidermutlus with annotation: cni.spidernet.io/network-resource-inject", Label("M00031"), func() { + var smcName string = "ann-network-resource" + common.GenerateString(10, true) + smc := &spiderpoolv2beta1.SpiderMultusConfig{ + ObjectMeta: metav1.ObjectMeta{ + Name: smcName, + Namespace: namespace, + Annotations: map[string]string{ + constant.AnnoNetworkResourceInject: "test", + }, + }, + Spec: spiderpoolv2beta1.MultusCNIConfigSpec{ + CniType: ptr.To(constant.MacvlanCNI), + MacvlanConfig: &spiderpoolv2beta1.SpiderMacvlanCniConfig{ + Master: []string{common.NIC1}, + EnableRdma: true, + RdmaResourceName: "test", + SpiderpoolConfigPools: &spiderpoolv2beta1.SpiderpoolPools{ + IPv4IPPool: []string{"test"}, + }, + }, + EnableCoordinator: ptr.To(true), + CoordinatorConfig: &spiderpoolv2beta1.CoordinatorSpec{ + PodRPFilter: nil, + }, + }, + } + GinkgoWriter.Printf("spidermultus cr: %+v \n", smc) + err := frame.CreateSpiderMultusInstance(smc) + Expect(err).NotTo(HaveOccurred(), "create spiderMultus instance failed: %v\n", err) + }) + + It("return an err if resoucename is set without ippools config when spidermutlus with annotation: cni.spidernet.io/network-resource-inject", Label("M00032"), func() { + var smcName string = "ann-network-resource" + common.GenerateString(10, true) + smc := &spiderpoolv2beta1.SpiderMultusConfig{ + ObjectMeta: metav1.ObjectMeta{ + Name: smcName, + Namespace: namespace, + Annotations: map[string]string{ + constant.AnnoNetworkResourceInject: "test", + }, + }, + Spec: spiderpoolv2beta1.MultusCNIConfigSpec{ + CniType: ptr.To(constant.MacvlanCNI), + MacvlanConfig: &spiderpoolv2beta1.SpiderMacvlanCniConfig{ + Master: []string{common.NIC1}, + EnableRdma: true, + RdmaResourceName: "test", + }, + EnableCoordinator: ptr.To(true), + CoordinatorConfig: &spiderpoolv2beta1.CoordinatorSpec{ + PodRPFilter: nil, + }, + }, + } + GinkgoWriter.Printf("spidermultus cr: %+v \n", smc) + err := frame.CreateSpiderMultusInstance(smc) + Expect(err).To(HaveOccurred()) + }) + It("set disableIPAM to true and see if multus's nad has ipam config", Label("M00017"), func() { var smcName string = "disable-ipam-multus-" + common.GenerateString(10, true)