feature: support infiniband with ib-sriov and ipoib cni #2815

weizhoublue · 2023-12-11T12:23:02Z

for RDMA with infiniband, support
(1) https://github.com/k8snetworkplumbingwg/ib-sriov-cni
(2) https://github.com/Mellanox/ipoib-cni

codecov · 2023-12-11T12:26:35Z

Codecov Report

Merging #2815 (48022ca) into main (510b670) will decrease coverage by 0.51%.
Report is 4 commits behind head on main.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2815      +/-   ##
==========================================
- Coverage   81.60%   81.09%   -0.51%     
==========================================
  Files          49       49              
  Lines        5333     5333              
==========================================
- Hits         4352     4325      -27     
- Misses        821      852      +31     
+ Partials      160      156       -4

Flag	Coverage Δ
unittests	`81.09% <ø> (-0.51%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

cyclinder · 2023-12-12T02:03:14Z

cmd/spiderpool-init/cmd/config.go

 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/utils/pointer"
+	"os"


sort 一下 imports

IDE 自动排的，save 就这样

goland 的自动排序没 vscode 好使...

cyclinder · 2023-12-12T02:08:31Z

docs/usage/rdma-ib-zh_CN.md

+          namespace: kube-system
+        spec:
+          cniType: ib-sriov
+          sriov:


Suggested change

sriov:

ibsriov:

cyclinder · 2023-12-12T02:11:05Z

pkg/k8s/apis/spiderpool.spidernet.io/v2beta1/spidermultus_types.go

@@ -28,7 +28,7 @@ type SpiderMultusConfigList struct {
 // MultusCNIConfigSpec defines the desired state of SpiderMultusConfig.
 type MultusCNIConfigSpec struct {
 	// +kubebuilder:validation:Required
-	// +kubebuilder:validation:Enum=macvlan;ipvlan;sriov;ovs;custom
+	// +kubebuilder:validation:Enum=macvlan;ipvlan;sriov;ovs;ib-sriov;custom


这里是 ib-sriov, 47行 json tag 是 ibsriov, 不一致

cni bin 叫 ib-sriov，为了 CNItype 和最终multus 中的 type 等一致，golang中避免2个变量，这里一致引用 ib-sriov
而 spec.ibsriov 又不合适为 ib-sriov

cyclinder · 2023-12-12T02:12:33Z

pkg/multuscniconfig/utils.go

+type IBSRIOVNetConf struct {
+	Type                string                    `json:"type"`
+	Pkey                *string                   `json:"pkey,omitempty"`
+	LinkState           *string                   `json:"link_state,omitempty"`


json tag 统一下，都为驼峰法

这个是 CNI 最终的 config，没法改

他们官方怎么又又下划线，又有驼峰

Signed-off-by: [email protected] <[email protected]>

cyclinder · 2023-12-14T10:29:08Z

docs/usage/rdma-ib-zh_CN.md

+
+2. 基于 [IPoIB CNI](https://github.com/Mellanox/ipoib-cni) 给 POD 提供 IPoIB 的网卡，它并不提供 RDMA 网卡通信能力，适用于需要 TCP/IP 通信的常规应用，因为它不需要提供 SRIOV 资源，因此能让主机上运行更多 POD
+
+另外，在 RDMA 通信场景下，对于基于 clusterIP 进行通信的应用，为了让 RDMA 流量通过 underlay 网卡转发，可在容器网络命名空间内基于 cgroup eBPF 实现 clusterIP 解析，具体可参考 [cgroup eBPF 解析 clusterIP](./underlay_cni_service-zh_CN.md)


所以这种模式下，不需要 coordinator?

coordinator 依然可解决 POD 健康检查等问题

cyclinder · 2023-12-14T10:30:12Z

docs/usage/rdma-ib-zh_CN.md

+          name: ib-sriov
+          namespace: kube-system
+        spec:
+          cniType: ib-sriov


这种模式下仍然需要 coordinator?

存在不妨碍，可保底

cyclinder · 2023-12-14T10:35:53Z

pkg/k8s/apis/spiderpool.spidernet.io/v2beta1/spidermultus_types.go

@@ -43,6 +43,12 @@ type MultusCNIConfigSpec struct {
 	// +kubebuilder:validation:Optional
 	OvsConfig *SpiderOvsCniConfig `json:"ovs,omitempty"`

+	// +kubebuilder:validation:Optional
+	IbSriovConfig *SpiderIBSriovCniConfig `json:"ibsriov,omitempty"`


为了与上面保持统一，tag 改为 ib-sriov? 更和谐一些，golang 本身也支持 "-"

https://stackoverflow.com/questions/57478256/how-to-unmarshal-a-json-string-with-a-hyphen-in-key-to-a-struct

没有项目的 spec 中的 key 是带 - 号的把，非常奇怪

cyclinder · 2023-12-14T10:38:55Z

pkg/multuscniconfig/utils.go

+type IBSRIOVNetConf struct {
+	Type                string                    `json:"type"`
+	Pkey                *string                   `json:"pkey,omitempty"`
+	LinkState           *string                   `json:"link_state,omitempty"`


他们官方怎么又又下划线，又有驼峰

Signed-off-by: [email protected] <[email protected]>

weizhoublue requested review from Icarus9913 and windsonsea as code owners December 11, 2023 12:23

cyclinder reviewed Dec 12, 2023

View reviewed changes

weizhoublue added pr/not-ready not ready for merging release/feature-new release note for new feature labels Dec 12, 2023

weizhoublue changed the title ~~suport ib-sriov cni~~ new feature: suport infiniband with ib-sriov and ipoib cni Dec 12, 2023

weizhoublue changed the title ~~new feature: suport infiniband with ib-sriov and ipoib cni~~ feature: support infiniband with ib-sriov and ipoib cni Dec 12, 2023

weizhoublue removed the pr/not-ready not ready for merging label Dec 12, 2023

weizhoublue force-pushed the pr/welan/asdf branch from d81fa63 to 61793a0 Compare December 12, 2023 12:49

weizhoublue had a problem deploying to release-base-images December 12, 2023 12:52 — with GitHub Actions Failure

weizhoublue temporarily deployed to release-base-images December 12, 2023 13:01 — with GitHub Actions Inactive

weizhoublue temporarily deployed to release-base-images December 12, 2023 13:43 — with GitHub Actions Inactive

weizhoublue added 17 commits December 14, 2023 11:15

suport ib-sriov cni

071763c

Signed-off-by: [email protected] <[email protected]>

fix

b07d4c1

Signed-off-by: [email protected] <[email protected]>

fix

1cb4a70

Signed-off-by: [email protected] <[email protected]>

fix

dc8ca75

Signed-off-by: [email protected] <[email protected]>

fix

420d28c

Signed-off-by: [email protected] <[email protected]>

fix

da8cb38

Signed-off-by: [email protected] <[email protected]>

d

fd34d60

Signed-off-by: [email protected] <[email protected]>

ipoib

912899e

Signed-off-by: [email protected] <[email protected]>

ipoib

69147dc

Signed-off-by: [email protected] <[email protected]>

ipoib

7e90743

Signed-off-by: [email protected] <[email protected]>

ipoib

b9cb791

Signed-off-by: [email protected] <[email protected]>

ipoib

f9d1286

Signed-off-by: [email protected] <[email protected]>

ipoib

c826404

Signed-off-by: [email protected] <[email protected]>

ipoib

f4f8d03

Signed-off-by: [email protected] <[email protected]>

ipoib

7131cae

Signed-off-by: [email protected] <[email protected]>

ipoib

ff9cf38

Signed-off-by: [email protected] <[email protected]>

ipoib

c3c3ba5

Signed-off-by: [email protected] <[email protected]>

weizhoublue added 3 commits December 14, 2023 11:18

ipoib

e1f19d3

Signed-off-by: [email protected] <[email protected]>

d

f71d3c9

Signed-off-by: [email protected] <[email protected]>

d

434d3d9

Signed-off-by: [email protected] <[email protected]>

weizhoublue temporarily deployed to release-base-images December 14, 2023 04:15 — with GitHub Actions Inactive

weizhoublue temporarily deployed to release-base-images December 14, 2023 04:24 — with GitHub Actions Inactive

weizhoublue force-pushed the pr/welan/asdf branch from ea87c0d to 434d3d9 Compare December 14, 2023 04:36

weizhoublue temporarily deployed to release-base-images December 14, 2023 04:37 — with GitHub Actions Inactive

weizhoublue added 3 commits December 14, 2023 13:41

d

cce480f

Signed-off-by: [email protected] <[email protected]>

d

569a136

Signed-off-by: [email protected] <[email protected]>

d

4e6078d

Signed-off-by: [email protected] <[email protected]>

weizhoublue had a problem deploying to release-base-images December 14, 2023 05:47 — with GitHub Actions Error

weizhoublue temporarily deployed to release-base-images December 14, 2023 05:50 — with GitHub Actions Inactive

d

5f1c3c6

Signed-off-by: [email protected] <[email protected]>

weizhoublue temporarily deployed to release-base-images December 14, 2023 06:16 — with GitHub Actions Inactive

d

7a38732

Signed-off-by: [email protected] <[email protected]>

weizhoublue temporarily deployed to release-base-images December 14, 2023 07:21 — with GitHub Actions Inactive

d

1bc4218

Signed-off-by: [email protected] <[email protected]>

weizhoublue temporarily deployed to release-base-images December 14, 2023 08:00 — with GitHub Actions Inactive

d

4d1268c

Signed-off-by: [email protected] <[email protected]>

cyclinder reviewed Dec 14, 2023

View reviewed changes

weizhoublue added 8 commits December 14, 2023 19:11

d

2c604e0

Signed-off-by: [email protected] <[email protected]>

d

3d6e000

Signed-off-by: [email protected] <[email protected]>

d

bace082

Signed-off-by: [email protected] <[email protected]>

d

110125e

Signed-off-by: [email protected] <[email protected]>

d

deaab5c

Signed-off-by: [email protected] <[email protected]>

d

1b89e23

Signed-off-by: [email protected] <[email protected]>

d

e3f831a

Signed-off-by: [email protected] <[email protected]>

d

48022ca

Signed-off-by: [email protected] <[email protected]>

weizhoublue merged commit 00156ed into main Dec 14, 2023
41 checks passed

weizhoublue deleted the pr/welan/asdf branch December 14, 2023 15:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: support infiniband with ib-sriov and ipoib cni #2815

feature: support infiniband with ib-sriov and ipoib cni #2815

weizhoublue commented Dec 11, 2023 •

edited

Loading

codecov bot commented Dec 11, 2023 •

edited

Loading

cyclinder Dec 12, 2023

weizhoublue Dec 12, 2023

cyclinder Dec 14, 2023

cyclinder Dec 12, 2023

cyclinder Dec 12, 2023

weizhoublue Dec 12, 2023

cyclinder Dec 12, 2023

weizhoublue Dec 12, 2023

cyclinder Dec 14, 2023

cyclinder Dec 14, 2023

weizhoublue Dec 14, 2023

cyclinder Dec 14, 2023

weizhoublue Dec 14, 2023

cyclinder Dec 14, 2023

weizhoublue Dec 14, 2023

cyclinder Dec 14, 2023


		2. 基于 [IPoIB CNI](https://github.com/Mellanox/ipoib-cni) 给 POD 提供 IPoIB 的网卡，它并不提供 RDMA 网卡通信能力，适用于需要 TCP/IP 通信的常规应用，因为它不需要提供 SRIOV 资源，因此能让主机上运行更多 POD

		另外，在 RDMA 通信场景下，对于基于 clusterIP 进行通信的应用，为了让 RDMA 流量通过 underlay 网卡转发，可在容器网络命名空间内基于 cgroup eBPF 实现 clusterIP 解析，具体可参考 [cgroup eBPF 解析 clusterIP](./underlay_cni_service-zh_CN.md)

feature: support infiniband with ib-sriov and ipoib cni #2815

feature: support infiniband with ib-sriov and ipoib cni #2815

Conversation

weizhoublue commented Dec 11, 2023 • edited Loading

codecov bot commented Dec 11, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weizhoublue commented Dec 11, 2023 •

edited

Loading

codecov bot commented Dec 11, 2023 •

edited

Loading