Skip to content

Commit

Permalink
d
Browse files Browse the repository at this point in the history
  • Loading branch information
weizhoublue committed Dec 14, 2023
1 parent 3d6e000 commit bace082
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 21 deletions.
4 changes: 2 additions & 2 deletions docs/usage/rdma-ib-zh_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@

另外,在 RDMA 通信场景下,对于基于 clusterIP 进行通信的应用,为了让 RDMA 流量通过 underlay 网卡转发,可在容器网络命名空间内基于 cgroup eBPF 实现 clusterIP 解析,具体可参考 [cgroup eBPF 解析 clusterIP](./underlay_cni_service-zh_CN.md#基于-cgroup-ebpf-实现-service-访问)

### 基于 IB-SRIOV 提供 RDMA 网卡
## 基于 IB-SRIOV 提供 RDMA 网卡

以下步骤演示在具备 2 个节点的集群上,如何基于 [IB-SRIOV](https://github.com/k8snetworkplumbingwg/ib-sriov-cni) 使得 Pod 接入 SRIOV 网卡,并提供网络命名空间隔离的 RDMA 设备:

Expand Down Expand Up @@ -235,7 +235,7 @@
2 1000 1.84 12.20 1.90 1.97 0.47 2.24 12.20
---------------------------------------------------------------------------------------

### 基于 IPoIB 的常规网卡
## 基于 IPoIB 的常规网卡

以下步骤演示在具备 2 个节点的集群上,如何基于 [IPoIB](https://github.com/Mellanox/ipoib-cni) 使得 Pod 接入常规的 TCP/IP 网卡,使得应用能够在 Infiniband 网络中进行 TCP/IP 通信,但是应用不能进行 RDMA 通信

Expand Down
24 changes: 12 additions & 12 deletions docs/usage/rdma-ib.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,29 +8,29 @@ This chapter introduces how POD access network with the infiniband interface of

## Features

Different from RoCE, Infiniband network cards are proprietary devices based on Infiniband networks, and the Spiderpool offers two CNI options:
Different from RoCE, Infiniband network cards are proprietary devices for the Infiniband network, and the Spiderpool offers two CNI options:

1. [IB-SRIOV CNI](https://github.com/k8snetworkplumbingwg/ib-sriov-cni) provides SR-IOV network card with the RDMA device. It is suitable for workloads that require RDMA communication.
1. [IB-SRIOV CNI](https://github.com/k8snetworkplumbingwg/ib-sriov-cni) provides SR-IOV network card with the RDMA device. It is suitable for workloads requiring RDMA communication.

It provide two RDMA modes:
It offers two RDMA modes:

- Shared mode, Pod will have a SR-IOV network interface with RDMA feature, but all RDMA devices will be seen by all PODs running in the same node. POD may be confused for which RDMA device it should use
- Shared mode: Pod will have a SR-IOV network interface with RDMA feature, but all RDMA devices cloud be seen by all PODs running in the same node. POD may be confused for which RDMA device it should use.

- Exclusive mode, Pod will have a SR-IOV network interface with RDMA feature, and POD just enable to see its own RDMA devices.
- Exclusive mode: Pod will have a SR-IOV network interface with RDMA feature, and POD just enable to see its own RDMA device.

For isolate RDMA network cards, at least one of the following conditions must be met:
For isolated RDMA network cards, at least one of the following conditions must be met:

(1) Kernel based on 5.3.0 or newer, RDMA modules loaded in the system. rdma-core package provides means to automatically load relevant modules on system start

(2) Mellanox OFED version 4.7 or newer is required. In this case it is not required to use a Kernel based on 5.3.0 or newer.

2. [IPoIB CNI](https://github.com/mellanox/ipoib-cni) provides an IPoIB network card for POD, without RDMA device. It is suitable for conventional applications that require TCP/IP communication, as it does not require an SRIOV network card, allowing more PODs to run on the host

Moreover, for applications using clusterIP for RDMA communication, it must take the underlay network card to forwarded RDMA traffic, so it needs to implement the clusterIP by cgroup eBPF in the container network namespace. For specific details, please refer to [cgroup eBPF Resolving ClusterIP](./underlay_cni_service zh-CN.md#access-service-by-cgroup-ebpf)
Moreover, for applications using clusterIP for RDMA communication, it must take the underlay network card to forwarded RDMA traffic, so it needs to implement the clusterIP by cgroup eBPF in the container network namespace. For specific details, please refer to [cgroup eBPF Resolving ClusterIP](./underlay_cni_service.md#access-service-by-cgroup-ebpf)

### RDMA based on IB-SRIOV
## RDMA based on IB-SRIOV

The following steps demonstrate how to use [IB-SRIOV](https://github.com/k8snetworkplumbingwg/ib-sriov-cni) on a cluster with 2 nodes. It enables POD to own SR-IOV network card and RDMA devices with network namespace isolation:
The following steps demonstrate how to use [IB-SRIOV](https://github.com/k8snetworkplumbingwg/ib-sriov-cni) on a cluster with 2 nodes. It enables POD to own SR-IOV network card and the RDMA devices of isolated network namespace

1. Ensure that the host machine has an Infiniband card installed and the driver is properly installed.

Expand All @@ -51,7 +51,7 @@ The following steps demonstrate how to use [IB-SRIOV](https://github.com/k8snetw
~# ibstat mlx5_0 | grep "Link layer"
Link layer: InfiniBand

Make sure that the RDMA subsystem on the host is operating in exclusive mode. If not, switch to shared mode.
Make sure that the RDMA subsystem of the host is in exclusive mode. If not, switch to shared mode.

~# rdma system set netns exclusive
~# echo "options ib_core netns_mode=0" >> /etc/modprobe.d/ib_core.conf
Expand Down Expand Up @@ -125,7 +125,7 @@ The following steps demonstrate how to use [IB-SRIOV](https://github.com/k8snetw
...
]

4. create the CNI configuration of IB-SRIOV, and the ippool resource
4. Create the CNI configuration of IB-SRIOV, and the ippool resource

cat <<EOF | kubectl apply -f -
apiVersion: spiderpool.spidernet.io/v2beta1
Expand Down Expand Up @@ -233,7 +233,7 @@ The following steps demonstrate how to use [IB-SRIOV](https://github.com/k8snetw
2 1000 1.84 12.20 1.90 1.97 0.47 2.24 12.20
---------------------------------------------------------------------------------------

### IPoIB
## IPoIB

The following steps demonstrate how to use [IPoIB](https://github.com/mellanox/ipoib-cni) on a cluster with 2 nodes, it enables Pod to own a regular TCP/IP network cards without RDMA device.

Expand Down
4 changes: 2 additions & 2 deletions docs/usage/rdma-roce-zh_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ RDMA 网卡,也可以基于 SR-IOV CNI 来使用 exclusive 模式的网卡。

另外,对于基于 clusterIP 进行 RDMA 通信的应用,为了实现让 RDMA 流量通过 underlay 网卡转发,可在容器网络命名空间内基于 cgroup eBPF 实现的 clusterIP 的解析,具体可参考 [cgroup eBPF 解析 clusterIP](./underlay_cni_service-zh_CN.md)

### 基于 macvlan 或 ipvlan 共享使用 RDMA 网卡
## 基于 macvlan 或 ipvlan 共享使用 RDMA 网卡

以下步骤演示在具备 2 个节点的集群上,如何基于 macvlan CNI 使得 Pod 共享使用 RDMA 设备:

Expand Down Expand Up @@ -221,7 +221,7 @@ RDMA 网卡,也可以基于 SR-IOV CNI 来使用 exclusive 模式的网卡。
2 1000 6.88 16.81 7.04 7.06 0.31 7.38 16.81
---------------------------------------------------------------------------------------

### 基于 SR-IOV 隔离使用 RDMA 网卡
## 基于 SR-IOV 隔离使用 RDMA 网卡

以下步骤演示在具备 2 个节点的集群上,如何基于 SR-IOV CNI 给 Pod 分配隔离的 RDMA 设备:

Expand Down
10 changes: 5 additions & 5 deletions docs/usage/rdma-roce.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,19 @@ This chapter introduces how POD access network with the RoCE interface of the ho

RDMA devices' network namespaces have two modes: shared and exclusive. Containers can either share or exclusively access RDMA network cards. In Kubernetes, shared cards can be utilized with macvlan or ipvlan CNI, while the exclusive one can be used with SR-IOV CNI.

- Shared mode. Spiderpool leverages macvlan or ipvlan CNI to expose RoCE network cards on the host machine for Pod. The [RDMA shared device plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin) is employed for exposing RDMA card resources and scheduling Pods.
- Shared mode. Spiderpool leverages macvlan or ipvlan CNI to expose RoCE network cards on the host machine for all Pods. The [RDMA shared device plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin) is employed for exposing RDMA card resources and scheduling Pods.

- Exclusive mode. Spiderpool utilizes [SR-IOV CNI](https://github.com/k8snetworkplumbingwg/sriov-network-operator) to expose RDMA cards on the host machine for Pods, providing access to RDMA resources. [RDMA CNI](https://github.com/k8snetworkplumbingwg/rdma-cni) is used to ensure isolation of RDMA devices.

For isolate RDMA network cards, at least one of the following conditions must be met:
For isolated RDMA network cards, at least one of the following conditions must be met:

(1) Kernel based on 5.3.0 or newer, RDMA modules loaded in the system. rdma-core package provides means to automatically load relevant modules on system start

(2) Mellanox OFED version 4.7 or newer is required. In this case it is not required to use a Kernel based on 5.3.0 or newer.

Moreover, for applications using clusterIP for RDMA communication, it must take the underlay network card to forwarded RDMA traffic, so it needs to implement the clusterIP by cgroup eBPF in the container network namespace. For specific details, please refer to [cgroup eBPF Resolving ClusterIP](./underlay_cni_service zh-CN. md)

### Shared RoCE NIC with macvlan or ipvlan
## Shared RoCE NIC with macvlan or ipvlan

The following steps demonstrate how to enable shared usage of RDMA devices by Pods in a cluster with two nodes via macvlan CNI:

Expand All @@ -39,7 +39,7 @@ The following steps demonstrate how to enable shared usage of RDMA devices by Po
~# ibstat mlx5_0 | grep "Link layer"
Link layer: Ethernet

Make sure that the RDMA subsystem on the host is in shared mode. If not, switch to shared mode.
Make sure that the RDMA subsystem of the host is in shared mode. If not, switch to shared mode.

~# rdma system
netns shared copy-on-fork on
Expand Down Expand Up @@ -220,7 +220,7 @@ The following steps demonstrate how to enable shared usage of RDMA devices by Po
2 1000 6.88 16.81 7.04 7.06 0.31 7.38 16.81
---------------------------------------------------------------------------------------

### Isolated RoCE NIC with SR-IOV
## Isolated RoCE NIC with SR-IOV

The following steps demonstrate how to enable isolated usage of RDMA devices by Pods in a cluster with two nodes via SR-IOV CNI:

Expand Down

0 comments on commit bace082

Please sign in to comment.