Improve documentation, especially for Manila CSI (#354)

README.md

@@ -15,12 +15,13 @@ helm install my-release capi/<chartname> [...options]
 To list the available versions for the charts:
 
 ```sh
-helm search repo capi --devel --versions
+helm search repo capi --versions
 ```
 
 Currently, the following charts are available:
 
 | Chart | Description |
 | --- | --- |
 | [cluster-addons](./charts/cluster-addons) | Deploys addons into a Kubernetes cluster, e.g. CNI. |
+| [etcd-defrag](./charts/etcd-defrag/) | Installs a `CronJob` for running [etcd defragmentation](https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation). |
 | [openstack-cluster](./charts/openstack-cluster) | Deploys a Kubernetes cluster on an OpenStack cloud. |

charts/cluster-addons/README.md

@@ -14,6 +14,9 @@ see how addons are defined.
 
 - [Container Network Interface (CNI) plugins](#container-network-interface-cni-plugins)
 - [OpenStack integrations](#openstack-integrations)
+  - [Cinder CSI and storage class](#cinder-csi-and-storage-class)
+  - [Manila CSI and storage class](#manila-csi-and-storage-class)
+  - [Keystone authenticating webhook](#keystone-authenticating-webhook)
 - [Ingress controllers](#ingress-controllers)
 - [Metrics server](#metrics-server)
 - [Monitoring and logging](#monitoring-and-logging)
@@ -33,7 +36,13 @@ cni:
   type: cilium
 ```
 
-And to disable the installation of a CNI completely:
+> **NOTE**
+> 
+> When Cilium is used, the
+> [Cilium kube-proxy replacement](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/)
+> is configured by default with no further action required.
+
+To disable the installation of a CNI completely, use:
 
 ```yaml
 cni:
@@ -50,12 +59,13 @@ underlying infrastructure, for example
 [Container Storage Interface (CSI) implementations](https://kubernetes-csi.github.io/docs/)
 and [authenticating webhooks](https://kubernetes.io/docs/reference/access-authn-authz/webhook/).    
 
-This chart is able to deploy the CCM and the Cinder CSI plugin from the
+This chart is able to deploy the CCM, the Cinder and Manila CSI plugins and the Keystone
+authenticating webbook from the
 [Kubernetes OpenStack cloud provider](https://github.com/kubernetes/cloud-provider-openstack),
 which allows your Kubernetes cluster to integrate with the OpenStack cloud on which it is deployed.
 This enables features like automatic labelling of nodes with OpenStack information (e.g. server ID
 and flavor), automatic configuration of hostnames and IP addresses, managed load balancers for
-services and dynamic provisioning of RWO volumes.
+services and dynamic provisioning of RWO and RWX volumes.
 
 By default, the OpenStack integrations are not enabled. To enable OpenStack integrations on the
 target cluster, use the following in your Helm values:
@@ -65,6 +75,11 @@ openstack:
   enabled: true
 ```
 
+> **TIP**
+>
+> When using the [openstack-cluster chart](../openstack-cluster/), the OpenStack integrations
+> are enabled by default in the values for the chart.
+
 To configure options for the `[Networking]`, `[LoadBalancer]`, `[BlockStorage]` and `[Metadata]`
 sections of the cloud-config file, you can use Helm values, e.g.:
 
@@ -93,6 +108,95 @@ and the
 Additional configuration options are available for the OpenStack integrations - see
 [values.yaml](./values.yaml) for more details.
 
+### Cinder CSI and storage class
+
+The [Cinder service](https://docs.openstack.org/cinder/latest/) in an OpenStack cloud provides
+[block volumes](https://en.wikipedia.org/wiki/Block-level_storage) for workloads. These volumes
+can only be attached to a single pod at once, referred to as read-write-one (RWO).
+
+Cinder is available on the vast majority of OpenStack clouds, and so the Cinder CSI is installed
+by default whenever the OpenStack integrations are enabled. As part of this, a default
+[storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) is installed that
+allows Cinder volumes to be requested and attached to pods using
+[persistent volume claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims).
+This storage class uses the default Cinder volume type and the `nova` availability zone, and
+is configured as the default storage class for the cluster.
+
+To change the Cinder availability zone or volume type for the default storage class, use the
+following values:
+
+```yaml
+openstack:
+  csiCinder:
+    defaultStorageClass:
+      availabilityZone: az1
+      volumeType: fast-ssd
+```
+
+### Manila CSI and storage class
+
+In contrast to Cinder, the [Manila service](https://docs.openstack.org/manila/latest/) provides
+shared filesystems for cloud workloads. These volumes can be attached to multiple pods
+simultaneously, referred to as read-write-many (RWX).
+
+Because Manila is often not deployed on OpenStack clouds, it is not enabled by default even when
+the OpenStack integrations are enabled. To enable the Manila CSI, set the following variable:
+
+```yaml
+openstack:
+  csiManila:
+    enabled: true
+```
+
+Manila supports multiple backends, but currently only the [CephFS](https://docs.ceph.com/en/latest/cephfs/)
+backend is supported in the CAPI Helm charts. To utilise the CephFS support in the Manila CSI,
+the CephFS CSI plugin must also be enabled:
+
+```yaml
+csi:
+  cephfs:
+    enabled: true
+```
+
+By default, this will result in the Manila CSI creating volumes using the `cephfs` share type. If
+you need to use a different share type, use the following:
+
+```yaml
+openstack:
+  csiManila:
+    defaultStorageClass:
+      parameters:
+        type: cephfs_type
+```
+
+Any of the storage class parameters
+[specified in the Manila CSI docs](https://docs.ceph.com/en/latest/cephfs/) can be given under
+`openstack.csiManila.defaultStorageClass.parameters`. For example, to use the `kernel` mounter
+rather than the default `fuse` mounter, which can help performance, use the following:
+
+```yaml
+openstack:
+  csiManila:
+    defaultStorageClass:
+      parameters:
+        cephfs-mounter: kernel
+```
+
+### Keystone authenticating webhook
+
+The  [k8s-keystone-auth](https://github.com/heytrav/helm-charts/tree/main/charts/k8s-keystone-auth) 
+webhook can be installed by enabling the `k8sKeystoneAuth` subchart. Note that you will need to provide
+the **auth url** and **project id** for the Openstack tenant where you are deploying your cluster.
+
+```yaml
+openstack:
+  k8sKeystoneAuth:
+    enabled: true
+    values:
+      openstackAuthUrl: $OS_AUTH_URL
+      projectId: $OS_PROJECT_ID
+```
+
 ## Ingress controllers
 
 Running an
@@ -163,18 +267,3 @@ By default, Grafana is only available from within the cluster and must be access
 ```sh
 kubectl -n monitoring-system port-forward svc/kube-prometheus-stack-grafana 3000:80
 ```
-
-## Keystone Authentication Webhook
-
-The  [k8s-keystone-auth](https://github.com/heytrav/helm-charts/tree/main/charts/k8s-keystone-auth) 
-webhook can be installed by enabling the `k8sKeystoneAuth` subchart. Note that you will need to provide 
-the **auth url** and **project id** for the Openstack tenant where you are deploying your cluster.
-
-```yaml
-  k8sKeystoneAuth:
-    enabled: true
-    values:
-      openstackAuthUrl: $OS_AUTH_URL
-      projectId: $OS_PROJECT_ID
-
-```

charts/etcd-defrag/README.md

@@ -4,4 +4,4 @@ This chart installs a [CronJob](https://kubernetes.io/docs/concepts/workloads/co
 that will periodically defragment the etcd keyspace for a cluster that is managed using
 [kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/).
 
-It is installed as part of the [cluster-addons](../cluster-addons).
+It is installed as part of the [cluster-addons](../cluster-addons) by default.

charts/openstack-cluster/README.md

@@ -1,4 +1,4 @@
-# openstack-cluster chart
+# openstack-cluster chart  <!-- omit in toc -->
 
 This [Helm chart](https://helm.sh/) manages the lifecycle of a [Kubernetes](https://kubernetes.io)
 cluster on an [OpenStack](https://www.openstack.org/) cloud using
@@ -20,6 +20,22 @@ This README describes some of the basic options, however there are many other op
 available. Check out the [values.yaml](./values.yaml) (commented) and the chart
 templates for more details.
 
+## Contents  <!-- omit in toc -->
+
+- [Prerequisites](#prerequisites)
+- [OpenStack images](#openstack-images)
+- [OpenStack credentials](#openstack-credentials)
+- [Managing a workload cluster](#managing-a-workload-cluster)
+- [Multiple external networks](#multiple-external-networks)
+- [Volume-backed instances](#volume-backed-instances)
+- [Etcd configuration](#etcd-configuration)
+- [Load-balancer provider](#load-balancer-provider)
+- [Cluster addons](#cluster-addons)
+- [Accessing a workload cluster](#accessing-a-workload-cluster)
+- [Advanced](#advanced)
+  - [Flatcar support](#flatcar-support)
+  - [Keystone Authentication Webhook](#keystone-authentication-webhook)
+
 ## Prerequisites
 
 First, you must set up a
@@ -71,7 +87,8 @@ OpenStack credentials are required for two purposes:
   1. For Cluster API to manage OpenStack resources for the workload cluster, e.g. networks, machines.
   2. For OpenStack integrations on the workload cluster, e.g. OpenStack CCM, Cinder CSI.
 
-By default, this chart uses the same credentials for both.
+By default, this chart uses the same credentials for both, ensuring that the credential used for
+Cluster API operations is propagated to the workload cluster.
 
 The recommended way to do this is using an
 [Application Credential](https://docs.openstack.org/keystone/latest/user/application_credentials.html)
@@ -141,7 +158,7 @@ nodeGroups:
 To install or upgrade a cluster, use the following Helm command:
 
 ```sh
-helm upgrade my-cluster capi/openstack-cluster --devel --install -f ./clouds.yaml -f ./cluster-configuration.yaml
+helm upgrade my-cluster capi/openstack-cluster --install -f ./clouds.yaml -f ./cluster-configuration.yaml
 ```
 
 This will create a cluster on its own network with a three node, highly-available (HA)
@@ -174,7 +191,110 @@ command again. Some examples of updates that can be performed are:
     Once a new image is available, change the machine image and Kubernetes version
     as required to trigger a rolling upgrade of the cluster nodes.
 
-### Cluster addons
+## Multiple external networks
+
+When there is only one external network available to a project, Cluster API will attach
+its router to that network and use it to allocate any floating IPs.
+
+If the project you are using has multiple external networks available, you will need to
+explicitly tell Cluster API which one to use:
+
+```yaml
+clusterNetworking:
+  externalNetworkId: "<UUID of network>"
+```
+
+## Volume-backed instances
+
+Flavors with significant root disks (> 40GB, ideally 100GB) are recommended for Kubernetes
+nodes, as this is where downloaded images are stored which can be substantial in size.
+
+If flavors with large root disks are not available, it is possible to use volume backed instances
+instead.
+
+> **WARNING**
+>
+> [etcd requires fast disks](https://etcd.io/docs/v3.5/op-guide/hardware/#disks) in order to
+> operate reliably, so volume-backed instances are only appropriate if the storage system uses
+> SSDs - **network-attached spinning disk will not be fast enough**.
+>
+> The recommended OpenStack configuration is to use local disk on the hypervisor for ephemeral
+> root disks if possible.
+
+To configure Kubernetes nodes to use volume-backed instances, use the following options:
+
+```yaml
+controlPlane:
+  machineRootVolume:
+    diskSize: 100
+    volumeType: fast-ssd
+    availabilityZone: nova
+
+nodeGroupDefaults:
+  machineRootVolume:
+    diskSize: 100
+    volumeType: fast-ssd
+    availabilityZone: nova
+```
+
+## Etcd configuration
+
+If you do not have much SSD capacity, it is possible to configure Kubernetes nodes so that
+etcd is on a separate block device, using a different volume type. This allows the block
+device that requires SSD to be much smaller.
+
+To do this, use the following options:
+
+```yaml
+etcd:
+  blockDevice:
+    size: 10
+    volumeType: fast-ssd
+    availabilityZone: nova
+```
+
+This can be used in combination with volume-backed instances using a volume type backed
+by spinning disk, e.g.:
+
+```yaml
+etcd:
+  blockDevice:
+    size: 10
+    volumeType: fast-ssd
+    availabilityZone: nova
+
+controlPlane:
+  machineRootVolume:
+    diskSize: 100
+    volumeType: slow-platters
+    availabilityZone: nova
+
+nodeGroupDefaults:
+  machineRootVolume:
+    diskSize: 100
+    volumeType: slow-platters
+    availabilityZone: nova
+```
+
+## Load-balancer provider
+
+If the target cloud uses the
+[OVN Octavia driver](https://docs.openstack.org/ovn-octavia-provider/latest/admin/driver.html),
+Kubernetes clusters should be configured to use OVN for any load-balancers that are created,
+either by Cluster API or by the OpenStack Cloud Controller Manager for `LoadBalancer` services:
+
+```yaml
+apiServer:
+  loadBalancerProvider: ovn
+
+addons:
+  openstack:
+    cloudConfig:
+      LoadBalancer:
+        lb-provider: ovn
+```
+
+## Cluster addons
 
 The cluster addons are enabled by default. You can configure which addons are deployed
 and the configuration of those addons by specifying values for the addons Helm chart:
@@ -213,15 +333,17 @@ clusterctl get kubeconfig my-cluster > kubeconfig.my-cluster
 kubectl --kubeconfig=./kubeconfig.my-cluster get po -A
 ```
 
-## Flatcar
+## Advanced
+
+### Flatcar support
 
 To deploy clusters which use Ignition such as Flatcar, you will need to override the following setting in your local `values.yaml`:
 
 ```yaml
 osDistro: flatcar 
 ```
 
-## Keystone Authentication Webhook
+### Keystone Authentication Webhook
 
 To deploy with the k8s-keystone-auth webhook enabled, set `authWebhook`
 to "k8s-keystone-auth".