Has anyone tried to get OpenEBS cStor working for storage

[ShaunMaher]

Hi all.

I'm trying to get a production ready Talos cluster build working. I'm a bit stuck on storage and backups. I need to have backups go to storage external to the cluster (e.g. S3). I think I have rounded the options down to:

• Longhorn
• OpenEBS cStor
• Rook/Ceph

But why not...

• Mayastor: "Mayastor has no snapshot or cloning capabilities"
  • Reference: https://mayastor.gitbook.io/introduction/quickstart/known-limitations
• OpenEBS-Jiva: "Jiva is a light weight storage engine that is recommended
  to use for low capacity workloads. The snapshot and storage management
  features of the other cStor engine are more advanced and is recommended when
  snapshots are a need." - GRRRR
  • Reference: https://openebs.io/docs/user-guides/jiva/jiva-prerequisites
• Piraeus: Yeah, maybe. Doco mentions snapshots but nothing about sending
  said snapshots to remote storage.

I have used Longhorn on Rancher on RKE2 on Ubuntu and it was quite good. It has it's own built-in backup facility to send to S3 storage. There are active bug reports where people are working on getting Longhorn working on Talos so I haven't yet tried it myself.

I spent a little time getting Rook/Ceph working on Talos and, for storage, it seems like a great solution. The issue I had though was backups. Velero is able to tell Rook to create snapshots, which rook dutifully does but there is no mechanism (yet) for taking those snapshots and sending them to S3 storage.

Next test is with OpenEBS cStor. I am able to get cStor installed from the Helm charts. I have modified and applied the iscsiadm ConfigMap from the OpenEBS-Jiva instructions (https://www.talos.dev/v1.3/kubernetes-guides/configuration/replicated-local-storage-with-openebs-jiva/). I have patched the daemonset with hostPID: true. I can create a storage pool and a storage class. I try to provision a pod that consumes a PV and... no good.

I can see that a zfs volume is created for the PV. It looks to me like the iSCSI Target is being created. The iSCSI client doesn't seem to do it's part though. I can't work out why. The closest thing to an error message I can find in the logs is:

E0125 03:29:11.539991       1 controller_base.go:321] error syncing 'openebs/pvc-1559bdd7-811e-405a-ab94-3f3b81f8d4fd': failed to create PDB for volume: pvc-1559bdd7-811e-405a-ab94-3f3b81f8d4fd: failed to list PDB belongs to pools with selector openebs.io/cstor-disk-pool-9ffx=true,openebs.io/cstor-disk-pool-gwv8=true,openebs.io/cstor-disk-pool-vjw6=true: the server could not find the requested resource, requeuing
I0125 03:29:11.540053       1 event.go:282] Event(v1.ObjectReference{Kind:"CStorVolumeConfig", Namespace:"openebs", Name:"pvc-1559bdd7-811e-405a-ab94-3f3b81f8d4fd", UID:"6b8089b3-b4e0-4ff1-ae6c-b72d31052d08", APIVersion:"cstor.openebs.io/v1", ResourceVersion:"149987", FieldPath:""}): type: 'Warning' reason: 'Provisioning' failed to create PDB for volume: pvc-1559bdd7-811e-405a-ab94-3f3b81f8d4fd: failed to list PDB belongs to pools with selector openebs.io/cstor-disk-pool-9ffx=true,openebs.io/cstor-disk-pool-gwv8=true,openebs.io/cstor-disk-pool-vjw6=true: the server could not find the requested resource

This looks ridiculous for a show stopper. I can't find any pods that match the selector "openebs.io/cstor-disk-pool-9ffx=true,openebs.io/cstor-disk-pool-gwv8=true,openebs.io/cstor-disk-pool-vjw6=true", so, the message is correct. Is my issue simply that some pods didn't get labels when they were provisioned?

What have others done for storage and backups? Has anyone gotten OpenEBS cStor working on Talos?

Any help that can be provided would be much appreciated, and, if possible, reciprocated.

Cheers.
Shaun.

[ShaunMaher]

Update: Apparently the issue I'm having is a known cStor issue on >= K8s 1.25: openebs-archive/cstor-operators#435

[vladimirfx]

We have exaclty same issue. Did you able to deploy v3.4.0 of cStore Operator ?

[ShaunMaher]

I ended up deploying the version 3.4.0 chart from here: https://github.com/niladrih/cstor-operators/tree/helm-3.4.0 and it worked.

I'm still working on getting Velero backups working, but the OpenEBS cStor stuff worked.

[trunet]

Could you kubectl top pod -n [YOUR CSTOR NAMESPACE]; kubectl resource-capacity --pods -n mayastor (you'll need https://github.com/robscott/kube-capacity for the second command) to see what's the limits and consumption of cStor components? I have mayastor deployed, but would like snapshots for velero and I'm thinking on going cStor as well, but I have a modest i3 16GB NUCs and not much spare capacity, I need something lightweight.