kustomize-controller: SOPS support

[stefanprodan]

Flux v1 has partial support for decrypting secrets with SOPS. Flux v1 uses SOPS as a library when not using .flux.yaml. Flux v1 is able to detect YAMLs encrypted with SOPS and decrypt them before applying them on the cluster.

Unlike Flux v1, kustomize-controller runs as non-root inside an Alpine container with readonly rootfs and with ca-certificates being the only OS package installed. While this brings the attack surface to a minimum, it also makes impossible to use SOPS as a package or as an executable since SOPS shells-out to gpg and other executables.

Another concern is around service accounts and IAM roles, as each repository could have its own KMS entry, this means we have to run sops under different accounts.

I see a couple of options to integrate SOPS:

• use SOPS as an executable, and install all the OS packages it needs such as gnupg, glib, gnutls (frequent CVEs, increased attack surface, root user needed?, all IAM roles for the KMS must bind to the kustomize-controller service account breaking multi-tenancy)
• use SOPS as a package (all the above downsides plus go modules nightmare as we need to override sops deps that conflict with the toolkit deps)
• create a dedicated service/job that spins up a pod with sops installed, the keys mounted in volumes and the right service account with access to KMS/Vault/etc. This service/job would be controlled by kustomize-controller, it runs the decryption and return the individual YAMLs over HTTP to kustomize-controller.

[plod]

Option three seems to be the best option to me

[stefanprodan]

We've implemented SOPS using option 2. In the future we could use SOPS gRCP and run the decryption in a dedicated pod for KMS isolation.

[sbaildon]

Is there a design decision behind this note? When I started migrating to v2, I was expecting to wholly encrypt my manifests

Note that you should encrypt only the data section, encrypting the Kubernetes secret
metadata, kind or apiVersion is not supported by kustomize-controller.

[stefanprodan]

We replace Flux v1 code that was parsing YAMLs with the upstream Kubernetes library kyaml, while this brought major benefits, it made impossible for YAMLs to have the metadata, kind or apiVersion encrypted as it fails to parse.