How to reference secrets from different namespace in Kustomization?

[iavkam]

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: gotk-system
  namespace: gotk-system
spec:
  interval: 1m0s
  url: my-cluster-config.git
...

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: workloads
  namespace: default
spec:
  interval: 2m0s
  path: ./path/to/sync
  prune: true
  sourceRef:
    kind: GitRepository
    name: gotk-system
    namespace: gotk-system
...

This don't work

[stefanprodan]

What version are you using? What's the logged error for the above config?

[vladimirfx]

There is no 'namespace' attribute in Kustomization.spec.sourceRef . In 'decryption' section namespace is absent too.

[stefanprodan]

We don't allow cross-namespace secrets ref in the toolkit APIs. As for the sourceRef it does have a namespace field.

[vladimirfx]

Ok, in case of same repository (GitRepository reference) users should provide copy of SOPS secret? Or there is another, idiomatic way?

[stefanprodan]

I've tested this with gotk v0.1.8 and works fine

$ cat << EOF | kubectl apply -f -
> apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
> kind: Kustomization
> metadata:
>   name: workloads
>   namespace: default
> spec:
>   interval: 2m0s
>   path: ./path/to/sync
>   prune: true
>   sourceRef:
>     kind: GitRepository
>     name: gotk-system
>     namespace: gotk-system
> EOF
kustomization.kustomize.toolkit.fluxcd.io/workloads created

[iavkam]

Sorry, we were wrong. We meant in the section "decryption":

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: workloads
  namespace: default
spec:
  interval: 2m0s
  path: ./path/to/sync
  prune: true
  sourceRef:
    kind: GitRepository
    name: gotk-system
    namespace: gotk-system
  decryption:
    provider: sops
    secretRef:
      name: sops-pgp
      namespace: gotk-system
...

[stefanprodan]

As I said, we don't allow secrets to be pulled across namespaces as it would make multi-tenancy impossible. You have to keep secrets in the same namespace as the custom resource.

[iavkam]

It turns out that we need to copy the same secrets to different namespaces. Are there any best practices on this issue?

[stefanprodan]

You can avoid this by having the Kustomization objects in the system namespace, where the sources are.

[stealthybox]

@iavkam you have several options for syncing secrets across namespaces:

1. config generation in your repo:

ex: create a Makefile that allows you to make copy-sops-secrets
run it in CI (ex: GitHub Actions)

For your use-case, this might be hard to bootstrap depending on your setup

2a. controller that copies some existing secret from inside the cluster to other namespaces:

https://cert-manager.io/docs/faq/kubed/
https://github.com/appscode/kubed

https://github.com/mittwald/kubernetes-replicator

2b. hierarchical namespace controller

HNC can copy Secrets from a parent NS into all child NS's
https://kubernetes.io/blog/2020/08/14/introducing-hierarchical-namespaces/
https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/hnc

( I think this is a particularly good option, but it's very new )

3. some external secret manager that syncs things from outside (AWS/GCP KMS)

ex:
https://github.com/external-secrets/kubernetes-external-secrets