Best practice for directory structure of GitOps repository

[haraldatbmw]

I'm testing around the optimal directory structure of the GitOps repo for my current project. My application is fairly simple consisting out of two containers (frontend and backend) which gets built and pushed to a registry by Jenkings CI. The frontend depends on the backend. The app needs to get deployed to 3 clusters: dev, test, prod.

By now I think it's the easiest way to put everything into one GitOps repository with the following structure.
What do you think about this approach?

1. All application-specific resources
2. Kustomize base resources (separated for common, backend and frontend)
3. Kustomize overlays for all environments (dev, test, prod)
4. GitOps toolkit resources
5. Toolkit kustomizations for the application with dependencies (backend depends-on common, frontend depends-on backend)
6. Toolkit components

.
├── app                             (1)
│   ├── base                        (2)
│   │   ├── backend
│   │   │   ├── kustomization.yaml
│   │   │   └── *.yaml
│   │   ├── common
│   │   │   ├── kustomization.yaml
│   │   │   └── *.yaml
│   │   └── frontend
│   │       ├── kustomization.yaml
│   │       └── *.yaml
│   └── overlays                    (3)
│       ├── dev
│       │   ├── backend
│       │   │   └── kustomization.yaml
│       │   └── frontend
│       │       └── kustomization.yaml
│       ├── test
|       |   └── ...
│       └── prod
|           └── ...
|
└── infra                           (4)
    ├── dev
    │   ├── app                     (5)
    │   │   ├── backend-kustomization.yaml
    │   │   ├── common-kustomization.yaml
    │   │   └── frontend-kustomization.yaml
    │   ├── gotk-system             (6)
    │   │   ├── kustomization.yaml
    │   │   ├── toolkit-components.yaml
    │   │   ├── toolkit-kustomization.yaml
    │   │   └── toolkit-source.yaml
    ├── test
    |   └── ...
    └── prod
        └── ...

[stefanprodan]

This kind of structure is what I had in mind when I wrote the bootstrap procedure. It works great when you have tens of services all managed by a single team. For multi-teams with hundreds of services you may want to create a repo per team and delegate the reconciliation to those repos, so each team can managed its own manifests. Another option is to place the manifests in each app repo and reference them in the main gitops repo.

[haraldatbmw]

@stefanprodan Thanks for your answer!
If I place the manifests in each app repo should I place it with all the kustomize base and overlays or just the naked kubernetes manifests and leave the kustomize stuff in the gitops repo? I know everything is possible but I'm looking for good experiences :-)

[tobbbles]

I have a general question about the management of sources + releases, compared to a kustomized 'install'

Given the above example, how would you manage git/helm sources + releases of aforementioned sources? I'm trying to achieve as much of a single source of truth for a cluster as possible.

It is unclear to me where, for example, a cert-manager release would reside, along with it's jetstack HelmRepository.

[stefanprodan]

Here is an example on how to organise a gitops repository for multi-env deployments https://github.com/fluxcd/flux2-kustomize-helm-example

[tobbbles]

Thanks for the really insightful example repo. I spot just one downfall (or potentially a lack of understanding from me.)

Take the following example for the current layout:

I want to patch my production redis release to ensure the usePassword: false value is indeed false; however for my staging release, I want to patch this to be true. With the current setup in the example, I can't see any possibility to set it up this way.

I'm trying to migrate from v1 -> v2 and hitting this barrier, where my v1 workloads (external + internal sources) would be patched per-environment; I'm struggling to replicate this setup with Flux v2

[stefanprodan]

You can follow the apps overlay structure and apply it to infra items.

[stefanprodan]

I can't see any possibility to set it up this way.

I guess you didn't read the docs on that repo, in case you want to do that here it is https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/README.md