IaC

[james-otten]

• Terraform for creating node VMs in proxmox
• Helm chart for creating resources
• Incomprehensible instructions for creating a new environment (still includes manual steps)

Overview

• Proxmox
• K3s
• Metallb
• Longhorn

Github secrets to be added

• DEV0_KEY
• DEV0_KNOWN_HOSTS
• DEV0_SSH_TARGET
• DEV0_PROJECT_PATH

TODO

• Add github secrets (above)
• Add CD
• Setup certs - Not something I can do
• Zero checkov findings

[james-otten]

I think this is ready for initial review. dev0 isn't "done done" due to TLS, but these changes are additive and I'd like to test CD.

[WillNilges]

So we gotta fill out the example.tfvars?

[james-otten]

You need to fill out the values in whatever var file you reference

[WillNilges]

Can that be added/explained in the docs?

[james-otten]

Good call, updated step 2

[WillNilges]

You might want to call this out in the README as well

[james-otten]

This one doesn't need to exist (the vm already exists)
The other ones are mentioned as ../../../../values.yaml and ../../../../secret.values.yaml

[WillNilges]

Some instructions on how to get the kubeconfig might be good.

[WillNilges]

scp -i tf/<private key> ubuntu@<control node>:/etc/rancher/k3s/k3s.yaml ./

[Andrew-Dickinson]

Woah helm is insane. Y'all can run this show, I'll sit this one out

[rossbannerman]

LGTM! A couple of optional/nice to have things we might want to consider a bit further down the track:

• Using resource type StatefulSet instead of Deployment for redis and postgres
• Leaning on ingress-nginx (or another ingress controller) as opposed to a static nginx config
• Having Terraform bootstrap the k3s cluster with ArgoCD and then have it manage rendering and applying charts/manifests

[WillNilges]

Getting this when I try to run stage 2 again.

null_resource.mgr_stage_two: Provisioning with 'file'...
╷
│ Error: file provisioner error
│
│   with null_resource.mgr_stage_two,
│   on stage2config.tf line 9, in resource "null_resource" "mgr_stage_two":
│    9:   provisioner "file" {
│
│ Error connecting to SSH_AUTH_SOCK: dial unix /tmp/wilnil.agent: connect: no such file or directory

What exactly is the stage 2? It looks like it's setting up metallb and such? The stage2 script is running terraform... where exactly? I see a terraform zip on the k3s nodes... that seems a bit cursed. I have no idea if this is the kind of thing we should be doing with terraform.

Update: Looks like it wanted my ssh agent. When I start my ssh agent, it seems to be able to SSH, and I get this:

null_resource.mgr_stage_two: Provisioning with 'remote-exec'...
null_resource.mgr_stage_two (remote-exec): Connecting to remote host via SSH...
null_resource.mgr_stage_two (remote-exec):   Host: 10.70.90.230
null_resource.mgr_stage_two (remote-exec):   User: debian
null_resource.mgr_stage_two (remote-exec):   Password: false
null_resource.mgr_stage_two (remote-exec):   Private key: true
null_resource.mgr_stage_two (remote-exec):   Certificate: false
null_resource.mgr_stage_two (remote-exec):   SSH Agent: true
null_resource.mgr_stage_two (remote-exec):   Checking Host Key: false
null_resource.mgr_stage_two (remote-exec):   Target Platform: unix
null_resource.mgr_stage_two (remote-exec): Connected!
null_resource.mgr_stage_two: Still creating... [1m10s elapsed]
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 4: cd: meshdb: No such file or directory
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 5: git: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 6: cd: infra/cluster: No such file or directory
null_resource.mgr_stage_two: Still creating... [1m20s elapsed]
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 10: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 11: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 12: terraform: command not found
null_resource.mgr_stage_two (remote-exec): sed: can't read metallb_extra.yaml: No such file or directory
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 15: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 16: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 17: terraform: command not found
null_resource.mgr_stage_two: Still creating... [1m30s elapsed]
null_resource.mgr_stage_two: Still creating... [1m40s elapsed]
null_resource.mgr_stage_two: Still creating... [1m50s elapsed]
null_resource.mgr_stage_two (remote-exec): error: the path "/opt/meshdb_mgmt/meshdb/infra/cluster/metallb_extra.yaml" does not exist
╷
│ Error: remote-exec provisioner error
│
│   with null_resource.mgr_stage_two,
│   on stage2config.tf line 14, in resource "null_resource" "mgr_stage_two":
│   14:   provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_1068678211.sh": Process exited with status 1
╵

Confused as to why the SSH agent is necessary, but that aside, it seems like stage2 is expecting some stuff that isn't present. Could be due to me not having the agent set up when I was originally configuring.

Update

Got logs from a complete from-scratch run (which I think is really the way to do it). I think we gotta wait for the dpkg lock. Experimenting right now...

Interesting: null_resource.mgr_config_files (remote-exec): cp: cannot stat 'meshdb/infra/helm/meshdb/secret.values.yaml': No such file or directory. I guess that's because the branch is different. I wonder if we should just copy them over from the computer via terraform.

Gonna track stuff here: #368

[WillNilges]

one sec cooking