cockroachTerraformAnsible Deploy CockroachDB on kubernetes using terraform and ansible

Purpose

Demo deployments of CockroachDB with kubernetes using terraform and ansible. Has GKE (Google), EKS (AWS) and AKS (Azure) terraform deployments. The Ansible code is deployed from a terraform provisioner subdirectory for each cloud provider. Ansible code differences between cloud providers are minimal. The ansible code could also be used in on-premise situations.

Outline

• Useful Links
• K8s and ansible on macOS
• Single Region
  • Single Region GKE
    • Terraform Creation GKE
  • Single Region EKS
    • Terraform Creation GKE
  • Single Region AKS
    • Terraform Creation AKS
• Multi region
  • GKE
• Ansible Documentation

Useful Links

• Deploy CockroachDB on kubernetes using manual config files
• Single Region CockroachDB
• Multi-Cluster Deployment
• Python Virtual Environment

k8s ansible on mac

Prior to Mac OS 14, using a global python3 worked with ansible. It no longer works well-using venv instead. easiest to install ansible with brew install ansible

python3 -m venv ansible-venv
source ansible-venv/bin/activate
pip3 install requests
pip3 install google-auth
pip3 install kubernetes
pip3 install psycopg2
brew install ansible

Make sure to run terraform and ansible with this environment sourced

Single Region

GKE

• set the parameters in the main terraform job file
  • main.tf file includes version information such as gke_release_channel, gcp credentials, region and zone
  • no need to set the variables in main parameter file
  • this parameter file is only needed for reruns to disable parts of the operation

Terraform creation GKE

kick off terraform creation-the gke creation takes a very long time-over 10 minutes

source ansible-venv/bin/activate
cd terraform/test
terraform init
terraform apply --auto-approve

• The terraform and ansible will deploy using the steps referenced in this cockroachDB documentation
• To connect with the GCP need to run gcloud container clusters get-credentials
  • The appropriate parameters for this are available in the GCP cloud interface by selecting the cluster and clicking on the connect link near the top of the UI
  • Set the context to the cockroach namespace with kubectl config set-context --current --namespace=cockroach
• To finish the steps referenced above, continue at Step 3. Use the Built-in SQL Client at substep 1. NOTE: the client.yaml is already created so that step is not needed

Terraform creation EKS

kick off terraform creation-the eks creation takes a long time-over 10 minutes

source ansible-venv/bin/activate
cd terraform/test
terraform init
terraform apply --auto-approve

• The terraform and ansible will deploy using the steps referenced in this cockroachDB documentation
• To connect with the GCP need to run gcloud container clusters get-credentials
  • The appropriate parameters for this are available in the GCP cloud interface by selecting the cluster and clicking on the connect link near the top of the UI
  • Set the context to the cockroach namespace with kubectl config set-context --current --namespace=cockroach
• To finish the steps referenced above, continue at Step 5

EKS

• set the parameters in the main terraform job file
  • main.tf file includes version information such as cluster_prefix, credentials, IP addresses, region, vm_size, versions
  • no need to set the variables in main parameter file
  • this parameter file is only needed for reruns to disable parts of the operation

Terraform creation AKS

kick off terraform creation but must re-initialize aks if a cluster t

source ansible-venv/bin/activate
cd terraform-azure/test
terraform init
terraform apply --auto-approve

• The terraform and ansible will deploy using the steps referenced in this cockroachDB documentation
• To finish the steps referenced above, continue at Step 5

Multi-region

GKE multi-cluster

• set the parameters in the main terraform job file
  • main.tf file includes version information such as gke_release_channel, gcp credentials, region and zone
  • no need to set the variables in main parameter file
  • this parameter file is only needed for reruns to disable parts of the operation

Ansible documentation

• In each of the terraform directories, the ansible is initiated using provisioning.tf
  • For Single Region AKS, provisioning.tf
  • For Single Region GKE, provisioning.tf
  • For Multi-Cluster GKE, provisioning.tf
• The common single region ansible code is ansible
  • There are directories here for the certificates and the ca cert
  • The cockroach directory has the ansible and k8s code to deploy cockroach on kubernetes
  • The common playbook.yml starts kubernetes ansible
• The multi-cluster ansible code is ansible-multiregion
  • This code implements this multi-cluster documentation
    • However, step 7 in these steps which defines RBAC roles are not done.
  • The cockroach directory has the ansible and k8s code