Orchestrating Cloud Resources with ArgoCD and Terraform
Automating ArgoCD using Terraform
In recent years, GitOps has become a hot topic, and ArgoCD has been at the forefront of this conversation. While I’m a big fan of GitOps, my day-to-day work also involves managing cloud infrastructures on GPC, AWS, and on-premises, where Terraform is my go-to tool for orchestrating a large number of cloud resources.
I’ve recently started working on a project utilizing a RKE2 Kubernetes Cluster with Cilium as the preferred networking solution. On top of that, to handle multiple tenants and different projects efficiently, I’ve chosen ArgoCD for application orchestration. It’s not only a popular choice, but I also appreciate its approach of declarative application setup, which allows configurations to be scattered across various Version Control Systems like GitLab, GitHub, and Bitbucket — be it either public or private. As you might imagine, setting up a comprehensive orchestration system involves considering many components. In this post, I’ll primarily focus on the integration of ArgoCD and Terraform, highlighting how these tools work together seamlessly for effective application orchestration. I have successfully incorporated this approach into various projects, demonstrating its ease of understanding and smooth management.
ArgoCD
For newcomers, ArgoCD is an innovative and open-source tool designed for declarative continuous delivery of Kubernetes applications. As a GitOps continuous delivery tool, ArgoCD helps streamline and automate the deployment and management of applications in Kubernetes clusters. It operates on the principle of defining the desired state of your applications in a Git repository, and ArgoCD ensures that the actual state of the applications in the cluster converges to the declared state.
When examining the ArgoCD infrastructure, it’s evident that it’s designed with modularity in mind, comprising several dynamic components. On one side, you have either the user or the CI pipeline initiating the deployment of a new application, and on the other, there’s the Kubernetes cluster where the actual deployment is targeted. ArgoCD, alongside your preferred Version Control System (VCS), serves as the intermediary, offering the flexibility to configure it according to your preferences.
This flexibility allows you to decide which parts and how to automate and control the access levels for different teams within your company. The setup we’ll delve into has proven effective in a production environment, and it’s been well-received by developers who joined the project later.
That covers the design choices; now, let’s dive into technical details.
Requirements
This guide assumes you have a functional Kubernetes cluster and have already installed the following prerequisites:
The specific Kubernetes distribution you’re using is not a critical factor, as long as you have the necessary access to deploy applications on it. I choose Rancher Kubernetes Engine 2 (RKE2) since it is a lightweight and versatile Kubernetes distribution known for its simplicity and ease of use, while complying with latest security standards. For deeper insights into RKE2, you can explore my earlier article, which also outlines how seamlessly it integrates with the features of Cilium:
The decision to use Sealed Secrets is influenced by the on-premises deployment, distinct from managed solutions offered by GCP, AWS, or Azure. If leveraging a cloud provider, SOPS stands out as a notable alternative for encrypting and securely storing deployment-related secrets. However, in our on-premises scenario, we’ve embraced a more Kubernetes-native solution:
Last but not least, while OpenTofu or Ansible could theoretically achieve similar outcomes, I’ve found no issues with HashiCorp BSL licensing change, leading me to stick with Terraform. For practical guidance on selecting between Terraform and Ansible, check out this resource:
Terraform
We’ll kick off with Terraform, as it typically serves as the starting point for infrastructure automation — the backbone for the applications. In our specific case, it will reference our pre-deployed Kubernetes cluster and deploy the ArgoCD Helm chart. Since we will also deploy some applications on top of it, we also need to specify the location for additional ArgoCD configurations, which will, in turn, reference the source repositories for applications.
terraform {
required_version = ">= 0.13"
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.18.1"
}
kubectl = {
source = "gavinbunney/kubectl"
version = "1.14.0"
}
helm = {
source = "hashicorp/helm"
version = "2.11.0"
}
}
}
# Create a namespace for ArgoCD deployments
resource "kubernetes_namespace" "argo" {
metadata {
name = var.argo_namespace
}
}
# Install ArgoCD Helm chart
resource "helm_release" "argocd" {
depends_on = [kubernetes_namespace.argo]
name = "argocd"
repository = "https://argoproj.github.io/argo-helm"
chart = "argo-cd"
version = "5.36.7"
namespace = var.argo_namespace
create_namespace = false
wait = true
wait_for_jobs = true
}
# Install sealed-secrets for secret encryption
resource "helm_release" "sealed_secrets" {
name = "sealed-secrets"
repository = "https://bitnami-labs.github.io/sealed-secrets"
chart = "sealed-secrets"
version = "2.13.2"
namespace = "kube-system"
create_namespace = false
wait = true
wait_for_jobs = true
}
# Apply SSH key for usage by private GitHub repositories
resource "kubectl_manifest" "gh_private_repo_key" {
yaml_body = file("sealed-ssh-secret.yaml")
}
# Now we add the repository where the argo config lies -
# only needed because it is private. In this way, we have a repo
# dedicated to argo config and everybody can file PRs to it.
# NOTE: ArgoCD Repository is defined as a Kubernetes secret - don't be confused by that
resource "kubernetes_secret_v1" "argo_config_repo" {
metadata {
name = "argocd-config-repo"
namespace = var.argo_namespace
labels = {
"argocd.argoproj.io/secret-type" = "repository"
}
}
data = {
name = "argocd-config"
type = "git"
url = "<argo-config-repository-url>"
sshPrivateKey = "${file("~/.ssh/id_ed25519")}"
}
}
# Create an application which in turn applies the ArgoCD files
# from the config repo.
resource "kubectl_manifest" "application_argo_config" {
yaml_body = yamlencode({
apiVersion = "argoproj.io/v1alpha1"
kind = "Application"
metadata = {
name = "argo-config"
namespace = var.argo_namespace
}
spec = {
destination = {
# This parameter should be changed in case applications should be
# deployed to an external cluster.
# External clusters need to be registered beforehand
server = "https://kubernetes.default.svc"
}
project = "default"
sources = [
{
path = "./"
repoURL = "<argo-config-repository-url>"
targetRevision = "main"
}
],
syncPolicy = {
automated = {
prune = true
selfHeal = false
}
}
}
})
}This configuration is intentionally made as simple as possible and makes presumption that the SSH private key for your VCS like GitHub is stored in .ssh/id_ed25519 and that you’re able to access your cluster using the Kubernetes config located in .kube/config.
In summary of the code comments, the process involves:
- Installing ArgoCD and Sealed-Secrets Helm charts.
- Referencing and connecting to the repository where the complete ArgoCD configuration is stored.
- Applying the configuration from the ArgoCD config repository.
❓ Now, one might wonder: Why is the ArgoCD configuration stored in a separate repository?
The rationale behind this lies in the potential complexity of the ArgoCD configuration. Organizing it into a separate repository allows for a more structured approach, as we’ll delve into in the following section. Also considering that the repository containing Terraform configuration might involve not only ArgoCD but also various other infrastructure-related deployments, opting for a distinct ArgoCD configuration repository contributes to clarity and manageability. For example, consider the deployment of Teleport using Terraform, which I thoroughly discuss in this post:
Secret Management
An important detail highlighted in the Terraform script is the installation and utilization of Sealed-secrets for encrypting the SSH private key. This key is crucial for ArgoCD to be able to establish a connection and retrieve your application configuration from a designated GitHub repository. There are various approaches to configuring this, but we chose to employ the label argocd.argoproj.io/secret-type: repo-creds on the Kubernetes Secret object. This label informs ArgoCD that any repository beginning with a specific prefix, such as git@github.com:<your-github-username>, can use this secret—in our case, the SSH private key—for authentication. For a clearer understanding, here is the complete configuration of the (temporary) file ssh-secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: argoproj-ssh-creds
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repo-creds
stringData:
url: git@github.com:<your-github-username>
type: git
data:
sshPrivateKey: <your-super-secret-ssh-key>However, pushing this directly to our Version Control System (VCS) poses a security risk as the SSH private key becomes visible to everyone. To address this concern, we take a precautionary step by encrypting it using Sealed-secrets, resulting in the creation of sealed-ssh-secret.yaml. This encrypted file is then actually applied by the Terraform script mentioned earlier and the ssh-secret.yamlcan be deleted. While delving into the intricacies of how Sealed Secrets work is beyond the scope here, you can refer to this post for a detailed exploration:
With these steps, the essential tasks of Terraform are completed. Yet, the final puzzle piece lies in understanding the content within the ArgoCD configuration repository that this script references.
ArgoCD Configuration Repository
Having a declarative configuration for ArgoCD is recommended for potential redeployments. This repository acts as the primary source for Argo’s configuration. Notably, if you create your apps through the UI, keep in mind that they won’t be automatically recreated during an ArgoCD redeployment.
The repository adheres to the following structure, a subjective choice that I personally find compelling and straightforward:
├── apps
│ ├── demo
│ │ ├── config.yaml
├── config
│ ├── add-config.yaml
├── projects
│ ├── demo.yaml
├── repositories
│ ├── demo-app-config.yaml
├── argo-config.yaml
├── argo-projects.yaml
└── argo-repositories.yamlTo clarify the rationale behind individual files and why I recommend this structure, let’s start with the root directory:
./apps: This directory manages the configuration for an ArgoCD Application object and points to an external repository where the application configuration is stored. An example of the/apps/demo/config.yamlfile would be:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: demo
spec:
destination:
name: 'in-cluster'
namespace: <app-namespace>
source:
repoURL: '<URL-of-the-GitHub-repository>'
targetRevision: <target-branch>
path: './'
# ArgoCD project to which this application belongs
project: demo./config: This directory is designated for additional configurations specific to ArgoCD itself. An example includes the configuration of a notification policy, such as connecting the company Slack channel to inform developers about application-related issues../projects: This directory oversees the creation of ArgoCD projects, serving as a collective space for various applications to belong to../repositories: This directory functions as a whitelist, specifying repositories from which application configurations can be pulled../argo-config.yaml,./argo-projects.yamland./argo-repositories.yaml: ArgoCD Application objects executed by Terraform. They, in turn, call the ArgoCD objects defined in the previously described directories. Here’s an example of theargo-config.yamlbut similar configurations apply to the others:
# The base config for Argo. This is the app which is installed
# by the helm/terraform deployment.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argo-config
namespace: argocd
spec:
destination:
namespace: argocd
server: https://kubernetes.default.svc
project: default
# Apply what is in the ./config directory of respective repository
source:
path: ./config
repoURL: <argocd-config-repository-url>
targetRevision: <target-branch>For the purposes of the demo application, configuration files have been added to these folders:
projects/demo.yaml:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: demo
namespace: argocd
# Finalizer that ensures that project is not deleted until it is not referenced by any application
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
description: Demo Application
# Allow manifests to deploy from any Git repos
sourceRepos:
- '*'
# Allow all applications to deploy to all namespaces in all available clusters
destinations:
- namespace: '*'
server: '*'
# Policies to determine which Kubernetes objects can be deployed by the app
clusterResourceWhitelist:
- group: ''
kind: Namespace
- group: ''
kind: Deployment
- group: ''
kind: Service
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: demo-apps
namespace: argocd
spec:
destination:
namespace: argocd
server: https://kubernetes.default.svc
project: default
sources:
- path: ./apps/demo
repoURL: <application-repository-url>
targetRevision: <target-branch>repositories/demo-app-config.yaml:
apiVersion: v1
kind: Secret
metadata:
labels:
argocd.argoproj.io/secret-type: repository
name: repo-kubernetes-config
namespace: argocd
stringData:
name: "demo-app-config"
type: "git"
url: <application-repository-url>
# No need to specify SSH Private Key since
# this is done by repo-creds in Terraform and applicable to allThis explanation sheds light on why we opted for this repository structure and why it was separated from the Terraform repository, even for a relatively simple example.
Utilizing a distinct Git repository for your Kubernetes manifests, separate from your application source code, is highly recommended for several reasons:
- Clean Separation: It enables a clear distinction between application code and configuration, allowing isolated modifications to manifests without triggering unnecessary CI builds.
- Audit Log Clarity: A configuration-only repository ensures a cleaner Git history for audit purposes, eliminating noise from regular development activities.
- Microservices Deployment: For applications composed of services from multiple repositories but deployed as a single unit, storing manifests in a central configuration repository accommodates varying versioning schemes and release cycles.
- Access Control: Separating source code and configuration repositories allows for distinct access controls, ensuring that developers working on the application might not have direct access to production environments.
- CI Pipeline Stability: Avoiding an infinite loop of build jobs and Git commit triggers is achieved by pushing manifest changes to a separate repository, maintaining stability in the CI pipeline.
And that’s the entire setup — one repository for what we commonly refer to as infrastructure resources, separate repositories for configurations specific to each infrastructure component, and a third one for the application manifests themselves. Pretty cool, isn’t it?
Conclusion
In the realm of cloud infrastructure management, the GitOps approach, notably championed by ArgoCD, has gained significant traction. In this journey through orchestrating cloud resources, Terraform emerges as the linchpin for handling a multitude of resources. Delving into a real-world project utilizing an RKE2 Kubernetes Cluster, the focus narrows onto ArgoCD for streamlined application orchestration, exploring its integration with Terraform for a seamless, declarative setup. The post provides a practical guide, shedding light on the rationale behind design choices, the necessity of a clean repository structure, and the harmonious interplay between ArgoCD and Terraform in the orchestration landscape.
To stay current with the latest cloud technologies, make sure to subscribe to my weekly newsletter, Cloud Chirp. 🚀
