Backstage: Infrastructure Automation Using Templates and Terraform

Leveraging Backstage Templates and Terraform for Infrastructure Automation

As I was grasping Backstage for the first time last year, one notable application pattern stood out: the focus on simplifying initial application setup. Whether configuring an HTTP Server or a Docker project, users could input parameters via a Template form, leading to the creation of a new GitHub repository with pre-generated code. However, I wondered if Backstage Templates could extend beyond their typical use, given their role in bridging the gap between application developers and infrastructure expertise. In this post, we’ll explore a scenario where Backstage Templates are used to streamline the generation of GCP service accounts within an existing Terraform-managed infrastructure. This approach aims to provide a seamless experience for application developers while reducing the workload for infrastructure administrators.

Source: Backstage Docs

First off, why automate this?

To be honest, the true advantages of this approach become apparent at scale, where it significantly boosts productivity for application developers and saves precious time for infrastructure administrators. By automating repetitive tasks, we alleviate the burden on administrators who are constrained by finite resources. With streamlined processes, such as reducing the time to review requests to a mere 10 seconds and a single click, Backstage templates make managing requests manageable, regardless of their volume.

Let’s delve into a typical scenario encountered by infrastructure administrators and the steps required to address it.

When There’s NO Automation

As an application developer or a user of cloud infrastructure, obtaining permissions for accessing specific resources like databases or PubSub topics typically involves several steps:

1. Requesting Permissions: The need for access prompts you to reach out to an infrastructure admin.
2. Waiting for Response: The admin may take hours or even days to respond to your request.
3. Frustration and Delay: During the waiting period, frustration builds as you’re unable to progress with your task and are forced to postpone it.
4. Revisiting the Issue: After some time has passed, you return to the request to follow up.
5. Received Permissions: Finally, you receive the permissions, but they may not be sufficient due to either underestimating the required permissions or administrative errors.
6. Repeating the Cycle: You’re forced to go through the cycle again, requesting additional permissions or corrections.
7. Resolution: After multiple iterations and potentially a week or more, the task is finally resolved.

These occurrences, though not ideal, are common, especially when administrators are overwhelmed or rushed. However, implementing a solution like Backstage can help streamline and expedite this process, making it more efficient for all parties involved.

Spotify’s Backstage: OIDC Authentication using Keycloak

When There IS Automation

Let’s walk through the same scenario with the proposed solution:

1. Requesting Permissions: You realize the need for permission to a specific cloud resource.
2. Configuring Permission Resource: Instead of reaching out to an admin, you configure the permission resource yourself using Backstage Templates.
3. Pipeline Validation: Your request passes through a pipeline that validates whether it’s accessing a highly secure resource.
4. Automatic Fulfillment or Admin Approval: If the resource isn’t highly secure, your request is automatically fulfilled, allowing you to proceed with your work. However, if the resource is highly secure, your request requires approval from the admin.
5. Resolution: Depending on the outcome, your request is either automatically fulfilled or awaits approval from the admin. If approved, you’re granted the necessary permissions, and you can continue with your tasks.

While this process still involves some waiting, the workload for the admin is significantly reduced, mainly limited to reviewing requests. If your request is fulfilled automatically, it benefits everyone involved. Now, let’s delve into the code to understand how this solution can be implemented in practice.

⚠️ Note: The following code requires a basic understanding of Backstage, Terraform and GCP, which we can’t fully cover here due to space constraints. This implementation serves as a practical demonstration of the discussed solution and can be customized to meet your specific needs.

Backstage Templates

I won’t dive too much into Backstage, but essentially one of the features it offers on the GUI is to fill a form which behind the scenes generates some code in whatever language you prefer — in our case, Terraform. This code can then be arbitrarily fed into any sort of pipeline, or just a simple PR, which, depending on the platform, can automatically trigger a GitHub Action, CloudBuild, or whatever you configure. Configuring stuff on the VCS platform side is slightly off-topic in this post, but let’s look at a simple Backstage template that generates a GCP service account with required permissions or roles.

Programmatically, it looks like this — Backstage really does a good job of providing a convenient way of configuring the Templates with many different options.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: gcp-service-account-tmpl
  title: GCP Service Account Template
  description: This template generates a Terraform file for GCP Service Account and opens a PR in BitBucket
  tags:
  - gcp
  - terraform
  links:
  - title: Documentation
    url: https://cloud.google.com/security/products/iam
    icon: docs
spec:
  # The main purpose of this field is for display purposes in Backstage, so that 
  # people looking at catalog items can get an understanding of to whom this Template belongs. 
  # It is not to be used by automated processes to for example assign authorization in runtime systems. 
  owner: you
  # The type of component created by the template, e.g. website. This is used for filtering templates, and 
  # should ideally match the Component spec.type created by the template.
  type: gcp_resource

  # Ref: https://backstage.io/docs/features/software-templates/writing-templates#specparameters---formstep--formstep
  parameters:
  - title: Fill in some steps
    required:
    - name
    - display_name
    properties:
      name:
        title: Name
        type: string
        pattern: '^[a-z-]+$' # Only lower letters and "-"
        description: Unique name for your Service Account
      display_name:
        title: Description
        type: string
        maxLength: 100
        description: Provide a short description
      roles:
        title: Roles
        type: array
        pattern: '^[a-zA-Z.]+$' # Only upper/lower letters and "."
        ui:options:
          addable: true
          orderable: false
          removable: true
        items:
          type: string
        description: "Reference: https://cloud.google.com/iam/docs/understanding-roles"
      permissions:
        title: Permissions
        type: array
        pattern: '^[a-zA-Z.]+$' # Only upper/lower letters and "."
        ui:options:
          addable: true
          orderable: false
          removable: true
        items:
          type: string
          title: permission
        description: "Reference: https://cloud.google.com/iam/docs/permissions-reference"
      bindings:
        title: Bindings
        description: "Reference: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity"
        type: array
        ui:options:
          addable: true
          orderable: false
          removable: true
        items:
          type: object
          title: Binding
          required:
            - role
            - members
          properties:
            role:
              type: string
              default: "iam.workloadIdentityUser"
              description: "Reference: https://cloud.google.com/iam/docs/understanding-roles"
            members:
              type: array
              title: Members
              items:
                type: string
  - title: Location & Ownership 
    required:
    - project
    - owner
    properties:
      project:
        title: Project
        type: string
        description: Choose a GCP Project
        enum: [
          "project-a",
          "project-b"
        ]
      owner:
        title: Owner
        type: string
        description: Owner of the component
        ui:field: OwnerPicker
        ui:options:
          catalogFilter:
          kind: User

  steps:
  - id: tf-template
    name: Create a Terraform file for GCP Service Account
    action: fetch:template
    input:
      url: ./skeleton
      targetPath: ./outputs
      values:
        name: ${{ parameters.name }}
        display_name: ${{ parameters.display_name }}
        roles: ${{ parameters.roles }}
        permissions: ${{ parameters.permissions }}
        bindings: ${{ parameters.bindings }}
        owner: ${{ parameters.owner }}
      
  - id: rename
    name: Rename template files
    action: fs:rename
    input:
      files:
      - from: ./outputs/template.tf
        to: ./outputs/${{ parameters.name }}_instance.tf
      - from: ./outputs/catalog-info.yaml
        to: ./outputs/${{ parameters.name }}-component-info.yaml

  - id: github_pr
    name: Create GitHub PR
    action: publish:github:pull-request
    input:
      repoUrl: <repository-url>
      branchName: '${{ parameters.environment }}'
      # Ref: https://github.com/backstage/backstage/pull/20733 - but apparently not yet available
      update: true
      title: '🔩 Create ${{ parameters.name }} GCP Service Account'
      description: |
        ## Creating GCP Service Account ${{ parameters.name }}
        
        This is an initial pull request to create an GCP Service Account and was created based on the Backstage template.

        *created by: Backstage 👷‍♂️⚙️👷‍♀️
      sourcePath: ./outputs
      targetPath: 'projects/${{ parameters.environment }}/resources'

  - id: label_pr
    name: Add labels to PR
    action: github:issues:label
    input:
      repoUrl: <repository-url>
      number: '${{ steps.github_pr.output.pullRequestNumber }}'
      labels:
      - terraform
      - created-by-backstage
      - ${{ parameters.environment }}
      - gcp


  output:
    links:
    - title: 'Go to pull request :D'
      url: ${{ steps.github_pr.output.remoteUrl }}
      icon: github

Visually, this concept would manifest as follows:

By completing the form above you’d can generate the following terraform file:

module "${{ values.name }}_service_account" {
  source       = "../../../modules/service_account"
  project      = local.project
  name         = "${{ values.name }}"
  display_name = "${{ values.display_name }}"
  roles        = ${{ values.roles }}
  permissions = ${{ values.permissions }}
  bindings = ${{ values.bindings }}
}

(Optional) If you wish to track the generated resources in Backstage, you can also create the following file alongside:

apiVersion: backstage.io/v1alpha1
kind: Resource
metadata: "${{ values.name }}-service-account"
  name: 
spec:
  type: service-account
  owner: ${{ values.owner }}
  system: terraform-infra

Templates, aside from inputting specific variables, offer a range of actions, including generating code with your variables, submitting a PR to your preferred VCS, or even creating a webhook. This flexibility means you’re not limited to initializing new repositories or projects from the template. Instead, you can open a PR to create and add a file in a certain directory of your existing repository. This capability essentially allows you to template any independent component on top of your deployed infrastructure.

Conclusion

The example with Backstage Templates may seem somewhat specific, but I believe it holds immense potential to enhance the productivity of developer teams. We’re only scratching the surface of its various applications, yet it has already proven to save significant time and has become an indispensable tool for large-scale software companies striving to maximize productivity.

To stay current with the latest cloud technologies, make sure to subscribe to my weekly newsletter, Cloud Chirp. 🚀