..

creating proxmox resources with opentofu

Proxmox web console is awesome. It’s highly configurable, fast and very easy to use. After a year of creating resources by clicking around with my tiny mouse, I just had to find a way to automate VM/CT creation.

OpenTofu (a well known terraform fork) is a perfect way to do this. One of the main reason is to use for_each feature which is pay walled in terraform. I am just shocked how such a crucial and important feature is not available to everyone.

Essentially, it helps you reduce the config length by not specifying the same provider instance over and over again. For example, let’s say we’ve got 5 proxmox nodes. There’s no way you can write a for loop to create few VMs in every node. You have to repeat the same configuration code again and again for each new node.

provider selection

The most famous proxmox terraform provider is undoubtedly telmate. Unfortunately, it was quite difficult for me to properly use it.

Just when I was about to give up, I found out about bgp/proxmox provider. It’s extremely easy to use and the developer is very helpful and actively maintaining it!

I am not gonna go over the basics of terraform/opentofu as I feel like there are already great articles on it.

My main goal is to show how to properly use for_each loop for reducing code length and deploy lots of VMs across different proxmox nodes without any repetitive configuration.

So, I have a k3s setup so what we will do is create 2 worker VMs and 1 master VM on each node.

Let’s first create vars.tf:

variable "proxmox_hosts" {
  type = list(object({
    name     = string
    endpoint = string
    username = string
    password = string
  }))
}

variable "vm_count_worker" {
  type    = number
  default = 2
}

variable "vm_count_master" {
  type    = number
  default = 1
}

After that, create terraform.tfvars where we will put our credentials:

proxmox_hosts = [
  {
    name     = "host1"
    endpoint = "https://host1.example.xyz/api2/json"
    username = "root@pam"
    password = "password"
  },
  {
    name     = "host2"
    endpoint = "https://host2.example.xyz/api2/json"
    username = "root@pam"
    password = "password"
  },
  {
    name     = "host3"
    endpoint = "https://host3.example.xyz/api2/json"
    username = "root@pam"
    password = "password"
  }
]

Sweet. Now, let’s create main.tf. First we specify the provider:

terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "0.70.0"
    }
  }
}

And then we loop over the proxmox nodes using for_each:

provider "proxmox" {
  alias     = "by_name"
  for_each = {
    for h in var.proxmox_hosts : h.name => h
  }

  endpoint = each.value.endpoint
  username = each.value.username
  password = each.value.password
  insecure = true
}

Most of it is quite self-explanatory. Now, let’s add the VM resource for k3s workers. One thing to note is that I am cloning an existing ubuntu 22.04 template.

If you don’t have one, you can first download the minimal cloud image version from here with .img extension and then run the following script for creating a vm, importing and resizing the disk and configuring boot settings:

qm create 5000 --memory 2048 --core 2 --name ubuntu-22 --net0 virtio,bridge=vmbr0
qm disk import 5000 <os-image> local-lvm
qm set 5000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-5000-disk-0
qm set 5000 --ide2 local-lvm:cloudinit
qm set 5000 --boot c --bootdisk scsi0
qm set 5000 --serial0 socket --vga serial0
qm disk resize 1000 local-lvm +20G

Go to cloud-init section of the VM in the console, set your username and password along with an ssh key. After that, click regenerate image and then right click on the VM and click on convert to template.

Setting minimal resources for a template is advised because you can easily change the resources of a cloned VM afterwards.

Alright, now we create the VM resource for k3s_worker:

resource "proxmox_virtual_environment_vm" "k3s_worker" {
  count     = length(var.proxmox_hosts) * var.vm_count_worker  
  provider  = proxmox.by_name[var.proxmox_hosts[floor(count.index / var.vm_count_worker)].name]

  name      = format("worker-%02d", count.index + 1)
  node_name = var.proxmox_hosts[floor(count.index / var.vm_count_worker)].name

  vm_id     = 120 + count.index

  description = format("k3s worker %02d", count.index + 1)
  tags = ["k3s"]

  started = false

  cpu {
    cores = 2
    type = "x86-64-v2-AES"
    architecture = "x86_64"
  }

  memory {
    dedicated = 4096
  }

  clone {
    full = true
    vm_id = 1000
    node_name = var.proxmox_hosts[floor(count.index / var.vm_count_worker)].name
  }

  disk {
     datastore_id = "local-lvm"
     size         = "40"
     interface    = "scsi0"
  }

  initialization {
    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }

  stop_on_destroy = true
}

provider specifies which proxmox provider configuration to use for creating the VM.

proxmox.by_name is an alias for the proxmox provider, defined earlier in the configuration.

var.proxmox_hosts[floor(count.index / var.vm_count_worker)].name calculates the index of the proxmox host to use for the current VM. The floor function ensures that the index is an integer.

Now, we write the same config for k3s master:

resource "proxmox_virtual_environment_vm" "k3s_master" {
  count     = length(var.proxmox_hosts) * var.vm_count_master  
  provider  = proxmox.by_name[var.proxmox_hosts[floor(count.index / var.vm_count_master)].name]

  name      = format("master-%02d", count.index + 1)
  node_name = var.proxmox_hosts[floor(count.index / var.vm_count_master)].name

  vm_id     = 130 + count.index

  description = format("k3s master %02d", count.index + 1)
  tags = ["k3s"]

  started = false

  cpu {
    cores = 4
    type = "x86-64-v2-AES"
    architecture = "x86_64"
  }

  memory {
    dedicated = 4096
  }

  clone {
    full = true
    vm_id = 1000
    node_name = var.proxmox_hosts[floor(count.index / var.vm_count_master)].name
  }

  disk {
     datastore_id = "local-lvm"
     size         = "40"
     interface    = "scsi0"
  }

  initialization {
    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }

  stop_on_destroy = true
}

Let’s run it:

tofu apply

You should see a total of 9 VM to be created. That’s it, you can experiment around with creating CTs as well which is quite easy.