You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
lramm cad3cd738c added helm chart longhorn 1 year ago
.github/workflows Fix CI with new ansible 2.10 package 2 years ago
inventory/sample added port to registry mirror 1 year ago
pics update pics 2 years ago
roles added helm chart longhorn 1 year ago
terraform-hetzner added hetzner terraform equal to proxmox 1 year ago
terraform-proxmox fixed path for proxmox hosts.ini 1 year ago
.gitignore updated gitignore because of terraform extensions 1 year ago
.yamllint Fixes #3: Fix linting issues, add ansible-lint and yamllint configuration. 3 years ago
LICENSE Update LICENSE 1 year ago added helm chart longhorn 1 year ago
ansible.cfg added static ip address and copy the kubeconfig file 2 years ago
cloudinit.yml fixed broken shell commands 1 year ago
reset.yml reorganize roles 1 year ago
site.yml added helm chart longhorn 1 year ago

Original work was extended to

  • Hardening for OS (and K3s Cluster)
  • Storage Nodes
  • Caching Server for Container Images and Package Caching
  • Bandwidth reduction due to proxy usage
  • Added Playbook for VM-Template creation
  • Terraform can now create an equal cluster on Proxmox and in Hetzner Cloud
  • Added Helm deploy for Longhorn

Currently only on Debian 11 tested


cp terraform-proxmox/ terraform-proxmox/
cp -R inventory/sample inventory/my-proxmox-cluster


cp terraform-hetzner/ terraform-hetzner/
cp -R inventory/sample inventory/my-hetzner-cluster

Proxmox setup

The "cloudinit.yml" contains a simple Ansible Workflow to create a template VM for the Proxmox k3s.

ansible-playbook -i proxmoxServerIp, cloudinit.yml

Terraform setup

To get started with the Terraform part you need to provide the Credentials for your Proxmox Server/ Cluster. If you did not know how, take a look here.

Setup your k3s Cluster Config in the, after you apply, the ansible inventory file will be created. The Terraform Config uses the template vm from the cloudinit.yml, which has to be present on each node you want to interact with. Terraform is going to set the sshkey, -user, -password to the VMs it is going to create.

cd terraform*

terraform init
terraform plan
terraform apply

Ansible setup

One finished you can run the following code, but be sure to update the yml in the group_vars folder.

ansible-playbook site.yml -i inventory/my-proxmox-cluster/hosts.ini


ansible-playbook site.yml -i inventory/my-hetzner-cluster/hosts.ini


At the end, the kube.conf will be copied to your local machine to ~/.kube/{{ cluster_name }}-config

Original README below

Build a Kubernetes cluster using k3s on Proxmox via Ansible and Terraform

This is based on the great work that done with Ansible, all I left to do is to put it all together with terraform and Proxmox!

System requirements

  • The deployment environment must have Ansible 2.4.0+
  • Terraform installed
  • Proxmox server

How to

for updated documentation check out my medium.

Proxmox setup

This setup is relaying on cloud-init images.

Using cloud-init image save us a lot of time and it's work great! I use ubuntu focal image, you can use whatever distro you like.

to configure the cloud-init image you will need to connect to a Linux server and run the following:

install image tools on the server (you will need another server, these tools cannot be installed on Proxmox)

apt-get install libguestfs-tools

Get the image that you would like to work with. you can browse to and select any other version that you would like to work with. for Debian, got to it can also work for centos (R.I.P)


update the image and install Proxmox agent - this is a must if we want terraform to work properly. it can take a minute to add the package to the image.

virt-customize focal-server-cloudimg-amd64.img --install qemu-guest-agent

now that we have the image, we need to move it to the Proxmox server. we can do that by using scp

scp focal-server-cloudimg-amd64.img Proxmox_username@Proxmox_host:/path_on_Proxmox/focal-server-cloudimg-amd64.img

so now we should have the image configured and on our Proxmox server. let's start creating the VM

qm create 9000 --name "ubuntu-focal-cloudinit-template" --memory 2048 --net0 virtio,bridge=vmbr0

for ubuntu images, rename the image suffix

mv focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.qcow2

import the disk to the VM

qm importdisk 9000 focal-server-cloudimg-amd64.qcow2 local-lvm

configure the VM to use the new image

qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0

add cloud-init image to the VM

qm set 9000 --ide2 local-lvm:cloudinit

set the VM to boot from the cloud-init disk:

qm set 9000 --boot c --bootdisk scsi0

update the serial on the VM

qm set 9000 --serial0 socket --vga serial0

Good! so we are almost done with the image. now we can configure our base configuration for the image. you can connect to the Proxmox server and go to your VM and look on the cloud-init tab, here you will find some more parameters that we will need to change.

alt text

you will need to change the user name, password, and add the ssh public key so we can connect to the VM later using Ansible and terraform. update the variables and click on Regenerate Image

Great! so now we can convert the VM to a template and start working with terraform.

qm template 9000

terraform setup

our terraform file also creates a dynamic host file for Ansible, so we need to create the files first

cp -R inventory/sample inventory/my-cluster

Rename the file terraform/vars.sample to terraform/ and update all the vars. there you can select how many nodes would you like to have on your cluster and configure the name of the base image. to run the Terrafom, you will need to cd into terraform and run:

terraform init
terraform plan
terraform apply

it can take some time to create the servers on Proxmox but you can monitor them over Proxmox. it shoul look like this now:

alt text

Ansible setup

First, update the var file in inventory/my-cluster/group_vars/all.yml and update the user name that you're selected in the cloud-init setup.

after you run the Terrafom file, your file should look like this:

[master] Ansible_ssh_private_key_file=~/.ssh/proxk3s

[node] Ansible_ssh_private_key_file=~/.ssh/proxk3s Ansible_ssh_private_key_file=~/.ssh/proxk3s Ansible_ssh_private_key_file=~/.ssh/proxk3s Ansible_ssh_private_key_file=~/.ssh/proxk3s


Start provisioning of the cluster using the following command:

Ansible-playbook site.yml -i inventory/my-cluster/hosts.ini


To get access to your Kubernetes cluster just

scp debian@master_ip:~/.kube/config ~/.kube/config