Proxmox cluster with dishomogeneous cpu types

TL;DR: #

NO:
for live migration within a proxmox cluster build it with same cpus

Preface: #

I have a proxmox cluster built with what I find lying around or is cheap enough to be justifiable,
so I have different cpu types in the cluster nodes: Amd Ryzen (node1) and Intel i5 (node2).

Using the default cpu type, that has a limited set of instructions, everything work as intended, particularly live migration between the hosts.

Tests: #

I've then tried to edit the custom cpu types (/etc/pve/virtual-guest/cpu-models.conf) adding a custom one that has more cpu instructions thatn the base kvm64 cpu.

But I did not find a reliable way to tell which flags were compatible with the underlying host.

I know that the best scenario is to run a cluster with homogeneous hardware, but I also know that this is what I have available :D

So, my aim is to create a custom cpu profile that contains the flags that are common to both the nodes cpus, so I can have better performance thank the standard kvm64.

So I wrote myself a script to try to match the flags that are common to both the nodes:

#clean the working file:
echo "" > result.txt

cat host1.txt | tr " " "\n" > host1_list.txt
cat host2.txt | tr " " "\n" > host2_list.txt

#loop the list of flags from node 2 against list of flags of node1 and display only the commone ones:
while read p; do
  grep -E "(^| )$p( |$)" host1_list.txt >> result.txt
done < host2_list.txt

#format the flags to be used in a proxmox custom cpus file, see: https://pve.proxmox.com/pve-docs/cpu-models.conf.5.html
# sed ':a;N;$!ba;s/\n/;+/g' : substitutes the carriage returns with ;+
# sed 's/;//' : removes the first occurrence of ; from the line
flags=$(cat result.txt | sed ':a;N;$!ba;s/\n/;+/g' | sed 's/;//')

#echo the configurations to write to: /etc/pve/virtual-guest/cpu-models.conf
echo "##############################################"
echo "cpu-model: test"
echo "    flags $flags"
echo "    phys-bits host"
echo "    hidden 0"
echo "    hv-vendor-id proxmox"
echo "    reported-model kvm64"
echo "##############################################"

#remove workfiles not needed anymore:
rm host1_list.txt host2_list.txt result.txt

Usage: #

# Proxmox_Cpu_Flags
trying to find minimum cpu flags common to 2 nodes in a cluster

## Usage:
execute lscpu on your proxmox nodes, and take the output of the line "Flags:", for example:

```
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
```
and save the content without the "Flags:" part in the files:
* host1.txt
* host2.txt

I've used host2.txt with the older cpu but it should be the same

Then execute compare.sh, you should have an output similar to:
```
##############################################
cpu-model: test
    flags +fpu;+vme;+de;+pse;+tsc;+msr;+pae;+mce;+cx8;+apic;+sep;+mtrr;+pge;+mca;+cmov;+pat;+pse36;+clflush;+mmx;+fxsr;+sse;+sse2;+ht;+syscall;+nx;+pdpe1gb;+rdtscp;+lm;+constant_tsc;+rep_good;+nopl;+nonstop_tsc;+cpuid;+aperfmperf;+pni;+pclmulqdq;+monitor;+ssse3;+fma;+cx16;+sse4_1;+sse4_2;+movbe;+popcnt;+aes;+xsave;+avx;+f16c;+rdrand;+lahf_lm;+abm;+3dnowprefetch;+ssbd;+ibpb;+fsgsbase;+bmi1;+avx2;+smep;+bmi2;+rdseed;+adx;+smap;+clflushopt;+xsaveopt;+xsavec;+xgetbv1;+xsaves;+arat
    phys-bits host
    hidden 0
    hv-vendor-id proxmox
    reported-model kvm64
##############################################
```

this should be inserted on your proxmox node (one is enough) in the file: /etc/pve/virtual-guest/cpu-models.conf

## Nota bene:
right now it's not working as intended (I'm posting on r/proxmox to ask for help)

But it's not working as hoped:

• I've tried to boot the VM on Node1 using the new cpu profile gives me many errors on cpu flags incompatible:
• I had to remove:aperfmperfconstant_tsccpuidnonstop_tscnoplrep_goodhtmonitor
• once I removed the incompatible cpu flags on AMD it works ok
• I've tried to migrate to Node2 the vm and it crashed due to non compatible cpu flag:
• I had to remove:ssbd

so I tried again to start the vm on Node1 and migrate to Node2, this time I got a nice kernel panic, which is the same output I get when I just start the vm on Node2 without the migration.

image of kernel panic:

So there's obviously something wrong in my process, but I am not understanding what, maybe the compatibility is not supported among different cpu types or I am starting from some very wrong assumptions.

I've found another thread where OP selected Ivy-Bridge or Sandy-Bridge to enhance compatibility for migration between the hosts he/she had (Amd and Intel),my question is:is there a reliable way to determine the highest cpu flags/cpu profile supported by both machines?(other than trial and error selecting the various cpu types and trying a live migration)

Reddit post and suggestions: #

I've posted a questione on r/proxmox: post
and u/elettronik gave me some insights I flew over.

Live migration for me is not essential, since most of the projects I host are not that sensitive to a soft and controlled shutdown and re-power on, so it definitely could be a way.

After another bit of fiddling and trying I've managed to fix what I think are the address size in the config file, setting it to the lowest value:

cpu-model: test3
flags +fpu;+de;+pse;+tsc;+msr;+pae;+mce;+cx8;+apic;+sep;+mtrr;+pge;+mca;+cmov;+pat;+pse36;+clflush;+mmx;+fxsr;+sse;+sse2;+syscall;+nx;+pdpe1gb;+rdtscp;+lm;+pni;+pclmulqdq;+ssse3;+cx16;+sse4_1;+sse4_2;+movbe;+popcnt;+aes;+xsave;+rdrand;+lahf_lm;+abm;+fsgsbase;+bmi1;+smep;+bmi2;+adx;+smap;+clflushopt;+xsaveopt;+xgetbv1;+arat;+hypervisor
phys-bits 39
hidden 0
hv-vendor-id proxmox
reported-model kvm64

The flags are tested with:

qemu-system-x86_64 -cpu qemu64,check,+fpu,+de,+pse,+tsc,+msr,+pae,+mce,+cx8,+apic,+sep,+mtrr,+pge,+mca,+cmov,+pat,+pse36,+clflush,+mmx,+fxsr,+sse,+sse2,+syscall,+nx,+pdpe1gb,+rdtscp,+lm,+pni,+pclmulqdq,+ssse3,+cx16,+sse4_1,+sse4_2,+movbe,+popcnt,+aes,+xsave,+rdrand,+lahf_lm,+abm,+fsgsbase,+bmi1,+smep,+bmi2,+adx,+smap,+clflushopt,+xsaveopt,+xgetbv1,+arat,+hypervisor

which with the flag "check" outputs errors in case the flags are not supported on the host, so this seems a reliable way to determine which flags are supported and which not.

So:
dual cpu works ok also on Ryzen, live migration is ok to Intel but the other way around the kernel panics.

Conclusions: #

Right now I've decided to stick with the default kvm64 for all the vms until I'll need some more performance or on the specific vms that will benefit from a broader set of cpu instructions, in the near future I will evaluate the possibility to apply a broader set of instructions but losing live migrations.

So, for reliable production hardware stick with Proxmox's manual and create cluster with same cpu types on the hosts,
otherwise you know what your problems might be.

Bonus: #

benchmark of different cpu flags:

128/256 F = with AES flag
128/256 N = without AWS flag

test run as

openssl speed -evp aes-128-cbc
openssl speed -evp aes-256-cbc

obviously it's a single test but one can see the improvement.

• Next: Kindle 4 NT Jailbreaking and converting to a dashboard
• Previous: Surface Go Dual Boot Secureboot