How to Check and Restore AES-NI for VPS

Wetin AES-NI for VPS really dey give you

AES-NI for VPS na group of six x86 instructions wey dey perform one round of AES (advanced encryption standard) for hardware. If your provider CPU model hide dem, the silicon underneath still get dem, but OpenSSL no fit see dem and e go fall back to software implementation wey dey cost roughly ten times more cycles per byte. You fit check the feature with one command, measure the difference with two, and many times turn the fast path back on with one environment variable.

The instructions na AESENC, AESENCLAST, AESDEC, AESDECLAST, AESIMC and AESKEYGENASSIST. Intel ship dem for 2010 and AMD follow, so any server CPU wey you likely rent get the silicon. One companion instruction, PCLMULQDQ, dey do carry-less multiplication, and na wetin GCM (Galois/counter mode) need to build its authentication tag. AES-GCM dey fast only when both dey available, because the cipher and the tag na separate work.

Four places for VPS wey you go see this for your monitoring:

• TLS (transport layer security) termination. Web server wey dey serve AES-128-GCM or AES-256-GCM dey spend most of its bulk-crypto time inside AES.
• Encrypted volumes. LUKS (Linux unified key setup) and dm-crypt dey run aes-xts for every read and every write, inside the kernel, for the CPU.
• AES-based VPN traffic. OpenVPN with AES-256-GCM and IPsec with AES-GCM both dey depend on am.
• Encrypted backups. Anything wey dey encrypt stream with AES before e comot from the server dey pay the same cost.

One common workload no dey affected at all. WireGuard dey use ChaCha20-Poly1305 for its data and e no dey touch AES, so self-hosted WireGuard VPN go run at the same speed for host wey mask the flag. This difference na practical reason to weigh WireGuard against OpenVPN before you choose tunnel for cheap VPS.

How to check whether your VPS get AES-NI

Kernel dey copy the CPUID feature bits go inside /proc/cpuinfo, so one grep fit answer the question.

grep -m1 -o '\baes\b' /proc/cpuinfo
lscpu | grep -i -o '\baes\b'

Any command wey print aes mean say CPU dey advertise AES-NI to this guest. If e print nothing, e no dey. lscpu dey read the same flags, so both commands always agree. Use the one wey dey installed.

Now check the CPU wey host claim say you dey run on.

grep -m1 'model name' /proc/cpuinfo

A real model string like Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz or AMD EPYC 7443P 24-Core Processor mean say host pass the physical CPU model through give you. QEMU Virtual CPU version 2.5+ or Common KVM processor mean say another thing dey happen, and na that case wey worth understand.

Wetin make flag miss when silicon get am

CPUID na instruction wey program dey use ask CPU wetin e support. Inside virtual machine, CPUID always dey trap go hypervisor, so na hypervisor dey decide wetin guest go see. Most panels dey show this decision as guest CPU model. qemu64 and kvm64 na generic baseline models, and none of dem get AES-NI or SSSE3 for feature set, so guest no dey see aes flag even when physical host na current EPYC. VPS na guest for another person hardware, so every feature wey e report na decision wey happen one level up. If this layering still new to you, start with wetin VPS be.

Hosts dey choose generic model intentionally, because live migration between machines wey get different processors only go work if guest never hear about feature wey destination no get. Na you dey pay the cost. Your kernel and your own copy of OpenSSL both dey read that masked CPUID once when dem start, then both dey choose slow code path for the whole life of the process.

The fix for source na host-side setting: -cpu host for QEMU terms, a named model wey include AES-NI, or explicit +aes wey dem add to the model. You no fit set any of dem from inside guest. Open support ticket, or choose plan wey hypervisor dey pass CPU model through, na the durable answer.

Measure the gap with openssl speed

No just believe published benchmark. Run the cipher wey your own server actually dey negotiate.

openssl version
openssl speed -evp aes-128-gcm

The result row get label AES-128-GCM and e give throughput for six block sizes, with unit of 1000 bytes per second. Read the 8192-byte column for bulk transfer, because per-call overhead dey dominate the 16-byte column and e no tell you anything about file download.

Now run the same command with AES-NI and PCLMULQDQ switched off for software:

OPENSSL_ia32cap="~0x200000200000000" openssl speed -evp aes-128-gcm

That value come from OpenSSL own capability vector documentation. When ~ dey for beginning, e mean “clear these bits”. Bit 57 na AES-NI and bit 33 na PCLMULQDQ, so 0x200000200000000 name exactly those two and nothing else. If the second number far below the first one, your box get working AES-NI and you don finish. If both numbers match, OpenSSL already dey use software path, because the flag no dey there to clear.

[
  {
    "label": "AES-NI and PCLMULQDQ",
    "mb_per_sec": "4,850",
    "cycles_per_byte": 0.7
  },
  {
    "label": "Software fallback",
    "mb_per_sec": 310,
    "cycles_per_byte": 11.0
  }
]

Those figures na representative published results for modern x86 core near 3.4 GHz, no be measurement from any particular host. Read dem as pattern. Hardware path dey run around 0.7 cycles per byte, while software fallback dey run around 11.0, which come to about 4,850 MB/s against 310 MB/s for one core. The two commands above na the only numbers wey describe your server. Use the same discipline for the rest of the machine, so pair this with repeatable way to benchmark VPS before you draw conclusion about a plan.

Force the bits make dem turn back on with OPENSSL_ia32cap

Na dis part wey dey surprise people. AES-NI instructions no need privilege, and hypervisor no dey trap dem. Na only CPUID dey trapped. So host fit tell your guest say AES-NI no dey, while AESENC still dey execute natively at full speed. Software skip the fast path because e ask CPUID and get wrong answer. The instruction itself never stop to work.

OpenSSL let you answer on behalf of the CPU. Plain hexadecimal value for OPENSSL_ia32cap dey overwrite the capability vector instead of masking am.

OPENSSL_ia32cap="0x0200020207000000" openssl speed -evp aes-128-gcm

If that run fast pass the plain run by several times, the silicon get AES-NI and your host dey hide am. First, na diagnosis. For OpenSSL, e also happen to be the fix.

Two warnings dey apply, and you fit trigger both intentionally.

If you set only the first vector, later vectors remain zero. This one turn off AVX2 and AVX-512 code paths. Na deliberate action for here. No try force AVX bits on masked guest, because AVX registers need operating system to enable extended state for XCR0, and your kernel refuse to do am based on the same masked CPUID. VEX-encoded instruction go then raise undefined-opcode fault and the process go die.

If you force AES-NI on core wey truly no get am, the process go die immediately:

Illegal instruction (core dumped)

Na AESENC dey raise undefined-opcode fault, because that core no get that instruction to execute. Some server firmware fit also disable AES-NI for hardware until the next reset, and the symptom go be the same. Either way, the answer na different host, no be different environment variable.

To keep the override for service wey go run for long time, use systemd drop-in.

sudo systemctl edit nginx

[Service]
Environment="OPENSSL_ia32cap=0x0200020207000000"

sudo systemctl daemon-reload
sudo systemctl restart nginx
sudo systemctl show nginx -p Environment

The last command suppose print the variable back to you. Understand wetin you dey agree to: if dem ever migrate that server to host wey CPU truly no get AES-NI, nginx go die with illegal instruction on the first TLS connection. Put am for your runbook, or keep the override comot from production entirely and use am only to prove the point when you open ticket.

Wetin the override no fit fix

OPENSSL_ia32cap only reaches OpenSSL and nothing else. Every other software dey do its own feature detection and no dey read that variable.

The kernel na the important case. dm-crypt and LUKS dey use the kernel crypto API, and the aesni_intel module no go load when the CPU feature bit dey missing:

modprobe: ERROR: could not insert 'aesni_intel': No such device

No user-space variable dey for this one. The kernel dey read CPUID once during boot, and that decision go remain until you reboot for another host. So your encrypted volume go remain on the software cipher, no matter wetin OpenSSL dey do. Measure wetin you dey actually get:

sudo cryptsetup benchmark -c aes-xts -s 256

The aes-xts 256b row dey reach thousands of MiB/s with hardware AES, but e dey for low hundreds without am. Language runtimes wey get their own detection, including Go and Java, dey equally outside the reach of this override. Go's crypto/aes dey check CPUID directly and quietly use its constant-time software implementation when the bit no dey set. If na Go binary dey terminate your TLS, the OpenSSL variable no go change anything for am.

AES-NI no dey available, prefer ChaCha20

Dem design ChaCha20-Poly1305 make e fast for plain software. For core wey no get usable AES-NI, e usually faster pass AES-GCM by wide margin. So the sensible thing for that kind host na to stop preferring AES.

For nginx 1.19.4 and newer, wey build against OpenSSL 1.1.1 or newer:

ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
ssl_conf_command Ciphersuites TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384;

ssl_ciphers cover TLS 1.2. ssl_conf_command Ciphersuites cover TLS 1.3. For TLS 1.3, nginx no get dedicated directive, and e pass the string straight go OpenSSL without checking am. Because of this, typo there fit pass silently. Reload and confirm wetin client dey offer:

sudo nginx -t && sudo systemctl reload nginx
openssl s_client -connect example.com:443 -tls1_3 </dev/null 2>/dev/null | grep -i cipher

Healthy result go name TLS_CHACHA20_POLY1305_SHA256. Before you commit the change, run openssl speed -evp chacha20-poly1305 beside the AES run for the same box, then make the two numbers decide.

ARM hosts dey use different extensions

AES-NI na for x86 only. ARM VPS dey use ARMv8 cryptographic extensions, wey be separate instruction set wey dey do the same work. For aarch64, the flags dey under Features instead of flags:

grep -m1 Features /proc/cpuinfo

Look for aes and pmull. pmull na the ARM counterpart of PCLMULQDQ, and GCM need am for the same reason. OpenSSL override variable for ARM na OPENSSL_armcap, with im own bit layout defined for crypto/arm_arch.h inside OpenSSL source. So the x86 hex value for this guide no mean anything there. For real-world use, ARM server cores wey dem dey sell as VPS hosts dey expose these extensions, so masked-feature problem mostly concern x86.

Failure modes, with the strings you go see

No aes for /proc/cpuinfo, and the forced run dey much faster. Host dey mask CPUID. Confirm say model name generic, then ask your provider which guest CPU model their hypervisor dey present.

No aes for /proc/cpuinfo, and the forced run dey print Illegal instruction. The instructions genuinely no dey, or firmware don disable dem. Move the workload go another host.

aes dey present but throughput still low. Check say you dey read the 8192-byte column, and say nothing else dey use the core. For shared plan, noisy neighbour fit look exactly like missing CPU feature until you run the test twice for different times of day.

The masked run and the normal run dey give the same number. OpenSSL don already dey for the software path. That result na the finding, no be mistake for the test.

Nested virtualisation dey change the answer one level down. Guest wey dey inside another guest go get whichever CPUID the middle layer choose to pass on, and AES-NI easy to lose there without noticing. If you run nested virtual machines for VPS, check the flag inside the inner guest and for the machine wey you rent.

FAQ

Why my VPS no get aes flag for /proc/cpuinfo?

Na because hypervisor dey present generic guest CPU model. qemu64 and kvm64 no include AES-NI for their feature sets, so CPUID report say e no dey, no matter the physical processor wey dey underneath. Hosts dey do this so dem fit migrate guest wey dey run between machines wey get different CPUs. Run grep -m1 'model name' /proc/cpuinfo: string like QEMU Virtual CPU version 2.5+ or Common KVM processor na the sign, while real Xeon or EPYC model string mean say CPU model pass through, and the flag truly no dey for the silicon.

OPENSSL_ia32cap really dey enable AES-NI, or e just dey pretend?

E dey enable the real instructions. AES-NI instructions no need privileged access, and hypervisor no dey trap dem, so AESENC dey execute natively no matter wetin CPUID report. Na only CPUID instruction hypervisor intercept. If you set OPENSSL_ia32cap to plain hexadecimal value, e go replace the answer wey OpenSSL get from CPUID. OpenSSL go then select its hardware code path, and the hardware go run am at full speed. If the silicon truly no get the instructions, process go die with Illegal instruction (core dumped) for the first AES operation.

Override go make my LUKS encrypted volume faster?

No. OPENSSL_ia32cap na OpenSSL dey read am, and nothing else. LUKS and dm-crypt dey use kernel crypto API. For there, aesni_intel module fail to load with modprobe: ERROR: could not insert 'aesni_intel': No such device when feature bit clear. Kernel dey read CPUID during boot, and no user-space variable fit change that. Measure the real figure with sudo cryptsetup benchmark -c aes-xts -s 256, then compare the aes-xts 256b row with host wey dey report the flag.

Missing AES-NI flag dey slow down WireGuard?

No. WireGuard dey use ChaCha20-Poly1305 for all data, and e never use AES instructions. So throughput go remain the same for masked host and unmasked host. OpenVPN and IPsec wey you configure with AES-GCM go lose throughput for host wey no get AES-NI. Because of this, two tunnels for the same VPS fit behave very differently. E good make you know this before you blame the network.

How I fit check AES-NI for ARM VPS?

ARM cores no get AES-NI. Dem get ARMv8 cryptographic extensions, wey dey do the same work with different instructions. Run grep -m1 Features /proc/cpuinfo and look for aes and pmull. aarch64 dey list dem under Features instead of flags. The x86 OPENSSL_ia32cap value no get meaning for ARM. OpenSSL equivalent variable for there na OPENSSL_armcap, and crypto/arm_arch.h for OpenSSL source define how its bits dey arranged.