Are avx and avx2 the **only** CPU flags needed to run MongoDB 5 and higher?

[loganmarchione]

I'll start by saying I'm sorry this is long and I don't know if this is an issue with my hypervisor, or Mongo in general, or the Mongo Docker image...

I run a hypervisor called Proxmox. On the Proxmox box, I have a virtual machine running Debian 11. On that VM, I have Docker installed and I'm trying to run mongo:5.0.13-focal.

|---------------------------------|
|           Proxmox               |
|                                 |
|  |---------------------------|  |
|  |         Debian (VM)       |  |
|  |                           |  |
|  |  |---------------------|  |  |
|  |  |    Mongo (Docker)   |  |  |
|  |  |                     |  |  |
|  |  |                     |  |  |
|  |  |                     |  |  |
|  |  |                     |  |  |
|  |  |                     |  |  |
|  |  |                     |  |  |
|  |  |                     |  |  |
|  |  |---------------------|  |  |
|  |                           |  |
|  |---------------------------|  |
|                                 |
|---------------------------------|

By default, Proxmox doesn't expose the avx and avx2 instructions to VMs (the default CPU type is called kvm64 and it only has a limited set of flags). Below is what those flags look like inside my Debian 11 VM.

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti

When I run this command on my Debian 11 VM, running the default kvm64 CPU:

docker run -it --rm mongo:5.0.13-focal grep avx /proc/cpuinfo

I receive this error (as intended because of this PR in the Docker image):

WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that!
  see https://jira.mongodb.org/browse/SERVER-54407
  see also https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610/2
  see also https://github.com/docker-library/mongo/issues/485#issuecomment-891991814

The Proxmox team has a way to create custom CPUs with specific flags. See here for documentation. Basically, I added the correct file, shutdown my Debian 11 VM, changed the CPU type, started the VM, and can now see the AVX flags from Debian 11.

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic avx hypervisor lahf_lm cpuid_fault pti avx2

When running that same docker run... command as before, it doesn't crash and I now see the CPU flags.

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic avx hypervisor lahf_lm cpuid_fault pti avx2

However, when I try to actually run the container (e.g., docker run -it -d mongo:5.0.13-focal), I get this error in dmesg.

[  455.345445] traps: mongod[4514] trap invalid opcode ip:5589c390e15a sp:7fff6288d1f0 error:0 in mongod[5589bf891000+51e1000]

Question one: Why is this happening? I have AVX and AVX2 support, so the container should run, right?

To make matters more interesting, the Proxmox support team suggests (one, two, three) to just pass the host CPU directly to the VMs (instead of creating a custom CPU type).

When I do that, and run docker run -it --rm mongo:5.0.13-focal grep avx /proc/cpuinfo, I get these flags (this is from an Intel Core i5-10400).

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip pku ospke md_clear arch_capabilities

However, this fixes me running docker run -it -d mongo:5.0.13-focal. No errors in dmesg and the container stays up.

CONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS          PORTS       NAMES
9224eee47e6b   mongo:5.0.13-focal   "docker-entrypoint.s…"   25 seconds ago   Up 24 seconds   27017/tcp   upbeat_bhaskara

Question two: Are avx and avx2 the only CPU flags needed to run MongoDB 5 and higher? Should the PR mentioned earlier be checking for other CPU flags?

[yosifkit]

avx and avx2 are the broad answer to "what was added in sandy bridge" (the target chosen by MongoDB). For physical CPUs, if it has those it should have everything else from that micro-architecture. But it is technically every opcode that gcc could use for that target:

‘sandybridge’
Intel Sandy Bridge CPU with 64-bit extensions, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, CX16, SAHF, FXSR, AVX, XSAVE and PCLMUL instruction set support.

[loganmarchione]

@yosifkit, thanks for that link! I compared my kvm64 CPU and Sandy Bridge and there were only a handful of different instructions. Did some experimenting and found that I needed the xsave flag. Below is my /etc/pve/virtual-guest/cpu-models.conf file from Proxmox.

cpu-model: avx
    flags +avx;+avx2;+xsave
    phys-bits host
    hidden 0
    hv-vendor-id proxmox
    reported-model kvm64

I'm good now, so you can close this out. Not sure if you want to update the logic here to check for xsave, or assume most people will have something more generic like host or Sandy Bridge or something else that already has xsave.

[tianon]

Glad you got it figured out! 👍

The warning we added was intended as a hint for users running into issues with the new builds (not to be exhaustive) -- coming up with an exhaustive list of opcodes/features that the compiled binaries provided by MongoDB use is a somewhat complicated proposition, which is why we chose to just use one or two "sentinel" features from the new chipset they're targeting vs the old for showing this warning.

[adelynx]

@yosifkit, thanks for that link! I compared my kvm64 CPU and Sandy Bridge and there were only a handful of different instructions. Did some experimenting and found that I needed the xsave flag. Below is my /etc/pve/virtual-guest/cpu-models.conf file from Proxmox.

cpu-model: avx
    flags +avx;+avx2;+xsave
    phys-bits host
    hidden 0
    hv-vendor-id proxmox
    reported-model kvm64

I'm good now, so you can close this out. Not sure if you want to update the logic here to check for xsave, or assume most people will have something more generic like host or Sandy Bridge or something else that already has xsave.

Hi @loganmarchione, I did create etc/pve/virtual-guest/cpu-models.conf with given flags but still can't see the avx flag in my VM. Could you please share exact steps you did to solve this issue? Thanks

[loganmarchione]

@adelynx Did you shutdown the VM, change the CPU to the new custom CPU type, then start the VM? That's all I had to do.

[adelynx]

@loganmarchione I did shutdown the VM. I tried all what you mentioned and too many other solutions none of them worked so I gave up installing MongoDB in Proxmox tbh.

[lucidnx]

@loganmarchione I did shutdown the VM. I tried all what you mentioned and too many other solutions none of them worked so I gave up installing MongoDB in Proxmox tbh.

as of now, you can just use x86-64-v3, which include all these flags