VMFailInvalid returned by VMXON

[strictlymike]

Hello,

I am trying to use VMX ona 2xCPU Intel Xeon 5130 @ 2.00GHz machine. After I issue the VMXON instruction, I find that CF is set, which I understand is the indication of a VMfailInvalid result [1]. Here are the setup steps I have taken:

I have verified that the CPU supports VMX by checking ECX.VMX after issuing CPUID.1;
I have checked that CR4.VMXE is cleared (i.e., nothing else has entered VMX operation);
I have set CR4.VMXE;
I have checked that CR0 and CR4 comply with the IA32_VMX_CR*_FIXED* MSRs
I have read size from IA32_VMX_BASIC[bits 44:32] (On my system, 2048 bytes)
I have secured the required 2048 bytes of page-aligned memory (e.g., starting at 0xd0519000)
I have written revid from IA32_VMX_BASIC[bits 31:0] to my VMXON region
I have passed the physical address of my VMXON region to the VMXON instruction

I don't know how to check, but based on the what I have read about A20M#, I highly doubt it is a factor--is that a good assumption? I wonder if I am correctly passing in the m64 operand of the VMXON instruction using gcc inline assembly. Can someone provide an example of the correct inline assembly for this? Or, better yet, tell me something I am doing wrong? My output is duplicated below for ease of diagnosis.

Thanks..
-M.

Footnotes:
1. Developer's Manual vol. 2B, sec. 5.2 - Conventions

Output:
VMX capable
Feature control checks
IA32_FEATURE_CONTROL.lock[bit 0] = 1: yes
IA32_FEATURE_CONTROL.VMXINSMX[bit 1] = 0
IA32_FEATURE_CONTROL.VMXOUTSIDESMX[bit 2] = 1: yes
Linux KVM check: IA32_FEATURE_CONTROL & 5 == 5
CR4.VMXE set successfully
CR0 checks:
CR0.NE = 1: yes
CR0.PE = 1: yes
CR0.PG = 1: yes
CR0 valid for VMXON
CR4 checks:
CR4.SMXE = 0: yes
CR4.VMXE = 1: yes
CR4 valid for VMXON
Other conditions worth noting:
CR4.PAE = 0
IA32_VMX_BASIC.REVID[bits 31:0] = 0xb (11)
IA32_VMX_BASIC.VMXONBYTES[bits 44:32] = 2048 bytes
IA32_VMX_BASIC.WIDTH32[bit 48] = 0
VMXON region address width limited only to the physical address width
Physical address width: 38 bits
Preparing VMXON region
vmxon_region = 0xee96f000
vmxon_pa = 0x00 0xf0 0x96 0x2e 0x00 0x00 0x00 0x00
VMXON returned VMFailInvalid

[hellfire · Accepted solution]

The result of VMXON may be checked incorrectly.

vmxon_pa = __pa(vmxon_region);
...
/* VMXON */
asm volatile (
"clcnt" // Clear CF
"vmxon (%%eax)nt" // VMXON does not affect EAX. EAX == vmxon_pa after its execution. It is not zero.
"jnc endnt"
"movl $1, %%eaxnt" // Err if CF set
// Nothing changed: EAX is not zero at all
"end:nt"
: "=a" (err)
: "a" (&vmxon_pa), "m" (vmxon_pa)
: "memory", "cc"
);
...
// err is always != 0 here
return err;

Good examples of VMX initialization code can be found in Xen (http://xen.org) and KVM (http://www.linux-kvm.org) sources.

[strictlymike]

Thank you for your keen eyes! After fixing that omission and taking into account multiple cores, VMXON succeeds on both cores. Subsequent VMXOFF instructions (per core) execute withoutany #UD, further confirming that VMX operation was engaged on each.

Thanks again!

-M.