I have written a simple VMM that executes a small program under VTx. Everything is working fine with the exception of one item, if I receive an NMI while running the code that executes after the vmexit, the system crashes (this is on Windows). The issue is that before calling vmlaunch/vmresume I perform some modifications to the host state such as setting IA32_LSTAR msr to facilitate the code running under VTx as well as restoring register state, etc for the code to continue executing.
I am guessing the issue is that I have not fully restored the host to a state where the NMI can be processed and my code returned to successfully. So my question is how do I ensure that I can successfully perform all the necessary changes to get the host back to a working state before I receive an NMI?
Found a couple of links related to VM entry and exit that may provide some hints for you:
One thing to consider is that you shouldn't be thinking about it like "how should i get back to the host state before i receive an NMI" it should be "how should I handle the NMI i receive when in various states".
If you have a full OS running in the VM and you don't care to emulate anything / capture any information from the NMI, you may just not want to exit for the NMI. If you don't you'll have to exit and let the Host OS handle the NMI - see "24.6.1 Pin-Based VM-Execution Controls", specifically for the NMI exiting control bit.
Then you'll need to handle the VM Exit appropriately from there. Edit: If the error is in this state (VMX Root operation), and you've created a bad host state for NMI processing, make sure your IRQ levels are correct and that you're restoring host state correctly *during VM exit* (not manually after the exit occurs).
Don't manually change MSR's the host requires for proper execution, there is a VMCS areas for doing this ([Entry|Exit] [Host|Guest] [Save|Load] MSR areas).