Why AVX do not support real/V86 mode

[yuhong2]

Why doesn't AVX support real/V86 mode when all older SSE extensions did?
You may know that the VEX prefix in non 64-bit modes is basically based on illegal operands of the LDS/LES instructions.
What you may not know is that NTVDM uses the same illegal operands in V86 mode as a trap to protected mode, which would conflict with the VEX prefix.
Trivia: The Windows/386 series used ARPL, which was an illegal instruction in V86 mode, as it's trap.

[Quoc-Thai_L_Intel]

Here are some additional details relating to this topic:

AVX and FMA instructions comprises of 256-bit and 128-bit instructions that operates on YMM states via VEX prefix encoding. SIMD instructions operating on XMM states (i.e. not accessing the upper 128 bits of YMM) generally do not use VEX prefix. For processors that support YMM states, the YMM state exists in all operating modes. However, the available interfaces to access YMM states may vary in different modes. The processor's support for instruction extensions that employ VEX prefix encoding isindependent of the processor's support for YMM state. Instructions requiring VEX prefix encoding generally are supported in 64-bit, 32-bit modes, and 16-bit protected mode. They are not supported in Real mode, Virtual-8086 mode or entering into SMM mode.

Note that bits 255:128 of YMM register state are maintained across transitions into and out of these modes. Because, XSAVE/XRSTOR instruction can operate in all operating modes, it is possible that the processor's YMM register state can be modified by software in any operating mode by executing XRSTOR. The YMM registers can be updated by XRSTOR using the state information stored in the XSAVE/XRSTOR area residing in memory.

-Thai