On page 3-378 of the 64-ia-32 documentation set, volume 2A, order number 325383 (April 2016), the instruction FXAM is described as "FXAM --- Examine ModR/M". The instruction's documentation does not show the instruction takes ModR/M bytes. In fact, the instruction is described as operating on a fixed register, i.e. ST(0).
Yeah, that's an old doc bug. Should say something like "Examine Floating Point". Hopefully it will be fixed in the next revision of the SDM presumably in a quarter or so.