is ISA (x86_64/AVX/AMX) and precision (fp32/fp16/bf16/i8) controllable?

[CFR]

I'm looking to do some studies about running transformer/LLM on CPU resources.  I'd like to take a single Sapphire Rapids box and, in a controlled way, vary ISA/precision.  I see the ability to give OpenVINO a "hint" about precision but it's unclear how compelling that "hint" is to the underlying software (oneDNN?).  I don't see any OpenVINO API that controls ISA nor do I see any diagnostics that would tell me what an actually run ended up using.

Does OpenVINO have some way to control the things I want so I can do controlled studies? Is there a better choice (like maybe extensions for pytorch)?

[Aznie_Intel]

Hi CFR,

Thanks for reaching out. I listed the documentation that might be helpful for your reference.

• CPU Dispatcher Control for OpenVINO™ Inference Runtime Execution - to learn the implementation of ISA extensions and how to change the ISA extensions’ optimized kernel function 
• LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO
• Precision Control - How to control inference precision and limitation

Hope this helps.

Regards,

Aznie

[CFR · Accepted solution]

Thanks.  Looks like that gets me going in the right direction.

[Aznie_Intel]

Hi CFR,

Glad to hear that. This thread will no longer be monitored since this issue has been resolved. If you need any additional information from Intel, please submit a new question.

Regards,

Aznie