Get Started Accelerating AI on Intel® Xeon®

Author: 

Mihika Nerurkar, Cloud Solutions Architect

Introduction

If you're aiming to deploy AI/ML applications in the cloud, you can benefit from AI acceleration on Xeon® CPU-powered instances from various vendors. Intel® Xeon® Scalable processors feature Intel® Advanced Matrix Extensions (Intel® AMX), which accelerate the matrix arithmetic at the heart of AI computations. But how do you know if your cloud instance supports AMX? This detection tool provides the answer.

This Blog Will Show You How to Automatically Detect Intel® AMX Features

This blog will introduce a new Intel® AMX Detection Tool that tells you when your platform supports AMX and whether your OS has enabled it. It’s a useful tool when you’re developing AI applications on Intel® Xeon® enabled instances. This tool is most useful when you're deploying production-scale ML inference pipelines, conducting enterprise IT audits of AMX-capable hardware, or validating that your developer workstation is AMX-enabled before tuning models.

Why AMX Matters

If you’re building or scaling AI applications, you know matrix multiplications sit at the core of machine learning and deep learning workloads. From inference pipelines to large-scale training, performance bottlenecks often come down to how efficiently your hardware can handle those dense linear algebra operations. That’s where Intel® Advanced Matrix Extensions (Intel® AMX) comes in.

Intel® AMX accelerates AI inferencing, deep learning, and high-performance computing (HPC) applications. It was first introduced in 4th Gen Intel® Xeon® Scalable processors, and further tuned for future generations like Xeon® 6. This diagram below shows how AMX works with 2-D register files and matrix multiply functions:

The datatypes supported by Intel® AMX:

• BF16, FP16: acceleration for training/inference
• INT8: optimized 8-bit integer performance for inference

For developers, the challenge is knowing when your platform supports AMX and whether your OS has enabled it. That’s exactly what the Intel® AMX Detection Tool solves.

The Problem Developers Face

You spin up a new VM, provision bare-metal servers, or test on your local dev machine. You want to take advantage of Intel® oneDNN, TensorFlow with Intel® extensions, or PyTorch tuned for AMX. But do you really know if AMX is enabled? Without visibility, you could be running workloads in a suboptimal mode, missing out on significant acceleration.

How the Intel® AMX Detection Tool Works

The AMX Detection Tool provides a lightweight, cross-platform way to verify AMX readiness in both hardware and software.

1. CPU Support:
   • The tool executes the cpuid instruction to check if AMX bits are set.
   • Specifically:
     
     • CPUID.07H.00H:EDX[24] → AMX-TILE
     • CPUID.07H.00H:EDX[22] → AMX-BF16
     • CPUID.07H.00H:EDX[25] → AMX-INT8
2. OS Support:
   • It uses the XGETBV instruction to query the XCR0 register.
   • This confirms whether your operating system has enabled AMX state management (critical for actually using AMX in workloads).
3. Cross-Platform Implementation:
   
   • On Linux, inline assembly and syscalls provide low-level access.
   • On Windows, Visual Studio intrinsics handle the checks. 

The Python script ties it all together by compiling the C-based detection logic, running it, and presenting a clear pass/fail summary of your AMX capabilities. Example output:

Why It’s a Game-Changer for Developers

• Zero Guesswork: No more digging through processor manuals or running trial workloads to check AMX.
  
• Cloud-Ready: Validate AMX enablement in AWS, Azure, or GCP instances before committing resources.
• Confidence in Tuning: When using Intel-optimized frameworks, you’ll know you’re hitting the AMX fast paths.
  
• Future-Proof: With Xeon® 6 and beyond, AMX expands its role in accelerating ML and AI inferencing. This tool makes those differentiators immediately visible. 
  

Deploy the AMX Detection tool on Linux:

In this section we will go over how to run the Intel® AMX Detection Tool on your CPU of choice.

Prerequisites

1. Python 3.6+ and Git installed on your system
   
2. Access to the target machine where AMX support needs to be checked
3. Clone or download the tool from GitHub:
    git clone https://github.com/intel/Intel-AMX-Detection.git
4. cd Intel-AMX-Detection

Linux Usage:

1. Open a terminal and navigate to the tool’s directory.
2. Compile both amx_os_linux.c and amx_detection.c together:
   gcc -o amx_detection amx_detection.c amx_os_linux.c
3. Run the detection script:
   python3 amx-detect-linux.py
   
4. Output will display whether AMX instructions are available and supported on your CPU as well as OS.

For detailed instructions on using the tool as well usage instructions for Windows, please refer to the README: https://github.com/intel/Intel-AMX-Detection 

How to Get Started

Ready to check your system? The tool is open source and simple to use. Clone, compile, and run it in just a few steps.

• Visit the GitHub repo here: https://github.com/intel/Intel-AMX-Detection
• Developer Tip: Pair the AMX Detection Tool with Intel® oneAPI libraries and frameworks to unlock the full acceleration potential of AMX for AI inferencing.

Notices and Disclaimers

Performance varies by use, configuration, and other factors. Learn more on the Performance Index site. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for configuration details. No product or component can be absolutely secure.

Your costs and results may vary.

Intel technologies may require enabled hardware, software, or service activation.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.