How to use MCDRAM in Hybrid Mode

[Lu__Xiaoyang]

Hi,

I have a question about how to how to use MCDRAM in Hybrid Mode. For example, when using MCDRAM in Hybrid Mode, if I call the Cache path that MCDRAM uses as a cache, and call the HBM path that MCDRAM uses as addressable memory. Can I only allocate the data on Cache Path or only allocate the data on HBM Path by using numactl -m like Flat Mode? I assume by default when using the MCDRAM in Hybrid Mode, the data will be only allocated on Cach Path. And if we adding the tag numactl -m, the data can be allocated in HBM path only.  I don't know if my guess is right or not. Any suggestions or commands are welcome.   

I appreciate all your feedback, Thank you! 

[McCalpinJohn]

If the system is configured correctly, you should be able to allocate data in the "flat" portion of MCDRAM using numactl, the libnuma API, or the "memkind()" API.   Any data that is not specifically allocated in MCDRAM will be allocated in DDR4 memory, and accesses to DDR4 memory will be cached by the remaining MCDRAM.

[Carlos_R_Intel]

As per John's response, the hybrid mode in Theta works the same way as in any other Intel® Xeon Phi™ based system. A "numactl -H" command will give you output where numa node 0 corresponds to the 192 GB of DDR4 in the system and is the default location. Your flat space will be in numa node 1 and have different sizes (8 GB or 12 GB) depending on which option ("equal" or "split") you choose on job submission. The remaining MCDRAM will be dedicated to caching DDR4 accesses in a transparent manner and you will have no direct control over that process.

[Lu__Xiaoyang]

Thank you for your reply.

Are there any tools can help me to change the cluster and memory mode by using the system/OS/command line?