topic Hi in Software Archive

Intel Xeon Phi - Find Cluster & Memory Mode

CPati2 — Tue, 05 Sep 2017 21:03:28 GMT

Hi all,

Is there a way to validate via system/OS/command line whether the cluster and memory mode selected from BIOS is what the system booted with?

System Details:

Operating System: CentOS Linux 7 (Core)
Kernel: Linux 3.10.0-514.10.2.el7.x86_64
Architecture: x86-64 Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz

Thanks.

Hi

aazue — Tue, 05 Sep 2017 22:33:12 GMT

Hi
If is not ghost , and the two choice are tried , Normally, the command line (dmesg) must
show a difference if your option is selected or no. if I remember correctly,exist
also (dmidecode)
Regards

You can use

jimdempseyatthecove — Wed, 06 Sep 2017 12:37:54 GMT

You can use

numactl -H

and/or numa API to determine number of nodes.

It will show you CPU nodes as well as memory-only nodes.

Jim Dempsey

Hi Jim

CPati2 — Wed, 06 Sep 2017 16:32:47 GMT

Hi Jim

It does show nodes, but currently for quadrant and flat mode how come following is true? Shouldn't there be 5 nodes: 4 for each cpu quadrant and 1 for MCDRAM?

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
node 0 size: 98178 MB
node 0 free: 90328 MB
node 1 cpus:
node 1 size: 16384 MB
node 1 free: 15905 MB
node distances:
node 0 1
0: 10 31
1: 31 10

Thanks.

The above shows you are not

jimdempseyatthecove — Wed, 06 Sep 2017 17:33:55 GMT

The above shows you are not in quadrant mode.

Instead it appears you have All-to-All with the MCDRAM set as 1 memory node.

My system with SNC-4 with MCDRAM set as cache:

numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207
node 0 size: 24450 MB
node 0 free: 1351 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223
node 1 size: 24576 MB
node 1 free: 6442 MB
node 2 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239
node 2 size: 24576 MB
node 2 free: 10859 MB
node 3 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
node 3 size: 24576 MB
node 3 free: 20643 MB
node distances:
node   0   1   2   3
0: 10 21 21 21
    1: 21 10 21 21
   2: 21 21 10 21
   3: 21 21 21 10

I suggest you check your BIOS settings.

Jim Dempsey

When I had the system

jimdempseyatthecove — Wed, 06 Sep 2017 17:39:18 GMT

When I had the system configured as 4 CPU nodes + 4 memory nodes, the report listed 8 nodes.

Jim Dempsey

Hi Jim,

CPati2 — Wed, 06 Sep 2017 17:57:19 GMT

Hi Jim,

I will check BIOS setting.

What do you mean by 4 CPU nodes + 4 memory nodes? Which cluster and memory mode it's?

Thanks.

After (fiddling with the BIOS

JJK — Thu, 07 Sep 2017 10:25:00 GMT

After (fiddling with the BIOS and rebooting) about 10 times I end up with the following results:

you can find out whether you're using the MCRAM in flat. cache or hybrid mode by looking at the "memory only" NUMA nodes; the size of memory available to the "memory-only" NUMA nodes will tell you whether you're running in flat (16 GB memory-only numa), hybrid (8 GB memory-only numa) or cache (no memory-only numa)
as a minor detail, check /var/log/messages to see if the non-cached memory is considered hot-swappable or not:

< knl kernel: SRAT: Node 4 PXM 4 [mem 0x640000000-0x73fffffff] hotplug
< knl kernel: SRAT: Node 5 PXM 5 [mem 0xd40000000-0xe3fffffff] hotplug
< knl kernel: SRAT: Node 6 PXM 6 [mem 0x1440000000-0x153fffffff] hotplug
< knl kernel: SRAT: Node 7 PXM 7 [mem 0x1b40000000-0x1c3fffffff] hotplug
---
> knl kernel: SRAT: Node 4 PXM 4 [mem 0x640000000-0x73fffffff]
> knl kernel: SRAT: Node 5 PXM 5 [mem 0xd40000000-0xe3fffffff]
> knl kernel: SRAT: Node 6 PXM 6 [mem 0x1440000000-0x153fffffff]
> knl kernel: SRAT: Node 7 PXM 7 [mem 0x1b40000000-0x1c3fffffff]

as stated before, use 'numactl -H' to determine the number of available NUMA nodes:
- 1 node -> all2all/quadrant/hemisphere , MCDRAM = cache
- 2 nodes, one with cpus, other without -> all2all/quadrant/hemisphere , MCDRAM = flat/hybrid
- 2 nodes with cpus in them -> SNC-2 , MCDRAM = cache
- 4 nodes, 2 with cpus, others without -> SNC-2, MCDRAM = flat/hybrid
- 4 nodes with cpus in them -> SNC-4 , MCDRAM = cache
- 8 nodes, 4 with cpus, others without -> SNC-4, MCDRAM = flat/hybrid

the number of available nodes is also present in /var/log/messages (this is for SNC4 + flat):

knl kernel: smpboot: Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 OK
knl kernel: smpboot: Booting Node   3, Processors  #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 OK
knl kernel: smpboot: Booting Node   1, Processors  #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 OK
knl kernel: smpboot: Booting Node   2, Processors  #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 OK
knl kernel: smpboot: Booting Node   0, Processors  #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 OK
knl kernel: smpboot: Booting Node   3, Processors  #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 OK
knl kernel: smpboot: Booting Node   1, Processors  #96 #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 OK
knl kernel: smpboot: Booting Node   2, Processors  #112 #113 #114 #115 #116 #117 #118 #119 #120 #121 #122 #123 #124 #125 #126 #127 OK
knl kernel: smpboot: Booting Node   0, Processors  #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141 #142 #143 OK
knl kernel: smpboot: Booting Node   3, Processors  #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156 #157 #158 #159 OK
knl kernel: smpboot: Booting Node   1, Processors  #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171 #172 #173 #174 #175 OK
knl kernel: smpboot: Booting Node   2, Processors  #176 #177 #178 #179 #180 #181 #182 #183 #184 #185 #186 #187 #188 #189 #190 #191 OK
knl kernel: smpboot: Booting Node   0, Processors  #192 #193 #194 #195 #196 #197 #198 #199 #200 #201 #202 #203 #204 #205 #206 #207 OK
knl kernel: smpboot: Booting Node   3, Processors  #208 #209 #210 #211 #212 #213 #214 #215 #216 #217 #218 #219 #220 #221 #222 #223 OK
knl kernel: smpboot: Booting Node   1, Processors  #224 #225 #226 #227 #228 #229 #230 #231 #232 #233 #234 #235 #236 #237 #238 #239 OK
knl kernel: smpboot: Booting Node   2, Processors  #240 #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251 #252 #253 #254 #255 OK
knl kernel: Brought up 256 CPUs
knl kernel: smpboot: Total of 256 processors activated (665537.53 BogoMIPS)
knl kernel: node 4 initialised, 507840 pages in 33ms
knl kernel: node 5 initialised, 507840 pages in 33ms
knl kernel: node 6 initialised, 507840 pages in 33ms
knl kernel: node 7 initialised, 507723 pages in 33ms
knl kernel: node 0 initialised, 4880203 pages in 311ms
knl kernel: node 1 initialised, 5666684 pages in 360ms
knl kernel: node 2 initialised, 5666686 pages in 369ms
knl kernel: node 3 initialised, 5666685 pages in 369ms

Hi JJK,

CPati2 — Thu, 07 Sep 2017 14:51:41 GMT

Hi JJK,

Thank you for the detailed response. I also did similar analysis, but only with flat memory mode. The only question I have is regarding quadrant cluster mode and how the system sees it. As of now, it seems All-2-All and Quadrant/Hemisphere are visually similar based on "numactl -H".

I found another way to log which cluster and memory mode is active. Below log shows it's quadrant mode with flat memory. But still not sure about the number of nodes. Shouldn't the number of nodes for Quadrant/Hemisphere be 4?

On side note: What is the difference between Quadrant and Hemisphere?

hwloc-dump-hwdata
  Dumping KNL SMBIOS Memory-Side Cache information:
  File = ///sys/firmware/dmi/entries/14-0/raw, size = 4096
  Found KNL type = 160
  Found KNL type = 161
  Seeking dir ̀`160-' 4
  File = ///sys/firmware/dmi/entries/160-0/raw, size = 4096
  Getting general KNL info
  Seeking dir ̀`161-' 4
  File = ///sys/firmware/dmi/entries/161-0/raw, size = 4096
  Getting MCDRAM KNL info. Count=8 struct size=12
  MCDRAM controller 0
  Size = 2048 MB
  MCDRAM controller 1
  Size = 2048 MB
  MCDRAM controller 2
  Size = 2048 MB
  MCDRAM controller 3
  Size = 2048 MB
  MCDRAM controller 4
  Size = 2048 MB
  MCDRAM controller 5
  Size = 2048 MB
  MCDRAM controller 6
  Size = 2048 MB
  MCDRAM controller 7
  Size = 2048 MB
  Total MCDRAM 16384 MB
  Cluster Mode: Quadrant Memory Mode: Flat
  MCDRAM total = 17179869184 bytes, cache = 0 bytes
  MCDRAM total = 17179869184 bytes, cache = 0 bytes per node

numactl -H
  available: 2 nodes (0-1)
  node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
  node 0 size: 98178 MB
  node 0 free: 94617 MB
  node 1 cpus:
  node 1 size: 16384 MB
  node 1 free: 15927 MB
  node distances:
  node   0   1
    0:  10  31
    1:  31  10

Thanks.

Quadrant and All-to-All modes

McCalpinJohn — Thu, 07 Sep 2017 21:54:00 GMT

Quadrant and All-to-All modes are indistinguishable from a normal user perspective. The only things that change are related to the (undocumented) hash that controls the mapping of physical addresses to CHA units and MCDRAM controllers (EDCs).

From looking at the EDC performance counters, it looks like All-to-All mode has a simple round-robin interleaving of consecutive cache lines across the 8 EDC controllers. Quadrant mode modifies the physical address to EDC mapping so that consecutive physical addresses are mapped to pseudo-random EDCs. I think this was done so that the same physical address to CHA hash could be used in both modes, but to make this work in Quadrant mode, the mapping of addresses to EDCs had to be changed so that each address would map to an EDC in the same quadrant as the CHA that owns that address.

There is almost no difference in average idle memory latency between Quadrant and All-to-All modes, but All-to-All shows about 7% lower STREAM bandwidth. This is likely due to the increased contention on the links due to the higher average number of hops required for transfers in All-to-All mode.

I've checked my Phi 7210 in

JJK — Fri, 08 Sep 2017 07:47:00 GMT

I've checked my Phi 7210 in both modes; numactl and /var/log/messages show no noticable differences. The only difference that I can find is in the DMI info:

hemisphere:
Handle 0x0027, DMI type 160, 12 bytes
OEM-specific Type
    Header and Data:
        A0 0C 27 00 01 00 01 1F 02 07 01 04
    Strings:
        Member: Knights Landing General Information

quadrant
Handle 0x0027, DMI type 160, 12 bytes
OEM-specific Type
    Header and Data:
        A0 0C 27 00 01 00 01 1F 01 07 01 04
    Strings:
        Member: Knights Landing General Information

but as you can see, this is OEM-specific and hence does not apply to all platforms.

Hi,

Lu__Xiaoyang — Fri, 08 Mar 2019 18:54:11 GMT

Hi,

I want to know how to select cluster and memory mode using the command line? Are there any tools or commands?

Thanks.

Cluster and memory mode are

JJK — Mon, 11 Mar 2019 07:33:59 GMT

Cluster and memory mode are BIOS settings - you will need to change them in the BIOS and then do a (cold) reboot. Depending on the BIOS of your system , you may be able to change these settings using a commandline tool, but that is highly system dependent.