- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi all,
Is there a way to validate via system/OS/command line whether the cluster and memory mode selected from BIOS is what the system booted with?
System Details:
- Operating System: CentOS Linux 7 (Core)
- Kernel: Linux 3.10.0-514.10.2.el7.x86_64
- Architecture: x86-64 Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JJK,
Thank you for the detailed response. I also did similar analysis, but only with flat memory mode. The only question I have is regarding quadrant cluster mode and how the system sees it. As of now, it seems All-2-All and Quadrant/Hemisphere are visually similar based on "numactl -H".
I found another way to log which cluster and memory mode is active. Below log shows it's quadrant mode with flat memory. But still not sure about the number of nodes. Shouldn't the number of nodes for Quadrant/Hemisphere be 4?
On side note: What is the difference between Quadrant and Hemisphere?
hwloc-dump-hwdata Dumping KNL SMBIOS Memory-Side Cache information: File = ///sys/firmware/dmi/entries/14-0/raw, size = 4096 Found KNL type = 160 Found KNL type = 161 Seeking dir ̀`160-' 4 File = ///sys/firmware/dmi/entries/160-0/raw, size = 4096 Getting general KNL info Seeking dir ̀`161-' 4 File = ///sys/firmware/dmi/entries/161-0/raw, size = 4096 Getting MCDRAM KNL info. Count=8 struct size=12 MCDRAM controller 0 Size = 2048 MB MCDRAM controller 1 Size = 2048 MB MCDRAM controller 2 Size = 2048 MB MCDRAM controller 3 Size = 2048 MB MCDRAM controller 4 Size = 2048 MB MCDRAM controller 5 Size = 2048 MB MCDRAM controller 6 Size = 2048 MB MCDRAM controller 7 Size = 2048 MB Total MCDRAM 16384 MB Cluster Mode: Quadrant Memory Mode: Flat MCDRAM total = 17179869184 bytes, cache = 0 bytes MCDRAM total = 17179869184 bytes, cache = 0 bytes per node numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 0 size: 98178 MB node 0 free: 94617 MB node 1 cpus: node 1 size: 16384 MB node 1 free: 15927 MB node distances: node 0 1 0: 10 31 1: 31 10
Thanks.
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi
If is not ghost , and the two choice are tried , Normally, the command line (dmesg) must
show a difference if your option is selected or no. if I remember correctly,exist
also (dmidecode)
Regards
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can use
numactl -H
and/or numa API to determine number of nodes.
It will show you CPU nodes as well as memory-only nodes.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Jim
It does show nodes, but currently for quadrant and flat mode how come following is true? Shouldn't there be 5 nodes: 4 for each cpu quadrant and 1 for MCDRAM?
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
node 0 size: 98178 MB
node 0 free: 90328 MB
node 1 cpus:
node 1 size: 16384 MB
node 1 free: 15905 MB
node distances:
node 0 1
0: 10 31
1: 31 10
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The above shows you are not in quadrant mode.
Instead it appears you have All-to-All with the MCDRAM set as 1 memory node.
My system with SNC-4 with MCDRAM set as cache:
numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207
node 0 size: 24450 MB
node 0 free: 1351 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223
node 1 size: 24576 MB
node 1 free: 6442 MB
node 2 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239
node 2 size: 24576 MB
node 2 free: 10859 MB
node 3 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
node 3 size: 24576 MB
node 3 free: 20643 MB
node distances:
node 0 1 2 3
0: 10 21 21 21
1: 21 10 21 21
2: 21 21 10 21
3: 21 21 21 10
I suggest you check your BIOS settings.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When I had the system configured as 4 CPU nodes + 4 memory nodes, the report listed 8 nodes.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Jim,
I will check BIOS setting.
What do you mean by 4 CPU nodes + 4 memory nodes? Which cluster and memory mode it's?
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
After (fiddling with the BIOS and rebooting) about 10 times I end up with the following results:
- you can find out whether you're using the MCRAM in flat. cache or hybrid mode by looking at the "memory only" NUMA nodes; the size of memory available to the "memory-only" NUMA nodes will tell you whether you're running in flat (16 GB memory-only numa), hybrid (8 GB memory-only numa) or cache (no memory-only numa)
- as a minor detail, check /var/log/messages to see if the non-cached memory is considered hot-swappable or not:
< knl kernel: SRAT: Node 4 PXM 4 [mem 0x640000000-0x73fffffff] hotplug < knl kernel: SRAT: Node 5 PXM 5 [mem 0xd40000000-0xe3fffffff] hotplug < knl kernel: SRAT: Node 6 PXM 6 [mem 0x1440000000-0x153fffffff] hotplug < knl kernel: SRAT: Node 7 PXM 7 [mem 0x1b40000000-0x1c3fffffff] hotplug --- > knl kernel: SRAT: Node 4 PXM 4 [mem 0x640000000-0x73fffffff] > knl kernel: SRAT: Node 5 PXM 5 [mem 0xd40000000-0xe3fffffff] > knl kernel: SRAT: Node 6 PXM 6 [mem 0x1440000000-0x153fffffff] > knl kernel: SRAT: Node 7 PXM 7 [mem 0x1b40000000-0x1c3fffffff]
- as stated before, use 'numactl -H' to determine the number of available NUMA nodes:
- 1 node -> all2all/quadrant/hemisphere , MCDRAM = cache
- 2 nodes, one with cpus, other without -> all2all/quadrant/hemisphere , MCDRAM = flat/hybrid
- 2 nodes with cpus in them -> SNC-2 , MCDRAM = cache
- 4 nodes, 2 with cpus, others without -> SNC-2, MCDRAM = flat/hybrid
- 4 nodes with cpus in them -> SNC-4 , MCDRAM = cache
- 8 nodes, 4 with cpus, others without -> SNC-4, MCDRAM = flat/hybrid
the number of available nodes is also present in /var/log/messages (this is for SNC4 + flat):
knl kernel: smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 OK knl kernel: smpboot: Booting Node 3, Processors #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 OK knl kernel: smpboot: Booting Node 1, Processors #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 OK knl kernel: smpboot: Booting Node 2, Processors #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 OK knl kernel: smpboot: Booting Node 0, Processors #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 OK knl kernel: smpboot: Booting Node 3, Processors #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 OK knl kernel: smpboot: Booting Node 1, Processors #96 #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 OK knl kernel: smpboot: Booting Node 2, Processors #112 #113 #114 #115 #116 #117 #118 #119 #120 #121 #122 #123 #124 #125 #126 #127 OK knl kernel: smpboot: Booting Node 0, Processors #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141 #142 #143 OK knl kernel: smpboot: Booting Node 3, Processors #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156 #157 #158 #159 OK knl kernel: smpboot: Booting Node 1, Processors #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171 #172 #173 #174 #175 OK knl kernel: smpboot: Booting Node 2, Processors #176 #177 #178 #179 #180 #181 #182 #183 #184 #185 #186 #187 #188 #189 #190 #191 OK knl kernel: smpboot: Booting Node 0, Processors #192 #193 #194 #195 #196 #197 #198 #199 #200 #201 #202 #203 #204 #205 #206 #207 OK knl kernel: smpboot: Booting Node 3, Processors #208 #209 #210 #211 #212 #213 #214 #215 #216 #217 #218 #219 #220 #221 #222 #223 OK knl kernel: smpboot: Booting Node 1, Processors #224 #225 #226 #227 #228 #229 #230 #231 #232 #233 #234 #235 #236 #237 #238 #239 OK knl kernel: smpboot: Booting Node 2, Processors #240 #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251 #252 #253 #254 #255 OK knl kernel: Brought up 256 CPUs knl kernel: smpboot: Total of 256 processors activated (665537.53 BogoMIPS) knl kernel: node 4 initialised, 507840 pages in 33ms knl kernel: node 5 initialised, 507840 pages in 33ms knl kernel: node 6 initialised, 507840 pages in 33ms knl kernel: node 7 initialised, 507723 pages in 33ms knl kernel: node 0 initialised, 4880203 pages in 311ms knl kernel: node 1 initialised, 5666684 pages in 360ms knl kernel: node 2 initialised, 5666686 pages in 369ms knl kernel: node 3 initialised, 5666685 pages in 369ms
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JJK,
Thank you for the detailed response. I also did similar analysis, but only with flat memory mode. The only question I have is regarding quadrant cluster mode and how the system sees it. As of now, it seems All-2-All and Quadrant/Hemisphere are visually similar based on "numactl -H".
I found another way to log which cluster and memory mode is active. Below log shows it's quadrant mode with flat memory. But still not sure about the number of nodes. Shouldn't the number of nodes for Quadrant/Hemisphere be 4?
On side note: What is the difference between Quadrant and Hemisphere?
hwloc-dump-hwdata Dumping KNL SMBIOS Memory-Side Cache information: File = ///sys/firmware/dmi/entries/14-0/raw, size = 4096 Found KNL type = 160 Found KNL type = 161 Seeking dir ̀`160-' 4 File = ///sys/firmware/dmi/entries/160-0/raw, size = 4096 Getting general KNL info Seeking dir ̀`161-' 4 File = ///sys/firmware/dmi/entries/161-0/raw, size = 4096 Getting MCDRAM KNL info. Count=8 struct size=12 MCDRAM controller 0 Size = 2048 MB MCDRAM controller 1 Size = 2048 MB MCDRAM controller 2 Size = 2048 MB MCDRAM controller 3 Size = 2048 MB MCDRAM controller 4 Size = 2048 MB MCDRAM controller 5 Size = 2048 MB MCDRAM controller 6 Size = 2048 MB MCDRAM controller 7 Size = 2048 MB Total MCDRAM 16384 MB Cluster Mode: Quadrant Memory Mode: Flat MCDRAM total = 17179869184 bytes, cache = 0 bytes MCDRAM total = 17179869184 bytes, cache = 0 bytes per node numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 0 size: 98178 MB node 0 free: 94617 MB node 1 cpus: node 1 size: 16384 MB node 1 free: 15927 MB node distances: node 0 1 0: 10 31 1: 31 10
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Quadrant and All-to-All modes are indistinguishable from a normal user perspective. The only things that change are related to the (undocumented) hash that controls the mapping of physical addresses to CHA units and MCDRAM controllers (EDCs).
From looking at the EDC performance counters, it looks like All-to-All mode has a simple round-robin interleaving of consecutive cache lines across the 8 EDC controllers. Quadrant mode modifies the physical address to EDC mapping so that consecutive physical addresses are mapped to pseudo-random EDCs. I think this was done so that the same physical address to CHA hash could be used in both modes, but to make this work in Quadrant mode, the mapping of addresses to EDCs had to be changed so that each address would map to an EDC in the same quadrant as the CHA that owns that address.
There is almost no difference in average idle memory latency between Quadrant and All-to-All modes, but All-to-All shows about 7% lower STREAM bandwidth. This is likely due to the increased contention on the links due to the higher average number of hops required for transfers in All-to-All mode.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've checked my Phi 7210 in both modes; numactl and /var/log/messages show no noticable differences. The only difference that I can find is in the DMI info:
hemisphere: Handle 0x0027, DMI type 160, 12 bytes OEM-specific Type Header and Data: A0 0C 27 00 01 00 01 1F 02 07 01 04 Strings: Member: Knights Landing General Information quadrant Handle 0x0027, DMI type 160, 12 bytes OEM-specific Type Header and Data: A0 0C 27 00 01 00 01 1F 01 07 01 04 Strings: Member: Knights Landing General Information
but as you can see, this is OEM-specific and hence does not apply to all platforms.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I want to know how to select cluster and memory mode using the command line? Are there any tools or commands?
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Cluster and memory mode are BIOS settings - you will need to change them in the BIOS and then do a (cold) reboot. Depending on the BIOS of your system , you may be able to change these settings using a commandline tool, but that is highly system dependent.

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page