Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

SANDY BRIDGE ADDRESS MAPPING SCHEME

sarkar__saptarshi
448 Views

Hi,

    I am investigating the memory address mapping scheme in Sandy Bridge MC.

    My platform has two sockets and total system RAM of 64 GB. 

    My question is regarding experimental values obtained from memory controller PCI registers

    which help in system address decoding by the MC

    Below are the readings of DRAM RULE register & DRAM INTERLEAVE register

###################################
  PCI DEVICE 12, FN : 0 BUS 1 (There are 10 DRAM Rules)
###################################

DRAM RULE REGISTER 0 (0x82c3)
--------------------------------

LIMIT                                      :       0x20b (64 MB Granularity )  ~ 32 GB
ATTRIBUTE FOR DRAM RULE :       DRAM
INTERLEAVE MODE                :       0    Address Bits <8,7,6> ^ Address Bits <18,17,16>
RULE_ENABLE                        :       1

DRAM INTERLEAVE LIST 0 REGISTER (0)
---------------------------------

PACKAGE 8               :       0
PACKAGE 7               :       0
PACKAGE 6               :       0
PACKAGE 5               :       0
PACKAGE 4               :       0
PACKAGE 3               :       0
PACKAGE 2               :       0
PACKAGE 1               :       0

DRAM RULE REGISTER 1 (102c3)
--------------------------------

LIMIT                                      :       0x40b (64 MB Granularity)  ~ 64 GB
ATTRIBUTE FOR DRAM RULE :       DRAM
INTERLEAVE MODE                :       0  Address Bits <8,7,6> ^ Address Bits <18,17,16>
RULE_ENABLE                        :       1

DRAM INTERLEAVE LIST 1 REGISTER (0x9090909)
-----------------------------------------

PACKAGE 8               : 1
PACKAGE 7               : 1
PACKAGE 6               : 1
PACKAGE 5               : 1
PACKAGE 4               : 1
PACKAGE 3               : 1
PACKAGE 2               : 1
PACKAGE 1               : 1

All the rest registers are diabled.

The DRAM Interleave Register for First 32 GB has only 0 entries. I beleive it indicates socket No 1.

The DRAM Interleave Register for the next 32 GB (till 64 GB ) has only 1 as entry. I believe it indicates socket No 2.

Does this mean that there is no interleaving across Nodes for either the first 32 GB or the next 32 GB  ?

However If I use the Performance Monitor Software, I observe interleaving across nodes even within the first 32 GB.

I am not able to explain the results.

Any suggestions are welcome.

Thank you

0 Kudos
1 Reply
Patrick_F_Intel1
Employee
448 Views

Hello Saptarshi,

I'm not familiar with all the dram setup registers but I have a question. Is NUMA enabled?

If NUMA is enabled, then (if I understand correctly) the memory attached to node 0 should all belong to node 0 and node 1's memory belongs to node 1.

If NUMA is not enabled then the memory is interleaved at the cacheline level.

Even if NUMA is enabled, depending on how you run (for example, if you are not pinned to the node on which you allocate memory) you can still see cross node memory accesses.

Pat

0 Kudos
Reply