How to use MTRR on machines with Skylake processors?

[XWang41]

I'm trying to do some memory management on xen-4.5.0 on machines with Skylake processors, and get confused with something about MTRR(Memory-type-range-register). While I was trying to get the value of IA32_MTRRCAP register, I got value 1d0a.

However, in the Intel manual released in December 2016 ( https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf, Figure 11.5, page 3092), bits 12-63 of IA32_MTRRCAP register are all marked reserved, while I got an 1 on bit 12. Why?

Meanwhile, when I was trying to read memory information from MTRRs, I could not find most of memory addresses in them.

I've tried these on several machines with Intel Skylake processors, and their situations are the same. But on machines with Intel Haswell processors, the value of IA32_MTRRCAP has nothing strange, I can also read all memory information from MTRRs. The model number for Haswell is Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz, for Skylake is Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz.

The code for getting value of IA32_MTRRCAP and get memory info from MTRRs:

rdmsrl(MSR_MTRRdefType, msr_deftype);
    sprintk("MTRRdefType=%lx\n", msr_deftype);
    rdmsrl(MSR_MTRRcap, msr_cap);
    sprintk("MTRRcap=%lx\n", msr_cap);
    vcnt = (int)(msr_cap & MTRR_VCNT_MASK);
    for (idx = 0; idx < vcnt; idx += 1) {
        rdmsrl(MSR_IA32_MTRR_PHYSBASE(idx), base);
        rdmsrl(MSR_IA32_MTRR_PHYSMASK(idx), mask);
        if (!(mask & MTRR_VRRP_MASK_MASK)) continue;
        start_mfn = base >> 12;
        num_mfn = (~(mask >> 12) & ((1UL << (phys_addr_size - 12)) - 1)) + 1;
        mem_type = base & 0xff;
        sprintk("===MTTR base=%lx mask=%lx start=%lx num_mfn=%d type=%d\n",
                base, mask, start_mfn, (int)num_mfn, mem_type);
    }

And the output on Haswell:

     (XEN) [0] vmx.c:806 MTRRdefType=c00

    (XEN) [0] vmx.c:811 MTRRcap=d0a

     (XEN) [0] vmx.c:855 ===MTTR base=6 mask=7800000800 start=0 num_mfn=8388608 type=6

     (XEN) [0] vmx.c:855 ===MTTR base=800000006 mask=7fc0000800 start=800000 num_mfn=262144 type=6

     (XEN) [0] vmx.c:855 ===MTTR base=c0000000 mask=7fc0000800 start=c0000 num_mfn=262144 type=0

     (XEN) [0] vmx.c:855 ===MTTR base=bc000000 mask=7ffc000800 start=bc000 num_mfn=16384 type=0

     (XEN) [0] vmx.c:855 ===MTTR base=bb800000 mask=7fff800800 start=bb800 num_mfn=2048 type=0

     (XEN) [0] vmx.c:855 ===MTTR base=83f800000 mask=7fff800800 start=83f800 num_mfn=2048 type=0

     (XEN) [0] vmx.c:855 ===MTTR base=83f600000 mask=7fffe00800 start=83f600 num_mfn=512 type=0

Everything is ok here. I can get the correct value of MTRRcap register and read all the memory info from MTRR.

Output on Skylake:

     (XEN) [0] vmx.c:839 MTRRdefType=c06

     (XEN) [0] vmx.c:844 MTRRcap=1d0a // got 1 on bit 12

     (XEN) [0] vmx.c:892 ===MTTR base=c0000000 mask=7fc0000800 start=c0000 num_mfn=262144 type=0

     (XEN) [0] vmx.c:892 ===MTTR base=a0000000 mask=7fe0000800 start=a0000 num_mfn=131072 type=0

     (XEN) [0] vmx.c:892 ===MTTR base=90000000 mask=7ff0000800 start=90000 num_mfn=65536 type=0

Where I cannot find most of memory on Skylake, and all found memory is UC type.

Did the programming way of MTRRs change in Skylake? I can only guess in this way.

Thanks.

[McCalpinJohn]

I don't know any details on the use of bit 12 in IA32_MTRR_CAP, but it seems likely that it relates to a capability present in the Skylake processor that is not present in earlier processors.  Given the need for backward compatibility, it seems extremely unlikely that the use of previously reserved bits would change the interpretation of the previously-defined lower-order bits....

The reason you can't find most of the memory in the variable-range MTRRs on your Skylake system is that it is not there!    The Haswell system has an IA32_MTRR_DEF_TYPE of '00', which maps to Uncached (and is the "traditional" way to program the MTRRs), while the Skylake system has an IA32_MTRR_DEF_TYPE of '06', which maps to the WriteBack type.   So on the Skylake, any address range that is not mapped to an MTRR will be assumed to be a WriteBack type, and there is no need for the MTRRs to explicitly map system memory.  (See Section 11.11 of Volume 3 of the Intel Architectures SW Developer's Manual, document 325384).