topic Hi! in Software Tuning, Performance Optimization & Platform Monitoring

pcm tools - does pcm-memory.x show memory bandwidth to the core(s), or also to the PCIe devices?

Adrian_C_1 — Tue, 03 Sep 2013 19:09:50 GMT

Hi,

The PCM toolset incldues a command, pcm-memory.x, which gives a top-level overview of memory bandwidth read, write and total. What constitutes this? Does it include the DMA to/from PCIe devices? CPU core/LLC access? Or both? Anything else?

Thanks!

-adrian

Interesting question.My bet

Bernard — Thu, 19 Sep 2013 12:49:02 GMT

Interesting question.My bet is that this could be total memory usage.Can you look at source code, does it give you some clue?

Hi,

Adrian_C_1 — Thu, 19 Sep 2013 17:59:33 GMT

Hi,

There's only 64gig of RAM. It's FreeBSD-10 serving lots of video content. So there's lots and lots of kernel structures ((network) mbuf entries, (vm/cache) vm_page_t entries) churning through memory. Since they're spread all throughout memory at the moment, I'm seeing 10-15% of CPU cycles spent page table walking. It scales up we increase traffic.

When I started grouping these allocations to occur in a smaller region, I saw a noticable (5-7%) drop in CPU utilisation.

Hence why I'd like to drill down further to fully understand what's going on. In my scenario, the direct map is mostly 1GB entries but there are regions of memory that can't be mapped with a 1GB page, so there will be some 2MB and 4KB page table entries. I'd like to drill down to understand where the bulk of the current TLB walking is.

Thanks!

-a

Regarding video content it is

Bernard — Fri, 20 Sep 2013 08:02:46 GMT

Regarding video content it is probably stored as Ethernet frame prior to the sending by NIC driver to Ethernet card itself so unless jumbo frames are used I suppose that standard size will be 1492 bytes for one frame and from the perspective of DTLB and in order to reduce pressure of frequently walking 4kb pages I think that best option will be usage of larger pages.

Can you set maybe programmatically your networking stack to use only larger pages?

the memory bandwidth metrics

Roman_D_Intel — Fri, 20 Sep 2013 14:27:02 GMT

the memory bandwidth metrics in PCM include all traffic: DMA, PCIe, core, etc.

Roman

Hi!

Adrian_C_1 — Fri, 20 Sep 2013 14:27:28 GMT

Hi!

We're sending large frames to the hardware and using TSO offload in the TX path. So it's going to be larger frames anyway (up to 64k segments.)

Roman - thankyou. I'll go

Adrian_C_1 — Fri, 20 Sep 2013 14:30:45 GMT

Roman - thankyou. I'll go digging through the uncore documentation for Sandy Bridge Xeon to see if I can better understand how the counters work.

I'd like to try and differentiate memory bus transactions from memory<->PCIe (and PCIe<->LLC as part of DDIO) and memory<->core (for instruction/data fetches/data stores.) Is this possible?

Thanks!

-a

Thank you Roman.

Bernard — Fri, 20 Sep 2013 15:04:37 GMT

Thank you Roman.

Hi Adrian!

Bernard — Fri, 20 Sep 2013 15:06:27 GMT

Hi Adrian!

What do you mean by writing 64k segments?

Thanks in advance

The NIC supports something

Adrian_C_1 — Sun, 22 Sep 2013 04:35:06 GMT

The NIC supports something called TSO - transmit segment offload. We hand the NIC 64k of TCP data. THe NIC takes care of breaking it up into MSS chunks and sending it to the client.

-adrian

Thanks Adrian.

Bernard — Sun, 22 Sep 2013 07:43:42 GMT

Thanks Adrian.

Initially I thought that you mean 64k of segments.