topic DPDK: Set hugepage affinity for specific NUMA node? in Embedded Intel® Core™ Processors

DPDK: Set hugepage affinity for specific NUMA node?

MCoop2 — Thu, 20 Nov 2014 00:22:26 GMT

Hello,

I am looking for clarification as to whether it is possible to assign a specific number of 1G hugepages to a dedicated NUMA node. Our hardware platform has 2 NUMA nodes however our DPDK application runs only on the cores of a single NUMA node. We are finding we need to double our hugepage allocation given the kernel is uniformly distributing the hugepages across the 2 NUMA nodes. We have modified the kernel boot line in grub.conf as follows:

"default_hugepagesz=1G hugepagesz=1G hugepages=8"

My question is similar to that posed in the following link, however I do not see in the response how to change nr_hugepages.

/message/8787?_ga=1.189585131.1372410714.1393984191# 8787 https://embedded.communities.intel.com/message/8787?_ga=1.189585131.1372410714.1393984191# 8787

Any attempts at modifying /sys//devices/system/node/node[0|1]/hugepages/hugepages-1048576kB/nr_hugepages has been unsuccessful.

[root@box]# echo 0 > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages

bash: echo: write error: Invalid argument

Any insight as to whether this is possible would be greatly appreciated,

-Marc

Re: DPDK: Set hugepage affinity for specific NUMA node?

Gabriel_T_Intel — Thu, 20 Nov 2014 00:24:57 GMT

Hello Mark

Welcome to the Embedded Community. We are going to work in this request and will get back to you soon.

Best Regards.

Gabriel.

Re: DPDK: Set hugepage affinity for specific NUMA node?

Muthurajan_J_Intel — Thu, 20 Nov 2014 23:31:18 GMT

Chapter 6 of http://dpdk.org/doc/intel/dpdk-start-linux-1.7.0.pdf http://dpdk.org/doc/intel/dpdk-start-linux-1.7.0.pdf has Quick Start Setup Script

Can you use 2 Meg instead of 1 G?

Because you will see at the end of 2.3.2.1 note saying with 1 G pages, it is not possible to reserve the hugepage memory after the system has booted.

And Note in section 2.3.2.1 indicates about in general equal reservation - can you see Documentation/Kernel-parameters.txt file in your Linux source tree for further details of these - "In the case of a dual-socket NUMA system, the number of hugepages reserved at boot time is generally divided equally between the two sockets (on the assumption that sufficient memory is present on both sockets).

However, in case you can use 2 Meg page size, in that case, the script can be useful to meet your requirement.

In the script, when you choose the option for "Setup hugepage mappings for NUMA systems" , it will ask you

Input the number of 2 MB pages for each node

then you will be able to reserve separately for node 0 and separately for node 1

Re: DPDK: Set hugepage affinity for specific NUMA node?

MCoop2 — Fri, 21 Nov 2014 02:02:38 GMT

I'm familiar with that section. Our application runs on a 64-bit platform and as such we have configured for 1G hugepages as is recommended.

-Marc

Re: DPDK: Set hugepage affinity for specific NUMA node?

Muthurajan_J_Intel — Fri, 21 Nov 2014 18:43:04 GMT

Thanks. With 64 bit system, one can use either 2 Meg or 1 Gig.

In case your performance requirement if it can be met with 2 Meg pages, then with 2 Meg pages you can achieve what you are looking for.

Because with 1 Gig, Because you will see at the end of 2.3.2.1 note saying with 1 G pages, it is not possible to reserve the hugepage memory after the system has booted.

Thanks