Is it possible keep my memory after reboot using Optane DC?

[hamj]

I'm trying a experiment that using kernel memory space as key-value storage with Optane DC. What I first thought is if I store key-value data in kernel memory space in Optane DC via system call I implemented, then I can load key-value from memory even after the reboot.

But, unfortunately, it doesn't work what i expected. I installed Optane DC following intel installation guide but every time I reboot, I can't load key-value that I inserted before reboot. It doesn't seem like keep memory data before reboot.

Is it possible what I thought using Optane DC?

If it is, is there specific setting for these kind of operation?

Following is my Optane DC installation commands: (root mode)

ipmctl create -goal

(reboot)

ndctl create-namespace

mkfs.ext4 /dev/pmem0

mkdir /dev/ext4-pmem0

mount -o dax /dev/pmem0 /dev/ext4-pmem0

(i tried "pmempool create /dev/ext4-pmem0/pool0" command follwing after last command following the installation guide, but it doesn't work)

[Emeth_O_Intel]

Hi,

Thank you for contacting Intel Optane DC Persistent Memory Community Support.

Could you please so kind and provide us the operational mode you set up on your system?

App Direct (AD) Mode, Memory Mode (MM) or Mixed Mode?

The App Direct Mode is the combination of DRAM (1LM) + App Direct (AD)

The Memory Mode is volatile with DRAM caching.

The Mixed Mode is the memory mode + The App Direct (AD)

I will be waiting for your answer in order to proceed with the next step.

Regards,

Emeth O.

Intel Customer Support Technician.

Under Contract to Intel Corporation.

[hamj]

Hello,

Thank you for your quick reply.

I set up Optane DC as AppDirectNotInterleaved mode which is default setup of "ipmctl create -goal"

so I created default goal(appdirect) then reboot.

After reboot, I create namespace, make file system(ext4) on /dev/pmem0 which is Optane DC I installed, and "mount -o dax /dev/pmem0 /dev/ext4-pmem0".

(/dev/ext4-pmem0 is directory I made(mkdir))

Following the installation guide, in AppDirect Mode, My memory should be sum of Optane and RAM I installed. In this case, I installed two 16GB DDR4 RAM, and 128 GB Optane DC so it should be approximately 150GB. But I can see only 32GB memory via "free" command.

Optane works fine in MemoryMode=100. I can check my memory using "free" command, and it show me 126GB free memory in system.

So.... is there anything I miss? or Optane doesn't work what i thought?

[Emeth_O_Intel]

Hello,

I was reviewing the information provided and I noticed you mentioned the Intel Optane DC Persistent Memory works fine as "MemoryMode=100" because it was showing you the total amount of Memory between the system memory and the Persistent Memory (As Memory).

I am wondering if perhaps, with the command "#Free" the Operating system is showing only the Memory of your system and for that reason when you configure the Intel DCPMM in AppDirect Mode is not showing you the total of "150GB".

Have you tried to verify the information using the following command: "#ipmctl show -memoryresources" with this command you will able to see how much system memory and persistent memory is present on your server.

As additional information; I would like to gather more details from your system; could you please so kind and share with me the following outputs:

A. ">ipmctl show -topology"

B. ">ipmctl show -firmware"

C. ">ipmctl show -dimm"

I will be waiting for your outcome in order to figure out the root cause of your issue.

Have a wonderful day,

Regards,

Emeth O.

Intel Customer Support Technician.

Under Contract to Intel Corporation.

[Emeth_O_Intel]

Hello,

By the way, could you please so kind and provide to me which option did you select on the BIOS Settings/Memory Configuration/Volatile Memory Mode?

Auto= Use on-DIMM Config

1LM=AppDirect

2LM=Memory Mode.

Please let me know the outcome in order to verify the information.

Regards,

Emeth O.

Intel Customer Support Technician.

Under Contract to Intel Corporation.

[hamj]

Hello! I apologize my late reply. I had to check my configuration and install all over again.

First, I solved the problem I mentioned. I modified fstab and now my OS mount /mnt/ext4-pmem0 on /dev/pmem0 automatically. I can make any file in it, and I can see them after reboot. So now, I can sure Optane DC is persistent perfectly.

But unfortunately, still it doesn't work what i expected. I can see only 32GB when i using "free" command. In BIOS(supermicro, Aptio Setup Utility), I can see there are total 162176MB Memory (32GB RAM + 128GB Optane DC).

And it is clearly AppDirect mode.

And these are additional info what you asked me

$ sudo ipmctl show -topology

 DimmID | MemoryType         | Capacity | PhysicalID| DeviceLocator

==============================================================================

 0x0020 | Logical Non-Volatile Device | 126.4 GiB | 0x0020  | P1-DIMMC1

 N/A  | DDR4            | 16.0 GiB | 0x001c  | P1-DIMMA1

 N/A  | DDR4            | 16.0 GiB | 0x001e  | P1-DIMMB1

$ sudo ipmctl show -firmware

 DimmID | ActiveFWVersion | StagedFWVersion

============================================

 0x0020 | 01.02.00.5355  | N/A

~$ sudo ipmctl show -dimm

 DimmID | Capacity | HealthState | ActionRequired | LockState | FWVersion

==============================================================================

 0x0020 | 126.4 GiB | Healthy   | 0       | Disabled | 01.02.00.5355

$ sudo ipmctl show -memoryresources

Capacity=126.4 GiB

MemoryCapacity=0.0 GiB

AppDirectCapacity=126.0 GiB

UnconfiguredCapacity=0.0 GiB

InaccessibleCapacity=0.4 GiB

ReservedCapacity=0.0 GiB

Is there something i missed? Thank you for your help.

[hamj]

ps) My final plan is put at least 1GB to kernel memory space, and check it still there after reboot.

[Emeth_O_Intel]

Hello, 

Thank you for letting us know the information. 

I was reviewing your post and I noticed you were able to find the solution by editing the fstab and mounting the operating system in /mnt/ext4-pmem0 on /dev/pmem0. It is great that you could find the correct process. 

Could you please so kind and provide to me the outcome of the following command in order to verify the namespace created: 

"#ndctl check -namespace" 

On the other hand, when you enter the command "#free" and you only see 32GB is because the output will show you only the memory of the system, not the persistent memory itself. 

For instance, let see the outputs below: 

$ sudo ipmctl show -memoryresources

Capacity=126.4 GiB

MemoryCapacity=0.0 GiB

AppDirectCapacity=126.0 GiB

UnconfiguredCapacity=0.0 GiB

InaccessibleCapacity=0.4 GiB

ReservedCapacity=0.0 GiB

This output is showing us that persistent memory has been configured as 100% AppDirect and 0% Memory Mode.

$ sudo ipmctl show -topology

 DimmID | MemoryType     | Capacity | PhysicalID| DeviceLocator

==============================================================================

 0x0020 | Logical Non-Volatile Device | 126.4 GiB | 0x0020 | P1-DIMMC1

 N/A | DDR4      | 16.0 GiB | 0x001c | P1-DIMMA1

 N/A | DDR4      | 16.0 GiB | 0x001e | P1-DIMMB1

Now on this output, we can see the total amount of the DCPMM that it equals to 126.4 GiB and as a DDR4 there are 16.0 GiB x2.

About your plan to configure 1GB to Kernel Memory Space, let me verify this information and do some researches about it because the kernel is found in an elevated system state, which includes protected memory space and full access to the device’s hardware.

As soon as possible I will get back to you with the information of the next step.

Regards,

Emeth O.

Intel Customer Support Technician.

Under Contract to Intel Corporation.

[hamj]

Hello!

Before I write the information you asked, I have to check something what you said.

What is the meaning "mounting the operating system in /mnt/ext4-pmem0 on /dev/pmem0"? Following is my fstab

# /etc/fstab: static file system information.

#

# Use 'blkid' to print the universally unique identifier for a

# device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).

#

# <file system> <mount point>  <type> <options>    <dump> <pass>

# / was on /dev/sda1 during installation

UUID=7f5488e7-80db-4c87-b545-d518a8f54c70 /        ext4  discard,errors=remount-ro 0    1

10.1.1.1:/share/    /share     nfs       auto,nofail,soft,timeo=120   0    0

/dev/disk/by-id/pmem-8439d9a5-d99c-4587-ac1c-910a31754a67 /mnt/ext4-pmem0 ext4 defaults,dax   1    2

As you can see, my original storage is /dev/sda1. So my working files are in /dev/sda1 I guess (and it mounted on "/")

I think optane is working like additional storage for now. Is this what you mean or something wrong in here?

==============================================================================================

~$ sudo ndctl list -N

[

 {

  "dev":"namespace0.0",

  "mode":"fsdax",

  "map":"mem",

  "size":135289372672,

  "uuid":"8439d9a5-d99c-4587-ac1c-910a31754a67",

  "blockdev":"pmem0"

 }

]

When I reboot after I created goal for AppDirect, I created namespace0.0. It take size 128GB automatically (same as Optane DC)

$ sudo ndctl check-namespace namespace0.0 -v

namespace0.0: namespace_check: checking namespace0.0

namespace0.0: namespace_check: namespace0.0: check aborted, namespace online

error checking namespaces: Device or resource busy

checked 0 namespaces

And this is what you asked. For sure, I add -v option.

Thank you

[Emeth_O_Intel]

Hi,

Thank you for the information provided.

I will double-check this information and do some research about the information you are asking.

However, could you please so kind and provide to me with more details about the project you are trying to accomplish with this type of configuration in order to have a better understanding of it.

Regards,

Emeth O.

Intel Customer Support Technician.

Under Contract to Intel Corporation.

[hamj]

What I want is simply check my Optane DC working what I expected.

Following the explanation of "https://itpeernetwork.intel.com/intel-optane-dc-persistent-memory-operating-modes/",

For example, a platform with 1.536 TB of Intel® Optane™ DC Persistent Memory and 192 GB of DRAM would register with the OS as 1.728 TB of total memory in App Direct, but only appear as 1.536 TB in Memory Mode.

My system has two 16GB DRAM and 128GB Optane DC Persistent Memory.

Following this explanation, My system must show me it has 160 GB of total memory(approximately) in App Direct ( 16 + 16 + 128), and 128GB in Memory Mode.

I checked it works exactly what I want in Memory Mode. My system (Debian 9.0 + linux 5.0.3) showed me there is 126GB memory and I checked it.

But in App Direct Mode, My system tells me only 32GB is in(16 + 16). Look below logs.

$ sudo free -h

         total  used  free  shared buff/cache available

Mem:   31Gi  3.2Gi  27Gi  9.0Mi  336Mi  27Gi

Swap:   0B    0B     0B

top - 21:08:06 up 2 days, 6:21, 2 users, load average: 0.00, 0.00, 0.00

Tasks: 249 total, 1 running, 248 sleeping, 0 stopped, 0 zombie

%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

MiB Mem : 31817.8 total, 28235.3 free, 3244.4 used, 338.1 buff/cache

MiB Swap:  0.0 total,  0.0 free,  0.0 used. 28177.4 avail Mem

$ sudo vmstat -s

  32581384 K total memory

  3322104 K used memory

  145024 K active memory

  174100 K inactive memory

  2813084 K free memory

  20736 K buffer memory

  325460 K swap cache

   0 K total swap

   0 K used swap

   0 K free swap

   444 non-nice user cpu ticks

   31 nice user cpu ticks

  19048 system cpu ticks

 391423993 idle cpu ticks

   151 IO-wait cpu ticks

   0 IRQ cpu ticks

   266 softirq cpu ticks

   0 stolen cpu ticks

  283802 pages paged in

  29704 pages paged out

   0 pages swapped in

   0 pages swapped out

  4238669 interrupts

  6268488 CPU context switches

 1567057595 boot time

   1677 forks

I tried to access physical address of Optane DC using devmem2 and mmap() but both can't access because of permission denied. (It worked fine on DRAM)

I tried put more than 32GB meaning less data to memory programatically because there is a little possibility that Optane DC working fine, just not showing itself as memory to OS, but my system failed when it took more data than its capacity (32GB).

I followed every instruction from intel, and I tried everything to using Persistent Memory.

But in this situation, Memory mode is just big volatile memory and App Direct Mode is just 128GB additional Storage, not Memory.

I want using it as "Persistent Memory" not a storage.

[hamj]

I create a simple program that save data on memory.

Every data points next data using pointers. So I thought it can be possible implement a kind of in-memory Database.

What I have to do is simply allocate root address directly. If Optane DC working perfect, then I can access to in-memory Database simply access the root address I allocated.

But, I have difficulty to check that makes Optane DC working as persistent memory. In Memory mode, it is just a memory, In Add Direct mode, it is just a stroage (for now)

[Emeth_O_Intel]

Hi,

Sorry for the delay, I was doing some research about it.

I would like to gather a clarification about the kernel memory space, Is this to boot from DCPMM or just to store OS there?

Generally, a cheap storage device would be used for OS. One can boot from DCPMM instead of a storage device. But, this is not recommended.

Moreover, I have some links for you that may help you; please check the information below:

https://software.intel.com/en-us/videos/provisioning-intel-optane-dc-persistent-memory-modules-in-linux

https://software.intel.com/en-us/articles/quick-start-guide-configure-intel-optane-dc-persistent-memory-on-linux

https://www.intel.com/content/www/us/en/products/docs/memory-storage/optane-persistent-memory/optane-dc-persistent-memory-brief.html

Have a wonderful day.

Regards,

Emeth O.

Intel Customer Support Technician

A Contingent Worker at Intel

[Emeth_O_Intel · Accepted solution]

Hi,

I was doing some research and I found the following website that can help you to have a better understanding about Persistent Memory Programming.

I hope this information helps to have better understanding about this type of coding, I would like to remark that this is not recommended. Moreover, you can test it but we are not responsible for any hardware damaged. 

Have a wonderful day.

Regards,

Emeth O.

Intel Customer Support Technician

A Contingent Worker at Intel

[Emeth_O_Intel]

Hi,

I was reviewing this thread and I have not seen any activity recently from your end.

If you have any other questions, do not hesitate and contact us back and we will be more than happy to assist you.

Emeth O.

Intel Customer Support Technician

A Contingent Worker at Intel

[hamj]

Hi!

First, I'm sorry for my late reply.

Thanks for your help, now I figure out what am I can do.

Now, I know I can get the starting address of optane DC from kernel, drivers/nvdimm/pmem.c.

And I check I can access that virt_addr so I can write and read directly from Optane!

Thank you so much

[Emeth_O_Intel]

Hi,

I am very glad to see you could find the proper configuration of this specific project you have in mind.

Please if you have any other questions do not hesitate and let us know and I will be more than happy to assist you.

Have a wonderful day.

Regards,

Emeth O.

Intel Customer Support Technician

A Contingent Worker at Intel