Server Products
Data Center Products including boards, integrated systems, Intel® Xeon® Processors, RAID Storage, and Intel® Xeon® Processors
4829 Discussions

I want to use DSA, but can't set CONFIG_INTEL_IOMMU_DEFAULT_ON

HaoqiLu
Beginner
2,737 Views

My CPU is Intel(R) Xeon(R) Gold 6438N, and I want to use Intel® Data Streaming Accelerator (DSA).

I set my configuration by User Guide. One of the steps requires setting  CONFIG_INTEL_IOMMU_DEFAULT_ON or the CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON.

I use command "cat /boot/config-$(uname -r) | grep CONFIG_INTEL_IOMMU ", which shows:

 

1.png

 so I checked the information and did the following operations:

1. sudo vim /etc/default/grub

and I added intel_iommu=on,sm_on into the end of line GRUB_CMDLINE_LINUX_DEFAULT

2. sudo update-grub

3. now I use command:  cat /proc/cmdline

it shows:
BOOT_IMAGE=/boot/vmlinuz-6.5.0-27-generic root=UUID=4fa8d2bb-7f5b-4d02-90d6-1a40cfed0e84 ro quiet splash intel_iommu=on,sm_on vt.handoff=7

4. I reboot the machine
Now the problem is: the result of cat /boot/config-$(uname -r) | grep CONFIG_INTEL_IOMMU doesn't change, it's still the same.
What can I do to set the CONFIG_INTEL_IOMMU_DEFAULT_ON and CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON equal y as the User Guide said.

I hope to get a reply as soon as possible, I will be grateful !

Labels (1)
0 Kudos
40 Replies
HaoqiLu
Beginner
1,776 Views

help! Anyone knows how to solve this problem? Or how to get relevant Intel employees to notice this post?

0 Kudos
ACarmona_Intel
Moderator
1,751 Views

Hello HaoqiLu,


Thank you for posting in our communities.


We have a forum category for specific issues related to your product, which is the Xeon Processor. This category is monitored by our experts, who can provide you with the best solution. To help you get your answer more quickly, we will now move your post to that category.


Thank you, and have a great day ahead!


Best regards,

Carmona A.

Intel Customer Support Technician


0 Kudos
Azeem_Intel
Employee
1,687 Views

Hi HaoqiLu,



Greetings for the day!



Hope you're well. Could you please specify the Linux distribution you're running? Also, is this running on a virtual machine? Additionally, could you share the output of the command: dmesg | grep idxd?


You can find information about checking the Intel® Data Streaming Accelerator on the OS in the https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html?DocID=759709 Please check and let us know.




Best Regards,


Azeem_Intel



0 Kudos
HaoqiLu
Beginner
1,663 Views

Thanks for your reply. Here are the information:

❯ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
❯ sudo dmesg | grep idxd
[  817.158224] idxd 0000:75:01.0: Unable to turn on user SVA feature.
[  817.165738] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)
[  817.165936] idxd 0000:f2:01.0: Unable to turn on user SVA feature.
[  817.168988] idxd 0000:f2:01.0: Intel(R) Accelerator Device (v100)

I am configuring it on my physical machine now, but I also hope to be able to use DSA in containers in k8s in the future.

Looking forward to your next reply!

0 Kudos
Dineshbabu
Employee
1,626 Views

Hi HaoqiLu,


Greetings!


We don't have enough information about working DSA with Kubernetes. As per the user guide, it can be used in the Linux kernel.


Regards,

Dinesh


0 Kudos
HaoqiLu
Beginner
1,619 Views

OK, sir. But my problem now is that I can't use it in host, just with the error show before.

I followed the user manual, but something unexpected happened.

How can I solve this?

0 Kudos
Dineshbabu
Employee
1,606 Views

Hi HaouqiLu,


Greetings!


Please allow us some time to review the details and we will get back with an update shortly.


Regards,

Dinesh


0 Kudos
HaoqiLu
Beginner
1,605 Views

Thank you very much! Looking forward to your reply!

0 Kudos
Irwan_Intel
Moderator
1,467 Views

Hi HaoqiLu,


Since you are using Ubuntu 22.04.3 LTS Release: 22.04. I would like to know if you have checked the compatibility of the OS with the Intel DSA accelerator. Please refer to page 13 and Table 2-1 of this link https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html?DocID=759709 

The DSA is compatible with SUSE Linux Enterprise Server SLES 15 SP4, please let us know if you have further questions.


Regards,

Irwan_Intel


0 Kudos
HaoqiLu
Beginner
1,457 Views

Sorry sir, did you mean that only SUSE Linux Enterprise Server SLES 15 SP4 has complete IDXD driver and i can't use DSA on Ubuntu 22.04.3 LTS? Is there any other way to instasll this driver on Ubuntu or other support? After all this is a very popular linux distribution.

0 Kudos
Sachinks
Employee
1,410 Views

Hello HaoqiLu,


Thank you for your reply!


May I ask you to confirm Which Linux Distribution are you running?

Is it a virtual machine?


Also May I ask you to run command : Run dmesg | grep idxd  

and share the output with us?



You can also check the DSA on the OS using the steps given in this link : https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html?DocID=759709


Regards,

Sachin KS


0 Kudos
Azeem_Intel
Employee
1,337 Views

Hi HaoqiLu,



Greetings for the day!


This is the first follow-up regarding the issue (I want to use DSA, but can't set CONFIG_INTEL_IOMMU_DEFAULT_ON) you reported to us.


We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided.



Best Regards,


Azeem_Intel


0 Kudos
HaoqiLu
Beginner
1,300 Views

Hello sir,

I am not sure what the action plan you are referring to is. I have responded to all the above replies one by one and given the required information, such as: 

i'm using Ubuntu 22.04.3 LTS, and the result of sudo dmesg | grep idxd is:

 [ 817.158224] idxd 0000:75:01.0: Unable to turn on user SVA feature.

[ 817.165738] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)

[ 817.165936] idxd 0000:f2:01.0: Unable to turn on user SVA feature.

[ 817.168988] idxd 0000:f2:01.0: Intel(R) Accelerator Device (v100)

you said the DSA is compatible with SUSE Linux Enterprise Server SLES 15 SP4. However, I cannot change the system due to limited conditions. My current problem is how to use DSA on my Ubuntu22.04 system.

And according to the documentation on Intel’s official website:

Technology Guide | Intel® Data Streaming Accelerator (DSA) - Calico VPP with Intel® DSA on 5th Gen Intel® Xeon® Scalable

Processor

It also uses Ubuntu22.04 system. I want to know why I can’t use it.

0 Kudos
Azeem_Intel
Employee
1,283 Views

Hi HaoqiLu,



Greetings for the day!


Thank you for your response. Kindly grant us some time as we need to check with our internal team. We will provide you with a resolution as soon as possible.



Best Regards,


Azeem_Intel


0 Kudos
Azeem_Intel
Employee
1,206 Views

Hi HaoqiLu,



Greetings for the day!


1 – There is a misunderstanding on the compatibility of DSA Accelerator and Operational Systems. On page 13 and Table 2-1 of this link https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html?DocID=759709, mentions: “Distribution kernel versions with complete IDXD driver support are shown in Table 2-1.” Which shows: “SUSE Linux Enterprise Server 15 SP4”.

That means the only Kernel including all IDXD driver support is “SLES 15 SP4”. Other OSs (like Ubuntu 22.04 and others) can be used with DSA, however, the Kernel needs to be updated including IDXD drivers.

For this enablement on Ubuntu 22.04, please check with the OS vendor. We do not provide on Operational systems.

 

Is good to highlight also that Intel® Xeon® Gold 6438N is a 4th Generation of Xeon and per Operating System Compatibility for Xeon the compatibility for SUE is only “SLES 15 SP4”.

Again, that does not mean that the feature will not work, but the kernel maybe does not have the full drivers. For not included Drivers on the OS, needs to contact OS vendor.

 

2 – The mention “Technology Guide | Intel® Data Streaming Accelerator (DSA) - Calico VPP with Intel® DSA on 5th Gen Intel® Xeon® Scalable Processor It also uses Ubuntu22.04 system.” This is valid for 5th Gen Xeon (not 4th Gen Xeon) as the customer is using in this case.

 

3 – The error the you reporting is: “0000:f2:01.0: Unable to turn on user SVA feature”. This error could be related to missing drivers parameters on the OS.

You needs to analyze the information from DSA and Linux commands together and look for inconsistencies or missing elements related to VT-d and PRICtl functionalities.

 Kernel Boot Parameters: 

 


  1. Run grep -i vtd /etc/default/grub and grep -i vtx /etc/default/grub
  2.  
  3. Look for options like intel_vtd.active=off or intel_vtd_cmt_mode=off that might disable VT-d functionalities.

 

VT-d Status: 

  1. Run grep -i vtd /proc/cpuinfo and grep -i vtx /proc/cpuinfo
  2.  
  3. Look for entries like "VT-d: active" or "VT-d: disabled" to confirm VT-d activation.

 

Driver Status: 

  1. Run dmesg | grep -i prictl or journalctl | grep -i prictl
  2.  
  3. Look for any error messages or information about PRICtl driver loading or initialization issues. If VT-d is disabled, investigate the cause through BIOS settings or kernel boot parameters.
  4.  

 

BIOS

On the BIOS settings, please make sure the following options are correct:

enabled "Opt-Out Illegal MSI Mitigation", enabled "PRS Capability for PCIe" and disabled "Limit CPU PA to 46 bits".

 

Note: Please also consult with the system or motherboard documentation for specific BIOS settings related to VT-d and VT-x and make sure no additional config is needed on BIOS level.

 

On the Intel® Data Streaming Accelerator User Guide Page 15, there is a description about checking the DSA on the OS:

 


  1. Run lspci -vvv -s 6a:01.0

 

The output should be like the following:

  1. Capabilities: [220 v1] Address Translation Service (ATS)
  2. ATSCap: Invalidate Queue Depth: 00
  3. ATSCtl: Enable+, Smallest Translation Unit: 00

 


  1. Run dmesg | grep -i idxd
  2.  
  3. If you see “Unable to turn on SVA feature”, VT-d scalable mode may not be enabled by default, reboot with “intel_iommu=on,sm_on” added to the kernel command line to enable

 

VT-d scalable mode.

  1. Run dmesg | grep idxd
  2.  
  3. Received: Unable to turn on SVA

 


  1. Add this grub config command in /etc/default/grub file:
  2.  
  3. GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on,sm_on iommu=on"
  4.  
  5. Run update-grub
  6.  
  7. Reboot the system.
  8.  

 

If the commands above do not show the issue, please share OS Logs, and below Logs

 

  1. Authorization log. Location: /var/log/auth.log.
  2. Daemon Log. Location: /var/log/daemon.log.
  3. Debug log. Location: /var/log/debug.
  4. Kernel log. Location: /var/log/kern.log.
  5. System log. Location: /var/log/syslog.

I hope this helps.


Best Regards,


Azeem_Intel



0 Kudos
HaoqiLu
Beginner
1,178 Views

Hello Azeem_intel:

Thank you very much for your thoughtful reply!

First of all I want to apologize, the example I gave is indeed about 5th Gen Xeon (not 4th Gen Xeon) . But there are still many examples using 4th Gen Xeon. Such as:

"Intel® Data Streaming Accelerator (DSA)DPDK-DMA Packet Copying with IntelDSA on 4th Gen Intel® Xeon® ScalableProcessors"

"Intel® Data Streaming Accelerator (DSA) –Offloading Memory Copying in VPP by Shared Memory Packet Interface (memif) Plugin with Intel® DSA on 4th Gen Intel® Xeon® Scalable Processors"

"Intel® Data Streaming Accelerator (Intel® DSA) - Calico VPP Multinet with Intel® DSA on 4th Gen Intel® Xeon® Scalable Processor"

According to the System Configurations, the first two used Ubuntu-Server 22.04 and the third one used Ubuntu 22.04.1 LTS.

So I think Ubuntu 22.04 can also use DSA smoothly.

You said I needs to contact OS vendor. But I don't know how to do that, because I am using a free distribution on the Internet, I don’t know how to get technical support.

I think the problem is that I don't have idxd driver as needed.

I found intel/idxd-driver on github, but it said this project will no longer be maintained by Intel.

So I want to ask if this  project can be used to install the driver.

And tou give me many command to check the problem, here is the result:

1- Run grep -i vtd /etc/default/grub and grep -i vtx /etc/default/grub

2- Run grep -i vtd /proc/cpuinfo and grep -i vtx /proc/cpuinfo

3- Run dmesg | grep -i prictl or journalctl | grep -i prictl

The result of these three commands is that there is no reply. That is, there is no explicit command indicating that VT-d is disabled.

4- Run lspci -vvv -s 6a:01.0

There is no reply, too.

5- Run dmesg | grep -i idxd

it shows:

[ 817.158224] idxd 0000:75:01.0: Unable to turn on user SVA feature.
[ 817.165738] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)
[ 817.165936] idxd 0000:f2:01.0: Unable to turn on user SVA feature.
[ 817.168988] idxd 0000:f2:01.0: Intel(R) Accelerator Device (v100)

I reboot with “intel_iommu=on,sm_on” added to the kernel command line to enable, the reuslt of "cat /etc/default/grub" is:

 

❯ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on,sm_on iommu=on"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

 

 

So the method(update /etc/default/grub file and reboot) doesn't solve the issue.

In general, I still want to ask about how to complete DSA configuration on Ubuntu22.04 system, including information about updating the idxd driver.
I have basically tried the above commands, but they seem to have no results.
It is difficult to filter out the relevant information from the log files you provided earlier. From my general inspection, I feel that there is no content related to idxd.

Thank you again for your help, and I hope you can continue to follow up, thank you very much!

 

 

Best Regards.

0 Kudos
Pintu
Employee
1,150 Views

Hi HaoqiLu,


Greeting for the Day!


Please allow us some time to review the details and we will get back with an update shortly.


Regards,

Manoranjan Das.


0 Kudos
Azeem_Intel
Employee
895 Views

Hi HaoqiLu,



Greetings for the day!


Thank you for providing the update. While we review the information, could you please share the OS logs with us? This will help us proceed further with resolving the issue.

 

  1. Authorization log. Location: /var/log/auth.log.
  2. Daemon Log. Location: /var/log/daemon.log.
  3. Debug log. Location: /var/log/debug.
  4. Kernel log. Location: /var/log/kern.log.
  5. System log. Location: /var/log/syslog.




Best Regards,

Azeem_Intel


0 Kudos
Dineshbabu
Employee
835 Views

Hello HaoqiLu,


Greetings!


We are checking in to see if you were able to find the information we requested. Could you please let us know, so we can proceed with finding a resolution? We look forward to hearing back from you.


Best regards,

Dinesh


0 Kudos
HaoqiLu
Beginner
817 Views

Surely!

I found there are no 

  1. Daemon Log. Location: /var/log/daemon.log.
  2. Debug log. Location: /var/log/debug.

And I copy other logs you need in /var/log,hope that can help you to find a resolution!

https://drive.google.com/drive/folders/1fxP3gzv0VB9cnK7A62E1icTKSh69oI86?usp=drive_link 

The above are some logs I found that you may need. Note that because I am also working on this machine (using k8s), there may be irrelevant information interference, so please try your best to filter it out.

0 Kudos
Reply