I want to use DSA, but can't set CONFIG_INTEL_IOMMU_DEFAULT_ON

[HaoqiLu]

My CPU is Intel(R) Xeon(R) Gold 6438N, and I want to use Intel® Data Streaming Accelerator (DSA).

I set my configuration by User Guide. One of the steps requires setting  CONFIG_INTEL_IOMMU_DEFAULT_ON or the CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON.

I use command "cat /boot/config-$(uname -r) | grep CONFIG_INTEL_IOMMU ", which shows:

 so I checked the information and did the following operations：

1. sudo vim /etc/default/grub

and I added intel_iommu=on,sm_on into the end of line GRUB_CMDLINE_LINUX_DEFAULT

2. sudo update-grub

3. now I use command:  cat /proc/cmdline

it shows:
BOOT_IMAGE=/boot/vmlinuz-6.5.0-27-generic root=UUID=4fa8d2bb-7f5b-4d02-90d6-1a40cfed0e84 ro quiet splash intel_iommu=on,sm_on vt.handoff=7

4. I reboot the machine
Now the problem is: the result of cat /boot/config-$(uname -r) | grep CONFIG_INTEL_IOMMU doesn't change, it's still the same.
What can I do to set the CONFIG_INTEL_IOMMU_DEFAULT_ON and CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON equal y as the User Guide said.

I hope to get a reply as soon as possible, I will be grateful !

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

Thank you for your quick response. Please allow us some time to review the details and we will get back with an update shortly.

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi OwenG05,

Greetings for the day!

Thank you for sharing the information. We've checked with our internal team. As mentioned, Ubuntu 22.04 may require loading the IDXD drivers onto the Kernel.

Ubuntu 22.04 uses Kernel version 5.15, and the IDXD driver was introduced around Kernel version 6.2, if I'm not mistaken. Please refer to the OS vendor documentation for confirmation.

Please also check the BIOS options to enable IDXD with the board manufacturer.

Regarding the IDXD repo, please check the following links:

GitHub - intel/idxd-config: Accel-config / libaccel-config

Release accel-config: release 4.1.6 · intel/idxd-config · GitHub

The output below clearly indicates that the DSA is present on the system. However, due to driver issues (IDXD), it is not possible to turn on SVA and continue with the DSA configuration 

❯ sudo dmesg | grep idxd

[ 817.158224] idxd 0000:75:01.0: Unable to turn on user SVA feature.

[ 817.165738] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)

[ 817.165936] idxd 0000:f2:01.0: Unable to turn on user SVA feature.

[ 817.168988] idxd 0000:f2:01.0: Intel(R) Accelerator Device (v100)

Best Regards,

Azeem_Intel

[HaoqiLu]

Hi Azeem_Intel,

Greetings for the day!

It was too late yesterday and I went to rest. I didn't reply in time, sorry.
Based on your reply, do I just need to go to that link  to download and install accel-config and the project will help me install IDXD?

I have now successfully installed the idxd-config project according to the project introduction, what should I do next?
After I restarted the machine, I found that the output was still：

❯ sudo dmesg | grep idxd
[ 68.776021] idxd 0000:75:01.0: Unable to turn on user SVA feature.
[ 68.787714] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)
[ 68.787880] idxd 0000:f2:01.0: Unable to turn on user SVA feature.
[ 68.789908] idxd 0000:f2:01.0: Intel(R) Accelerator Device (v100)

Looking forward to your reply!

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

Thank you for the update. We are currently checking with our internal team. Please allow us some time, and we will get back to you shortly.

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

Hope you are doing well. We're currently investigating this issue internally. It seems to be related to the OS configuration in the BIOS, so we're putting our best effort into resolving it. It's important to note that Intel doesn't provide support for the OS, so we recommend you reaching out to the OS vendor for further assistance. Thank you for your understanding."

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

We appreciate your participation. Upon reviewing the kernel logs, it appears that all DSA is available, but the feature cannot be enabled due to OS configuration issues related to IDXD and SVA. Could you please confirm that the following options are enabled in the BIOS of the system?

Best Regards,

Azeem_Intel

[HaoqiLu]

Hi Azeem_Intel,

Greetings for the day!

I checked these BIOS settings. After updating the new version of bios, I confirmed that these settings are turned on.

The result of the command is:

❯ sudo dmesg | grep idxd
[ 825.345963] idxd 0000:75:01.0: Unable to turn on user SVA feature.
[ 825.369938] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)
[ 825.370060] idxd 0000:f2:01.0: Unable to turn on user SVA feature.
[ 825.393042] idxd 0000:f2:01.0: Intel(R) Accelerator Device (v100)

What else could be the cause of this problem?

Here is the picture:

 

 

  

[Pintu]

Hi Lu Haoqi,

Greetings for the Day!

As per the issue, with enabling the User SVA feature for the Intel Accelerator Device, start by reviewing the BIOS settings to ensure that all relevant options, especially those related to Shared Virtual Addressing (SVA), are properly configured. Next, ensure that you have the latest drivers installed for the Intel Accelerator Device by visiting the Intel website or your system manufacturer's support page. Additionally, confirm hardware compatibility by consulting documentation provided by Intel or your system manufacturer to verify support for the User SVA feature. Check for any resource conflicts or limitations that may be preventing the feature from being enabled, and examine system logs and diagnostic tools for relevant errors or warnings. Lastly, consider updating the firmware of the Intel Accelerator Device, as firmware updates can often address compatibility issues and improve functionality

Please don't hesitate to ask any further questions in the future. 

Thank you for using Intel products and services.

Regards,

Manoranjan.

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

My apologies for the delay. Please allow us some time to review the details, and we will get back to you with an update shortly.

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

Working internally, the error is pointing to something related to the BIOS and/or OS compatibility.

We can see the hardware DSA is assigned on the system per the logs:

```

May 8 22:59:43 node13 kernel: [  68.212591] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)

```

Can you please share the following information:

1. Have you tried running the default kernel "5.15" to see if the issue reproduces?

2. Please share the output of the command: `lspci -vvvv -s 0x75:1.0`

3. Please check page 38 of the Doc#:759709 Intel® Data Streaming Accelerator User Guide for steps to enable DSAA on your project for troubleshooting.

4. As this CPU is Tray, can you confirm us if the CPU was repurposed from another system?

Best regards,

Azeem_Intel

[HaoqiLu]

Hi Azeem_Intel,

Sorry it took so long to reply。

1. I currently have a small number of machines using this CPU, and all of them have tasks to run, so it is not convenient to reinstall the system. In addition, my system kernel version is 6.5.0-35-generic. Do you think it is a problem with the updated system?

2. the result is:
❯ sudo lspci -vvvv -s 0x75:1.0
75:01.0 System peripheral: Intel Corporation Device 0b25
Subsystem: Device 1f24:6000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
NUMA node: 0
IOMMU group: 1
Region 0: Memory at d6ffff20000 (64-bit, prefetchable) [size=64K]
Region 2: Memory at d6ffff00000 (64-bit, prefetchable) [size=128K]
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ RBE+ FLReset+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+ LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
Capabilities: [80] MSI-X: Enable+ Count=9 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP+ FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [150 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [160 v1] Transaction Processing Hints
Device specific mode supported
Steering table in TPH capability structure
Capabilities: [170 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=1
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=fd
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=1 ArbSelect=Fixed TC/VC=02
Status: NegoPending- InProgress-
Capabilities: [200 v1] Designated Vendor-Specific: Vendor=8086 ID=0005 Rev=0 Len=24 <?>
Capabilities: [220 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [230 v1] Process Address Space ID (PASID)
PASIDCap: Exec- Priv+, Max PASID Width: 14
PASIDCtl: Enable+ Exec- Priv+
Capabilities: [240 v1] Page Request Interface (PRI)
PRICtl: Enable- Reset-
PRISta: RF- UPRGI- Stopped+
Page Request Capacity: 00000200, Page Request Allocation: 00000200
Kernel driver in use: idxd
Kernel modules: idxd

3. as typing the command：
❯ sudo dmesg -t | grep ACPI | grep DMAR
ACPI: DMAR 0x00000000773E9000 000390 (v01 XF EGLSTRM 00000001 ACPI 00040000)
ACPI: Reserving DMAR table memory at [mem 0x773e9000-0x773e938f]

❯ sudo dmesg | grep -i dmar
[ 0.010658] ACPI: DMAR 0x00000000773E9000 000390 (v01 XF EGLSTRM 00000001 ACPI 00040000)
[ 0.010712] ACPI: Reserving DMAR table memory at [mem 0x773e9000-0x773e938f]
[ 0.963825] DMAR: IOMMU enabled
[ 0.963826] DMAR: Enable scalable mode if hardware supports
[ 2.015163] DMAR: Host address width 46
[ 2.015165] DMAR: DRHD base: 0x000000dc7fc000 flags: 0x0
[ 2.015172] DMAR: dmar0: reg_base_addr dc7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015175] DMAR: DRHD base: 0x000000e43fc000 flags: 0x0
[ 2.015180] DMAR: dmar1: reg_base_addr e43fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015182] DMAR: DRHD base: 0x000000ebffc000 flags: 0x0
[ 2.015186] DMAR: dmar2: reg_base_addr ebffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015188] DMAR: DRHD base: 0x000000f37fc000 flags: 0x0
[ 2.015199] DMAR: dmar3: reg_base_addr f37fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015201] DMAR: DRHD base: 0x000000faffc000 flags: 0x0
[ 2.015205] DMAR: dmar4: reg_base_addr faffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015207] DMAR: DRHD base: 0x000000d4bfc000 flags: 0x0
[ 2.015212] DMAR: dmar5: reg_base_addr d4bfc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015214] DMAR: DRHD base: 0x000000fb7fc000 flags: 0x0
[ 2.015219] DMAR: dmar6: reg_base_addr fb7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
[ 2.015220] DMAR: DRHD base: 0x000000a73fc000 flags: 0x0
[ 2.015224] DMAR: dmar7: reg_base_addr a73fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015226] DMAR: DRHD base: 0x000000b07fc000 flags: 0x0
[ 2.015230] DMAR: dmar8: reg_base_addr b07fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015232] DMAR: DRHD base: 0x000000b9bfc000 flags: 0x0
[ 2.015236] DMAR: dmar9: reg_base_addr b9bfc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015238] DMAR: DRHD base: 0x000000c2ffc000 flags: 0x0
[ 2.015242] DMAR: dmar10: reg_base_addr c2ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015244] DMAR: DRHD base: 0x000000cbffc000 flags: 0x0
[ 2.015248] DMAR: dmar11: reg_base_addr cbffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015250] DMAR: DRHD base: 0x000000cc7fc000 flags: 0x0
[ 2.015254] DMAR: dmar12: reg_base_addr cc7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
[ 2.015256] DMAR: DRHD base: 0x0000009dffc000 flags: 0x1
[ 2.015259] DMAR: dmar13: reg_base_addr 9dffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 2.015261] DMAR: RMRR base: 0x0000006e3e0000 end: 0x0000006e3e2fff
[ 2.015263] DMAR: ATSR flags: 0x0
[ 2.015265] DMAR: RHSA base: 0x0000009dffc000 proximity domain: 0x0
[ 2.015267] DMAR: RHSA base: 0x000000a73fc000 proximity domain: 0x0
[ 2.015268] DMAR: RHSA base: 0x000000b07fc000 proximity domain: 0x0
[ 2.015269] DMAR: RHSA base: 0x000000b9bfc000 proximity domain: 0x0
[ 2.015270] DMAR: RHSA base: 0x000000c2ffc000 proximity domain: 0x0
[ 2.015271] DMAR: RHSA base: 0x000000cbffc000 proximity domain: 0x0
[ 2.015272] DMAR: RHSA base: 0x000000cc7fc000 proximity domain: 0x0
[ 2.015273] DMAR: RHSA base: 0x000000dc7fc000 proximity domain: 0x1
[ 2.015274] DMAR: RHSA base: 0x000000e43fc000 proximity domain: 0x1
[ 2.015276] DMAR: RHSA base: 0x000000ebffc000 proximity domain: 0x1
[ 2.015277] DMAR: RHSA base: 0x000000f37fc000 proximity domain: 0x1
[ 2.015278] DMAR: RHSA base: 0x000000faffc000 proximity domain: 0x1
[ 2.015278] DMAR: RHSA base: 0x000000d4bfc000 proximity domain: 0x1
[ 2.015279] DMAR: RHSA base: 0x000000fb7fc000 proximity domain: 0x1
[ 2.015280] DMAR: SATC flags: 0x0
[ 2.015284] DMAR-IR: IOAPIC id 8 under DRHD base 0x9dffc000 IOMMU 13
[ 2.015285] DMAR-IR: HPET id 0 under DRHD base 0x9dffc000
[ 2.015287] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 2.018736] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 3.886313] DMAR: IOMMU feature pasid inconsistent
[ 3.886315] DMAR: IOMMU feature prs inconsistent
[ 3.886316] DMAR: IOMMU feature pasid inconsistent
[ 3.886317] DMAR: IOMMU feature prs inconsistent
[ 3.886318] DMAR: IOMMU feature pasid inconsistent
[ 3.886319] DMAR: IOMMU feature prs inconsistent
[ 3.886320] DMAR: IOMMU feature pasid inconsistent
[ 3.886320] DMAR: IOMMU feature prs inconsistent
[ 3.886321] DMAR: IOMMU feature pasid inconsistent
[ 3.886322] DMAR: IOMMU feature prs inconsistent
[ 3.886323] DMAR: IOMMU feature pasid inconsistent
[ 3.886324] DMAR: IOMMU feature prs inconsistent
[ 3.886324] DMAR: IOMMU feature pasid inconsistent
[ 3.886325] DMAR: IOMMU feature prs inconsistent
[ 3.886326] DMAR: IOMMU feature pasid inconsistent
[ 3.886326] DMAR: IOMMU feature prs inconsistent
[ 3.886327] DMAR: IOMMU feature pasid inconsistent
[ 3.886328] DMAR: IOMMU feature prs inconsistent
[ 3.886328] DMAR: IOMMU feature pasid inconsistent
[ 3.886329] DMAR: IOMMU feature prs inconsistent
[ 3.886330] DMAR: IOMMU feature pasid inconsistent
[ 3.886330] DMAR: IOMMU feature prs inconsistent
[ 3.886331] DMAR: IOMMU feature pasid inconsistent
[ 3.886332] DMAR: IOMMU feature prs inconsistent
[ 3.886333] DMAR: dmar12: Using Queued invalidation
[ 3.886340] dmar12 SVM disabled, incompatible paging mode
[ 3.886345] DMAR: dmar11: Using Queued invalidation
[ 3.886347] DMAR: dmar10: Using Queued invalidation
[ 3.886349] DMAR: dmar9: Using Queued invalidation
[ 3.886351] DMAR: dmar8: Using Queued invalidation
[ 3.886358] DMAR: dmar7: Using Queued invalidation
[ 3.886360] DMAR: dmar6: Using Queued invalidation
[ 3.886362] dmar6 SVM disabled, incompatible paging mode
[ 3.886364] DMAR: dmar5: Using Queued invalidation
[ 3.886367] DMAR: dmar4: Using Queued invalidation
[ 3.886372] DMAR: dmar3: Using Queued invalidation
[ 3.886374] DMAR: dmar2: Using Queued invalidation
[ 3.886376] DMAR: dmar1: Using Queued invalidation
[ 3.886377] DMAR: dmar0: Using Queued invalidation
[ 3.886382] DMAR: dmar13: Using Queued invalidation
[ 4.104308] DMAR: Intel(R) Virtualization Technology for Directed I/O

❯ sudo dmesg | grep -i idxd
[ 825.345963] idxd 0000:75:01.0: Unable to turn on user SVA feature.
[ 825.369938] idxd 0000:75:01.0: Intel(R) Accelerator Device (v100)
[ 825.370060] idxd 0000:f2:01.0: Unable to turn on user SVA feature.
[ 825.393042] idxd 0000:f2:01.0: Intel(R) Accelerator Device (v100)

Just to be clear, I have already set “intel_iommu=on,sm_on”.

4. No, these CPUs are purchased from suppliers.

Thank you for your support and look forward to your reply！

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

This is the first follow-up regarding the issue (I want to use DSA, but can't set CONFIG_INTEL_IOMMU_DEFAULT_ON) you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

This is the second follow-up regarding the reported issue. We're eager to ensure a swift resolution and would appreciate any updates or additional information you can provide.

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi HaoqiLu,

Greetings for the day!

This is the third follow-up regarding the reported issue. We're committed to ensuring a swift resolution and would greatly appreciate any updates or additional information you can provide.

If we don't hear back from you soon, we'll assume the issue has been resolved and will proceed to close the case

Best Regards,

Azeem_Intel