Intel X550-T2: no ASPM after NVM update 3.60

[andrebrait]

I recently got a no-brand (AFAICT) X550-T2 which shipped with (according to the NVM update tool) version 1.00 of the NVM firmware.

It did support ASPM, initially, but after I updated it, support for ASPM seems to be gone.

The NVM update tool identified the card immediately. I didn't have to do anything, so I assumed the card would be properly supported.

The NVM update tool also doesn't allow me to rollback. It simply states "Rollback blocked".

I have another card here, identical to the one I flashed, and I cannot see any differences that might result in ASPM being disabled.

Let me know what sort of details you need and I can provide them (logs, screenshots, backed up firmware, etc.)

It would be good to have a heads-up warning that not all updates are reversible even though you might have a backup, or a way to force the rollback.

[B_Y]

Hello @Andre,

Thank you for posting in Intel Ethernet Communities.

We need your following help to check on the Intel® Ethernet Converged Network Adapter X550-T2.

- May I seek your help to share a snapshot of the front and back of the X550-T2 that is having the issue? This would also help us identify if you are using an OEM or retail version of Intel Ethernet Adapter

- Please advise the model of your motherboard and operating system (OS) build.

Thank you.

Best regards,

BY_Intel

Intel Customer Support 

[andrebrait]

I'm running pfSense 2.7.0 CE (based on FreeBSD 14) on an ASRock N100DC-ITX.

The other X550-T2 I have that's on firmware 1.0 still works just fine and it has ASPM. pciconf displays it and it says it's enabled at L1, IIRC.

The one that was flashed with version 3.60 not only can't enable ASPM, but it seems ASPM is missing altogether. pciconf doesn't even display it anymore. Not even "ASPM Disabled". It's simply not there anymore, and power consumption went up by a watt or two, so I think it is indeed not being applied.

[andrebrait]

Front of the cardBack of the card

[B_Y]

Hello @Andre,

Thank you for the information and the snapshots of the problematic X550-T2.

May I check following details:

1. Have you try power cycle your system after the NVM update (while the X550-T2 with the issue is still connected)?

2. After completion of a power cycle or restart, please reboot into bios to check the status of PCH PCIE ASPM Support, and share with us on the status?

The path as per guide line <https://www.asrock.com/support/faq.it.asp?id=528>: In BIOS > Advanced > Chipset Configuration > PCH PCIE ASPM Support.

I believe the states are:

Moderate Power Savings = Allow L0 state

Maximum Power Savings = Allow L1 state (deepest power saving, highest exit latency)

3. Do you notice any latency issue with network besides the power consumption went up?

4. May I know if you are using:

a) Intel Non-Volatile Memory (NVM) Update Utility and follow the Intel® Ethernet NVM Update Tool: Quick Usage Guide for FreeBSD*

The download link: https://www.intel.com/content/www/us/en/download/19358/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-x550-series.html

b) Latest Intel® Network Adapters Driver for PCIe* 10 Gigabit Network Connections Under FreeBSD*:

https://www.intel.com/content/www/us/en/download/14303/intel-network-adapters-driver-for-pcie-10-gigabit-network-connections-under-freebsd.html

5. If the issue still persisted, may need your help to share pciconf displays and System Support Utility (SSU) Log

of the problematic X550-T2 here.

Steps to get SSU Log: https://www.intel.com/content/www/us/en/download/18895/26735/intel-system-support-utility-for-the-linux-operating-system.html?

Thank you.

Best regards,

BY_Intel

Intel Customer Support 

[andrebrait]

---

@B_Y wrote:

Hello @Andre,

Thank you for the information and the snapshots of the problematic X550-T2.

May I check following details:

1. Have you try power cycle your system after the NVM update (while the X550-T2 with the issue is still connected)?

2. After completion of a power cycle or restart, please reboot into bios to check the status of PCH PCIE ASPM Support, and share with us on the status?

The path as per guide line <https://www.asrock.com/support/faq.it.asp?id=528>: In BIOS > Advanced > Chipset Configuration > PCH PCIE ASPM Support.

I believe the states are:

Moderate Power Savings = Allow L0 state

Maximum Power Savings = Allow L1 state (deepest power saving, highest exit latency)

3. Do you notice any latency issue with network besides the power consumption went up?

4. May I know if you are using:

a) Intel Non-Volatile Memory (NVM) Update Utility and follow the Intel® Ethernet NVM Update Tool: Quick Usage Guide for FreeBSD*

The download link: https://www.intel.com/content/www/us/en/download/19358/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-x550-series.html

b) Latest Intel® Network Adapters Driver for PCIe* 10 Gigabit Network Connections Under FreeBSD*:

https://www.intel.com/content/www/us/en/download/14303/intel-network-adapters-driver-for-pcie-10-gigabit-network-connections-under-freebsd.html

5. If the issue still persisted, may need your help to share pciconf displays and System Support Utility (SSU) Log

of the problematic X550-T2 here.

Steps to get SSU Log: https://www.intel.com/content/www/us/en/download/18895/26735/intel-system-support-utility-for-the-linux-operating-system.html?

Thank you.

Best regards,

BY_Intel

Intel Customer Support 

---

1. Yes, I power cycled (including the times I disassembled the machine to switch between the 1.00 and 3.60 cards)

2. All BIOS options were the same between the two cards. All options pertaining to ASPM are enabled (the options are only Disabled, L1 and Auto, and it was set to Auto for both cards (the affected card had ASPM on 1.00, but not on 3.60 with the same BIOS settings). See reference manual here: https://download.asrock.com/Manual/Software/Intel%20N100/Software_BIOS%20Setup%20Guide_English.pdf

3. No, I haven't noticed any added latency as far as I can tell, but I did notice increased CPU usage with regular TCP traffic (but I haven't tested it thoroughly). I can test that in a more controlled way later.

4. a) Yes, I used the NVM Update Package you linked, version 3.60, for FreeBSD.
b) I'm using whatever driver is bundled with FreeBSD 14.0-CURRENT as shipped in pfSense 2.7.0 CE, but I don't think the driver has much influence in this case?

5. I can't take the logs from SSU right now, but I do have the output from pciconf -lcv for both cards. Please find them attached. I also attached the result of diff pciconf.1.00.ASPM.txt pciconf.3.60.NO-ASPM.txt so the difference is more obvious. One note: note the complete absence of ASPM being mentioned for the affected card, whereas normally cards that do not support ASPM have ASPM disabled(L0s/L1). I attached a sample of pciconf output from the same machine when an QNAP QXG-2G2T-I225 is plugged, where you can see that, normally, you can find the corresponding ASPM mention even when the card does not support it. I also attached the resulting firmware backup from the NVM Update tool, in case that contains some information you might need about the original 1.00 firmware.

[B_Y]

Hello @Andre,

Thank you for the information.

We are checking on this issue and will get back as soon as possible. Thank you for your patience. 

Best regards,

BY_Intel

Intel Customer Support 

[andrebrait]

While we wait, is there any way I can rollback to the previous firmware?

Every time I try through the CLI, following the instructions on the manual, the rollback reports being blocked.

I know the NVM Update tool doesn't allow certain rollbacks due to critical fixes and whatnot, but the card without ASPM is worse for me due to it preventing the CPU Package power state from going below C3, causing it to use more power than normal. With ASPM (which worked in firmware 1.00, hence why I want to rollback) it goes all the way down to C8.

Is there any way I can force the rollback?

[B_Y]

Hello @Andre,

Thank you for your patience. We need your help for a dump file from the current version:

Use "modinfo [device] command to get the current driver version for the device.

Example:

# modifo bluetooth

or with below commend to dump information be a file

# modifo bluetooth >> bluetooth-info.

Note: The bluetooth is just the example naming of the device, it will be differ based on your configuration.

Secondly, the rollback procedure as below but please DO NOT roll back yet as we need your help on dump file as per instruction above:

Recovery Mode for Intel® Ethernet Products

https://www.intel.com/content/www/us/en/support/articles/000089896/ethernet-products.html

Please take note, downgrading firmware may not always be supported or recommended, and it may cause compatibility issues or unexpected behavior.

Steps to use the Recovery Devices mode in the NVM update tool:

1. Download the older firmware version that you want to install on your NIC. https://www.intel.com/content/www/us/en/download/19358/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-x550-series.html
2. Make sure your NIC is connected to your computer and powered on.
3. Open the command prompt as an administrator and navigate to the directory where the NVM update tool is located.
4. Type the following command to enter the Recovery Devices mode: nvmupdatew64e.exe -r
5. The tool will prompt you to enter the path to the recovery image file. Type the full path to the file and press Enter.
6. The tool will then start the recovery process. This may take a few minutes to complete. Once it is finished, the NIC should have the older firmware version installed..

If you encounter any issues with the above steps or if the Recovery Devices mode does not work, you can try using the Intel Ethernet Flash Firmware Utility (BootUtil). The BootUtil can be used to flash the NIC with a specific firmware version by following the instructions provided in the user guide.

Looking forward to your reply.

Thank you.

Best regards,

BY_Intel

Intel Customer Support 

[B_Y]

Hello @Andre ,

I hope this message finds you well. I am just sending a follow up if you have any questions from the previous post. If so, please let us know. Thank you.

Best regards,

BY_Intel

Intel Customer Support 

[andrebrait]

Hi there

I'm currently out of town, but once I'm back (in about 2 weeks) I'll get you the dump

[B_Y]

Hello @Andre ,

Greetings. Gentle reminder regarding your request, we are following up to find out if you were able to complete the actions we previously recommended.

Please reply to this post to confirm, so we can continue helping on a resolution. Looking forward to receiving your reply!

Best regards,

BY_Intel

Intel Customer Support 

[andrebrait]

Hi there,

No, I haven't been able to try them out yet. I'm still traveling. I'll be back this Tuesday, 5th of December. Then I may be able to try them out that week.

Please leave this open until I've gotten back to you.

[B_Y]

Hello @Andre ,

Greetings. Just a gentle reminder if you need additional assistance on this thread.

Best regards,

BY_Intel

Intel Customer Support

[B_Y]

Hello @Andre ,

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

Best regards,

BY_Intel

Intel Customer Support

[andrebrait]

@B_Y I managed to test it again and here's what I could find:

1. There is no modinfo on FreeBSD and, but as far as I can tell, here's the driver version (shouldn't matter, though, as that's what's up the most current FreeBSD upstream and I don't think the one from Intel's website can be installed on pfSense very easily)
2. Recovery Mode does not work to revert the NVM update. It does not allow the rollback, as stated before, and the instructions provided here don't work for that either.

# sysctl dev.ix.0 | grep -i version
dev.ix.0.fw_version: Option ROM V1-b3105-p0 eTrack 0x80001731 PHY FW V523
dev.ix.0.iflib.driver_version: 4.0.1-k

[Matt_s]

Where you able to solve the aspm issue on x550? I want to use it on opnsense box (freebsd). When i bought it, it came with firmware 2.0 no aspm, cpu state max c3 without card c10, i updated to firmware 3.6 still no aspm, c3 is max. Same thing on freebsd and linux

[andrebrait]

No, I ended up selling the cards.