82599ES second port flaping on Proxmox 7/Debian 11

[stuckoff]

Hi

I have two servers with  dual SFP+ 82599ES Controllers. The first port looks that links fine to the switch, as the second one flaps all the time (I already changed the cables and SFPs - it is not caused by them)

[ 3552.245953] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3553.434783] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3554.486002] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3556.381015] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3556.726064] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3563.166018] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3565.014225] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3567.469847] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3569.270372] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3572.450224] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3573.526467] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3573.910976] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3575.766523] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3576.423882] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3578.006599] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3578.557056] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Down
[ 3580.246624] ixgbe 0000:18:00.1 enp24s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

I tried with 2 drivers: 5.12.5 and 5.13.4 - no difference. Here are some details for the adapter:

root@pve-02:~# ethtool -i enp24s0f1
driver: ixgbe
version: 5.13.4
firmware-version: 0x800006da, 1.1824.0
root@pve-02:~# uname -a
Linux pve-02 5.11.22-5-pve #1 SMP PVE 5.11.22-10 (Tue, 28 Sep 2021 08:15:41 +0200) x86_64 GNU/Linux  

this is the log from the switch:

Oct 26 2021 00:30:22+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[0]:Interface XGigabitEthernet1/0/1 has turned into DOWN state.
Oct 26 2021 00:30:13+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[1]:Interface XGigabitEthernet1/0/1 has turned into UP state.
Oct 26 2021 00:30:11+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[2]:Interface XGigabitEthernet1/0/1 has turned into DOWN state.
Oct 26 2021 00:30:03+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[3]:Interface XGigabitEthernet1/0/1 has turned into UP state.
Oct 26 2021 00:30:01+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[4]:Interface XGigabitEthernet1/0/1 has turned into DOWN state.
Oct 26 2021 00:29:50+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[5]:Interface XGigabitEthernet1/0/1 has turned into UP state.
Oct 26 2021 00:29:43+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[6]:Interface XGigabitEthernet1/0/1 has turned into DOWN state.
Oct 26 2021 00:29:31+03:00 DST cpbs-hw-sw-01 %%01IFPDT/4/IF_STATE(l)[7]:Interface XGigabitEthernet1/0/1 has turned into UP state.

Could you please help me how to identify the root cause of this issue.

Regards

Stoyan

[Zigfreid_Intel]

Hello stuckoff,

Thank you for posting on Intel Ethernet Communities. 

We are now reviewing your case however we would also like to clarify if you are using a discreet Network card for your set up? If you are, can we ask for the part number of your adapter for us to validate. Please refer to the link below. 

https://www.intel.com/content/www/us/en/support/articles/000007022/ethernet-products/500-series-network-adapters-up-to-10gbe.html

If you have questions or other clarifications please let us know so we can help you further

Awaiting to hear from you soon, should there be no response from you, I’ll make sure to reach out after 3 business days. Thank you and have a nice day!

Best regards,

Zigfreid I.

Intel® Customer Support

[Zigfreid_Intel]

Hello stuckoff,

Good day! We hope that you are having a great one so far.

We'd like to follow up the requested information for us to proceed checking your query. Feel free to let us know if you have additional questions or clarifications on this request. 

Hoping to hear from you soon.

Should there be no response from you, I’ll make sure to reach out after 3 business days

Best regards,

Zigfreid I.

Intel® Customer Support

[Caguicla_Intel]

Hello stuckoff,

How are you doing?

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

May you have a great rest of the week!

Best regards,

Crisselle C.

Intel Customer Support

[stuckoff · Accepted solution]

Hi Team,

the issue was fixed by changing the SFP+ modules.

Thank you for your support

Regards

stuckoff