I/O request timeouts on Linux with Intel P3520/P4600 NVMe PCIe SSDs

[SGoel5]

Hi,

We are experiencing persistent I/O request timeouts on Linux with P3520/P4600 SSDs. We have tried multiple different kernels (3.10, 4.4, 4.9) and see the timeouts on all of them. The P4600 seems to be more prone to these than the P3520 though we see them on the latter as well. We have the latest firmware installed on both drives which are housed in the same machine (Supermicro 5018R-WR with X10SRW-F motherboard and E5-1650 V4 CPU). We can reproduce the timeouts by simply running mkfs -t xfs on the drive.

Here is the output from isdct (version isdct-3.0.9.400-17.x86_64):

- Intel SSD DC P3520 Series CVPF717100L01P2JGN -

Bootloader : MB1B0105

DevicePath : /dev/nvme0n1

DeviceStatus : Healthy

Firmware : MDV10271

FirmwareUpdateAvailable : The selected Intel SSD contains current firmware as of this tool release.

Index : 0

ModelNumber : INTEL SSDPEDMX012T7

ProductFamily : Intel SSD DC P3520 Series

SerialNumber : CVPF717100L01P2JGN

- Intel SSD DC P4600 Series BTLE736007F54P0KGN -

Bootloader : 0110

DevicePath : /dev/nvme1n1

DeviceStatus : Healthy

Firmware : QDV10150

FirmwareUpdateAvailable : The selected Intel SSD contains current firmware as of this tool release.

Index : 1

ModelNumber : INTEL SSDPEDKE040T7

ProductFamily : Intel SSD DC P4600 Series

SerialNumber : BTLE736007F54P0KGN

Here are the messages the 4.9 kernel prints when using the P4600

[ 151.297903] nvme nvme1: I/O 568 QID 1 timeout, aborting

[ 151.303130] nvme nvme1: I/O 569 QID 1 timeout, aborting

[ 151.308347] nvme nvme1: I/O 570 QID 1 timeout, aborting

[ 151.313562] nvme nvme1: I/O 571 QID 1 timeout, aborting

[ 151.355465] nvme nvme1: completing aborted command with status: 0000

[ 151.411273] nvme nvme1: completing aborted command with status: 0000

[ 151.466903] nvme nvme1: completing aborted command with status: 0000

[ 151.522609] nvme nvme1: completing aborted command with status: 0000

[ 151.578226] nvme nvme1: completing aborted command with status: 0000

...

[ 165.395295] nvme nvme1: Abort status: 0x0

[ 165.399296] nvme nvme1: Abort status: 0x0

[ 165.403299] nvme nvme1: Abort status: 0x0

[ 165.407304] nvme nvme1: Abort status: 0x0

We would appreciate your help in resolving this issue.

Regards,

Shantanu Goel

[idata]

Hello Shantanu Goel,

Thank you for your interest in the Intel® SSD P3520 Series and the Intel® SSD P4600 Series.I understand that your system is experiencing persistent I/O request timeouts.Could you please tell me which is the specific Linux* OS distribution that you are using, and also provide a brief description of the intended use of the SSDs?Additionally, in order to provide the adequate assistance, please share the report generated by the Intel® System Support Utility for the Linux* Operating System ( https://downloadcenter.intel.com/download/26735/).I'll be waiting for your response.Regards,Andres V.

[SGoel5]

Hi Andres,

We are using RHEL 6 and create a filesystem on the SSD to store data.

Please see the attached SSU output from the system.

Thanks,

Shantanu

[idata]

Hello Shantanu,

Thank you for providing the requested file. We will analyze the provided data and get back to you via this community thread as soon as we have relevant information. Thank you for your patience. Regards,Andres V.

[idata]

Hello Shantanu,

In order to further understand the issue, your system, and the troubleshooting that you have performed, could you please answer the following questions?

• How many SSDs of each model are experiencing the timeout issue? How many SSDs do you have of each series?
• Have you tried connecting the SSDs to different slots? Have you been able to test the drives in another motherboard or system?
• Are you using any king of RAID controller?

Regarding the Intel® SSD DC P4600 Series, a new firmware version will tentatively be available within the next couple of weeks as part of the latest Intel® Solid State Drive Data Center Tool version, so please keep checking the download link https://downloadcenter.intel.com/download/27248?v=t https://downloadcenter.intel.com/download/27248?v=t, update your firmware and test again.

I'll be waiting for your response. Regards,Andres V.