topic Re: 2.8 Gbyte/sec limitation on RMS25CB080 in Intel® Xeon® Processor and Server Products

2.8 Gbyte/sec limitation on RMS25CB080

idata — Sat, 26 Sep 2015 15:25:09 GMT

3 series of tests on Intel R2216GZ4GCLX server show maximum read speed about 2.8Gbyte/sec, then theoretical maximum is 6Gbyte/sec (6Gbit/s per port, 8 ports)

Drives: 10 Intel SATA SSD 730 240Gb (520Mbyte/sec sequential read)

Mainboard and SAS controller has latest firmwares, PCIe Gen3 enabled in BIOS.

Testing sequential read with 256K blocks, 64 outstanding I/O (iometer, SQLIO), Win2012R2, LSI drivers 6.708.7.0, 10 drives with SAS Expander and 8 drives direct controller to backplane.

1 drive - 520Mbyte/s, 2 drives - 1025, 3 - 1535, 4 - 2040, 5 - 2550, 6 - 2800, add more drives do not increase transfer speed.

Any ideas about bottle-neck?

Re: 2.8 Gbyte/sec limitation on RMS25CB080

AP16 — Mon, 28 Sep 2015 19:59:33 GMT

First, while LSI 2208 chip advertised with PCI-E 3.0 capability, Intel states for http://ark.intel.com/products/60283/Intel-Integrated-RAID-Module-RMS25CB080 RMS25CB080 only Gen2 PCI Express lanes, resulting in 4 GB/s peak. Second, the diagram of LSI 2208 shows a 4.8 GB/s chips internal bandwidth limit for all of SAS ports. Then, take a look at official LSI benchmarking guide for better testbed setups http://docs.avagotech.com/docs/12353177 LSI MegaRAID Controller Benchmark Tips (1891 KB)

Re: 2.8 Gbyte/sec limitation on RMS25CB080

idata — Tue, 29 Sep 2015 14:11:38 GMT

Thanks, is very useful document, however it doesn't explain value 2.8Gbyte/s. Besides, I don't understand restriction 4.8Gb/s on 8 SAS ports (page 15). If each port transfers 6Gbit/s, 8*6Gbit has to give 6Gbyte, right?

As far as I know, my controlers already have support of PCIe Gen3.

From Intel RMS25CB080 Datasheet:

Features:

...

x8 PCI Express Generation 3

interface for fast communication with the server board

At launch, compatibility will be limited to PCIe Gen2. PCIe Gen3 support is anticipated to be added by August 2012 and will require a module with a manufacture date that follows the

addition of PCIe Gen3 support.

At launch, compatibility will be limited to PCIe Gen2. PCIe Gen3 support is anticipated to be added by August 2012 and will require a module with a manufacture date that follows the

addition of PCIe Gen3 support.

Re: 2.8 Gbyte/sec limitation on RMS25CB080

AP16 — Wed, 30 Sep 2015 17:37:30 GMT

If each port transfers 6Gbit/s, 8*6Gbit has to give 6Gbyte, right?

You forgot about signal encoding. 6Gbit/s is a raw PHY level rate, at ATA protocol level you have only 4/5 of it, thus only 4800 Gbit/s, 600 MB/s. So eight ports = 4.8 GB/s.

Practically, data transfers are even slower, about 4/5 part from ATA level. The magic numbers 520 MB/s in all Intel SATA SSD`s spec comes from that factor - its maximum really achievable speed.

Again, follow the LSI benchmark guide to reach more speed from controller.

Re: 2.8 Gbyte/sec limitation on RMS25CB080

idata — Wed, 30 Sep 2015 21:32:34 GMT

OK, I agree with restriction in 520Mbyte/s on the port (to a disk). I studied LSI Benchmark Tips and found out that my tests completely correspond to them. Settings of the controler: No Read Ahead, Write Through, Direct IO. Setting up test: 256Kb I/O size, 0% random, 100% read, 64 queue depth. Values 2.5GByte/s received on 5 disks and 2.8GByte/s on 6 disks shows that I use both lines from the controler to expander, differently I would receive about 2.4. However I still don't see the reason for a limit in 2.8Gbyte/s.

Re: 2.8 Gbyte/sec limitation on RMS25CB080

AP16 — Thu, 01 Oct 2015 19:19:07 GMT

Well, the only idea is to create a support ticket to Intel and let them deal with LSI.

Re: 2.8 Gbyte/sec limitation on RMS25CB080

idata — Fri, 02 Oct 2015 21:08:32 GMT

Hello,

We strongly recommend contacting our support group directly for further assistance on this inquiry:

http://www.intel.com/p/en_US/support/contactsupport http://www.intel.com/p/en_US/support/contactsupport