[REGRESSION] Serious SAS slowdown after BIOS upgrade

VBerk1 — Thu, 12 Sep 2013 20:17:18 GMT

Hi,

We have a server and a storage system, as follows:

Intel R2304LH2HKC server with an http://www.intel.com/content/www/us/en/motherboards/server-motherboards/server-board-s4600lh-s4600lt.html S4600LH2 mobo (http://www.intel.com/p/en_US/support/highlights/server/ss-r2000lh2lt2 R2000LH2 series)
http://www.lsi.com/products/storagecomponents/Pages/MegaRAIDSAS9286-8e.aspx LSI MegaRAID SAS 9286-8e
http://ark.intel.com/products/74545/Intel-Storage-System-JBOD2224S2DP Intel JBOD2224S2DP (JBOD 2000 series)
2pcs http://www.ebay.co.uk/itm/DELL-FOXCONN-SAS-4X-8-PAIRS-30AWG-CABLE-E124936-D-AWN-20276-80c-30v-LOT-OF-2-/181197759785?clk_rvr_id=520105540782 Foxconn SFF-8088 SAS cable

We've managed to get out some 2800MBps from this setup but had serious issues with thermal management, it easily went over 100 Celsius degrees under heavy load, so I decided to upgrade the BIOS (i.e. BIOS, ME, FRU/SDR, BMC, everything) to the https://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3594&DwnldID=23008&ProductFamily=Server+Products&ProductLine=Intel%C2%AE+Server+Boards&ProductProduct=Intel%C2%AE+Server+Board+S4600LH+Family&lang=eng latest release. During the FRU/SDR upgrade it asked me several questions regarding if I want to enter a serial number and stuff, I answered all of those with no (if that matters in any way).

The good news is that since the upgrade thermal management seems to have been sorted out, now it greatly increases fan speed when it goes above 90C and cools down to 70-80ish, then the fans slow down and it starts over.

The bad news is that now I can't get out more than about 2100MBps from the storage. Many BIOS settings have been modified after the upgrade, so a related setting (maybe something with PCI-E) might be the cause. I'm yet to find it.

Anyway, we've performed several benchmarks before the upgrade, too. It produced similar speeds when I plugged in the 2 SAS cables into different converter boards in the JBOD (to test HA). So it may be that for some reason the RAID controller only speaks on one cable now. I have no idea how that could happen after a server mobo BIOS upgrade, but I can't rule it out either.

The other possible cause is, again, some bad PCI-E setting.

In any case, I'll try it out with the other JBOD (fortunately we have 2), with different cables, and in a different PCI-E slot.

A last resort option is a BIOS reset, but I'd prefer not to do that.

The system has 4x8 cores, so with HT it's 64 cores in total (0% typical CPU load, even during benchmarks) and 384GB RAM, so these aren't quite likely to be bottlenecks

Do you have any ideas what else may went wrong?

Re: [REGRESSION] Serious SAS slowdown after BIOS upgrade

VBerk1 — Fri, 13 Sep 2013 11:00:02 GMT

Oh guess what, it's proved to be a BIOS regression. After like a day of struggling with other things, finally (and fearfully) I downgraded to the https://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=22568&lang=eng penultimate release as a last resort, and now BOTH thermal management AND storage throughput is fine OOB.

Where can I report this bug to Intel? I don't want to mark this as an answer just yet, coz the problem's not solved, just worked around.

Re: [REGRESSION] Serious SAS slowdown after BIOS upgrade

VBerk1 — Fri, 13 Sep 2013 11:33:26 GMT

NVM, I've contacted Intel via the http://www.intel.com/support/feedback.htm official contact page.

Re: [REGRESSION] Serious SAS slowdown after BIOS upgrade

VBerk1 — Mon, 23 Sep 2013 20:43:40 GMT

It's a http://www.intel.com/support/motherboards/server/sb/CS-033827.htm known bug. Until a fix is released, I'm advised to stick with v7.

topic Re: [REGRESSION] Serious SAS slowdown after BIOS upgrade in Intel® Xeon® Processor and Server Products

[REGRESSION] Serious SAS slowdown after BIOS upgrade

Re: [REGRESSION] Serious SAS slowdown after BIOS upgrade

Re: [REGRESSION] Serious SAS slowdown after BIOS upgrade

Re: [REGRESSION] Serious SAS slowdown after BIOS upgrade