Embedded Server
Consolidate Considerations of Intel® Xeon and Atom server Hardware, Firmware, Software, and Tools
281 Discussions

PCH selectable ports (HSIO SATA/PCIe/USB3) issue on Xeon D-1500

MStar1
Beginner
2,838 Views

Hi!

I would like to report an issue I've been observed with selectable PCH ports on a Xeon D-1500 SoC.

 

The Xeon-D documentation states the SoC supports four configurable PCH ports:

  1. 2 x SATA or 2 x PCIe (Ports SATA 4/5, HSIO# 13/14), controlled via GPIO16/GPIO49
  2. 2 x PCIe or 2 x USB3 (Ports PCIe 1/2, HSIO# 5/6), controlled via GPIO71/70.

There are two ways to switch the interfaces - strictly assign them via FITc tool (PCH Straps 4 & 9) or set as "GPIO dependent" via FITc and then change the associated GPIO input level.

 

The problem was found with SATA/PCIe ports - they are both physically connected to miniPCIe/mSATA slots, and the active interface is always SATA - the mSATA SSD is always recognized by the BIOS, while it's impossible to detect the miniPCIe module.

Tried to set the PCH Strap4 bits [3:2] = 01b and PCH Strap9 bits [31:30] = 01b to strictly assign the interfaces - it doesn't work.

Also tried to set the strap bits to 11b (GPIO dependent) and them set the GPIO16/49 levels to GND - it doesn't work too.

 

At the same time, another switchable pair (PCIe/USB3 at PCIe 1/2 ports) works properly - it can be set strictly to PCIe, or to USB3, or set to "GPIO dependent" via GPIO71/70 and then the associated GPIO really changes the port operation mode.

 

I know the fact that PCIe port should be enabled only on one of the two interfaces at the same time to avoid conflicts - so I set PCIe at PCIe/SATA muxed port and strictly selected USB3 on PCIe/USB3 muxed port, there's no error here.

 

Does anyone have the succesful experience with PCIe interfaces on physical SATA 4/5 ports?

 

Best regards, Mikhail.

0 Kudos
10 Replies
CarlosAM_INTEL
Moderator
2,551 Views

Hello, @MStar1​:

 

Thank you for contacting Intel Embedded Community.

 

Could you please clarify if the affected project is your design or a third- party one? 

 

In case that it is a third-party device, could you please inform the name of the manufacturer, its model, the part number, and where its documentation is stated?

 

On the other hand, could you please let us know how many units of the project related to this circumstance have been manufactured? How many are affected? Could you please give the failure rate? Also, could you please list the sources that you have used to design it and if it has been verified by Intel?

 

Finally, could you please give pictures of the top side markings of the affected processors? By the way, please provide the part number of the processors associated to this situation.

 

We are waiting for your answer to these questions.

 

Best regards,

@Mæcenas_INTEL​.

0 Kudos
MStar1
Beginner
2,551 Views

Hi @Mæcenas_INTEL​ 

 

Thanks for your response!

 

Look, this design is our's, but I would like to say, it's not the only one.

 

In fact, this "muxed port" Xeon D-1500 feature has been known since the early processor documentation, and all our designs since 2016 have been done keeping this feature in mind.

Trying to count them: one microTCA Processor board, two COM Express Processor boards, two VPX 3U Processor boards and three VPX 6U Processor boards.

I guess it's about three hundreds of boards have been totaly produced.

 

The first design, where we really needed USB3 on PCIe lanes was microTCA Processor board in 2016, and everthing was okay there (I already mentioned that this muxed pair works clearly as described in EDS vol.3 and FITc).

Since that, all the other boards required primarily SATA interfaces on SATA4/5 muxed ports, so we simply didn't check their "muxable" nature and used them at their default settings (SATA 4/5), and it worked too.

But now we came up to the design with two miniPCIe/mSATA slots in VPX form-factor and suddenly failed trying to use both interfaces on the muxed ports SATA4/PCIe1 and SATA5/PCIe2.

 

Regarding the affected CPUs: nearly all of our designs were assembled with Xeon D-1527 V2 or Xeon D-1548 V2 SoCs, so I have been discovering this issue on the Xeon D-1548 board with the following SoC numbers:

-----------------------

Xeon D-1548 SR2DJ

J750B295

78LN657100745

-----------------------

 

I have two similar boards with Xeon D-1527 and Xeon D-1567 SoCs, I can replicate the issue here and report back.

 

Unfortunately, we don't have the CRB board for Xeon D platform (as there were some difficulties to get it quickly in 2016, so we had to bring up the first microTCA design from scratch using only the Xeon D documentation, Beverly Cove CRB schematics, and it's pre-built BIOS that was available at IBL), so I can't reproduce the issue myself on CRB, but I can tell how to do that.

 

All that you need is:

  1. Take the Xeon-D CRB (or another Xeon D-based board),
  2. Connect one or two SATA drives to ports SATA4 and/or SATA5,
  3. Set the SATA4/PCIe1 and SATA5/PCIe2 muxed port setting (FITc PCH strap9 bits [31:30] and strap4 bits [3:2]) both to 00b (SATA mode),
  4. Verify that one or both SATA drives are detected by the system.
  5. After that, change SATA4/PCIe1 and SATA5/PCIe2 muxed port setting both to 01b (PCIe mode), and PCIe/USB3 muxed port settings (strap9 bits [21:20] and [19:18] both to 01b (USB3 mode)),
  6. Leave the SATA drives connected and check if they have disappeared from the system (SATA/PCIe switched to PCIe mode, OK) or not (SATA/PCIe stuck in SATA mode, the issue).

 

Thanks and best regards,

Mikhail.

0 Kudos
CarlosAM_INTEL
Moderator
2,551 Views

Hello, @MStar1​:

 

Thanks for your update.

 

Could you please confirm the workarounds for the errata PCH9, PCH10, PCH11, PCH14, PCH15, PCH22, PCH23, and PCH24 have been implemented in the affected design? The errata details are stated on pages 57 through 61 of the Intel® Xeon® Processor D-1500 Product Family Processor and Intel(R) Xeon(R) Processor D-1500 NS Product Family NDA Specification Update document # 557970.

 

Could you please verify that the affected design achieves the guidelines stated in sections 7, 8, 10, and 12; on pages 159 through 185, from 227 to 233, and from 249 to 268 of the Grangeville with Intel(R) Xeon(R) Processor D 1500 Product Family Platform Design Guide [PDG] document # 543448?

 

These documents can be found when you are logged into your Resource & Design Center (RDC) privileged account at the following websites:

 

http://www.intel.com/cd/edesign/library/asmo-na/eng/557970.htm

http://www.intel.com/cd/edesign/library/asmo-na/eng/543448.htm

 

The RDC Account Support form is the channel to process your account update request or any inconvenience related to the provided websites. It can be found at:

 

https://www.intel.com/content/www/us/en/forms/design/contact-support.html

 

We are waiting for your answer to these questions.

 

Best regards,

@Mæcenas_INTEL​.

0 Kudos
MStar1
Beginner
2,551 Views

Hi @Mæcenas_INTEL​.

 

Thanks for the information.

 

Regarding the errata PCH9, PCH10, PCH11, PCH24: they describe protocol-specific things rather than physical layer-related, and I think it's not applicable to me issue. Moreover, I confirm the USB3 is working properly on a USB3/PCIe muxable pair, so no issues regarding USB3 have been observed and noticed.

 

Regarding the errata PCH14, PCH23: it's not related to me issue too, as I don't have any questions to PCIe 1/2 port operation when they work through the PCIe/USB3 muxable signal pair - and I confirm, PCIe works properly.

 

Regarding the errata PCH15, PCH22: it's applicable to the SATA operation mode, but when SATA/PCIe muxable pair work in SATA mode (and they work in SATA mode all of the time!), there're no issues to the connected SATA drives, they are detected properly and the data transfer speeds are as expected (according to the SSD specs), so no question here.

 

What about the design guidelines provided in PDG, the Xeon D is not our first design, we've done many successful designs since 2006, on Core 2 Duo/Quad/Core i7 2nd Gen/Atom E6xx/Atom E38xx, and finally, on Xeon D-1500 platforms. I'm familiar with the high speed layout requirements, so, please, trust me, there's no problem there.

 

As I promised I've performed some more testing, using another board (but very similar, with exactly the same muxable port layout as the first one) equipped with the Xeon D-1567 12-core SKU.

The SoC marking is below:

------------------------

Xeon D-1567 SR2M3

J621C063

76UG243603351

------------------------

I could replicate the issue and the board behavior is absolutely the same - both SATA/PCIe muxed ports always stay in SATA mode (and work fine with mSATA SSDs), while PCIe/USB3 muxed pairs switch to the desired mode and detect connected PCIe devices or USB3 pen-drives respectively.

 

Here's a complete set of PCH straps that are used in Description sections of both our BIOSes (the same on 8-Core and 12-Core SKUs).

=================

Strap0 = 0030D7C2h,

Strap1 = 050001FFh,

Strap2 = 91000000h,

Strap3 = 2D000000h,

Strap4 = FFC8E106h,

Strap5 = 00h,

Strap6 = 00h,

Strap7 = 00h,

Strap8 = 00h,

Strap9 = 40550F80h,

Strap10 = 00007000h,

Strap11 = 2E000090h,

Strap12 = 00h,

Strap13 = 00h,

Strap14 = 00h,

Strap15 = 01084A76h,

Strap16 = 00h,

Strap17 = 00000002h,

Strap18 = 00h,

Strap19 = 00000004h,

Strap20 = 00h.

==================

I compared them to the 556198 Xeon D-1500 SPI Programming Guide and doesn't see if there's something wrong here.

 

And here's another thing I would like to share:

There's a register at offset 410h in B0:D28:F0 PCI space that is somehow related to the muxed port operation.

Unfortunately, this register is not clearly described in any of available docs (EDS, BWG), but the bits that are checked in different cases make me think the register is quite important.

We've performed the register reading with different muxed port settings via PCH Straps 4 and 9, and output the register value at the early BIOS stages to the UART debug log.

Here what we got:

  1. Strap4 = FFC8E106h (SATA5/PCIe2 in PCIe mode)

Strap9 = 40550F80h (SATA4/PCIe1 in PCIe mode, both PCIe/USB3 in USB3 mode)

B0:D28:F0:410h = 00001010b

 

2. Strap4 = FFC8E102h (SATA5/PCIe2 in SATA mode)

Strap9 = 00550F80h (SATA4/PCIe1 in SATA mode, both PCIe/USB3 in USB3 mode)

B0:D28:F0:410h = 00001010b

 

3. Strap4 = FFC8E102h (SATA5/PCIe2 in SATA mode)

Strap9 = 00410F80h (SATA4/PCIe1 in SATA mode, both PCIe/USB3 in PCIe mode)

B0:D28:F0:410h = 00000101b

The bits [1:0] and [3:2] are responsible for PCIe/USB3 operation mode, and I confirm they change along with the physical operation mode change. It's OK there.

The bits [5:4] are expected to change when SATA/PCIe ports change their operation mode, but this doesn't happen, the resigter bits stay at 00b showing the SATA mode is always enabled, and my interface tests show the same, only mSATA devices are detected, while miniPCIe are not.

 

Do you have any more ideas regarding the issue, or maybe, what to check/to test?

 

Thanks and best regards,

Mikhail.

0 Kudos
MStar1
Beginner
2,551 Views

Hi @Mæcenas_INTEL​ ,

 

Let me ask, is there any update regarding the issue?

 

Thanks and best regards,

Mikhail.

0 Kudos
CarlosAM_INTEL
Moderator
2,551 Views

Hello, @MStar1​:

 

Thanks for your updates.

 

We want to address the following questions:

 

1.      Which BIOS are you using? In Customer Reference Board (CRB) BIOS, it will change soft strap setting during POST according to CRB. It is used for supporting multiple CRBs with different soft strap. So... You have to make sure your BIOS does not change soft strap first. Get your BIOS serial log and see if any soft strap setting procedure during BIOS. Or, consult your BIOS vendor to confirm this. Please check your BIOS.

 

2.      How many boards are affected?

 

3.      Does the pcie module always fail your test?

 

4.      Have you moved the pcie module to different boards? Different ports? Does the failure follow the pcie module or the board?

 

5.      Are you able to scope the signals and capture the transactions happening?

 

6.      Are you using SSC?

 

7.      What about common clocking for SATA?

 

8.      Can you reproduce this issue on a CRB?

 

We are waiting for your answer to these questions.

 

Best regards,

@Mæcenas_INTEL​.

0 Kudos
MStar1
Beginner
2,551 Views

Hi @Mæcenas_INTEL​ ,

 

Thanks for the update.

 

Regarding your questions:

 

1. We are using our own BIOS based on AMI Aptio V source code. I confirm there're no modifications to the PCH Strap settings during BIOS execution and the register values remain unchanged since the board power-up until the BIOS finishes it's routines, we checked the values in the debug log.

While we speak about the PCH Straps - could you check the values provided and confirm everything is OK?

There's one interesting moment here - the Xeon D-1500 PCH is very similar to C220 PCH series (and it's D31:F0 LPC DeviceID is 8C56h and equals to C224 LPC DeviceID). According to 486708 Lynx Point EDS, the C224 PCH doesn't support switchable ports at all, while other SKUs (Q87/H87/QM87 or C226, for example) fully support them. I checked the PCH Soft straps for 8 Series/C220 series in 489495 Intel 8 Series SPI Programming Guide, and found minor differencies comparing to Xeon D-1500 PCH Straps.

So, isn't there a mistake in Xeon D-1500 Straps?

I expect the switchable port functionality wasn't used in fact by any board vendors and some problems might have been discovered only now, four years since Xeon D-1500 release?

 

2. I think that any Xeon-D PCH is affected to the problem. I could reproduce the problem on modules with Xeon D-1527, Xeon D-1548 and Xeon D-1567 SoCs (they have different CPU parts under the IHS, two dies with V2 and third die with Y0 steppings, but have the similar PCH in all cases). I don't have other SKUs to try, but I guess all of them will demostrate the same behaviour as the PCH die is identical through all the Xeon D-1500 product line.

 

3. The miniPCIe card always fail the test. As I mentioned, I have two slots connected to ports SATA4/PCIe1 and SATA5/PCIe2 respectively. During my experiments, I occupied one slot with miniPCIe card and the other slot with mSATA SSD. Both ports are being switched via the straps, so both ports are expected to change the operating mode and have THE SAME OPERATION MODE at the same time!

So, I confirm the mSATA SSD is always detected regardless the PCH Strap settings, so I can say the muxable ports look like thay aren't muxable in fact and they always "look" to the SATA controller ports - that's why I always detect the mSATA SSD and doesn't detect the mini PCIe submodule in another slot.

 

4. I can swap the modules vice versa, and the situation will not change - the mSATA SSD will be detected on SATA5 instead of SATA4, while miniPCIe card won't be detected on PCIe1 port the same way as it failed on PCIe2 port earlier. The miniPCIe card itself works OK in another on miniPCIe slot connected to the Xeon D-1500 PCH PCIe lane directly (it's connected to PCIe Port6).

 

5. Well, I have the possibility to do that as I have an oscilloscope, but I don't have the PCIe protocol analyzer, so I could only check if the signal is a SATA 6Gb/s or PCIe 5GT/s. I can report back with the screenshots.

 

6. No, the SSC is disabled via the Silicon Enabling\ICC Settings in FITc as we have different PCIe devices across the backplane and they use local 100MHz clocking.

 

7. I'm sorry, but I don't catch what you mean here - the SATA connector doesn't provide a separate clock lines, only the TX and RX pairs. Truly speaking, the mSATA/miniPCIe connector provides the CLK+/- lines, but they are used only for miniPCIe devices, while the mSATA SSD have this pins as "N/C".

 

8. Unfortunately, I don't have the CRB to check. We had problems to get the CRB while we started the development of our first MicroTCA Xeon D processor module in 2016, so we had to bring up the board from scratch, and then there wasn't real need in CRB as the BIOS base was ready and we only modified it according to the future design needs.

 

Thanks and best regards,

Mikhail.

0 Kudos
CarlosAM_INTEL
Moderator
2,551 Views

Hello, @MStar1​:

 

Thanks for your reply.

 

The switchable ports seem to be unsupported by the C224 PCH.

 

As pointed out other SKUs fully support them.

 

The straps on the fitc tool are there but not functional.

 

Best regards,

@Mæcenas_INTEL​.

0 Kudos
MStar1
Beginner
2,551 Views

Hi @Mæcenas_INTEL​ ,

 

Thanks for the reply.

 

There's still some misunderstanding for me.

While C224 doesn't support muxable port (and this is clearly stated in EDS), the Xeon-D PCH definitely supports at least switching between PCIe and USB3 - and I confirm it works.

So, I suppose the Xeon D PCH Chipset DeviceID equals to C224 PCH for some other reasons - for example, because it doesn't support integrated CPU graphics (while full-functional C226 SKU supports int.graphics as well).

 

Could you tell me, do you have the possibility to check the SATA/PCIe muxable behavior on any Xeon D-1500 CRB or other Xeon D-1500 board at your site?

I already told you, we have several different board designs, and all the devices behave exactly the same - and I don't have the chance to test the ports elsewhere.

 

Maybe should I open a ticket at Resource and Design Center Support? They might have the hardware to confirm my issue or point me at my mistakes?

 

Sorry for impatience, but time goes on while the problem is still not solved.

 

Thanks for understanding.

Best regards, Mikhail.

0 Kudos
CarlosAM_INTEL
Moderator
2,551 Views

Hello, @MStar1​:

 

Thanks for your reply.

 

We have contacted you via email.

 

Best regards,

@Mæcenas_INTEL​.

0 Kudos
Reply