Atom x5-Z8350 RTC prevents booting/corrupts CMOS

bema_sh · ‎02-23-2024

Hello everyone,

my company is building machines which contain a single-board computer with an Atom x5-Z8350 processor. Since we started using this SBC we regularily have issues with the SBC not booting up in the field. Customers complain that the screen stays black and nothing happens, even after power-cycling the machine several times. Sometimes this happens after the customer has used the machine for some time, but we also had cases where completely new machines are dead-on-arrival at the customer site, even though the machines worked perfectly when they left our facility.

After analyzing the problem, it seems that the issue is caused by a corrupted CMOS. If we clear the CMOS using the button on the SBC, the SBC starts back up again and works as inteded. We also found out that this issue only appears on systems which use a coin cell battery for the RTC. Systems which were shipped without a coin cell battery are not affected. If the coin cell battery has a voltage of around 2.9 - 3.0 V, the issue happens the most. If the battery is either below or above this voltage range, the issue only occurs rarely.

My questions:

Is it possible that insufficient or unstable power from the RTC battery is causing a CMOS corruption?
In the Intel documentation AP-728, I read that there are two reset signals for the RTC circuitry (RTCRST# and SRTCRST#). SRTCRST# should only be pulled low while also pulling RTCRST# low to avoid the system entering an indeterminate state. Could an unwanted assertion of SRTCRST# (e.g. EMI) without also asserting RTCRST# cause the issue we are seeing?
I mapped out the circuitry on the SBC which supplies power to the RTC domain. Please see the picture below (hand-drawn schematic) for details. If it is not readable, please let me know and I will create a better schematic diagram. The second picture shows the recommended circuitry for the RTC domain in Intel AP-728.
Mapped out RTC power circuitry on the SBCRecommended RTC power circuitry in Intel AP-728

As can be seen, the manufacturer of the SBC decided to add an additional MOSFET to disconnect the battery as soon as the VccDSW3_3 power rail becomes available. Could this cause the issue we are seeing? I measured the voltage at C1 with an oscilloscope and even if I randomly disconnected the power supply from the SBC, I could not record a drop in the voltage. The coin cell battery took over as soon as the system power rail was gone.

Is there a way to permanently lock the access to the User RAM in the RTC domain? Can I use a software tool etc. to accomplish that? The BIOS of the SBC provides an option, but it only locks a part of the user RAM (address range 0x38 - 0x3F).

I now that this post is not that specific, but we really need to solve the problem and the manufacturer of the SBC is not cooperative at all. We went back and forth with them and all the solutions they proposed did not help, so I am now reaching out to you and digging deeper on my own to find the root cause of the issue.

If you need any more data or information please let me know and I will try to bring them to you ASAP. Any help would be very appreciated!

Best Regards

Mike_Intel · ‎02-25-2024

Hello bema_sh,

Thank you for posting in Intel community Forum.

Upon checking, please be informed that your query will be best answered by our Embedded Connectivity Support team. We will help you moving this post to the designated team.

Best regards,

Michael L.

Intel Customer Support Technician

CarlosAM_INTEL · ‎02-26-2024

Hello, @Mike_Intel:

Thank you for contacting Intel Embedded Community.

We need to clarify that the Intel Atom® x5-Z8350 Processor is a non-embedded device, as you may confirm in the yellow-highlighted information in the following screenshot:

This information is stated on the following website:

https://ark.intel.com/content/www/us/en/ark/products/93361/intel-atom-x5-z8350-processor-2m-cache-up-to-1-92-ghz.html

Due to this, you should consider this to transfer this forum to the proper support channel.

Best regards,

@CarlosAM_INTEL.

bema_sh · ‎02-27-2024

Hello @CarlosAM_INTEL and @Mike_Intel

thank you for taking the time to check my question. Could you please go ahead and move the topic to the correct support channel? At the moment the rate of service cases of our machines is going up, so we would really appreciate any help we can get.

I also did some experiments on my own. Here are the current results:

- Connecting a 3,67 V batterie (measured 3,57 V behind the diode) resulted in a defective RTC on one of the SBCs. The time still increments when Windows is running, but if I disconnect the AC power from the SBC, the time stops. When I boot Windows again, the time and date are equal to the time and date when I last shutdown the SBC. This means a 3.6 V lithium cell is not a route we are going to pursue further.

- I connected a variable power supply to the battery input of the SBC to simulate different battery voltages. After shutting down the SBC multiple times (>100) and turning it back on again, I was able to reproduce the issue once at a battery voltage of 2,13 V. After a CMOS reset, the SBC booted again, but I could not reproduce the issue afterwards.

The issue is that we are not fully convinced that the RTC or the RTC battery are really the culprit. Maybe we are on the wrong path at the moment. If you have any idea what could cause this behaviour or if there is a way to localize the issue, please let me know.

Looking forward to your reply. Thanks!

Best Regards

bema_sh · ‎02-26-2024

Hello Michael L.

thanks for moving the topic to the correct sub-community!

We are thinking that using a battery with a higher voltage (e.g. Lithium cell with 3.6 V) could be a possible solution, but I could not tell what maximum voltage the processor supports on the corresponding voltage rail. Could you also provide me with information on that?

Thanks!

Best Regards

Louis_DY · ‎05-27-2024

Hello bema_sh,

We are suffering same issue with you on the Amber Lake Y42 platform, so I wonder if you got final solution to solve this issue, could you please share here, thank you!

bema_sh · ‎05-27-2024

Hello @Louis_DY ,

we were able to fix the problem, but it actually had nothing to do with the RTC battery.

As mentioned in my first post, we are using an SBC with an Atom x5-Z8350 processor. The SBC features a header where you can connect a 3V lithium battery (CR2032) to keep the RTC running when the main power supply of the system is turned off. If a customer has the issue that the SBC is not booting, removing and reconnecting the battery or resetting the BIOS with the button on the SBC solved the problem. So at first we suspected a problem regarding the RTC or the battery.

After building a testbench which continuously power cycles the SBCs and checks if they are booting, we were able to reliably reproduce the problem we had at the customer site. During testing, we discovered by accident that the problem was actually the power supply we are using for the SBC. The SBC is powered via a micro-USB connector and is receiving 5V with a maximum current draw of 2A (according to the documentation of the SBC). Our power supply delivers a stable 5,15 V of power (to accomodate for voltage drops across the USB cable) and a maximum current of 3A. When we checked the voltage with an oscilloscope, everything was fine and we could not see any drops in the supply voltage. The power supply is built by us in-house (own design and PCB) and uses 2x10 µF ceramic capacitors on the output. It turned out that if we connected an electrolytic capacitor of at least 2200 uF to the output of the supply in parallel, the issue with the SBC not booting went away completely, even though the voltage already looked fine with only the ceramic capacitors.

I suppose what is happening is that the SBC constantly updates the RTC time in the BIOS. When the main power is removed at the same time the current time is written into the BIOS, the BIOS gets corrupted and the SBC is unable to boot again. By adding the big electrolytic capacitor on the output we increased the fall time of the supply voltage when turning off the SBC, which gives the SBC enough time to finish any write operations it is currently performing, leaving the BIOS in an healthy state. But this is just an assumption, since neither the manufacturer of the SBC or Intel did tell us how the RTC is working in detail.

In conclusion, what I can recommend to you is checking your power supply and maybe adding more capacitance or using a higher quality power supply. If there is anything more I can help you with or you need additional information, just let me know and I will try my best to answer the questions.

Louis_DY · ‎05-28-2024

Hello bema_sh,

Thanks for your share! Your information above is really helpful for us, we will adopt your ideal to try if this issue can be solved on our product as soon as possible and let you know any result.

Our product is a tablet with one large Li-on battery inside as supply for system, but without coin battery for RTC power rail. Instead of coin battery, a LDO will generate a stable 3.3V to supply RTC rail after Li-on battery installed in tablet, and this 3.3V through diode circuitry to supply VCCRTC. Very unfortunately we are very difficult to reproduce this phenomenon based on our system! So far we can't find out the way that can reproduce reliable this hang booting, therefore, I wonder how you do power cycling your testbench, auto S5 by some test tool? if so, we have this tool and can perform this testing on our system to check if this issue can be reliably reproduced.

On the other hand, our system have an EC which mange the system power on and off timing sequence on board, based on above your information, you finally solved your problem by adding the capacitor's capacitance to increase the delay of VCC to ground (Supposed it is ), we will also try to add some delay to shut down DSW and primary well supply after system power off by EC control, but first of all, we need find the way to reliably reproduce booting issue, then it can be verified if any solution is effective or not.

Highly appreciate your sharing!

bema_sh · ‎05-28-2024

Hello @Louis_DY ,

glad I am able to help!

Our product is a tablet with one large Li-on battery inside as supply for system, but without coin battery for RTC power rail. Instead of coin battery, a LDO will generate a stable 3.3V to supply RTC rail after Li-on battery installed in tablet, and this 3.3V through diode circuitry to supply VCCRTC

Ok, so the system your are working with is completely different from the SBC we are using, since we power the SBC from a power supply, have no lithium battery and an actual battery for the RTC instead of an LDO. We should also keep in mind that you are using a different processor architecture, but if I had to guess I would suspect that there are not that many changes in the RTC region, since the Intel AP-728 I mentioned above (documentation for the RTC) covers a whole range of processors.

Very unfortunately we are very difficult to reproduce this phenomenon based on our system! So far we can't find out the way that can reproduce reliable this hang booting, therefore, I wonder how you do power cycling your testbench, auto S5 by some test tool? if so, we have this tool and can perform this testing on our system to check if this issue can be reliably reproduced.

I do not know if you can use our concept with your tablet, but I can still explain you how we did it. We used a microcontroller connected to four MOSFETs to turn on and off four of the power supplies for the SBCs. We used the same power supplies I mentioned above (made in-house with own PCB design). We connected the SBCs to the power supplies, so that the microcontroller could turn the SBCs on and off by controlling the MOSFETs.

On the SBCs we installed our standard Windows Image and added a small powershell script to the autostart folder. We also connected an Arduino to each SBC via USB. When the SBC boots up, it runs the powershell script, which sends a command to the Arduino over the serial connection. When the Arduino receives the command, it sets one of its digital outputs to HIGH. This signal is connected to an input pin on the main microcontroller which controls the MOSFETs. If the input pin reads HIGH, the microcontroller knows that the SBC booted up correctly.

To finish it, we connected a small LCD to the main microcontroller to display the number of cycles and how many times the SBCs have booted.

The testing cycle is as follows:

1. Turn all SBCs on with the MOSFETs

2. Wait for 2 minutes so the SBCs can boot up

3. Check the state of the return signals from the SBCs. Increment the counters (successful boot counter and power cylce counter) accordingly and update the LCD.

4. Turn off the SBCs.

5. Small delay (15 seconds), then go back to 1

This way, we were able to reproduce the issue. On average, every 70-150 power cycles one of the SBCs stopped booting. After we added the electrolytic capacitor, we ran a total of 10,000 test cycles. Since we have four SBCs connected, this equals 40,000 total test cycles. During these cycles, we were not able to reproduce the problem, so we decided that adding the capacitor fixes the issue.

On the other hand, our system have an EC which mange the system power on and off timing sequence on board, based on above your information, you finally solved your problem by adding the capacitor's capacitance to increase the delay of VCC to ground (Supposed it is ), we will also try to add some delay to shut down DSW and primary well supply after system power off by EC control,

Exactly. We were able to confirm with an oscilloscope that the fall time when turning off the power supply is a bit longer with the capacitor connected. However, I cannot tell you how this affects the SBC. What I mentioned above regarding the writing of the BIOS is just an assumption on our end.

However, what you suggested regarding the delay on shutdown of DSW sound like a good idea, I would definitely give it a try.

Siyabonga · ‎05-28-2024

It is possible that unstable power from the RTC battery is causing CMOS corruption, especially if the battery voltage is around 2.9 - 3.0V. Unwanted assertion of SRTCRST# without RTCRST# due to EMI could indeed cause issues. The additional MOSFET in the RTC power circuit might also contribute to the problem, despite voltage measurements appearing stable. To permanently lock access to User RAM in the RTC domain, you can explore BIOS settings or use software tools, but options may be limited. Ensuring stable power to the RTC domain and further shielding against EMI could mitigate the issue.

bema_sh · ‎05-28-2024

Hello @Siyabonga ,

thank you for sharing your ideas and suggestions. As mentioned above we were able to fix the problem by adding a large electrolytic capacitor to the power rail of the SBC. Still, this is some helpful information and it will probably be helpful for everyone who finds this topic. Also, if the problem reoccurs on our site (which I hope it does not :D), I may look into your suggestions. Thank you!