Soft realtime with PREEMPT-RT for NIOS II

[Altera_Forum]

Dear all, 

 We have made an effort to port PREEMPT-RT on NIOS II. It is possible to use PREEMPT-RT for NIOS II (with MMU) under Linux.   PREEMPT-RT is a soft realtime extension for Linux.  The PREMPT-RT port for NIOS II is available here: http://uuu.enseirb.fr/~kadionik/nios2-preempt-rt/ (http://uuu.enseirb.fr/%7ekadionik/nios2-preempt-rt/)  We have written a quick and dirty tutorial for using PREEMPT-RT for NIOS II under Linux: http://uuu.enseirb.fr/~kadionik/nios2-preempt-rt/#howtos (http://uuu.enseirb.fr/%7ekadionik/nios2-preempt-rt/#howtos)  Be careful in your SoPC design. You must add these peripherals (and respect names): - NIOS softcore processor MMU. - 2 timers:  * 64-bit Timer hrclock for PREEMPT-RT.  * 32-bit Timer sys_clk_timer for Linux.  PREEMPT for NIOS II (with MMU) has been tested on these boards ( http://uuu.enseirb.fr/~kadionik/nios2-preempt-rt/#testing (http://uuu.enseirb.fr/%7ekadionik/nios2-preempt-rt/#testing) ) - Altera Stratix 1S10. - Altera Cyclone III 3C25.  We have added the hrtimer support to the initial Linux port.  With this initial PREEMPT-RT port, we have measured latency with cyclictest when we stress the board with tools like dd, hackbench, stress and ping flooding. We have until now extra max latency < 1500 µs when we stress the board with dd.  We found that the Linux port for NIOS II seems to be weak in the thread creation/destruction and needs to be improved.  If you make improvements to this initial port, please contact us (kadionik AT enseirb-matmeca.fr).  Sincerely Yours;  Patrice

[Altera_Forum]

This is great news! 

I will try it soon.

[Altera_Forum]

Hi, 

 Yes. The question was asked by you the first time here ;-) : http://www.alteraforum.com/forum/showthread.php?t=20748  I think this initial port can be improved for reducing latency time as explained before.  Patrice

[Altera_Forum]

This is absolutely GREAT as IMHO there will be lots of application for NIOS that will need a better realtime behavior and of course the MMU version of NIOS will be the way to go, anyway.  

 So I would very much like to see the "Realtime-Level" as a configurable option in the build system. AFAIK, at the moment there are two levels, your work might introduce a third one.  BTW.: what are the cons to _always_ using the realtime extensions ?   I understand that while the max latency is improved, the overall performance of the Kernel will be somewhat lower. Do you have numbers for that ?   BTW / 2.: did the lack of atomic instructions impose any problems to your work ?   -Michael

[Altera_Forum]

HI Michael, 

 -- BTW.: what are the cons to _always_ using the realtime extensions ?  With realtime, if you have realtimer threads running always, other standard threads will never run and you will have the impress that your system is frozen if you interact with him with normal program. In fact not.  -- I understand that while the max latency is improved, the overall performance of the Kernel will be somewhat lower. Do you have numbers for that ?  One I have loaded my system with "stress -i 20 -c 20" (40 process), the max latency is about 800 µs.  -- BTW / 2.: did the lack of atomic instructions impose any problems to your work ? For atomic operations, interrupts are disabled (it is done in the same manner with processor without atomic instructions). Just a cost in terms of latency.  Patrice.

[Altera_Forum]

--- Quote Start ---  -- BTW.: what are the cons to _always_ using the realtime extensions ?  With realtime, if you have realtimer threads running always, other standard threads will never run and you will have the impress that your system is frozen if you interact with him with normal program. In fact not..  --- Quote End ---    I understand that this is true with a standard system, independent of the Kernel configuration, as well: If you start a process or thread and assign a realtime Prioridy (SCHED_FIFO or SCHED_RR) to it, and this process or thread never goes to sleep on it's own, all "normal" user process will hang. forever.    --- Quote Start ---   One I have loaded my system with "stress -i 20 -c 20" (40 process), the max latency is about 800 µs.   --- Quote End ---   This does sound very good. But did you find that the overall Kernel performance (for time shared work) significantly degrades (with no realtime process running) ?     --- Quote Start ---  For atomic operations, interrupts are disabled (it is done in the same manner with processor without atomic instructions). Just a cost in terms of latency.  --- Quote End ---    I see. With the actual NIOS "hardware" there seems not to be another way to go. :( unfortunately this will not be possible with a future SMP implementation :( .  -Michael

[Altera_Forum]

Michael, 

   --- Quote Start ---  I understand that this is true with a standard system, independent of the Kernel configuration, as well: If you start a process or thread and assign a realtime Prioridy (SCHED_FIFO or SCHED_RR) to it, and this process or thread never goes to sleep on it's own, all "normal" user process will hang. forever.  --- Quote End ---  By default no because you have for the kernel "Real-Time group scheduling" enabled. See here: http://www.kernel.org/doc/documentation/scheduler/sched-rt-group.txt By default, you have : # cat /proc/sys/kernel/sched_rt_period_us  1000000 # cat /proc/sys/kernel/sched_rt_runtime_us  950000   That means that during 1 second, 95 % of CPU time is for RT threads (if they use it of course) and the 5 % remaining time for the non-RT threads.   If your RT threads need more than 950 ms of CPU time, they are stopped during the last 5 ms for non RT threads and rerun after. This behaviour is unacceptable for a realtime system and must be disabled by:# echo -1 > /proc/sys/kernel/sched_rt_runtime_us   In this cas, RT threads take all the CPU time they need and it is possible that non RT threads are not executed. That is what we want with a real hard realtime system. In this special case, what you say is correct.     --- Quote Start ---  This does sound very good. But did you find that the overall Kernel performance (for time shared work) significantly degrades (with no realtime process running) ?   --- Quote End ---  I use:# echo -1 > /proc/sys/kernel/sched_rt_runtime_us  to have with PREEMPT-RT the more realtime behaviour as possible.    --- Quote Start ---   I see. With the actual NIOS "hardware" there seems not to be another way to go. http://www.alteraforum.com/images/smilies/frown.gif unfortunately this will not be possible with a future SMP implementation http://www.alteraforum.com/images/smilies/frown.gif .  --- Quote End ---  Yes, you're right. No SMP support at this time...   Patrice.

[Altera_Forum]

Thanks for letting me know about "sched_rt_runtime_us" ! 

 I still don't understand what votes agains _always_ using your patch, though.   -Michael

[Altera_Forum]

--- Quote Start ---  Thanks for letting me know about "sched_rt_runtime_us" !  I still don't understand what votes agains _always_ using your patch, though.   -Michael  --- Quote End ---    Hi Michael,  Coming from the HW community, when I think realtime, I think hard realtime i.e. max latency is guaranteed in any cases. If the Real-Time group scheduling is enabled and I use PREEMPT-RT, my RT thread that has the max priority can be preempted by a non RT thread after 950 ms of CPU processing. And it will run after 5 ms. It is not acceptable because you add an extra latency. If I disable this with PREEMPT-RT, I can have something that is closer to a hard realtime OS. PREEMPT-RT is soft realtime and it is important to avoid mechanisms that introduce extra latencies. PREEMPT-RT must be tuned finally. See: Real-Time Failure. F. Rowand. http://elinux.org/images/b/be/real_time_linux_failure.pdf  Patrice.

[Altera_Forum]

Thus while improving the hard realtime capability (max latency) of the Kernel, the "soft" realtime capability (average latency) will be worse.  

 This it would be perfect to have a switch in the Kernel/System "make" procedure to select either option.   -Michael

[Altera_Forum]

--- Quote Start ---  Thus while improving the hard realtime capability (max latency) of the Kernel, the "soft" realtime capability (average latency) will be worse.   This it would be perfect to have a switch in the Kernel/System "make" procedure to select either option.   -Michael  --- Quote End ---    Hi Michael,  There is an interesting discussion on soft vs hard realtime at this time on the RT mailing list.  hard or soft realtime defines in fact a QoS that you offer to your system according to its needs.  Patrice