Nested interrupts again

[Altera_Forum]

Hello Linux masters, 

 I need to make a small patch inside uClinux kernel for NIOS2. I need to enable one specific IRQ to have absolute priority over all atomic processes running in Linux - over threads, over other ISRs, over instruction traps and so on - simply said - over any atomic section in uClinux means. I know, it is possible, but don't know exactly how to do it.  Fot the first try I want to enable my specific interrupt to nest all the other possible ISRs. I have modified function process_int() as follows:  asmlinkage void process_int(unsigned long vec, struct pt_regs *fp) {     //check if special nested interrupt     if (is_nested_interrupt)     {         //process my special interrupt     }     else     {         //enable special nested interupt         is_nested_interrupt = 1;         __asm__ __volatile__(             "wrctl    ienable,r0\n"             : /* No output */             : /* No input */             : /* No regs */);         local_irq_enable();         /* give the machine specific code a crack at it first */         irq_enter();         kstat_cpu(0).irqs++;         latency_cause(-99,~vec);              if (irq_list.handler) {             if ((irq_list.handler(vec, irq_list.dev_id))==IRQ_NONE)             ;         } else    # ifdef DEBUG             {             printk(KERN_ERR "No interrupt handler for level %ld\n", vec);     ////        asm("trap 5");             }    # else      # if 1             printk(KERN_ERR "Ignoring interrupt %ld: no handler\n", vec);      # else             panic("No interrupt handler for level %ld\n", vec);      # endif    # endif              irq_exit();              //disable special nested interrupt         local_irq_disable();         __asm__ __volatile__(             "ldw    r8,0(%0)\n"             "wrctl    ienable,r8\n"             : /* No output */             : "r" (&saved_ienable)             : "r8");         is_nested_interrupt = 0;     } }  There are two new global variables. "is_nested_interrupt" is used to distinguish if interrupt is nested or not. "saved_ienable" is used to hold ienable register changes made during interrupt exception handling to be sure, that at the end of non-nested interrupt there is correct value stored in ienable register prepared to catch any new interrupt source. These two variables are defined also in irq.c as follows:  unsigned long saved_ienable = 0; unsigned long is_nested_interrupt = 0;  Last change needed to be done is in "clrimr" and "setimr" functions. "saved_ienable" variable is allways updated according to new required mask, but "ienable" itself is updated only in case, when we are not in interrupt, see the code:  /* * Use a zero to clean the bit. */ static inline void clrimr(int mask) {     int flags;     local_irq_save(flags);     __asm__ __volatile__(     "ldw    r8,0(%1)\n"     "and    r8,r8,%0\n"     "stw    r8,0(%1)\n"         "ldw    r9,0(%2)\n"     "beq    r9,r0,1f\n"     "andi    r8,r8,0\n"     "1:     wrctl    ienable, r8\n"     : /* No output */     : "r" (mask), "r" (&saved_ienable), "r" (&is_nested_interrupt)     : "r8", "r9");     local_irq_restore(flags); } /* * Use a one to set the bit. */ static inline void setimr(int mask) {     int flags;     local_irq_save(flags);     __asm__ __volatile__(     "ldw    r8,0(%1)\n"     "or        r8,r8,%0\n"     "stw    r8,0(%1)\n"         "ldw    r9,0(%2)\n"     "beq    r9,r0,1f\n"     "andi    r8,r8,0\n"     "1:     wrctl    ienable, r8\n"         : /* No output */     : "r" (mask), "r" (&saved_ienable), "r" (&is_nested_interrupt)     : "r8", "r9");     local_irq_restore(flags); }  Finally the problem is, that even when ienable is set to ZERO during interrupt service routine, the Linux fails to continue somewhere inside the first interrupt routine. More preciselly I found that some first ISR is beiing called and of course "local_irq_enable()" is called to set-up PIE bit in status register, but note that at the same time ienable is set to zero. Then later during servicing this ISR some instruction trap execption is processed and some uknown time later, but very soon (because of no Linux booting console output) it somewhere hangs. I just can say, that it is almost impossible to debug where and what happened.  And now the question is, if PIE=1 and ienable=0, how much is it different from the situation when PIE=0 and ienable is non-zero..??? In both cases interrupts are disabled, but in the first one the Linux during boot hangs possibly after returning from instruction trap servicing. Thank you very much for the explanation.  jan

[Altera_Forum]

No need to do that patch? 

 You should assign irq number 0 in sopc builder to get highest priority. And you should set IRQF_DISABLED flag in request_irq(). Then this irq will always be processed first and won't get other lower irqs. It runs in interrupt context until it finishes. No other process will run at the same time.  - Hippo

[Altera_Forum]

<div class='quotetop'>QUOTE (jan @ Sep 8 2009, 02:03 PM) <{post_snapback}> (index.php?act=findpost&pid=23780)</div> 

--- Quote Start ---  I need to make a small patch inside uClinux kernel for NIOS2. I need to enable one specific IRQ to have absolute priority over all atomic processes running in Linux - over threads, over other ISRs, over instruction traps and so on - simply said - over any atomic section in uClinux means.[/b]  --- Quote End ---   You don&#39;t.   I suppose you don&#39;t want to hear this, but it&#39;s not a good idea to do such high priority stuff with the CPU that runs Linux.   You should either do this in FPGA hardware, or implement a second processor in the FPGA that independently handles the high-priority stuff.  When using a not modifiable hardware things like you describe are unavoidable, but with an FPGA processor, IMHO, this is not appropriate.  -Michael 

[Altera_Forum]

Hmmmm, 

 Thank you Michael, we also suggest solution with another Nios processing just fast interrupts. But yes, this is a complication, so I was thinking about the easiest way - nesting interrupt - but I know there can be also problems with the right content of stack when returning from such non-standard nested exception.  Hippo I dont know if you are right, I know, that "inthandler" (in entry.S) is the function put at interrupt vector address. And from the code investigation I did not find, that interrupts are somewhere re-enabled. But maybe I do not understand the right meaning of IRQF_DISABLED. Can you explain to me, how does it work? I know, that PIE (in "status" register) must be one and "ipending" register must be non-zero - this is the cause of any interrupt. But NIOS automatically sets PIE to zero on interrupt vector entry. So, in which place inside the standard interrupt handler sets PIE to 1 allowing nested interrupts?  jan

[Altera_Forum]

If you want to do something in a super-high-priority interrupt this needs to be something very short running and thus of low complexity.  

 Such a thing very likely can be easily done in hardware (HDL code).  If it needs informations directly from the CPU or sends some information directly to the CPU you can provide the hardware module with an Avalon slave interface with "registers" for status informations and/or a FIFO for data to be transferred.  If it needs to access the memory or some peripheral device, you can provide it with an Avalon master interface (have it be a DMA controller).   Of course you can do both interfaces if appropriate (see the Altera DMA controllers).  Of course accessing pins is "standard" for a HDL module.   So I don&#39;t suppose a coprocessor-CPU is a viable option here.  -Michael

[Altera_Forum]

<div class='quotetop'>QUOTE (jan @ Sep 8 2009, 09:56 PM) <{post_snapback}> (index.php?act=findpost&pid=23783)</div> 

--- Quote Start ---  Hippo I dont know if you are right, I know, that "inthandler" (in entry.S) is the function put at interrupt vector address. And from the code investigation I did not find, that interrupts are somewhere re-enabled. But maybe I do not understand the right meaning of IRQF_DISABLED. Can you explain to me, how does it work? I know, that PIE (in "status" register) must be one and "ipending" register must be non-zero - this is the cause of any interrupt. But NIOS automatically sets PIE to zero on interrupt vector entry. So, in which place inside the standard interrupt handler sets PIE to 1 allowing nested interrupts?  jan[/b]  --- Quote End ---    You can find irq flags in header linux-2.6/include/linux/interrupt.h . The interrupt processing in LInux allows nested interrupt by default. The interrupt will be reenabled if IRQF_DISABLED is not set. Please study some Linux books or trace kernel code.  - Hippo

[Altera_Forum]

Hippo, 

 Regarding IRQF_DISABLED flag, it has no effect in uClinux distribution for NIOS. Interrupt system is here rewritten into much simple form, that do not allow nested interrupts. As a complement to my statement, IRQF_DISABLED flag is used in generic IRQ handler implemented for i386, powerpc and such platforms, but definitelly not for NIOS.  Michael,  Neither DMA neither FIFO neither some other simple logic can be used to automatize our special HDL component interrupt routine, it is also impossible to make a workaround, since the HDL component is blackbox for us and we can use only the features provided by the component. So, there are only two possible solutions: 1) using nested interrupts 2) using second NIOS I am now interested in option 1) and if you think, that it is not a good idea, then I must say I must do it alone, or select option 2).  Thank you, Jan

[Altera_Forum]

Hi for the last time at this topic :-) 

 I tried a small test, not to re-enable interrupts immediatelly, but after first hundred interrupts. Means first hundred interrupts has been serviced with nested interrupts disabled and and all next are serviced with nested interrupts enabled. And exactly on 101. interrupt I got error message:  BUG: failure at kernel/posix-cpu-timers.c:1295/run_posix_cpu_timers()! Kernel panic - not syncing: BUG!  I checked this bug and run_posix_cpu_timers() is stated as function running with interrupts disabled, means if I want to have interrupts enabled (due to nesting), then I need to change this function to be reentrant in interrupt means. And I suggest there are lots such functions required to be changed in order to allow interrupt nesting. Ok Michael, you made me sure to select another solution than nested interrupts :-)  jan

[Altera_Forum]

<div class='quotetop'>QUOTE (jan @ Sep 9 2009, 12:40 PM) <{post_snapback}> (index.php?act=findpost&pid=23792)</div> 

--- Quote Start ---  the HDL component is blackbox[/b]  --- Quote End ---   Supposedly a management decision rather than a technical issue ?  That is really not an adequate way of working with NIOS, as it dumps any benefit this system offers. If you can&#39;t work together with the "hardware" designers, there is no decent help for this project.  But if you can add a second CPU in SOPC-Builder, using SOPC-Builder you also can add your own hardware component with an Avalon master interface that accesses the existing "black box" component via the bus (i.e. a kind of DMA controller).  -Michael

[Altera_Forum]

<div class='quotetop'>QUOTE (jan @ Sep 9 2009, 06:40 PM) <{post_snapback}> (index.php?act=findpost&pid=23792)</div> 

--- Quote Start ---  Hippo,  Regarding IRQF_DISABLED flag, it has no effect in uClinux distribution for NIOS. Interrupt system is here rewritten into much simple form, that do not allow nested interrupts. As a complement to my statement, IRQF_DISABLED flag is used in generic IRQ handler implemented for i386, powerpc and such platforms, but definitelly not for NIOS.[/b]  --- Quote End ---    Which nios2 uclinux distribution do you use? Did you get update? I introduced generic irq handling in 2008 May.  This the link to community supported nios2 uclinux,  http://www.nioswiki.com/installnios2linux (http://www.nioswiki.com/installnios2linux)  - Hippo

[Altera_Forum]

hippo 

 Hmmm, so then I must apologize, since it was my fault to not check the newest state. We use uClinux 2.6.23 from May 2007. Ok, I will do some testing with the newest one linux 2.6.31 and try to play with IRQF_DISABLED. Thank you for information. BTW, is there any link, that explains new interrupt handling?  Michael  Yes, you are also right, finally with the new arguments about Linux our management decided to create completelly new HDL components, that will fit all the needs. In our case we will use FIFO long enough to catch data until Linux is ready to read it. SG DMA is also possible, but experiences shows to HDL designers, that using proved components is allways much better option. Thank you,  jan

[Altera_Forum]

<div class='quotetop'>QUOTE (jan @ Sep 11 2009, 11:20 AM) <{post_snapback}> (index.php?act=findpost&pid=23815)</div> 

--- Quote Start ---  ... but experiences shows to HDL designers, that using proved components is allways much better option.[/b]  --- Quote End ---   I did not vote for not using the "proven HDL design". I just said that if it can be accessed by the CPU via the Avalon bus, it also can be accessed via a (new) dedicated Avalon-bus-mastering HDL (DMA-type) component.   What I maybe would do, is not enhance the proven hdl component (e.g. by FIFOs), but create an additional HDL component that accesses it via the Avalon bus.  The Avalon bus is clever enough to allow for multiple accesses from different masters to different slaves at the same time, so just by configuration done by software, you can have your the "DMA-Controller" use the normal Linux RAM (introducing bus conflict that the Avalon bus solves by delays) or a dedicated on-chip ram (no bus conflicts).   IMHO this is easier to do, better testable and more flexible than a design with dedicated FIFOs.  -Michael