kernel init problem

[Altera_Forum]

Hi 

 We have developed our own board with an Altera Stratix II and NIOS II Standard CPU. We download the kernel (2.6.16.11) to the SDRAM (32MB) using nios2-download, and nios2-terminal gives us the following output:  uClinux/Nios II Altera Nios II support © 2004 Microtronix Datacom Ltd. KERNEL -> TEXT=0x04000000-0x04173da8 DATA=0x04173da8-0x04182000 BSS=0x042520a0-0 x042520cf KERNEL -> MEM=0x4253000-0x6000000 STACK=0x6000000-0x6000000 No Command line passed Done setup_arch Built 1 zonelists Kernel command line: trap_init reached init_IRQ done PID hash table entries: 256 (order: 8, 4096 bytes) oom-killer: gfp_mask=0xd0, order=0 Stack from 04181e74:<0>  <0> 00000000<0> 04037d8c<0> 00000000<0> 00000042<0> 041765b0<0> 041765b0<0> 00000000<0> 00000000<0>  <0> 000000d0<0> 04173db0<0> 00000000<0> 0403937c<0> 00000000<0> deadbeef<0> deadbeef<0> 04181f04<0>  <0> deadbeef<0> 00000000<0> 00000000<0> 000000d0<0> 00000000<0> 04181f24<0> 040393e0<0> 04159b38<0>  <0> 00000000<0> 04159b38<0> 0406faec<0> 0424713e<0> deadbeef<0> deadbeef<0> 00000000<0> 04159b38<0>  <0> 00000000<0> 04159b30<0> 040706a0<0> 04181f40<0> 00000000<0> deadbeef<0> deadbeef<0> deadbeef<0>  <0> ffffffff<0> 00000000<0> deadbeef<0> 04000654<0> 00000000<0> 00000001<0> 00000000<0> 00000001<0> Call Trace:<0>  <0> [<04005728>]<0> [<04004c0c>]<0> [<04005740>]<0> [<04000110>]<0>  Mem-info: DMA per-cpu: cpu 0 hot: high 6, batch 1 used:0 cpu 0 cold: high 2, batch 1 used:0 DMA32 per-cpu: empty Normal per-cpu: empty HighMem per-cpu: empty Free pages: 0kB (0kB HighMem) Active:0 inactive:0 dirty:0 writeback:0 unstable:0 free:0 slab:0 mapped:0 pagetables:0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:32768kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 HighMem free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB DMA32: empty Normal: empty HighMem: empty 0 pages of RAM 0 free pages 0 reserved pages 0 pages shared 0 pages swap cached Kernel panic - not syncing: Out of memory and no killable processes...  We have found out that this has something to do with the call to local_irq_enable() in init/main.c (around line 500). If we comment out this line, the execution continues and stops after the "Memory available: " line. Reported memory on this line is correct.  We have a Altera Stratix II with Nios II evaluation board, and the kernel runs fine on this board.  We have successfully run memory tests using the memtest example included in the Nios II 6.0 examples.  Our board also have an SMC91111 ethernet controller.  Anyone has any ideas on how to solve this problem?  Regards Hein Gustavsen

[Altera_Forum]

yes, it is irq related. 

 Note, the interrupt vector (ie, exception vector) should be in uClinux program memory, ie sdram. Note in Linux, irq 0 used to be auto-detected and that is not supported in Nios II, so you should not use irq 0 for your devices. Only timer_0 may use irq 0.  so you have to update in sopc builder, change exception vector to sdram. and assign irq of smc91111 to non-zero value. Then regenerate the new sof.  see, http://nioswiki.jot.com/wikihome/operating...ms/%c2%b5clinux (http://nioswiki.jot.com/wikihome/operatingsystems/%c2%b5clinux)

[Altera_Forum]

Thank you. This solved our problem. 

 Hein    --- Quote Start ---  originally posted by hippo@Jul 12 2006, 03:37 PM yes, it is irq related.  note, the interrupt vector (ie, exception vector) should be in uclinux program memory, ie sdram. note in linux, irq 0 used to be auto-detected and that is not supported in nios ii, so you should not use irq 0 for your devices. only timer_0 may use irq 0.  so you have to update in sopc builder, change exception vector to sdram. and assign irq of smc91111 to non-zero value. then regenerate the new sof.  see, http://nioswiki.jot.com/wikihome/operating...ms/%c2%b5clinux (http://nioswiki.jot.com/wikihome/operatingsystems/%c2%b5clinux) <div align='right'><{post_snapback}> (index.php?act=findpost&pid=16833)  --- quote end ---    --- Quote End ---  

[Altera_Forum]

Hi Heing and Hippo, 

 I need some more understanding on this topic,    If my nios2_system.h has following <div class='quotetop'>QUOTE </div>  --- Quote Start ---  ---------------------------------------------------------------------------- # define na_high_res_timer                                    0x809209e0# define na_high_res_timer_irq                                          3 # define na_sys_clk_timer                                      0x80920940# define na_sys_clk_timer_irq                                          0   /* Redefining high_res_timer -> timer0 */# undef na_high_res_timer# undef na_high_res_timer_irq # define na_timer0                              ((np_timer*) 0x809209e0)# define na_timer0_irq                                                  3 ------------------------------------------------------------------------------------------[/b]  --- Quote End ---    Does this means even na_sys_clk_timer_irq number is 0 (Zero), na_sys_clk_timer_irq is not been used as timer0.   And due to same reason I am facing following crash   Linux version 2.6.11-uc0 (root@rdvd3) (gcc version 3.4.6)# 9 Thu Jul 13 18:29:12 IST 2006 uClinux/Nios II Altera Nios II support (C) 2004 Microtronix Datacom Ltd. setup_arch: No persistant network settings signature at 007F0000 Built 1 zonelists Kernel command line: PID hash table entries: 128 (order: 7, 2048 bytes) Dentry cache hash table entries: 4096 (order: 2, 16384 bytes) Inode-cache hash table entries: 2048 (order: 1, 8192 bytes) Memory available: 14296k/16384k RAM, 0k/0k ROM (1408k kernel code, 486k data) Mount-cache hash table entries: 512 (order: 0, 4096 bytes) scheduling while atomic: swapper/0x04000002/0 Stack from 0116fdf0:<0>       <0> 04000002<0> 01149fa4<0> 01162438<0> 011620c8<0> 01162434<0> 01160034< 0> 00000000<0> 000000d0<0>       <0> 00000001<0> 00000000<0> 04000000<0> ffffe000<0> 0116fe28<0> 0114b1a4< 0> 01162438<0> 00000000<0>       <0> 00000000<0> 0102fae0<0> 00000000<0> 00000000<0> 00000010<0> 00000010< 0> 00000000<0> fffffff4<0>       <0> 011cd0d0<0> 00000b00<0> ffffe000<0> 01160034<0> 00000b00<0> 0120ad60< 0> 00000001<0> 0102fcec<0>       <0> 01009568<0> 0121e7e0<0> 00000010<0> 00000001<0> 000000d0<0> ffffffff< 0> 0116ff28<0> 00000000<0>       <0> deadbeef<0> deadbeef<0> 00000000<0> 00000000<0> 0116ff28<0> 00000001< 0> 00000b00<0> ffffffff<0> Call Trace:<0>       <0> <0> <0> <0> <0>       <0> <0> <0> <0> <0>       <0> <0> <0> <0> <0>       <0> <0> <0> <0> <0> cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 Normal per-cpu: empty HighMem per-cpu: empty Free pages:       14220kB (0kB HighMem) Active:0 inactive:0 dirty:0 writeback:0 unstable:0 free:3555 slab:21 mapped:0 pa getables:0 DMA free:14220kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:16384k B pages_scanned:0 all_unreclaimable? no lowmem_reserve: 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pag es_scanned:0 all_unreclaimable? no lowmem_reserve: 0 0 0 HighMem free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pa ges_scanned:0 all_unreclaimable? no lowmem_reserve: 0 0 0 DMA: 2097664*4kB 1*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2 048kB 3*4096kB = 8404872kB Normal: empty HighMem: empty Kernel panic - not syncing: Out of memory and no killable processes...   OR some times I get following bootup crash if I change vmlinux size (add or remove files from my file system, or add-remove any device support)  oom-killer: gfp_mask=0xd0 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 Normal per-cpu: empty HighMem per-cpu: empty Free pages:       12148kB (0kB HighMem) Active:0 inactive:18755125 dirty:0 writeback:0 unstable:0 free:3037 slab:287 map ped:0 pagetables:0 DMA free:12148kB min:512kB low:640kB high:768kB active:0kB inactive:75020500kB p resent:16384kB pages_scanned:457899 all_unreclaimable? no lowmem_reserve: 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pag es_scanned:0 all_unreclaimable? no lowmem_reserve: 0 0 0 HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present: 0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve: 0 0 0 DMA: 1*4kB 2*8kB 2097664*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2 048kB 2*4096kB = 33574772kB Normal: empty HighMem: empty oom-killer: gfp_mask=0xd0 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 Normal per-cpu: empty HighMem per-cpu: empty Free pages:       12148kB (0kB HighMem) Active:0 inactive:18755125 dirty:0 writeback:0 unstable:0 free:3037 slab:287 map ped:0 pagetables:0 DMA free:12148kB min:512kB low:640kB high:768kB active:0kB inactive:75020500kB p resent:16384kB pages_scanned:457899 all_unreclaimable? no lowmem_reserve: 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pag es_scanned:0 all_unreclaimable? no lowmem_reserve: 0 0 0 HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present: 0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve: 0 0 0 DMA: 1*4kB 2*8kB 2097664*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2 048kB 2*4096kB = 33574772kB Normal: empty HighMem: empty oom-killer: gfp_mask=0xd0 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 Normal per-cpu: empty HighMem per-cpu: empty . . .. . .    Is this crash none other than because of same reason.  waiting for reply, thanks, Ankur

[Altera_Forum]

it is OK to use irq0 for timer, as there is no auto irq detection in the driver. 

but hwselect use hires timer for system timer, which may be not what you want. your problem may be the execption vector, it must be at sdram. check it in sopc builder.

[Altera_Forum]

--- Quote Start ---  originally posted by hippo@Jul 14 2006, 01:33 AM it is ok to use irq0 for timer, as there is no auto irq detection in the driver. but hwselect use hires timer for system timer, which may be not what you want. your problem may be the execption vector, it must be at sdram. check it in sopc builder. <div align='right'><{post_snapback}> (index.php?act=findpost&pid=16887)  --- quote end ---    --- Quote End ---     Exception vector is in sdram at offset 0x0000_0020. Sdram size is 16 MB, I think its enough memory to run kernel with minimal initramfs support + network, flash, jffs2, ps/2 kybd-mouse, and one my added device driver for fb ( crash comes with / without this driver). No apps till now.   flash and network are enable as per nioswiki.  thanks, Ankur

[Altera_Forum]

try remove hires timer in sopc builder.

[Altera_Forum]

--- Quote Start ---  originally posted by hippo@Jul 14 2006, 02:49 AM try remove hires timer in sopc builder. <div align='right'><{post_snapback}> (index.php?act=findpost&pid=16889)  --- quote end ---    --- Quote End ---    my hires timer is 1ms and sys_clk_timer is 10ms, does it cause any effect, Is there any way to force use sys_clk_timer as timer0.  thanks, Ankur

[Altera_Forum]

--- Quote Start ---  originally posted by ankur+jul 14 2006, 03:56 pm--><div class='quotetop'>quote (ankur @ jul 14 2006, 03:56 pm)</div>  --- quote start ---  <!--quotebegin-hippo@Jul 14 2006, 02:49 AM try remove hires timer in sopc builder. <div align='right'><{post_snapback}> (index.php?act=findpost&pid=16889)  --- quote end ---    --- Quote End ---    my hires timer is 1ms and sys_clk_timer is 10ms, does it cause any effect, Is there any way to force use sys_clk_timer as timer0.  thanks, Ankur <div align='right'><{post_snapback}> (index.php?act=findpost&pid=16890)</div> [/b]  --- Quote End ---   the sys clk timer should be a full featured timer. the kernel will program the timer period.  first try if it works without hires timer. then you need to modify the perl scripts called from arch/nios2nommu/scripts/hwselect.pl .

[Altera_Forum]

the sys clk timer should be a full featured timer. 

the kernel will program the timer period.  first try if it works without hires timer. then you need to modify the perl scripts called from arch/nios2nommu/scripts/hwselect.pl . <div align='right'><{post_snapback}> (index.php?act=findpost&pid=16893)</div>   --- Quote End ---     I change sys_clk_timer location in sopc builder (moved up) and now my timer irq is 0 ie same as sys_clk_timer.  I think which ever timer we will specify first in sopc builder will become timer0.  but but but Still I am facing the crash.... If I reduce size of my filesystem (removing some files or remove devices support) and booting comes upto busybox prompt# but once I do "ls" system hangs.....  I know if further I reduce kernel elf size, it will boot properly as earlier.  Some memory figures -------------------------- My vmlinux (elf executable) size is coming something around 13 MB also vmlinux.srec (srec) size is coming around 5 MB  For downloading vmlinux into sdram I am using "nios2-download -g vmlinux" command. Once download completes it gives me 1802 KB download confirmation. SDRAM is 16MB.  Hippo is it ok with respect to nios memory requirments for running kernel. You are more experienced person among us : ) Heing, how big is your kernel size can you share with us ???  Any idea where to head from here !!!!  Ankur

[Altera_Forum]

first, about the kernel size. 

elf is big, because it contains a lot of debug symbols. you can strip off the debug info, and the striped size will be around 2MB in your case. the srec is ascii hex, so it will be more than two times of the binary. the binary size is about 1.8MB in your case. if you compress the kernel with zImage or (gzip + u-boot) , it will be around 1MB.  it looks like problem of kernel size? I had kernel of 3.6MB binary, but did not have such problem.  check usr/initramfs_list   my suggestions, 1. always start with minimal hardware, and minimal kernel config . 2. then add more stuff step by step . I know this takes a long time. but this is what I do when I faced problem like this.  did you enable large memory >1MB allocation in kernel config?  try add some debug messages in init/initramfs.c .  other stuff that might (or might not) be helpful, 1. zImage 2. stand alone init, sash

[Altera_Forum]

Hi Hippo, 

 I followed as you suggested and I am getting some strange results.  I am using now "kernel.config" file given in download with buildroot package. Only change I made is Prints enable at nios-uart and jtag-uart is disable.  But still I am facing same issue.  Strange thing is when I add 2-3 printk statements, for changing size of the  kernel, it boots fine.  Both conditions boot prints are pasted below.  Some other observation I have made....  I added some device support for changing kernel size and same issue I have face with big vmlinux (elf executable) size also (17 MB, 16 MB, 15MB), just removing or adding a dummy printk make kernel boot perfectly fine.  Added support: network, mtd, jffs2, ext2, ext3, kernel hacking. (all with in hw design)  Kernel with same size with different support enable or disabled gives different crashing results or boot prefectly fine.  For certain build if work fine it "always" works fine, never get crashed. If certain build crash it always crashes.  For testing I am using Altera given hardware design with changed memeory map path: \altera\kits\nios2_51\examples\verilog\niosII_stratixII_2s60_ES\full_featured  In future I have to add support for other devices, I can&#39;t relly on adding printk fix. I need a proper solution or either why its happening. I put prints in initramfs.c and ramfs is in safe region. Also large memory >1MB allocation in kernel config is enable. Current kernel size is coming ~7 MB. I tested SDRAM also and its passing all tests.   Any Idea what is going wrong.  CRASHING PRINTS  <div class='quotetop'>QUOTE </div>  --- Quote Start ---  uClinux/Nios II Altera Nios II support © 2004 Microtronix Datacom Ltd. KERNEL -> TEXT=0x01000000-0x010adf1c DATA=0x010adf1c-0x010b8000 BSS=0x01117f20-0 x01117f34 KERNEL -> MEM=0x1118000-0x2000000 STACK=0x2000000-0x2000000 No Command line passed Done setup_arch Built 1 zonelists Kernel command line: PID hash table entries: 128 (order: 7, 2048 bytes) Dentry cache hash table entries: 4096 (order: 2, 16384 bytes) Inode-cache hash table entries: 2048 (order: 1, 8192 bytes) Memory available: 15072k/16384k RAM, 0k/0k ROM (695k kernel code, 424k data) Mount-cache hash table entries: 512 (order: 0, 4096 bytes) initramfs_START =0x10c1b20  initramfs_END =0x110d320   initramfs_size=0x4b800 Linux NoNET1.0 for Linux 2.6 NIOS serial driver version 0.0 ttyS0 (irq = 4) is a builtin NIOS UART io scheduler noop registered io scheduler deadline registered Freeing unused kernel memory: 340k freed (0x10b8000 - 0x110c000) oom-killer: gfp_mask=0xd0 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 Normal per-cpu: empty HighMem per-cpu: empty  Free pages:      13588kB (0kB HighMem) Active:62 inactive:251 dirty:0 writeback:0 unstable:0 free:3397 slab:120 mapped: 0 pagetables:0 DMA free:13588kB min:512kB low:640kB high:768kB active:248kB inactive:1004kB pre sent:16384kB pages_scanned:2097664 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pag es_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present: 0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 79*4kB 1*8kB 1*16kB 0*32kB 1*64kB 2097664*128kB 1*256kB 1*512kB 0*1024kB 0* 2048kB 3*4096kB = 268514452kB Normal: empty HighMem: empty oom-killer: gfp_mask=0xd0 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 Normal per-cpu: empty HighMem per-cpu: empty[/b]  --- Quote End ---      PASSING PRINTS  <div class='quotetop'>QUOTE </div>  --- Quote Start ---  uClinux/Nios II Altera Nios II support © 2004 Microtronix Datacom Ltd. KERNEL -> TEXT=0x01000000-0x010adf88 DATA=0x010adf88-0x010b8000 BSS=0x01117f40-0 x01117f54 KERNEL -> MEM=0x1118000-0x2000000 STACK=0x2000000-0x2000000 No Command line passed Done setup_arch Built 1 zonelists Kernel command line: PID hash table entries: 128 (order: 7, 2048 bytes)   bfr local_irq_enable Dentry cache hash table entries: 4096 (order: 2, 16384 bytes) Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)   aft local_irq_enable Memory available: 15072k/16384k RAM, 0k/0k ROM (695k kernel code, 423k data) Mount-cache hash table entries: 512 (order: 0, 4096 bytes) initramfs_START =0x10c1b40  initramfs_END =0x110d340   initramfs_size=0x4b800 Linux NoNET1.0 for Linux 2.6 NIOS serial driver version 0.0 ttyS0 (irq = 4) is a builtin NIOS UART io scheduler noop registered io scheduler deadline registered Freeing unused kernel memory: 340k freed (0x10b8000 - 0x110c000) main.c: aft free_initmem init started:  BusyBox v1.2.0-pre0 (2006.04.26-14:48+0000) multi-call binary   BusyBox v1.2.0-pre0 (2006.04.26-14:48+0000) Built-in shell (msh) Enter &#39;help&#39; for a list of built-in commands. # # # # # # # [/b]  --- Quote End ---  

[Altera_Forum]

it is really strange. 

 how do you run the kernel? do you use nios2-download? Or, is cache flushed properly in the boot loader?  if it crashes, will it fail at the same place?  check the kernel link script, and the link map output. check the required align of different sections. check if there are (junk) dos2unix in the make scripts.  I did find problems in cpu_idle loop of arch/nios2nommu/kernel/process.c for new kernels. I follow those codes of SH / MIPS to fix the problem.