Linux with MMU on NEEK

[Altera_Forum]

Hi, all. 

  I'm testing Linux MMU version, on my NEEK.  http://www.nioswiki.com/linux   It works fine and I can use "bash" shell. This is the evident proof that we are using the true 'fork' instead of 'vfork'.  May be this will depends on the version, but TSE driver claims an error and doesn't work on this design. The error is   ERROR: altera_tse.c:1666: request_mem_region() failed I think that this error is caused by misunderstanding of the usage for the function request_mem_region(). Inside of the request_mem_region(), the function __request_region() is called. If the resource has been already registered, this function returns a non-NULL value, that is the pointer for its resource. But the resource 'sgdma_rx_base' is already registered in the initialization process, so this function returns the 'conflict' and    if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) { is always true. So I made a dirty patch,   if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) { reg_resource = __request_region(&iomem_resource, sgdma_rx_base, sgdma_rx_size, "altera_tse", 0); if (reg_resource != NULL && reg_resource->flags & IORESOURCE_BUSY) { printk(KERN_ERR "ERROR: %s:%d: request_mem_region() failed\n", __FILE__, __LINE__); ret = -EBUSY; goto out_sgdma_rx; } } Moreover, the author is forgetting that the DMA is working in the physical address world, so we need to set the pointers of descripters like  // desc->source = read_addr; desc->source = virt_to_phys(read_addr); // desc->destination = write_addr; desc->destination = virt_to_phys(write_addr); // desc->next = (unsigned int *)next; desc->next = (unsigned int *)((unsigned long)next & 0x1fffffffUL); and so on.   Also the frame buffer fb0 will not work well, because the driver 'altfb.c' is not implemented for Linux with MMU version. So I put some codes for altfb_mmap(), like  /* We implement our own mmap to set MAY_SHARE and add the correct size */ static int altfb_mmap(struct fb_info *info, struct vm_area_struct *vma) { unsigned long phys_addr, phys_size; unsigned long addr; unsigned long size = vma->vm_end - vma->vm_start; unsigned long offset = vma->vm_pgoff << PAGE_SHIFT; // vma->vm_flags |= VM_MAYSHARE | VM_SHARED; // vma->vm_start = info->screen_base; // vma->vm_end = vma->vm_start + info->fix.smem_len; /* check range */ if (vma->vm_pgoff > (~0UL >> PAGE_SHIFT)) return -EINVAL; if (offset + size > altfb_fix.smem_len) return -EINVAL; vma->vm_flags |= VM_IO | VM_RESERVED; addr = vma->vm_start; phys_addr = altfb_fix.smem_start + offset; if ((offset + size) < altfb_fix.smem_len) phys_size = size; else phys_size = altfb_fix.smem_len - offset; vma->vm_page_prot = __pgprot(_PAGE_PRESENT|_PAGE_READ|_PAGE_WRITE); if (remap_pfn_range(vma, addr, phys_addr >> PAGE_SHIFT, phys_size, vma->vm_page_prot)) return -EAGAIN; return 0; } and rewrite the DMA descripters like  desc->next = (void *)virt_to_phys((desc + 1)); So now, I can evoke telnetd and control NEEK through ethernet, and use Nano-X on Linux MMU version, but can't enter ftp session, because 'getservbyname()' function will not work well.   I don't know the directory that the souce of 'getservbyname()' is included. Would anyone please tell me where is it?  Thank you, in advance.

[Altera_Forum]

I had the cache changed in the custom design to the minimum as in the NEEK design (it was 8+8k) and the ethernet works now. *headdesk* 

My theory is something isn't getting flagged properly for uncached IO and using the smaller cache just happens to avoid caching those bits.

[Altera_Forum]

Hi, 

   --- Quote Start ---  I had the cache changed in the custom design to the minimum as in the NEEK design (it was 8+8k) and the ethernet works now. *headdesk* My theory is something isn't getting flagged properly for uncached IO and using the smaller cache just happens to avoid caching those bits.  --- Quote End ---     I have a little bit doubt that Nios CPU with MMU can't have over 4Kbytes caches. There are several ways of connecting method for CPU, cache and MMU. For example,  1) CPU -- cache -- MMU -- Memory 2) CPU -- MMU -- cache -- Memory.   The first method has low latency, but also has 'synonym problems'. The second method accesses the cache by physical addresses, but has larger latency to the contrary. So I think that Nios CPU takes the next strategy i.e.  3) CPU |--cache --| -- Memory  ............|--MMU --|  by limiting the size of cache under page size.(Please refer Nios handbook n2cpu_nii5v1.pdf, page 2-10, Figure 2-2.) If so, we can't have both instruction and data caches larger than page size(=4Kbytes).  Kazu

[Altera_Forum]

--- Quote Start ---  If so, we can't have both instruction and data caches larger than page size(=4Kbytes).  --- Quote End ---   Is this confirmed by the hardware Gurus ? If yes this would slow down the CPU with dynamic memory greatly. I can't believe that Altera would choose such a silly implementation.   I do know that ARM uses the "low latency" CPU->cache->MMU->RAM way, which needs a cache flush with any MMU table update and thus any task switch, thus very bad with complex OSes, while the PC uses the "correct" higher latency CPU->MMU->cache->RAM way,  I was told NIOS features a "correct" implementation and with my first tests I found that the RAM latency in fact seems a lot greater than without an MMU and I took this for a confirmation.But of course I might be wrong.  -Michael

[Altera_Forum]

Does the MMU system still use the "uncached" bit 31 (0x80000000)?

[Altera_Forum]

No. The cacheable Bit is in the page descriptor of each virtual page. 

 -Michael

[Altera_Forum]

Hi, 

   --- Quote Start ---  Is this confirmed by the hardware Gurus ?    --- Quote End ---     No, not yet. But where is the nest of MMU's Gurus?    --- Quote Start ---   I can't believe that Altera would choose such a silly implementation.    --- Quote End ---     Yes, of course, I want to believe that Altera is NOT silly. But the bit order of tlbacc register is silly, there is no present bit, it's silly...   Now I tested the 'modprobe' and system call 'init_module'. It works, but unfortunately, the module is loaded in the 'vmalloc' area(0x80000000~). Nios instructions 'call' and 'jmpi' can't jump beyond the boundary of 256MBytes. So the kernel can't reach the module codes and module codes can't call kernel routines.   The easiest way to solve this problem is to change 'vmalloc' to 'kmalloc' in the function 'load_module' of the file '/kernel/module.c', as follows.   /* Allocate and load the module: note that size of section 0 is always zero, and we rely on this for optional sections. */ static noinline struct module *load_module(void __user *umod, unsigned long len, const char __user *uargs) { Elf_Ehdr *hdr; Elf_Shdr *sechdrs; char *secstrings, *args, *modmagic, *strtab = NULL; char *staging; unsigned int i; unsigned int symindex = 0; unsigned int strindex = 0; unsigned int modindex, versindex, infoindex, pcpuindex; unsigned int num_mcount; struct module *mod; long err = 0; void *percpu = NULL, *ptr = NULL; /* Stops spurious gcc warning */ unsigned long *mseg; mm_segment_t old_fs; DEBUGP("load_module: umod=%p, len=%lu, uargs=%p\n", umod, len, uargs); if (len < sizeof(*hdr)) return ERR_PTR(-ENOEXEC); /* Suck in entire file: we'll want most of it. */ /* vmalloc barfs on "unusual" numbers. Check here */ // if (len > 64 * 1024 * 1024 || (hdr = vmalloc(len)) == NULL) if (len > 64 * 1024 * 1024 || (hdr = kmalloc(len, GFP_KERNEL)) == NULL) return ERR_PTR(-ENOMEM); if (copy_from_user(hdr, umod, len) != 0) { err = -EFAULT; goto free_hdr; } /* Sanity checks against insmoding binaries or wrong arch, weird elf version */ if (memcmp(hdr->e_ident, ELFMAG, SELFMAG) != 0 || hdr->e_type != ET_REL || !elf_check_arch(hdr) || hdr->e_shentsize != sizeof(*sechdrs)) { err = -ENOEXEC; goto free_hdr; } ............................ err = mod_sysfs_setup(mod, mod->kp, mod->num_kp); if (err < 0) goto unlink; add_sect_attrs(mod, hdr->e_shnum, secstrings, sechdrs); add_notes_attrs(mod, hdr->e_shnum, secstrings, sechdrs); /* Get rid of temporary copy */ // vfree(hdr); kfree(hdr); /* Done! */ return mod; unlink: /* Unlink carefully: kallsyms could be walking list. */ list_del_rcu(&mod->list); synchronize_sched(); module_arch_cleanup(mod); cleanup: kobject_del(&mod->mkobj.kobj); kobject_put(&mod->mkobj.kobj); ftrace_release(mod->module_core, mod->core_size); free_unload: module_unload_free(mod);# if defined(CONFIG_MODULE_UNLOAD) && defined(CONFIG_SMP) free_init: percpu_modfree(mod->refptr);# endif module_free(mod, mod->module_init); free_core: module_free(mod, mod->module_core); /* mod will be freed with core. Don't access it beyond this line! */ free_percpu: if (percpu) percpu_modfree(percpu); free_mod: kfree(args); free_hdr: // vfree(hdr); kfree(hdr); return ERR_PTR(err); truncated: printk(KERN_ERR "Module len %lu truncated\n", len); err = -ENOEXEC; goto free_hdr; } Yes, OK, I'm sure. It's the file inside the directory /linux-2.6/kernel !:eek: But don't tell Linus about this tampering...  Kazu 

[Altera_Forum]

--- Quote Start ---  Now I tested the 'modprobe' and system call 'init_module'. It works, but unfortunately, the module is loaded in the 'vmalloc' area(0x80000000~). Nios instructions 'call' and 'jmpi' can't jump beyond the boundary of 256MBytes. So the kernel can't reach the module codes and module codes can't call kernel routines.  --- Quote End ---   Ahh. this is why Kernel Modules don't work with the MMU based distribution. (A "not usable" crit for me). I do hope Hippo and your are able to solve this issue.   Thanks for helpin out here !!!!  -Michael

[Altera_Forum]

Hi, 

  Maybe the dynamic linker 'ld.so.1' has some relocation problems, so I'm going to debug it on this weekend.  And do we have any other major problems around the kernel of Nios Linux with MMU version?  Now, should I tackle the problems of futex and SMP?:D  Kazu

[Altera_Forum]

Great ! Thanks :) 

 -Michael

[Altera_Forum]

Hi, 

  I'm testing the dynamic linker and shared libraries of Nios system. Once I experienced that the static library 'libnano-X.a' generates SIGSEGV and I believed that this was caused by the wrong relocation of 'ld.so.1'. For example,   00017328 <GrSetErrorHandler>: 17328: defffc04 addi sp,sp,-16 1732c: dc400015 stw r17,0(sp) 17330: 044000f4 movhi r17,3 17334: 8c63be04 addi r17,r17,-28936 17338: dd400215 stw r21,8(sp) 1733c: 202b883a mov r21,r4 17340: 8809883a mov r4,r17 17344: dfc00315 stw ra,12(sp) 17348: dcc00115 stw r19,4(sp) 1734c: 0008ac40 call 8ac4 <_init+0x408> <----- this address must be relocated as loading adr. + 0x8ac4. 17350: d4e29517 ldw r19,-30124(gp) 17354: 8809883a mov r4,r17 17358: d5629515 stw r21,-30124(gp) 1735c: 00087a00 call 87a0 <_init+0xe4> <----- this address must be relocated as loading adr. + 0x87a0. 17360: 9805883a mov r2,r19 17364: dfc00317 ldw ra,12(sp) 17368: dd400217 ldw r21,8(sp) 1736c: dcc00117 ldw r19,4(sp) 17370: dc400017 ldw r17,0(sp) 17374: dec00404 addi sp,sp,16 17378: f800283a ret but the result is as follows.   0x2aae3328: addi sp,sp,-16 0x2aae332c: stw r17,0(sp) 0x2aae3330: movhi r17,3 0x2aae3334: addi r17,r17,-28936 0x2aae3338: stw r21,8(sp) 0x2aae333c: mov r21,r4 0x2aae3340: mov r4,r17 0x2aae3344: stw ra,12(sp) 0x2aae3348: stw r19,4(sp) 0x2aae334c: call 0x20008ac4 <----- Wrong relocation? 0x2aae3350: ldw r19,-30124(gp) 0x2aae3354: mov r4,r17 0x2aae3358: stw r21,-30124(gp) 0x2aae335c: call 0x200087a0 <----- Wrong relocation? 0x2aae3360: mov r2,r19 0x2aae3364: ldw ra,12(sp) 0x2aae3368: ldw r21,8(sp) 0x2aae336c: ldw r19,4(sp) 0x2aae3370: ldw r17,0(sp) 0x2aae3374: addi sp,sp,16 0x2aae3378: ret .   But the real fact is that no one relocates these codes. These codes are already relocated by the (static-) linker, and the elf-header has no relocating information when it's loaded to the memory. With the switch '-shared', we can avoid this situation, but if we indicate an option switch '-prefer-non-pic' for shared libraries, the same thing will occur. Next code is the disassemble result of 'libmpeg2.so'.  00004660 <mpeg2_init>: 4660: d0a03017 ldw r2,-32576(gp) 4664: defffb04 addi sp,sp,-20 4668: dcc00315 stw r19,12(sp) 466c: dc800215 stw r18,8(sp) 4670: dfc00415 stw ra,16(sp) 4674: dc400115 stw r17,4(sp) 4678: dc000015 stw r16,0(sp) 467c: 04c00044 movi r19,1 4680: 0025883a mov r18,zero 4684: 10003226 beq r2,zero,4750 <mpeg2_init+0xf0> 4688: 000b883a mov r5,zero 468c: 0111a004 movi r4,18048 4690: 00019bc0 call 19bc <_init+0xbc> <---- here, we must relocate, but... 4694: 01802004 movi r6,128 4698: 000b883a mov r5,zero 469c: 10002426 beq r2,zero,4730 <mpeg2_init+0xd0> .  Of course, if these codes are linked as normal executables, these relocations will not make any troubles.  Kazu

[Altera_Forum]

--- Quote Start ---  No. The cacheable Bit is in the page descriptor of each virtual page.  -Michael  --- Quote End ---   As far as I can tell though, all ioremap_nocache does for physical addresses (that is, all addresses below 0x20000000) is change them to something in the kernel's virtual address range and set bit 31. The processor still understands addresses above 0x80000000 as mirrors of those below that, but uncached. ioremap_nocache breaks though for those addresses, so I made this change: --- a/arch/nios2/mm/ioremap.c +++ b/arch/nios2/mm/ioremap.c @@ -132,7 +132,7 @@ static int remap_area_pages(unsigned long address, return error; } -#define IS_MAPPABLE_UNCACHEABLE(addr) (addr < 0x20000000UL) +#define IS_MAPPABLE_UNCACHEABLE(addr) ((addr < 0x20000000UL) || (addr >= 0x80000000UL && addr < 0xA0000000UL)) /* * Map some physical address range into the kernel address space. @@ -172,10 +172,9 @@ void *__ioremap(unsigned long phys_addr, unsigned long size, int cacheflag) IS_MAPPABLE_UNCACHEABLE(last_addr) && !(cacheflag & _PAGE_CACHED)) { - return (void __iomem *)(0xe0000000UL + phys_addr); + return (void __iomem *)(0xe0000000UL + (phys_addr & 0x1fffffffUL)); }  And also added this (similar to Kazu's altera_tse patches): // This works on NIOS II MMU when ioremap is done as in line 175 of ioremap.c # define virt_to_phys2(vaddr) ((unsigned long) ((unsigned long)vaddr & 0x9fffffffUL))  This solved some other cache coherency problems I was having, but ethernet still doesn't work with 8k caches.  EDIT: this is probably more of a discussion for nios2-dev, but there doesn't seem to be any mail on it this month, is that list alive or has it moved?

[Altera_Forum]

Hmmm, AFAIR, the hardware manual states that with MMU, bit 31 is not use for cache enabling purpose, but that you need to create dedicated MMU pages for non-caching accesses (with the appropriate cacheable bit setting in the page descriptor). But of course nobody prevents you from defining all pages with the virtual address bit 31 set as non-cacheable. I have no idea how all this is done in the Kernel code. 

 -Michael

[Altera_Forum]

kazuyasu (http://www.alteraforum.com/forum/member.php?u=26618),  

 Re: relocation.  Why is all this done with relocatable code with direct calls and not with position independent code using indirect or relative calls ?   (In fact I need to look up what the mnemonic "call 0x2000870" in fact means, as the code dword can't specify a 32 Bit address, so maybe this in fact is the same as "call 0x870" as you maybe suggested. But I'm not in the Office right now...)  -Michael

[Altera_Forum]

--- Quote Start ---  Hmmm, AFAIR, the hardware manual states that with MMU, bit 31 is not use for cache enabling purpose, but that you need to create dedicated MMU pages for non-caching accesses (with the appropriate cacheable bit setting in the page descriptor). But of course nobody prevents you from defining all pages with the virtual address bit 31 set as non-cacheable. I have no idea how all this is done in the Kernel code.  -Michael  --- Quote End ---   OK, I found the manual (http://www.altera.com/literature/hb/nios2/n2cpu_nii51003.pdf), and you are right, bit 31 is not used that way. However, virtual addresses 0xe0000000 and above are are mapped for I/O, skipping the TLB and the cache. So the behavior of ioremap is correct.

[Altera_Forum]

Hi, 

   --- Quote Start ---  OK, I found the manual (http://www.altera.com/literature/hb/nios2/n2cpu_nii51003.pdf), and you are right, bit 31 is not used that way. However, virtual addresses 0xe0000000 and above are are mapped for I/O, skipping the TLB and the cache. So the behavior of ioremap is correct.   --- Quote End ---     Please be careful, because Wind River guys made a little bit confusing, meaningless and sometimes wrong macros around Nios MMU. For examples,   /* * This file is subject to the terms and conditions of the GNU General Public * License. See the file "COPYING" in the main directory of this archive * for more details. * * Copyright (C) 1994 - 1999, 2000, 03, 04 Ralf Baechle * Copyright (C) 2000, 2002 Maciej W. Rozycki * Copyright (C) 1990, 1999, 2000 Silicon Graphics, Inc. */# ifndef _ASM_NIOS2_GENERIC_SPACES_H# define _ASM_NIOS2_GENERIC_SPACES_H # include <asm/nios.h> # define IO_BASE 0xa0000000 <--- ??? #define UNCAC_BASE 0xa0000000 <--- ??? /* * This handles the memory map. * We handle pages at KSEG0 for kernels with 32 bit address space. */# define PAGE_OFFSET (KERNEL_REGION_BASE + DDR2_TOP_BASE) /* * Memory above this physical address will be considered highmem. */# ifndef HIGHMEM_START# define HIGHMEM_START 0x20000000UL# endif # endif /* _ASM_NIOS2_GENERIC_SPACES_H */ .  Kazu

[Altera_Forum]

Hi, 

   --- Quote Start ---   Why is all this done with relocatable code with direct calls and not with position independent code using indirect or relative calls ?    --- Quote End ---     In fact, the compiler will decide the use of direct calls 'call' and relative calls 'callr'. The linker only changes those addresses and can't change those instruction sequence. This means that once the compiler produces a code 'call ****', we don't have any method to change this code to a relative call. When we compile the next program without option switch like '-shared' or '-fPIC',   void func(void) { printf("hello Nios\n"); } the printf is called by a direct call. At its linking time, the linker finds that the 'printf' is included in a shared library, so the linker adds a GOT(global offset table) and the address resolving algorithm, and relocates the target adddress of direct call to the entrance of resolving algorithm. In this 'libnano-X.a' case, programs were compiled without '-shared', but it is linked to a shared library after. The value 0x870 is the entrance address of resolving algorithm, but the shared library is loaded to different addresses, so we need one more relocation at the loading time, but the relocation information has been already expired....    --- Quote Start ---   (In fact I need to look up what the mnemonic "call 0x2000870" in fact means, as the code dword can't specify a 32 Bit address, so maybe this in fact is the same as "call 0x870" as you maybe suggested.   --- Quote End ---     Yes, 'call 0x2000870' is only a mischief of the debugger. The real memory contents are '0008ac40', i.e. 'call 0x870'.  Kazu

[Altera_Forum]

--- Quote Start ---   Wind River guys made a little bit confusing, meaningless and sometimes wrong macros around Nios MMU.   --- Quote End ---   Are those really used somewhere ?????  Kazuyasu, Hippo and other "powers": can we get rid of this ?!?!?!? I feel that is a prominent issue, lest other community work might be affected negatively.   OTOH, even if the area 0xa0000000..0xdfffffff is not defined as "IO (default cacheability=no), but as "Kernel" (default cacheability=yes), maybe part of it might be set as not cacheable. Does not make any sense to me but...  -Michael

[Altera_Forum]

--- Quote Start ---   Yes, 'call 0x2000870' is only a mischief of the debugger. The real memory contents are '0008ac40', i.e. 'call 0x870'.  --- Quote End ---   ... which seems to be quite correct to me on the first view.  So: do you suggest that you are hit by a compiler, linker, or system setup bug here ?  -Michael

[Altera_Forum]

Hi, 

  --- Quote Start ---   Are those really used somewhere ?????   --- Quote End ---    'IO_BASE' is not used anywhere, but 'UNCAC_BASE' had been used before. Please see# 4 and# 5 of this thread. About the clearance of these confusion, I think it's not so difficult but not so easy... (The most difficult thing is 'who will do it?'.:D)    --- Quote Start ---   So: do you suggest that you are hit by a compiler, linker, or system setup bug here ?   --- Quote End ---     I think this is caused by the incomplete implementation of Nios's linker.  Kazu 

[Altera_Forum]

--- Quote Start ---  The most difficult thing is 'who will do it?'.:D  --- Quote End ---  I do hope for Yourself and Hippo ;). I also hope that I can get up to more speed soon after the holidays....   --- Quote Start ---   I think this is caused by the incomplete implementation of Nios's linker.  --- Quote End ---  That might be a disaster, as Hippo can impose pathes on the Linux build system configuration and supposedly on the libraries, but I don't think that anybody is working on the gnu gcc tools themselves, which I suppose are in the state MontVista left them in and nobody yet ever took a look at the internals, if at all the "official" (and verified) sources are available to the community. (Did anybody yet recompile the new gcc-nios ? AFAIK, the current official tool chain still is "binary" <other than the no-MMU-tool-chain>.)  -Michael

[Altera_Forum]

--- Quote Start ---  I have a little bit doubt that Nios CPU with MMU can't have over 4Kbytes caches. There are several ways of connecting method for CPU, cache and MMU. For example,  1) CPU -- cache -- MMU -- Memory 2) CPU -- MMU -- cache -- Memory.  --- Quote End ---   I asked Frank Storm our dealer's Altera FPGA. He said that the reference designs of Altera's do have 2*32K Cache and Altera recommends not not change the cache configuration. So the 4K limit does not seem to exist. Nonetheless I asked him to doublecheck the NIOS MMU/Cache "hardware" regarding this issue.  -Michael