Linux with MMU on NEEK

[Altera_Forum]

Hi, all. 

  I'm testing Linux MMU version, on my NEEK.  http://www.nioswiki.com/linux   It works fine and I can use "bash" shell. This is the evident proof that we are using the true 'fork' instead of 'vfork'.  May be this will depends on the version, but TSE driver claims an error and doesn't work on this design. The error is   ERROR: altera_tse.c:1666: request_mem_region() failed I think that this error is caused by misunderstanding of the usage for the function request_mem_region(). Inside of the request_mem_region(), the function __request_region() is called. If the resource has been already registered, this function returns a non-NULL value, that is the pointer for its resource. But the resource 'sgdma_rx_base' is already registered in the initialization process, so this function returns the 'conflict' and    if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) { is always true. So I made a dirty patch,   if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) { reg_resource = __request_region(&iomem_resource, sgdma_rx_base, sgdma_rx_size, "altera_tse", 0); if (reg_resource != NULL && reg_resource->flags & IORESOURCE_BUSY) { printk(KERN_ERR "ERROR: %s:%d: request_mem_region() failed\n", __FILE__, __LINE__); ret = -EBUSY; goto out_sgdma_rx; } } Moreover, the author is forgetting that the DMA is working in the physical address world, so we need to set the pointers of descripters like  // desc->source = read_addr; desc->source = virt_to_phys(read_addr); // desc->destination = write_addr; desc->destination = virt_to_phys(write_addr); // desc->next = (unsigned int *)next; desc->next = (unsigned int *)((unsigned long)next & 0x1fffffffUL); and so on.   Also the frame buffer fb0 will not work well, because the driver 'altfb.c' is not implemented for Linux with MMU version. So I put some codes for altfb_mmap(), like  /* We implement our own mmap to set MAY_SHARE and add the correct size */ static int altfb_mmap(struct fb_info *info, struct vm_area_struct *vma) { unsigned long phys_addr, phys_size; unsigned long addr; unsigned long size = vma->vm_end - vma->vm_start; unsigned long offset = vma->vm_pgoff << PAGE_SHIFT; // vma->vm_flags |= VM_MAYSHARE | VM_SHARED; // vma->vm_start = info->screen_base; // vma->vm_end = vma->vm_start + info->fix.smem_len; /* check range */ if (vma->vm_pgoff > (~0UL >> PAGE_SHIFT)) return -EINVAL; if (offset + size > altfb_fix.smem_len) return -EINVAL; vma->vm_flags |= VM_IO | VM_RESERVED; addr = vma->vm_start; phys_addr = altfb_fix.smem_start + offset; if ((offset + size) < altfb_fix.smem_len) phys_size = size; else phys_size = altfb_fix.smem_len - offset; vma->vm_page_prot = __pgprot(_PAGE_PRESENT|_PAGE_READ|_PAGE_WRITE); if (remap_pfn_range(vma, addr, phys_addr >> PAGE_SHIFT, phys_size, vma->vm_page_prot)) return -EAGAIN; return 0; } and rewrite the DMA descripters like  desc->next = (void *)virt_to_phys((desc + 1)); So now, I can evoke telnetd and control NEEK through ethernet, and use Nano-X on Linux MMU version, but can't enter ftp session, because 'getservbyname()' function will not work well.   I don't know the directory that the souce of 'getservbyname()' is included. Would anyone please tell me where is it?  Thank you, in advance.

[Altera_Forum]

Hi,  

   --- Quote Start ---   I do hope that this gets into the official distribution, so that we are able to use it just by setting the appropriate config option.    --- Quote End ---     I think that Hippo is the main maintainer of this distri, so he will decide it. (We need a kind of dictator for these things;).)     --- Quote Start ---   Moreover it then would be great to have a Wiki page describing the ways possibilities and how to do this using Eclipse as a gdb frontend.   (Do you know the book "The Art of Debugging with GDB, DDD, and Eclipse" ? )   --- Quote End ---     Of course, I know DDD and Eclipse well. And I think to connect these front-ends are not so hard task. But I have less interest for those things, because I'm not a software programmer. Or rather, I have much interest for 'futex' or 'SMP', though we have many problems for atomic codes and cache coherency. But is it possible?  Please let me know your opinion.   Kazu

[Altera_Forum]

Thank you for the wonderful works, Kazu.  

 Surely, I will commit them to the server. Though I am extremely busy on a hardware project this month.  About Eclipse and DDD, it is almost the same as what is in nommu wiki. Just change the compiler prefix and give the solib path. Threads (NPTL) work.  Cheers, Hippo

[Altera_Forum]

--- Quote Start ---   Or rather, I have much interest for 'futex' or 'SMP', though we have many problems for atomic codes and cache coherency. But is it possible?Kazu  --- Quote End ---  I'm very interested in threaded applications and thus in Futex and thus in atomic user land code.   I do know that Hippo is working on this for the MMU based Linux.  As NIOS does not have atomic operations, we need support by the OS for user-land atomic operations. Hippo is planing to do this by defining an "atomic area" to hold the appropriate supported atomic functions within the RAM page that is used for the MMU-cache and that is located within the FPGA as a TCM (and thus fast and not cacheable). You may want to take a look at the appropriate thread in the NIOS uClinux mailing list, where Hippo and I discussed that issue.   I do intend to help on that area, but right now I'm too busy with other (no-NIOS) stuff. At least I already did create a testing program that can be used to verify and to do some timing tests with atomic code and the various possible variants of doing a MUTETX / FUTEX.   I'm sure Hippo (and I) will be very happy to have you in the boat to make FUTEX work for NIOS / MMU.  SMP is a quite different beast. Hippo decided (IMHO wisely) do delay this issue and first provide a single CPU FUTEX.   I already did discuss (with Hippo and with an Altera FEA) doing SMP-save user-land atomic operations by introducing a "Hardware-FUTEX" by doing a multi-CPU-atomic "compare and xchange" custom instruction. One problem is that a custom instruction can't access memory through the cache and thus it only can decently handle a limited dedicated internal memory area for the "FUTEX"es and thus it only can be used a means to protect the user-code sequences handling the normal memory based Linux-FUTEXes. This two stage method of course asks for some "interesting" code ;).   Supposedly something like that is needed for SMP-Kernel-code, as well, but I did not take a look into that, yet.  At best, Altera would add some decent atomic support for NIOS (e.g. "load locked / store conditional" as introduced with the modern ARM CPUs). But I doubt that this will happen some day soon. Maybe Hippo has or can get some information on Altera's plans on this.  Maybe we should discuss all this on the (IMHO more appropriate) mailng list. Where I'm sure Hippo and others will jump in.  -Michael

[Altera_Forum]

Hi, 

 Thank you, Michael. I've read your post in the mailing list and learned a lot. The next is my humble opinion.   For the 'futex',   If we use no-MMU and non-SMP Nios uClinux system, I think the 'futex' is an easy problem, because we can control the Nios's interrupt enable bit from the userland. In this case, only interrupts and task switches are the factors which destroy the atomicity.  With MMU and non-SMP case. your plan to make custom instructions 'lock1' and 'lock2' are dangerous for the OS's security, because it means that we can control the interrupts from our userland. Moreover, we can't perceive the codes flows in the Nios's pipeline and get the exact timing of accepting interruptions.  It seems to me that the idea of 'Common Area for atomic functions' deprives the flexibility of 'futex'. From the viewpoint of userlands, 'futex' is only an int-type counter variable that is located in a shared memory. We can access it from anywhere (of course from our userland) and the kernel doesn't know it till the contention will occur.   Fundamentally, we should not revise the kernel's machine-independent codes or add new codes to there, and should keep the 'look & feel' of userland interfaces (not only for the 'futex').  So I think that the idea of emulations by 'hardware mutex' like Mr. Ben Twijnstra  http://sopc.et.ntust.edu.tw/pipermail/nios2-dev/2009-february/002439.html  is the best method, though we must change the futex's integer valuable to a pointer that points a 'hardware mutex'. I'm not sure how this will influence the kernel codes, but maybe, we can hide everything under the machine-dependent codes and their macros.    For the possibility of SMP by Nios CPU, I will write it later.  Kazu

[Altera_Forum]

Kazu,  

 With No-MMU, Futex is of very limited interest, as the no-MMU toolchain uses gcc3 while the MMU toolchain uses gcc4. Gcc4 can't compile for no-MMU hardware and gcc3 can't do TLS (Thread loacl storage: "_thread" variables in C) and thus is quite unusable for a decent threaded application, and does not support NPTL (Native Posix Thread Library).  As Linux allows the user land code to create as many Futexes as it desires, the Futex variables need to be located in normal RAM, and thus the standard Futex user-library and Kernel code needs to be used. The only thing that needs to be done architecture-specific is the userland atomic instructions.   Here Hippo suggested to follow the way that the BlackFin people, featuring the same problem, go: defining an "atomic region" that holds the appropriate functions and gets special handling by the Kernel's interrupt code.  This method is well proven and works without any hardware support.   Some hardware support (disabling the interrupt for a limited time that is automatically cut by the hardware) might be constructed at a later time if desired and optionally be enabled when configuring the system. But this should be not discussed right now.  -Michael

[Altera_Forum]

Hi, 

  Thank you for your quick reply.    --- Quote Start ---    As Linux allows the user land code to create as many Futexes as it desires, the Futex variables need to be located in normal RAM, and thus the standard Futex user-library and Kernel code needs to be used. The only thing that needs to be done architecture-specific is the userland atomic instructions.    --- Quote End ---     Of course, I'm thinking not only one 'hardware mutex' but special memory regions that emulate atomic codes. Each futex 'pointer' for their counters points one special 'hardware mutex' respectively. 'Hardware mutex' will limits the amount of futex counters, but do you need those more than one thousand? Of course, if there is a law that prohibits to change the type of futex counters, this method is impossible.     --- Quote Start ---    Here Hippo suggested to follow the way that the BlackFin people, featuring the same problem, go: defining an "atomic region" that holds the appropriate functions and gets special handling by the Kernel's interrupt code.   --- Quote End ---     If the main maintainer supports this plan, I will obey it.    --- Quote Start ---    Some hardware support (disabling the interrupt for a limited time that is automatically cut by the hardware) might be constructed at a later time if desired and optionally be enabled when configuring the system.    --- Quote End ---     I think that 'disabling the interrupt for a limited time' is quite difficult because we can't detect the instruction despatch and know what pipeline stage deals with the interruptions.  Kazu

[Altera_Forum]

--- Quote Start ---   I think that 'disabling the interrupt for a limited time' is quite difficult because we can't detect the instruction despatch and know what pipeline stage deals with the interruptions.  --- Quote End ---   With embedded software we can assume the user processes to be fair, and thus we just should try to avoid a hanging system due to blocked interrupts and so I think a simple timeout of - say - 1000 clock cycles should do here. Of course the user land process would need to re-enable the interrupt as soon as possible.   But I do see another major issue: The interrupt disabling by a custom instruction needs to be done outside of the CPU and thus the timing will be quite sloppy regarding the instructions. So I suppose 100 % atomicness can't be granted and I think the hardware support should be implemented on top of Hippo's suggestion, just in 99,9 % of the cases preventing the necessity for the ISR to bother with the atomic user code, as the ISR is normally not called any more if the user code is in the atomic region.  The mainline macro for an atomic operations would be   ___ custom disable interrupt ___ call appropriate_function_in_atomic_page ___ custom enable interrupt  instead of just  ___ call appropriate_function_in_atomic_page if no custom disable interrupt hardware is available  The Kernel code would stay the same.  This problem might make the SMP implementation even more interesting...  -Michael

[Altera_Forum]

--- Quote Start ---  Hi, all.   I'm testing Linux MMU version, on my NEEK.  http://www.nioswiki.com/linux   It works fine and I can use "bash" shell. This is the evident proof that we are using the true 'fork' instead of 'vfork'.  May be this will depends on the version, but TSE driver claims an error and doesn't work on this design. The error is   ERROR: altera_tse.c:1666: request_mem_region() failed I think that this error is caused by misunderstanding of the usage for the function request_mem_region(). Inside of the request_mem_region(), the function __request_region() is called. If the resource has been already registered, this function returns a non-NULL value, that is the pointer for its resource. But the resource 'sgdma_rx_base' is already registered in the initialization process, so this function returns the 'conflict' and    if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) { is always true. So I made a dirty patch,   if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) { reg_resource = __request_region(&iomem_resource, sgdma_rx_base, sgdma_rx_size, "altera_tse", 0); if (reg_resource != NULL && reg_resource->flags & IORESOURCE_BUSY) { printk(KERN_ERR "ERROR: %s:%d: request_mem_region() failed\n", __FILE__, __LINE__); ret = -EBUSY; goto out_sgdma_rx; } } Moreover, the author is forgetting that the DMA is working in the physical address world, so we need to set the pointers of descripters like  // desc->source = read_addr; desc->source = virt_to_phys(read_addr); // desc->destination = write_addr; desc->destination = virt_to_phys(write_addr); // desc->next = (unsigned int *)next; desc->next = (unsigned int *)((unsigned long)next & 0x1fffffffUL); and so on.   --- Quote End ---    Did you get any of these patches into the distribution? I am starting to work with the MMU and both of the Altera TSE drivers are broken with this version (more so than before ;)). I have fixed the atse.c driver to use the new macros for addresses, but need some more work on it to put ioremap()s in the right places because it uses phycal addresses for MAC control registers and descriptor memory. I am fairly new to driver programming so I am also a bit confused between ioremap, request_mem_region, phys_to_virt and all. Did you have to do anything else to get the altera_tse.c driver working?

[Altera_Forum]

Hi, 

   --- Quote Start ---    Did you have to do anything else to get the altera_tse.c driver working?   --- Quote End ---     Please see and try this.  Kazu  P.S. Why can't we attach *.c and *.h files directly?

[Altera_Forum]

If this code does work why not upload it as a patch ?  

 -Michael

[Altera_Forum]

Hi, 

   --- Quote Start ---   If this code does work why not upload it as a patch ?    --- Quote End ---    May I do such thing without permission? Of course, I'm using this patch in my NEEK without any trouble, e.g. telnet, ftp... but this is only a dirty and temporary patch and not well considered. It may be not robust.  Kazu  P.S. I'm a Japanese, hesitation and modesty are our key characters.:D

[Altera_Forum]

--- Quote Start ---  May I do such thing without permission?   --- Quote End ---   AFAIK, you need a git server account, which Hippo might provide.   --- Quote Start ---  Of course, I'm using this patch in my NEEK without any trouble, e.g. telnet, ftp... but this is only a dirty and temporary patch and not well considered. It may be not robust.  --- Quote End ---  Git is specially created to handle this. There are appropriate branches for this purpose in Hippo's git database or you can create a new branch.   --- Quote Start ---  P.S. I'm a Japanese, hesitation and modesty are our key characters.:D  --- Quote End ---  Seemingly very different from (at least some) Chinese's which I met in several technical forums. I thought they are your close neighbors :p.  -Michael

[Altera_Forum]

--- Quote Start ---  Hi,     Please see and try this.  Kazu  P.S. Why can't we attach *.c and *.h files directly?  --- Quote End ---   Thanks a lot! The driver seems to work at some level now. dhcpcd doesn't work, with either driver I get: dhcpcd: terminating on signal 7 and it takes down eth0 if it was previously brought up with ifconfig. It's a SIGBUS so probably another incorrect memory access issue.

[Altera_Forum]

TO_BE_DONE

[Altera_Forum]

Hi, 

   --- Quote Start ---   dhcpcd doesn't work, with either driver I get: dhcpcd: terminating on signal 7and it takes down eth0 if it was previously brought up with ifconfig. It's a SIGBUS so probably another incorrect memory access issue.  --- Quote End ---     I think that you are using the dhcpcd of 'dhcpcd-new' directory(Mr.Yoichi Hariguchi's dhcpcd will claim 'ioctl SIOCSIFBRDADDR (ifConfig): Cannot assign requested address' and not work).  So I debugged the new dhcpcd and found that the error is caused by calling a function ' memset' in the file 'client.c'  void *dhcpReboot() { dhcpStart(); memset(&DhcpOptions,0,sizeof(DhcpOptions)); // <- This calling will generate a signal SIGSEGV in my debugger. memset(&DhcpIface,0,sizeof(dhcpInterface)); if ( readDhcpCache() ) { This may be a dynamic linking error, but I don't know the reason exactly.  Kazu

[Altera_Forum]

--- Quote Start ---  Hi,     I think that you are using the dhcpcd of 'dhcpcd-new' directory(Mr.Yoichi Hariguchi's dhcpcd will claim 'ioctl SIOCSIFBRDADDR (ifConfig): Cannot assign requested address' and not work).  So I debugged the new dhcpcd and found that the error is caused by calling a function ' memset' in the file 'client.c'  void *dhcpReboot() { dhcpStart(); memset(&DhcpOptions,0,sizeof(DhcpOptions)); // <- This calling will generate a signal SIGSEGV in my debugger. memset(&DhcpIface,0,sizeof(dhcpInterface)); if ( readDhcpCache() ) { This may be a dynamic linking error, but I don't know the reason exactly.  Kazu  --- Quote End ---    Actually, it (dhcpcd-new) wasn't crashing there, but in udpipgen.c in in_cksum. However, dhclient (ISC) works in this version (it didn't compile in the nommu version) so I am using that now. I was trying to get DHCP working because I thought maybe I was just misconfiguring something, but I am still not seeing any communication in or out of the device, so I am back to trying to debug the driver. This hardware design was working before the MMU was added. Now I get no errors or hangs, it's just not sending or receiving anything.  Kazuyasu: in your altera_tse.c, you changed from using the .ndo functions to setting them somewhere else, was there a reason for that?

[Altera_Forum]

Hi, 

   --- Quote Start ---   I was trying to get DHCP working because I thought maybe I was just misconfiguring something, but I am still not seeing any communication in or out of the device, so I am back to trying to debug the driver. This hardware design was working before the MMU was added. Now I get no errors or hangs, it's just not sending or receiving anything.   --- Quote End ---     So I tested 'dhclient' and don't know whether it's fortunate or unfortunate, but it sends DHCP DISCOVER packets from my NEEK. The DHCP server returns an IP address. Of course, I encountered several bugs. 1) The compiler translates the function  memcpy (&from -> sin_addr, &ip -> ip_src, 4); in the file '/common/packet.c' to the next machine code  0x0000aeb0 <decode_udp_ip_header+152>: ldw r2,0(r21) 0x0000aeb4 <decode_udp_ip_header+156>: add r17,r16,r19 0x0000aeb8 <decode_udp_ip_header+160>: stw r2,4(fp) . Unfortunately, the address of 'from -> sin_addr' is word-aligned, so the code 'ldw r2,0(r21)' will evoke SIGBUS error. The detour is to rewrite the function as follows.  { int i; unsigned char *pd = &ip -> ip_src, *ps = &from -> sin_addr; for (i = 0; i < 4; i++) *pd++ = *ps++; } 2) Also the checksum is wrong, I only comment the code out.  // if (usum && usum != sum) { // udp_packets_bad_checksum++; // if (udp_packets_seen > 4 && // (udp_packets_seen / udp_packets_bad_checksum) < 2) { // note ("%d bad udp checksums in %d packets", // udp_packets_bad_checksum, udp_packets_seen); // udp_packets_seen = udp_packets_bad_checksum = 0; // } // return -1; // } Maybe , the function 'checksum' has some bugs. And 'dhclient-script' is not interpreted well. Maybe we need 'bash'.    --- Quote Start ---   in your altera_tse.c, you changed from using the .ndo functions to setting them somewhere else, was there a reason for that?   --- Quote End ---     I can't understand what you mean, but I think that these malfunctions are not caused by the driver.  Kazu

[Altera_Forum]

Ooops ?!?!?!? 

AFAIK, the memcpy() function/macro (builtin_memcpy)) is supposed to handle alignment issues automatically.   If from->sin_addr is a long (or pointer) type, it can assume it is aligned correctly, Thus the compiled code might be correct.   Maybe some erroneous memcpy() macro is active when the code sequence is compiled,.  If it is not, I would give memcpy the correct alignment. e.g.: memcpy ((char*)&from -> sin_addr, (char*)&ip -> ip_src, 4); If the compiled code in fact is erroneous, we would need to have the library macro or compiler fixed....  -Michael

[Altera_Forum]

Hi, 

   --- Quote Start ---   If from->sin_addr is a long (or pointer) type, it can assume it is aligned correctly, Thus the compiled code might be correct.    --- Quote End ---     Yes, you are right. And I think this source will not consider the machines that can not do unaligned-access.    --- Quote Start ---   If it is not, I would give memcpy the correct alignment. e.g.: memcpy ((char*)&from -> sin_addr, (char*)&ip -> ip_src, 4);   --- Quote End ---    Maybe, it's OK. And sorry, I made a mistake for the copy direction.   { int i; //unsigned char *pd = &ip -> ip_src, *ps = &from -> sin_addr; unsigned char *ps = &ip -> ip_src, *pd = &from -> sin_addr; for (i = 0; i < 4; i++) *pd++ = *ps++; } Kazu

[Altera_Forum]

--- Quote Start ---  Hi,     So I tested 'dhclient' and don't know whether it's fortunate or unfortunate, but it sends DHCP DISCOVER packets from my NEEK. The DHCP server returns an IP address. Of course, I encountered several bugs. 1) The compiler translates the function  memcpy (&from -> sin_addr, &ip -> ip_src, 4); in the file '/common/packet.c' to the next machine code  0x0000aeb0 <decode_udp_ip_header+152>: ldw r2,0(r21) 0x0000aeb4 <decode_udp_ip_header+156>: add r17,r16,r19 0x0000aeb8 <decode_udp_ip_header+160>: stw r2,4(fp) . Unfortunately, the address of 'from -> sin_addr' is word-aligned, so the code 'ldw r2,0(r21)' will evoke SIGBUS error. The detour is to rewrite the function as follows.  { int i; unsigned char *pd = &ip -> ip_src, *ps = &from -> sin_addr; for (i = 0; i < 4; i++) *pd++ = *ps++; } 2) Also the checksum is wrong, I only comment the code out.  // if (usum && usum != sum) { // udp_packets_bad_checksum++; // if (udp_packets_seen > 4 && // (udp_packets_seen / udp_packets_bad_checksum) < 2) { // note ("%d bad udp checksums in %d packets", // udp_packets_bad_checksum, udp_packets_seen); // udp_packets_seen = udp_packets_bad_checksum = 0; // } // return -1; // } Maybe , the function 'checksum' has some bugs. And 'dhclient-script' is not interpreted well. Maybe we need 'bash'.   I can't understand what you mean, but I think that these malfunctions are not caused by the driver.  Kazu  --- Quote End ---    That's interesting, I didn't run into a SIGBUS or any crashes with dhclient. dhclient-script didn't get installed by uClinux-dist and I didn't see one there, so I just took the one from my desktop and changed bash to hush, so that could be causing problems. Networking does work on the NEEK using your driver patch and the design on the wiki, contrary to what the wiki says about that design! I can get an address with DHCP using dhcpcd (-new) if I comment out the checksum part that causes the SIGBUS. So it does seem that my problem on the custom hardware is not entirely caused by the driver -- though it did work before adding the MMU. Maybe it's the PHY driver.

[Altera_Forum]

Hi, 

  I repaired my stupid copying bug and changed my shell to 'bash', 'dhclient' works well with UDP checksum, though 'dhclient-script' requires 'sed', 'expr', etc.:p   About the SIGBUS error of checksum, please give me time to debug. I'm only a Sunday programmer.:D But I would appreciate receiving the information of SIGBUS error, by setting  echo 1 > /proc/sys/kernel/print-fatal-signals . Kazu