Ethernet problem turned out to be a memory / DMA problem

[Altera_Forum]

So I have spent a couple weeks trying to troubleshoot Ethernet functionality on my custom board. I finally isolated the problem last night. Ethernet data is getting into the FPGA just fine. I can compare Signaltap data of the MII bus inputs to Wireshark data and they match perfectly. But when I dump the packet contents from RAM, they are garbage. So it appears that the SGDMA controllers are not playing nice with my SDRAM. 

 The interesting thing is my CPU can access RAM just fine without a hiccup. When I run the memory tester in the Eclipse tools, every bit passes. When I run Linux, the kernel is stable as a rock and code does not seem to ever crash. But ethernet data into the system is beautiful in Signaltap but garbage in RAM.   The interesting thing is I get the same behavior in Linux and Eclipse / Micrium / HAL. The TSE can negotiate a connection with the PHY just fine but packets are silently dropped because they are garbage.  So I am guessing I probably have my RAM timing numbers off. I am not using a RAM chip explicity supported in NIOS. I am using a Micron MT48LC16MA2.  How would you troubleshoot this problem?  David.  Do I need to put a scope on the memory pins?  All I know for sure is that Ethernet data is getting into the FPGA intact but not making it into SDRAM intact.

[Altera_Forum]

I would first try to reproduce the problem with a simple test system. 

 Does your memory tester use the SGDMA controller to generate transactions?  Its quite possible that your test cases are not generating the same burst transaction, or combination of burst transactions that the ethernet controller interface is generating.   Could you use SignalTapII to track the transaction of an invalid packet? Eg. put a hardware checksummer in the design, and use the failed checksum to trigger the analyzer. Then look at SignalTapII traces of the memory interface. If you can find the transaction corresponding to the data corruption, this should provide the memory transaction or sequence that leads to the problem. Eg. perhaps there is a refresh that occurs, screwing up the pipeline, or perhaps a problem in one of the FPGA controller FIFOs.  The other thing to try is to use the Micron device model in a testbench, and see if your controller timing generates any violation messages from the memory device model.  If you do have a high-speed scope, then sure, probe the pins. If you don't, then try slowing the memory controller clock down to see if the problem changes or goes away, or try increasing the clock to see if it gets worse (but the SDC analysis must still indicate the interface meets timing of course).  Cheers, Dave

[Altera_Forum]

I know it is a bit obvious - but are you sure you've got the buffer addresses used by the ethernet dma correct, and the cpu cache not interfering with the reads?

[Altera_Forum]

I'm pretty new at this stuff so I welcome any obvious hints... When I look at my memory map the TLB addresses are at a different address than the DMA stuff.  

 Where should I be comparing buffer addresses to? I noticed what appeared to be a bug to me in the Linux SGDMA drivers that they appear to grab a 0x400 wide memory range even though their actual control reigster address is much narrower. Hardcoding address offsets between the DMA's seemed to at least let me send data.  David

[Altera_Forum]

I've not run Linux nor used the ethernet on the fpga, but .... 

 The memory addresses give to the dma controller must be physical addresses, the linux virt_to_phys() function should give you this for memory malloced requesting 'DMA' (for other addresses it will return garbage!)  To get a virtual address for the dma controller itself (and other io) the driver needs to call io_remap() with the physical address.  On the ppc I was last writing linux drivers for, it was difficult to just allocate uncached memory (the calls wanted a dev_t which I didn't have). But the ring structures that describe the ethernet buffers (I think the altera MAC uses them - most do) will need to be in uncached memory (thinks - it might be worth using an M9K block that linux treats as io).  You'll probably need to allocate the transmit and receive buffers themselves from cached memory (although using uncached might help you get started!). That will mean you need to force the cache writes before transmits and invalidates before processing receive buffer. There are probably standard Linux functions to do this - and the driver might be doing them already - but it might be quicker (and teh code will run faster) if you write nios specific functions. After all, you probably don't need the code to be that portable!

[Altera_Forum]

Thank you very much but I just managed to solve my problem and get bidirectional ethernet communication for the first time after nearly a month of working on it. 

 Turns out to be a memory timing problem after all even though the Eclipse memory test project showed no errors.  Turns out that the original phase lead / lag calculations for our SDRAM was done incorrectly. But the interesting thing was it was close enough that normal processor access works just fine but only the DMA transfers from the Ethernet stack were corrupted.  I had initially ruled out RAM as a source of my problems because it passed all processor based RAM checks. In hindsight that was a mistake. Changing the phase shift of my SDRAM clock from -3 nS to -1.5nS solved the problem.   Thanks again, I have learned a lot troubleshooting this problem.  David

[Altera_Forum]

David, 

 I believe that memory test that you're referring to also makes use of a standard Avalon DMA, if you have one in your system. That ~might~ have caught the issue that you saw...  Learning is always good, BTW :-)  Cheers,  Slacker