Mysterious trouble sharing FPGA RAM with HPS

[Altera_Forum]

Hello all 

I'm a hobbyist and I'm trying to learn verilog and FPGA design by building a PDP-8 simulator. I've built the verilog for the PDP8 and it passes all the test code when run on a simulator.  I'm now trying to build it into the DE1-SoC so that the HPS can, via a program written in c, load PDP8 code into the RAM on the FPGA, start the PDP8 CPU which lives on the FPGA and communicate with the PDP8 on the FPGA via a terminal interface.  My problems are at the first integration step. I have a verilog module pdp8_RAM.v which I hope to have built to be a dual-port RAM that is shared by the HPS and the FPGA-PDP8. The code is below:   module pdp8_RAM ( // inputs: // from HPS bridge address, clk, chipselect, reset_n, write_n, writedata, // from PDP8 CPU pdp8_clk_from_CPU, pdp8_reset_from_CPU, pdp8_addr_from_CPU, pdp8_DI_from_CPU, pdp8_start_RD_from_CPU, pdp8_start_WR_from_CPU, // outputs: // to HPS bridge readdata, //, // to PDP8 CPU pdp8_DO_to_CPU, pdp8_RAM_CC_to_CPU ); input address; input clk; input chipselect; input reset_n; input write_n; input writedata; input pdp8_clk_from_CPU; input pdp8_reset_from_CPU; input pdp8_addr_from_CPU; input pdp8_DI_from_CPU; input pdp8_start_RD_from_CPU; input pdp8_start_WR_from_CPU; output reg readdata; output reg pdp8_DO_to_CPU; output wire pdp8_RAM_CC_to_CPU; reg ram ; reg busy; assign pdp8_RAM_CC_to_CPU = ~busy; // HPS RAM access always @ (posedge clk or negedge reset_n) begin if (!reset_n) readdata <= 0; else if (chipselect && !write_n) ram <= writedata; else readdata <= ram; end // PDP8 RAM access always @ (posedge pdp8_clk_from_CPU or posedge pdp8_reset_from_CPU) begin if (pdp8_reset_from_CPU) begin pdp8_DO_to_CPU <= 0; busy <= 1'b0; end else begin pdp8_DO_to_CPU <= ram; if (pdp8_start_RD_from_CPU) busy <= 1'b1; else if (pdp8_start_WR_from_CPU) begin ram <= {4'b0, pdp8_DI_from_CPU}; busy <= 1'b1; end else busy <= 1'b0; end end initial begin ram = 12'o1000; TAD 0 (ac = 1000) ram = 12'o7040; CMA (ac = 6777) ram = 12'o7402; HLT end endmodule   This is wired to the PDP8 CPU on the FPGA and I know these connections work because, if I run it at 1Hz, the LEDs and 7-segment display I've connected to the PDP8 CPU show that it nicely steps through and halts at the HLT at location 00002.  Things get weird when I try to alter the RAM from the HPS.  I used platform designer to hook it up to the GHRD as shown in the figure on the left.   I can build the system and the CPU runs the pre-loaded code in the RAM.  If I try to access the RAM from the HPS, using the code below (I'm only showing the relevant parts - please forgive the ugliness, I'm entirely self-taught on this stuff): First, the pointer to the RAM:  <snip> uint16_t volatile *ram_base_pointer; <snip> h2p_pdp8_ram_addr = virtual_base + ( ( unsigned long )( ALT_LWFPGASLVS_OFST + PDP8_RAM_0_BASE ) & ( unsigned long)( HW_REGS_MASK ) ); ram_base_pointer = (uint16_t volatile *)h2p_pdp8_ram_addr; <snip> void show_memory() { printf("\t 0\t 1\t 2\t 3\t 4\t 5\t 6\t 7\n"); int i; for (i = 0; i < 128; i++) { if ((i % 8) == 0) { printf("%.5o:\t", display_address_pointer); } printf("%.4o\t", *(ram_base_pointer + display_address_pointer)); display_address_pointer++; if ((i % 8) == 7) { printf("\n"); } } } <snip> uint16_t four_digit_ascii_to_octal(char *s, int offset) { int i; uint16_t result = 0; for (i = offset; i < (offset + 4); i++) { result = result + ((s - 48) << ((9 + (3 * offset)) - (3 * i))); } return result; } <snip> void edit_memory() { while(1) { printf("%.4o?", entry_address_pointer); scanf("%s", in_string); if (in_string == 'x') { return; } else { *(ram_base_pointer + entry_address_pointer) = four_digit_ascii_to_octal(in_string, 0); entry_address_pointer++; } } }   Immediately after programming the FPGA, if I run the code above - especially the function show_memory(), I see what I would expect to see - shown in the second image)https://alteraforum.com/forum/attachment.php?attachmentid=15855&stc=1 .  If I edit the memory using the function edit_memory(), it shows the memory as edited. HOWEVER - the CPU behaves as if the memory had not been edited! If I reboot linux and re-run the program, it reports the edited RAM contents. Oddly, if I leave the program running, re-program the FPGA, and read the RAM from the HPS, I now see the original RAM contents.  So, there's some strange disconnect between the RAM on the FPGA and the RAM that the HPS sees. Sometimes, the HPS reads it correctly - when the FPGA is freshly-programmed; other times, like after I've edited the "RAM", the HPS sees it as changed but the FPGA doesn't. It's making me crazy...  I tried adding the volatile qualifier to no effect - but I may be using it wrong.  Please forgive the amateur newbie question; I'm sorry for all the detail and for anything important that I left out.  Any suggestions would be greatly appreciated.  Thanks Brian   https://alteraforum.com/forum/attachment.php?attachmentid=15854&stc=1

[Altera_Forum]

It looks like a problem with the data cache in the HPS. After writing to the RAM you should flush the data cache, and before reading it you should invalidate it. How are you running your software on the HPS? If it is barebone, there should be some cache management functions with the hardware lib provided by Altera.

[Altera_Forum]

--- Quote Start ---  It looks like a problem with the data cache in the HPS. After writing to the RAM you should flush the data cache, and before reading it you should invalidate it. How are you running your software on the HPS? If it is barebone, there should be some cache management functions with the hardware lib provided by Altera.  --- Quote End ---    That sounds exactly like my problem - I had read about caches in processor design but it did not occur to me in this case. I'm not running 'bare metal'; I'm running the c-code through the Ubuntu linux that's part of the GHRD. I'm sorry for my ignorance, but some googling revealed several different ways to 'flush the data cache': cacheflush(char *s, int a, int b) as well as other, more platform-specific commands. It also looks like the cacheflush also invalidates the cache, but it's hard to be sure. What command would you suggest or where's a good place to look for more information?  Thank you for your quick reply! Have a good weekend  Brian

[Daixiwen]

Sorry I'm not familiar with Linux on the SoC platform. I hope someone else can answer that. Can you find any code examples for the HPS platform on Linux? If you find any code sample with a DMA it should be in there, it is also a common problem when dealing with DMAs.

[bwhit2]

Thanks; I did some unsuccessful digging: I found a library in the Quartus install called alt_cache.h but when I got it compiled, it threw an "illegal instruction" error. I the tried the built-in function __clear_cache() with no success.

I think had great success by using the Altera built-in IP megafunctions. I found that onchip_memory works just fine - an avalon bridge to the HPS works fine and controlling the RAM from the FPGA via an "External Bus to Avalon Bridge" had no "stale data" issues. I did not need even to use the volatile keyword in my code or try to clear the cache - it seems that these Altera modules deal properly with the cache. I hope this helps others with this problem. Please let me know if this is not clear or if I can be of further help.