Attaching DDR as external memory about vicuna HOT 12 CLOSED

Attaching an external memory is very board-specific, thus I have not included it in the demo section. In order to attach an external DDR memory and fill it with application data, you need to implement the following:

1. Create an interface for your external memory (such as by using Vivado's MIG IP). As far as I understand, you have already completed that step.
2. Attach that interface to the memory bus in addition to the existing onboard RAM (the onboard RAM is still needed because only that can be initialized). For that you need to edit demo_top.sv and add the external RAM interface along with the existing ram32 module (the onboard RAM) and the hwreg_iface module (the memory-mapped registers). Connect the input ports of your RAM interface to the mem_* signals in the same fashion as the ram32 and hwreg_iface modules. The request signal mem_req must be guarded by the address range you assign to your external RAM (for instance you could assign your RAM an address range starting from address 0x40000000 to 0x40FFFFFF, since the RAM on the Nexys 4 DDR appears to be 16 MB in size). Also, you need to add signals to receive read data from your external RAM and connect these to mem_rvalid and mem_rdata.
3. Write a small program (usually called a boot program) that can be used to receive data (e.g., via UART) and write it to the external memory. That small program goes into the onboard RAM and its only task will be to initialize the external memory with the data it receives. I have written a small boot program for RISC-V cores, you can find it here: https://github.com/michael-platzer/bootserdow. Unfortunately, there is no documentation but simply cloning that repository and running make inside it should produce a program called bootserdow with a memory initialization file called bootserdow.vmem that can be used to initialize the onboard RAM.
4. Compile the actual application using the Makefile of Vicuna (you will have to edit the linker script and change the RAM address and size to the values that you chose for your external memory in step 2), as you have already done so far, and download that to the board using the boot program. The bootserdow repository contains a Python script called serdow.py that communicates with the boot program and downloads an ELF file to the board (so this time you will use the *.elf file produced during compilation instead of the *.vmem file).

As an additional recommendation, you might want to test steps 3 and 4 (checking whether the boot program and serial downloader actually work by downloading an application to the onboard RAM) before attaching your external DDR memory.

from vicuna.

Thank you @michael-platzer plz keep the issue open so that after I read and search about all the great information you mentioned I can ask possible questions about the topic, and again thank you... You are a big help!

from vicuna.

Hi @michael-platzer I have completed this part as you mentioned and it is working very fine As an additional recommendation, you might want to test steps 3 and 4 (checking whether the boot program and serial downloader actually work by downloading an application to the onboard RAM) before attaching your external DDR memory. (I booted that hello world program you mentioned) Now I am trying to attach MIG to the core and complete the task, so far I have made the demo_top.sv like Mig.zip below and based on the Nexys 4 ddr I have created the Ip like Mig.zip below, by the way I think the ddr2 memory of nexys 4 ddr is 128MiB not 16MB as it says in datasheet but anyway am I in the right path now? I think with the roadmap you mentioned now I have to do second step and then it is done.
Mig.zip

from vicuna.

Hi @michael-platzer I have worked on MIG since previous comment and I have encountered an error when creating .elf file with new linker. I have edited the linker as you said:
MEMORY { boot : ORIGIN = 0x00000000, LENGTH = 0x00002000 /* 8 kB */ ram : ORIGIN = 0x40000000, LENGTH = 0x00FFFFFF /* 16 MB */ }
The test program is hello world as always and the error is like this:

Besides I have updated the demo_top.sv for MIG and it is like below in zip file:
demo_top_mig.zip
I really tried to follow your instructions but since the program has compiling error I couldn't test it...
Can you help on the error of compiler and take a look at demo_top edit?

from vicuna.

It seems that you are on the right track. The linking error is because you set the size of your external memory to 16 MB minus 1 byte. The stack is placed at the end of the RAM but needs to be aligned. Set the size of the RAM to 16 MB (i.e., 0x01000000), that should fix the linker error.

In your demo_top_mig.sv, the MEM_MASK_MIG should be set to 32'h00FFFFFF such that the expression mem_addr & ~MEM_MASK_MIG evaluates to 32'h40000000 (i.e., MEM_MASK_START) for all addresses in the range 32'h40000000 to 32'h40FFFFFF (the valid address range of your external RAM).

Be aware that mem_addr is a byte address that is always aligned to 32-bit (hence mem_addr[1:0] is always 0). The memory address used by your memory's application interface (app_addr) could be a word address (it seems that the RAM uses 64-bit memory words), so you might need to shift the address value accordingly. Have a look at the documentation of the MIG or your external memory to see whether that is true.

Finally, the current implementation assumes that memory access always has a latency of 1 cycle and thus sets mem_rvalid in the cycle following a request (see line 84 of demo_top.sv ). The latency of the external RAM is certainly higher, so you will need to change the logic for mem_rvalid. Also, memory read data must be returned in the order it has been requested. However, since the latencies of the different memory regions are all different, the sram_rvalid, hwreg_rvalid, and app_rd_data_valid might be asserted in a different order than the requests have actually been issued by the core. If the latency of your external memory is constant, then the easiest solution is probably to increase the latencies of the internal RAM and the hardware registers to the same value by buffering their results in a sequence of registers, such that all memory requests always have the same latency.

from vicuna.

Hi @michael-platzer sorry for answering a bit late I was trying to solve the issue but apparently I should ask again:
About linker I did what you said:
MEMORY { boot : ORIGIN = 0x00000000, LENGTH = 0x00002000 /* 8 kB */ ram : ORIGIN = 0x40000000, LENGTH = 0x01000000 /* 16 MB */ }
but it gives the other error (ram_region is solved but second one 'exc_wrapper' is still there):

do you know how can I solve it?
About app_addr the documentation for mig is really confusing but I think it is word address so I should change like below to have 2 lowest bits 0?
.app_addr (mem_addr[28:2]), // input [26:0] app_addr
And for the latency of 1 cycle problem I read the mig doc and it says nothing about constant latencies and I think I should use app_rdy and app_rd_data_valid but don't quit know how...

I know I have asked a lot but if you can suggest something for linker and this issue I will be very thankful

Here is the updated version I corrected some of the problems you implied...
demo_top_mig.zip

from vicuna.

The reason for the linker error is that the exception vectors cannot reach the code that is in the external RAM with a simple jump. Replace the .vectors section in crt0.S with the following code, that should fix the linker error:

    .section .vectors, "ax", @progbits
    .option norvc;

    # redirect all interrupts to the execption wrapper
    .rept 31
    nop
    .endr
    j call_exc_wrapper

    # the reset vector is redirected to _start
    tail _start

call_exc_wrapper:
    tail exc_wrapper

For word addressing the lowest two bits of mem_addr need to be dropped, so this seems fine.

If the external memory does not have constant latency or the documentation does not specify the latency, then you could initially only allow one outstanding memory request at a time.

from vicuna.

Thank you again @michael-platzer I did what you said for the crt0.s file and the error is gone... but just one weird thing:
After I change crt0.s file I compiled the test.c code which has that hello world program, and now the test.vmem file is 2.8GB!!!!!(before this changes on crt0.s and link.ld file the .vmem file was 23.8KB)
It is really a surprise to me , the .elf file is now 11KB which is normal and before all these things for ddr it was 11KB as well... Why .vmem file is much larger now? I know I should write .elf file with serdow.py but I just want to know if everything is right so that I just focus on ddr connections not other possible mistakes.

from vicuna.

Yes, that is normal. The reason is that *.vmem files do not have sections like *.elf files. Therefore, the entire range from address 0x00000000 up to 0x40FFFFFF is written to the *.vmem file (memory values are written as hexadecimal numbers, so the *.vmem file has roughly twice the size of the covered address range, plus some overhead, which works out to something > 2 GB in your case).

You can avoid creating the *.vmem by specifying the *.elf file as target when calling make, i.e.:

make test.elf

from vicuna.

Great I get it now...
By the way I considered most of the changes you mentioned (still not the latency problem unfortunately and I am asking this just to reduce the unknown areas for myself to finally solve this ddr problem) for ddr and I ran the new .elf file of hello world but after running the serdow.py and opening tera term I get this massage rather than hello world:
Python loader:

Tera Term output:

Is this because I haven't considered latency of ddr properly as you mentioned or something else?
demo_top_mig.zip

from vicuna.

Yes, the output you are seeing is from the exception handler in exceptions.c of the bootserdow repository, which reports a series of load access faults, probably as a result of an earlier attempt to write data to the external memory.

The signal mem_rvalid is asserted one cycle after the memory request was issued, and since the external memory has a longer latency a memory error is generated (i.e., mem_err is set).

I would recommend to try simulating the design in order to debug this.

from vicuna.

Thank you @michael-platzer.. I will try to do what you say, by the way I asked https://github.com/lowRISC/ibex/issues/1510 in Ibex repo (since I asked a lot from you I didn't want to bother you again) but they don't answer. I asked the question in stackoverflow https://stackoverflow.com/questions/70766820/ddr2-memory-actual-width-and-understanding as well but again no answers there either.. Do you know the answer of that question I asked? I can post it here as well...

Besides if in demo_top I set parameter real SYSCLK_PER = 10.0, parameter int unsigned PLL_MUL = 10, parameter int unsigned PLL_DIV = 20, then clk of all modules (vicuna,Ibex,URAT) will be 50MHz deducing from this formula:
localparam int unsigned CLK_FREQ = (1_000_000_000.0 / SYSCLK_PER) * PLL_MUL / PLL_DIV;? I am asking this since MIG gives ui_clk signal as clk for the ddr interface and in my settings it will be 150MHz which is higher than 50Mhz of all blocks in your design so 50Mhz can be used for memory bus including MIG as well?

from vicuna.