aolofsson / oh Goto Github PK

Reading from 0xf02XX regs results in timeout / external fetch abort.

Ok I know this is not likely to be hardware ... but I now have an updated epiphany driver running in my kernel that handles the interrupts from the mailbox. https://github.com/peteasa/examples/tree/parallella-oh/epiphany/mailbox

Right now I have a strange problem when using it:
I visit each core in the Epiphany chip and from that core write to the mailbox the core id of the core.
In the driver I map the ERX_CFG registers with devm_ioremap_resource() and in the interrupt handler I simply disable the interrupt in ERX_CFG. In the near future I would like a sysfile with the number of messages pending, but for now I keep it simple and from the command line monitor how many interrupts I have seen using the contents of /proc/interrupts.
Before visiting each core I enable the interrupt in the host application using a new ioctl call to the driver. After the core has sent the message back via the mailbox I read the number of messages with a call to ee_read_esys() and read the message to clear the source of the interrupt.
After the first core it seems that something goes wrong with the loading of the epiphany code. So I stopped reading the messages till the end of the run and this is what I get:

            int mbox_lo     = ee_read_esys(ELINK_MAILBOXLO);
            int mbox_hi     = ee_read_esys(ELINK_MAILBOXHI);
            int post_stat   = ee_read_esys(ELINK_MAILBOXSTAT);
            printf ("main(): PRE_STAT=%08x POST_STAT=%08x LO=%08x HI=%08x\n", pre_stat, post_stat, mbox_lo, mbox_hi);

/usr/epiphany/bin/mailbox
main: rows,cols: ( 4, 4)
main: 0: Message from eCore 0x808 ( 0, 0): main: Error in e_load 35476
"Hello World from core 0x808!"
main: 1: Message from eCore 0x809 ( 0, 1): main: Error in e_load 35476
"Hello World from core 0x809!"
main: 2: Message from eCore 0x80a ( 0, 2): main: Error in e_load 35476
...
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=00000808 HI=810f0320
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=00000809 HI=810f0320
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=0000080a HI=810f0320
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=0000080b HI=810f0320
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=00000848 HI=810f0320
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=00000849 HI=810f0320
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=0000084a HI=810f0320
main(): PRE_STAT=00000001 POST_STAT=00000001 LO=0000084b HI=810f0320
...

So for each of the cores I get a message via the mailbox and I also get a good collection of interrupts (actually 17 because I enable the interrupt before checking if I have visited each core)

So this proves that the interrupt handler correctly disables the interrupt and does not affect the user side memory mapping used to read the messages from the cores.

Not sure why when attempting to read the mailbox inside the loop after visiting each core results in failure:

/usr/epiphany/bin/mailbox
main: rows,cols: ( 4, 4)
main(): About to enable interrupt, RxCFG=03e0fe04
[ 95.915738] enable mailbox irq
[ 95.918780] enable_mailbox_irq: state: 0 rxcfg: 3e0fe04
[ 95.924109] enable_mailbox_irq: state: 0 rxcfg after: 13e0fe04
main(): After enable RxCFG=13e0fe04
main: 0: Message from eCore 0x808 ( 0, 0): "Hello World from core 0x808!"
main(): After Interrupt RxCFG=03e0fe04
main(): PRE_STAT=00000001 POST_STAT=00000000 LO=00000808 HI=810f0320
main(): Abo[ 96.109023] enable mailbox irq
ut to enable interrupt, RxCFG=03e0fe0c
[ 96.117587] enable_mailbox_irq: state: 0 rxcfg: 3e0fe0c
[ 96.126361] enable_mailbox_irq: state: 0 rxcfg after: 13e0fe0c
main(): After enable RxCFG=13e0fe0c
main: 1: Message from eCore 0x809 ( 0, 1): ""
main(): After Interrupt RxCFG=13e0fe0c
main(): ERROR: mailbox txfer stopped!

Notice that for some reason I dont get the error in e_load, but also the second e_load does not seem to work because the second core 0x809 does not get a chance to send a mailbox message... The only thing changed between these two runs is adding the ee_read_esys() for Mailbox Lo and Mailbox Hi at the end after all cores have been visited or after visiting each core.

Just to show how odd this problem is I have created a second version of the mailbox host program using ee_read_esys / ee_write_esys to enable the interrupts and now I get two complete cycles of my hello world loop before the e_load fails:

/usr/epiphany/bin/mailbox
main: rows,cols: ( 4, 4)
main: 0: Message from eCore 0x808 ( 0, 0): "Hello World from core 0x808!"
main(): PRE_STAT=00000001 POST_STAT=00000000 LO=00000808 HI=810f0320
main: 1: Message from eCore 0x809 ( 0, 1): "Hello World from core 0x809!"
main(): PRE_STAT=00000001 POST_STAT=00000000 LO=00000809 HI=810f0320
main: 2: Message from eCore 0x80a ( 0, 2): ""
main(): ERROR: mailbox txfer stopped!

Any ideas welcome!

as per fred-test

Consider path from wr_addr (clk_wr domain register) ---> oh_bin2gray (combinational logic) -> oh_dsync (clk_rd domain register)

Glitch may be generated in oh_bin2gray (logic deays) or due to clock skew in wr_addr register
(binary so multiple bits changing at the same time) and may be latched by clk_rd domain register causing failure (incorrect value of wr_addr_gray_sync).

The occurrence of the failure will depend on target device (clock/signal skews) and clock source (if all clocks come from common source via division it will be less likely)

Solution:
Gray code signal must be registered before being sent to clk_rd domain register.
It could be achieved by directly implementing Gray code counter using wr_addr_gray as registered signal and generating wr_addr via oh_gray2bin (both signals are in the same clock domain)

Relevant code:

always @ ( posedge wr_clk or negedge wr_nreset)
if(!wr_nreset)
wr_addr[AW:0] <= 'b0;
else if(wr_en)
wr_addr[AW:0] <= wr_addr[AW:0] + 'd1;
.
.
.
// convert to gray code (only one bit can toggle)
oh_bin2gray #(.DW(AW+1))
wr_b2g (.out (wr_addr_gray[AW:0]),
.in (wr_addr[AW:0]));

// synchronize to read clock
oh_dsync wr_sync[AW:0] (.dout (wr_addr_gray_sync[AW:0]),
.clk (rd_clk),
.nreset(rd_nreset),
.din (wr_addr_gray[AW:0]));

After I switched RX path to MMU from static remap, I started to see sporadic external fetch abort errors when running the mailbox test. (The only test that requires reading from the FPGA elink regs)
I think one error source was a flaky VDD_DSP so I took a step back and left the regulator always on, but there were still issues.
I've been all over the place but in the end I think it boils down to this one stupid line in the mailbox example. DOH!
Running exhaust test now, knock on wood.

e_open(&dev, 0, 0, platform.rows, platform.cols);
e_load_group("emain.elf", &dev, 0, 0, 1, 1, E_FALSE);
e_start_group(&dev);

e_open(&dev, 0, 0, 1, 1);
e_load_group("emain.elf", &dev, 0, 0, 1, 1, E_FALSE);
e_start_group(&dev);

So I loaded the program to core (0,0), but started all cores. Random SRAM data == random instructions == random off-chip access to an invalid address that causes the AXI slave to time out.
This wasn't visible with static remapping since all accesses was forced to [0x8e000000-0x90000000] by the mask.

At any rate, an invalid off-chip access from Epiphany should never bring down the entire system. It's an easy programming error, even I can do it ;)
("External Fetch Abort" in kernel when locks are held requires reboot)

// Ola

I have wired up the mailbox interrupt to the PS system to core0_nFIQ, but when I enable the interrupt he system hangs. To enable the interrupt, set bit [28] of RXCFG to 1. Assuming that we have to install an ISR in calling software from host or something?

An example of mailbox (with polling) can be found in elink/sw/mailbox

Would love to see this work finally:-))

cc @olajep @peteasa

Vivado version: 2015.4
Branch used: stable (the error persists on master)
OS: Centos 7.2

Details: When running the file oh/accelerator/fpga/build.sh a fatal error occurs whenthe command vivado -mode batch -source run.tcl is executed. As far as I know, system_i causes the error since vivado assumes that it is a blackbox.

I'm new to FPGAs and to Vivado SDK so I don't know what to do to reference that source and get rid of the error.

The error:

*** Running vivado
    with args -log system_wrapper.vdi -applog -m64 -messageDb vivado.pb -mode batch -source system_wrapper.tcl -notrace


****** Vivado v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

source system_wrapper.tcl -notrace
Design is defaulting to srcset: sources_1
Design is defaulting to constrset: constrs_1
INFO: [Project 1-479] Netlist was created with Vivado 2015.4
INFO: [Device 21-403] Loading part xc7z020clg400-1
INFO: [Project 1-570] Preparing netlist for logic optimization
INFO: [Opt 31-138] Pushed 0 inverter(s) to 0 load pin(s).
CRITICAL WARNING: [Project 1-486] Could not resolve non-primitive black box cell 'system' instantiated as 'system_i' [/home/kuky_nekoi/Desktop/oh/accelerator/fpga/system.srcs/sources_1/bd/system/hdl/system_wrapper.v:16]
INFO: [Project 1-111] Unisim Transformation Summary:
No Unisim elements were transformed.

link_design: Time (s): cpu = 00:00:06 ; elapsed = 00:00:06 . Memory (MB): peak = 1215.844 ; gain = 214.250 ; free physical = 675 ; free virtual = 12216
Command: opt_design
Attempting to get a license for feature 'Implementation' and/or device 'xc7z020'
INFO: [Common 17-349] Got license for feature 'Implementation' and/or device 'xc7z020'
Running DRC as a precondition to command opt_design

Starting DRC Task
INFO: [DRC 23-27] Running DRC with 2 threads
ERROR: [DRC 23-20] Rule violation (INBB-3) Black Box Instances - Cell 'system_i' of type 'system_i/system' has undefined contents and is considered a black box.  The contents of this cell must be defined for opt_design to complete successfully.
INFO: [Project 1-461] DRC finished with 1 Errors
INFO: [Project 1-462] Please refer to the DRC report (report_drc) for more information.
ERROR: [Vivado_Tcl 4-78] Error(s) found during DRC. Opt_design not run.

Time (s): cpu = 00:00:00.08 ; elapsed = 00:00:00.09 . Memory (MB): peak = 1245.855 ; gain = 22.008 ; free physical = 669 ; free virtual = 12210
INFO: [Common 17-83] Releasing license: Implementation
10 Infos, 0 Warnings, 1 Critical Warnings and 2 Errors encountered.
opt_design failed
ERROR: [Common 17-39] 'opt_design' failed due to earlier errors.

INFO: [Common 17-206] Exiting Vivado at Mon May  8 17:47:04 2017...
[Mon May  8 17:47:08 2017] impl_1 finished
wait_on_run: Time (s): cpu = 00:00:00.18 ; elapsed = 00:00:22 . Memory (MB): peak = 1101.141 ; gain = 0.000 ; free physical = 1032 ; free virtual = 12572
## launch_runs impl_1 -to_step write_bitstream
[Mon May  8 17:47:08 2017] Launched impl_1...
Run output will be captured here: /home/kuky_nekoi/Desktop/oh/accelerator/fpga/system.runs/impl_1/runme.log
## wait_on_run impl_1
[Mon May  8 17:47:08 2017] Waiting for impl_1 to finish...
[Mon May  8 17:47:08 2017] impl_1 finished
INFO: [Common 17-206] Exiting Vivado at Mon May  8 17:47:08 2017...
[ERROR]  : Bitstream parsing error !!!           Unsupported BIT file

This is the new issue that I am hoping to sort. Very odd because even without interrupts enabled the number of messages back from a core seem to be limited. I changed the e-task.c file:

int main(void){
  int *mailbox= (int *) MAILBOX_ADDR; 
  long long message = 0xfedcba9876543210;
  e_memcopy(mailbox, &message, sizeof(message))
  message++;
  e_memcopy(mailbox, &message, sizeof(message));
}

and the e-main process hangs after about 5 loops, with only one write the e-main process hangs after about 9 loops. I looked again at the simulations of the emailbox and can not see why this might happen. It is also possible to run as many other non-mailbox tests between the mailbox test runs.. still you get the same behaviour where on about the 8 or 9th mailbox write from the epiphany core the linux system just hangs with no additional log output and only reboot will fix.

 ./test.sh
INFO: loopcount: 0, row: 0, col: 0
main(0,0): Load core
main(0,0): Core Loaded
PRE_STAT=00000001 POST_STAT=00000001 LO=76543210 HI=fedcba98
PRE_STAT=00000001 POST_STAT=00000000 LO=76543211 HI=fedcba98
INFO: etx_status: 0xf8, etx_config: 0x1, erx_status: 0x8, erx_config: 0x300f004
INFO: erx_idelay0: 0x0, erx_idelay1: 0x0
/home/root/sw/mailbox/test.sh PASSED

 ./test.sh
INFO: loopcount: 0, row: 0, col: 0
main(0,0): Load core
main(0,0): Core Loaded
PRE_STAT=00000001 POST_STAT=00000001 LO=76543210 HI=fedcba98
PRE_STAT=00000001 POST_STAT=00000000 LO=76543211 HI=fedcba98
INFO: etx_status: 0xf8, etx_config: 0x1, erx_status: 0x8, erx_config: 0x300f004
INFO: erx_idelay0: 0x0, erx_idelay1: 0x0
/home/root/sw/mailbox/test.sh PASSED

... two more times then on the fifth run ->

./test.sh
INFO: loopcount: 0, row: 0, col: 0
main(0,0): Load core
main(0,0): Core Loaded
PRE_STAT=00000001 POST_STAT=00000001 LO=76543210 HI=fedcba98
PRE_STAT=00000001 POST_STAT=00000000 LO=76543211 HI=fedcba98
INFO: etx_status: 0xf8, etx_config: 0x1, erx_status: 0x8, erx_config: 0x300f004
INFO: erx_idelay0: 0x0, erx_idelay1: 0x0
/home/root/sw/mailbox/test.sh PASSED
root@parallella-hdmi:~/sw/mailbox# ./test.sh
INFO: loopcount: 0, row: 0, col: 0
main(0,0): Load core

it hangs up... reboot needed to recover

Tried sending two messages from each of 5 cores and the same behaviour occurs when the 5th core is loaded the system hangs with no log messages and you have to reboot to recover.

As mentioned running any of the tests (hello_world, idelay etc) after the 4th run has no effect. Next time you run the mailbox test and send messages from any core to the mailbox the system hangs up.

I created an axi mailbox dut and tried out the same read test I tried earlier and still found the same problems

Description: erx_cfgif raises mi_match correctly and mi_cfg_en is raised so that the mailbox read cycle is started, however the mailbox fifo only outputs the data on the rising edge of erx_mailbox.rd_clk and at the same time the erx_mailbox.mailbox_fifo_data with the old content gets sampled and sent to mi_dout. This can be seen in the attached diagram because the mailbox_fifo_data to the left of the marker gets sampled in the rising edge of the clock and output on mi_dout, however on the next clock cycle the correct data is available on mailbox_fifo_data.

Potential fix is to create a erx_mailbox valid signal emailbox_access_out that must then delay erx_cfgif.access_out so that the correct mailbox data gets delivered to the rxrr fifo. Last time I tried this fix the simulations failed because changing ecfg_if to take account of the emailbox_access_out affects the etx_cfgif block also. So it is a large change. Perhaps now it is time to introduce this valid signal for all the ecfg_if.mi_doutX channels?

ERROR: [Synth 8-439] module 'dsync' not found - sources_1/ipshared/www.parallella.org/parallella_base_v1_0/258848fc/src/emaxi.v:424]
ERROR: [Synth 8-285] failed synthesizing module 'emaxi'
ERROR: [Synth 8-285] failed synthesizing module 'axi_elink'
ERROR: [Synth 8-285] failed synthesizing module 'parallella_base'
ERROR: [Synth 8-285] failed synthesizing module 'system_parallella_base_0_0'
ERROR: [Synth 8-285] failed synthesizing module 'system'
ERROR: [Synth 8-285] failed synthesizing module 'system_wrapper'

(read on github)

Writes to mailbox when mailbox is full is destructive. Internal write pointer overflows?
I suggest we just drop writes to the mailbox when mailbox_full is asserted (that signal seems to work). Add a sticky (write to clear) overflow bit to E_MAILBOXSTATUS to detect. Might also need a mailbox_half_full interrupt. Driver can maintain a larger FIFO in RAM (if we need it).

FIFO depth = 32

Ignore 'hi'

32 x str = 32 entries can be read back from mailbox until status.not_empty == 0
As expected.

status: 0x00000003 lo: 00000000 hi: 0x00000660
status: 0x00000001 lo: 0x00000001 hi: 0x810f0730
status: 0x00000001 lo: 0x00000002 hi: 0x00007ff0
status: 0x00000001 lo: 0x00000003 hi: 0x00000003
...
status: 0x00000001 lo: 0x0000001e hi: 0x5555001e
status: 0x00000001 lo: 0x0000001f hi: 0x810f0730

33 x str = 33 entries read back from mailbox until status.not_empty == 0
As can be seen first entry (lo: 0x00000000) is overwritten by the last write. Also mailbox_full bit in status is set after first readout.

status: 0x00000001 lo: 0x00000020 hi: 0x810f0730
status: 0x00000003 lo: 0x00000001 hi: 0xffffffe0
status: 0x00000001 lo: 0x00000002 hi: 0x00000002
status: 0x00000001 lo: 0x00000003 hi: 0x00007ff0
status: 0x00000001 lo: 0x00000004 hi: 0x00000019
status: 0x00000001 lo: 0x00000005 hi: 0x00000005
status: 0x00000001 lo: 0x00000006 hi: 0x00000007
status: 0x00000001 lo: 0x00000007 hi: 0x00000007
status: 0x00000001 lo: 0x00000008 hi: 0x810f0730
status: 0x00000001 lo: 0x00000009 hi: 0xffffffe8
status: 0x00000001 lo: 0x0000000a hi: 0x0000000a
status: 0x00000001 lo: 0x0000000b hi: 0x00007ff0
status: 0x00000001 lo: 0x0000000c hi: 0x00000005
status: 0x00000001 lo: 0x0000000d hi: 0x0000000d
status: 0x00000001 lo: 0x0000000e hi: 0x0000000f
status: 0x00000001 lo: 0x0000000f hi: 0x0000000f
status: 0x00000001 lo: 0x00000010 hi: 0x810f0730
status: 0x00000001 lo: 0x00000011 hi: 0xfffffff0
status: 0x00000001 lo: 0x00000012 hi: 0x00000012
status: 0x00000001 lo: 0x00000013 hi: 0x00007ff0
status: 0x00000001 lo: 0x00000014 hi: 0x0000000d
status: 0x00000001 lo: 0x00000015 hi: 0x00000015
status: 0x00000001 lo: 0x00000016 hi: 0x00000017
status: 0x00000001 lo: 0x00000017 hi: 0x00000017
status: 0x00000001 lo: 0x00000018 hi: 0x810f0730
status: 0x00000001 lo: 0x00000019 hi: 0xfffffff8
status: 0x00000001 lo: 0x0000001a hi: 0x0000001a
status: 0x00000001 lo: 0x0000001b hi: 0x00007ff0
status: 0x00000001 lo: 0x0000001c hi: 0x00000015
status: 0x00000001 lo: 0x0000001d hi: 0x0000001d
status: 0x00000001 lo: 0x0000001e hi: 0x0000001f
status: 0x00000001 lo: 0x0000001f hi: 0x0000001f
status: 0x00000001 lo: 0x00000020 hi: 0x810f0730

63 x str = 63 entries read back from mailbox until status.not_empty == 0
First 31 entries (0x0-0x1e) is overwritten by the last 31 writes (0x20-0x3e)

status: 0x00000001 lo: 0x00000020 hi: 0x00000020
status: 0x00000001 lo: 0x00000021 hi: 0x00007ff0
status: 0x00000001 lo: 0x00000022 hi: 0x00000022
status: 0x00000001 lo: 0x00000023 hi: 0x0000001a
status: 0x00000001 lo: 0x00000024 hi: 0x00000024
status: 0x00000001 lo: 0x00000025 hi: 0x0000001c
status: 0x00000001 lo: 0x00000026 hi: 0x00000026
status: 0x00000001 lo: 0x00000027 hi: 0x0000001e
status: 0x00000001 lo: 0x00000028 hi: 0x00000028
status: 0x00000001 lo: 0x00000029 hi: 0x0000002a
status: 0x00000001 lo: 0x0000002a hi: 0x0000002a
status: 0x00000001 lo: 0x0000002b hi: 0x00007ff0
status: 0x00000001 lo: 0x0000002c hi: 0x0000002c
status: 0x00000001 lo: 0x0000002d hi: 0x00000024
status: 0x00000001 lo: 0x0000002e hi: 0x0000002e
status: 0x00000001 lo: 0x0000002f hi: 0x00000026
status: 0x00000001 lo: 0x00000030 hi: 0x00000030
status: 0x00000001 lo: 0x00000031 hi: 0x00000028
status: 0x00000001 lo: 0x00000032 hi: 0x00000032
status: 0x00000001 lo: 0x00000033 hi: 0x00000034
status: 0x00000001 lo: 0x00000034 hi: 0x00000034
status: 0x00000001 lo: 0x00000035 hi: 0x00007ff0
status: 0x00000001 lo: 0x00000036 hi: 0x00000036
status: 0x00000001 lo: 0x00000037 hi: 0x0000002e
status: 0x00000001 lo: 0x00000038 hi: 0x00000038
status: 0x00000001 lo: 0x00000039 hi: 0x00000030
status: 0x00000001 lo: 0x0000003a hi: 0x0000003a
status: 0x00000001 lo: 0x0000003b hi: 0x00000032
status: 0x00000001 lo: 0x0000003c hi: 0x0000003c
status: 0x00000001 lo: 0x0000003d hi: 0x0000003e
status: 0x00000001 lo: 0x0000003e hi: 0x0000003e
status: 0x00000003 lo: 0x0000001f hi: 0x00000020
status: 0x00000001 lo: 0x00000020 hi: 0x00000020
status: 0x00000001 lo: 0x00000021 hi: 0x00007ff0
status: 0x00000001 lo: 0x00000022 hi: 0x00000022
status: 0x00000001 lo: 0x00000023 hi: 0x0000001a
status: 0x00000001 lo: 0x00000024 hi: 0x00000024
status: 0x00000001 lo: 0x00000025 hi: 0x0000001c
status: 0x00000001 lo: 0x00000026 hi: 0x00000026
status: 0x00000001 lo: 0x00000027 hi: 0x0000001e
status: 0x00000001 lo: 0x00000028 hi: 0x00000028
status: 0x00000001 lo: 0x00000029 hi: 0x0000002a
status: 0x00000001 lo: 0x0000002a hi: 0x0000002a
status: 0x00000001 lo: 0x0000002b hi: 0x00007ff0
status: 0x00000001 lo: 0x0000002c hi: 0x0000002c
status: 0x00000001 lo: 0x0000002d hi: 0x00000024
status: 0x00000001 lo: 0x0000002e hi: 0x0000002e
status: 0x00000001 lo: 0x0000002f hi: 0x00000026
status: 0x00000001 lo: 0x00000030 hi: 0x00000030
status: 0x00000001 lo: 0x00000031 hi: 0x00000028
status: 0x00000001 lo: 0x00000032 hi: 0x00000032
status: 0x00000001 lo: 0x00000033 hi: 0x00000034
status: 0x00000001 lo: 0x00000034 hi: 0x00000034
status: 0x00000001 lo: 0x00000035 hi: 0x00007ff0
status: 0x00000001 lo: 0x00000036 hi: 0x00000036
status: 0x00000001 lo: 0x00000037 hi: 0x0000002e
status: 0x00000001 lo: 0x00000038 hi: 0x00000038
status: 0x00000001 lo: 0x00000039 hi: 0x00000030
status: 0x00000001 lo: 0x0000003a hi: 0x0000003a
status: 0x00000001 lo: 0x0000003b hi: 0x00000032
status: 0x00000001 lo: 0x0000003c hi: 0x0000003c
status: 0x00000001 lo: 0x0000003d hi: 0x0000003e
status: 0x00000001 lo: 0x0000003e hi: 0x0000003e

64 x str = 0 entries read back from mailbox until status.not_empty == 0
96 x str = 32 entries (0x40-0x5f) read back from mailbox until status.not_empty == 0
...

I notice you have a xilibs directory for "Xilinx simulation models". You should be pleased to find out that Xilinx have released their models under an Apache 2.0 licensed at https://github.com/Xilinx/XilinxUnisimLibrary and we are working to make them compatible with then open source simulation tools at https://github.com/SymbiFlow/XilinxUnisimLibrary

I can't find any documentation regarding ipx::add_bus_interface and Vivado only spits out XML from the gui... I have seen ADi using it but not clear on the syntax/use. Seems like fighting the tools...

I am ok with wiring up all of the wires explitly and won't be diving into this. If someone is interested in creating a more elegant solution with an "RX" interface instead of {rx_data_p[7:0], rx_data_n[7:0], rxi_frame_p, rxi_lclk_p, rxi_lclk_n, etc..} I am sure some people would appreciate.

My philosophy is to throw everything into "parallella_base" and to use the IP integrator to hook up AXI blocks only.

Until MMU table entries have a valid bit we must initialize all RXMMU table entries to point to a safe 1MB region.
When I try to do that, the system freezes.
[0x31a00000-0x31b00000] is a 1MB contiguous reserved region (dma_alloc_coherent()).
This is reproducible, always gets stuck on the same write.
Inserting sleeps doesn't make any difference.

Log:

[ 1173.299226] epiphany elink0: elink_reset: enter
[ 1173.303742] pkt: 00000000_00000003_810f0200_05_0000
[ 1173.309198] pkt: 00000000_00000000_810f0200_05_0000
[ 1173.314728] pkt: 00000000_ffffffff_810f0208_05_0000
[ 1173.319539] pkt: 00000000_aaaaaaaa_810f0310_05_0000
[ 1173.324386] pkt: 00000000_0000000a_810f0314_05_0000
[ 1173.329271] pkt: 00000000_00000001_810f0210_05_0000
[ 1173.334118] pkt: 00000000_00000000_810f0300_05_0000
[ 1173.338984] epiphany elink0: elink_update_mmu_mappings: enter
[ 1173.344717] epiphany elink0: elink_update_mmu_mappings: Initializing MMU table to safe 1MB region
[ 1173.353564] pkt: 00000000_000003a1_810e8000_05_0000
[ 1173.358430] pkt: 00000000_00000000_810e8004_05_0000
[ 1173.363278] pkt: 00000000_000003a1_810e8008_05_0000
[ 1173.368144] pkt: 00000000_00000000_810e800c_05_0000
[ 1173.373010] pkt: 00000000_000003a1_810e8010_05_0000
[ 1173.377858] pkt: 00000000_00000000_810e8014_05_0000
[ 1173.382724] pkt: 00000000_000003a1_810e8018_05_0000
[ 1173.387590] pkt: 00000000_00000000_810e801c_05_0000
[ 1173.392456] pkt: 00000000_000003a1_810e8020_05_0000
[ 1173.397304] pkt: 00000000_00000000_810e8024_05_0000
[ 1173.402170] pkt: 00000000_000003a1_810e8028_05_0000
[ 1173.407036] pkt: 00000000_00000000_810e802c_05_0000
[ 1173.411884] pkt: 00000000_000003a1_810e8030_05_0000
[ 1173.416750] pkt: 00000000_00000000_810e8034_05_0000
[ 1173.421616] pkt: 00000000_000003a1_810e8038_05_0000
[ 1173.426482] pkt: 00000000_00000000_810e803c_05_0000
[ 1173.431330] pkt: 00000000_000003a1_810e8040_05_0000
[ 1173.436196] pkt: 00000000_00000000_810e8044_05_0000
[ 1173.441062] pkt: 00000000_000003a1_810e8048_05_0000
[ 1173.445909] pkt: 00000000_00000000_810e804c_05_0000
[ 1173.450776] pkt: 00000000_000003a1_810e8050_05_0000
[ 1173.455642] pkt: 00000000_00000000_810e8054_05_0000
[ 1173.460508] pkt: 00000000_000003a1_810e8058_05_0000
[ 1173.465355] pkt: 00000000_00000000_810e805c_05_0000
[ 1173.470221] pkt: 00000000_000003a1_810e8060_05_0000
[ 1173.475088] pkt: 00000000_00000000_810e8064_05_0000
[ 1173.479935] pkt: 00000000_000003a1_810e8068_05_0000
[ 1173.484801] pkt: 00000000_00000000_810e806c_05_0000
[ 1173.489667] pkt: 00000000_000003a1_810e8070_05_0000
[ 1173.494515] pkt: 00000000_00000000_810e8074_05_0000
[ 1173.499381] pkt: 00000000_000003a1_810e8078_05_0000
[ 1173.504247] pkt: 00000000_00000000_810e807c_05_0000
[ 1173.509113] pkt: 00000000_000003a1_810e8080_05_0000
[ 1173.513961] pkt: 00000000_00000000_810e8084_05_0000
[ 1173.518827] pkt: 00000000_000003a1_810e8088_05_0000
[ 1173.523693] pkt: 00000000_00000000_810e808c_05_0000
[ 1173.528541] pkt: 00000000_000003a1_810e8090_05_0000
[ 1173.533407] pkt: 00000000_00000000_810e8094_05_0000
[ 1173.538273] pkt: 00000000_000003a1_810e8098_05_0000
[ 1173.543139] pkt: 00000000_00000000_810e809c_05_0000
[ 1173.547987] pkt: 00000000_000003a1_810e80a0_05_0000
[ 1173.552853] pkt: 00000000_00000000_810e80a4_05_0000
[ 1173.557719] pkt: 00000000_000003a1_810e80a8_05_0000
[ 1173.562585] pkt: 00000000_00000000_810e80ac_05_0000
[ 1173.567433] pkt: 00000000_000003a1_810e80b0_05_0000
[ 1173.572299] pkt: 00000000_00000000_810e80b4_05_0000
[ 1173.577165] pkt: 00000000_000003a1_810e80b8_05_0000
[ 1173.582013] pkt: 00000000_00000000_810e80bc_05_0000
[ 1173.586879] pkt: 00000000_000003a1_810e80c0_05_0000
[ 1173.591745] pkt: 00000000_00000000_810e80c4_05_0000
[ 1173.596592] pkt: 00000000_000003a1_810e80c8_05_0000
[ 1173.601458] pkt: 00000000_00000000_810e80cc_05_0000
[ 1173.606325] pkt: 00000000_000003a1_810e80d0_05_0000
[ 1173.611191] pkt: 00000000_00000000_810e80d4_05_0000
[ 1173.616038] pkt: 00000000_000003a1_810e80d8_05_0000
[ 1173.620904] pkt: 00000000_00000000_810e80dc_05_0000
[ 1173.625770] pkt: 00000000_000003a1_810e80e0_05_0000
[ 1173.630637] pkt: 00000000_00000000_810e80e4_05_0000
[ 1173.635484] pkt: 00000000_000003a1_810e80e8_05_0000
[ 1173.640350] pkt: 00000000_00000000_810e80ec_05_0000
[ 1173.645216] pkt: 00000000_000003a1_810e80f0_05_0000
[ 1173.650064] pkt: 00000000_00000000_810e80f4_05_0000
[ 1173.654930] pkt: 00000000_000003a1_810e80f8_05_0000
[ 1173.659796] pkt: 00000000_00000000_810e80fc_05_0000
[ 1173.664644] pkt: 00000000_000003a1_810e8100_05_0000
[ 1173.669510] pkt: 00000000_00000000_810e8104_05_0000
[ 1173.674376] pkt: 00000000_000003a1_810e8108_05_0000
[ 1173.679242] pkt: 00000000_00000000_810e810c_05_0000
[ 1173.684108] pkt: 00000000_000003a1_810e8110_05_0000
[ 1173.688956] pkt: 00000000_00000000_810e8114_05_0000
[ 1173.693822] pkt: 00000000_000003a1_810e8118_05_0000
[ 1173.698688] pkt: 00000000_00000000_810e811c_05_0000
[ 1173.703536] pkt: 00000000_000003a1_810e8120_05_0000
[ 1173.708402] pkt: 00000000_00000000_810e8124_05_0000
[ 1173.713268] pkt: 00000000_000003a1_810e8128_05_0000
[ 1173.718116] pkt: 00000000_00000000_810e812c_05_0000
[ 1173.722982] pkt: 00000000_000003a1_810e8130_05_0000
[ 1173.727848] pkt: 00000000_00000000_810e8134_05_0000
[ 1173.732714] pkt: 00000000_000003a1_810e8138_05_0000
[ 1173.737562] pkt: 00000000_00000000_810e813c_05_0000
[ 1173.742428] pkt: 00000000_000003a1_810e8140_05_0000
[ 1173.747294] pkt: 00000000_00000000_810e8144_05_0000
[ 1173.752141] pkt: 00000000_000003a1_810e8148_05_0000
[ 1173.757007] pkt: 00000000_00000000_810e814c_05_0000
[ 1173.761874] pkt: 00000000_000003a1_810e8150_05_0000
[ 1173.766740] pkt: 00000000_00000000_810e8154_05_0000
[ 1173.771587] pkt: 00000000_000003a1_810e8158_05_0000
[ 1173.776453] pkt: 00000000_00000000_810e815c_05_0000
[ 1173.781319] pkt: 00000000_000003a1_810e8160_05_0000
[ 1173.786167] pkt: 00000000_00000000_810e8164_05_0000
[ 1173.791033] pkt: 00000000_000003a1_810e8168_05_0000
[ 1173.795899] pkt: 00000000_00000000_810e816c_05_0000
[ 1173.800765] pkt: 00000000_000003a1_810e8170_05_0000
[ 1173.805613] pkt: 00000000_00000000_810e8174_05_0000
[ 1173.810479] pkt: 00000000_000003a1_810e8178_05_0000
[ 1173.815345] pkt: 00000000_00000000_810e817c_05_0000
[ 1173.820211] pkt: 00000000_000003a1_810e8180_05_0000
[ 1173.825059] pkt: 00000000_00000000_810e8184_05_0000
[ 1173.829925] pkt: 00000000_000003a1_810e8188_05_0000
[ 1173.834791] pkt: 00000000_00000000_810e818c_05_0000
[ 1173.839657] pkt: 00000000_000003a1_810e8190_05_0000
[ 1173.844486] pkt: 00000000_00000000_810e8194_05_0000
[ 1173.849371] pkt: 00000000_000003a1_810e8198_05_0000
[ 1173.854219] pkt: 00000000_00000000_810e819c_05_0000
[ 1173.859085] pkt: 00000000_000003a1_810e81a0_05_0000
[ 1173.863951] pkt: 00000000_00000000_810e81a4_05_0000
[ 1173.868817] pkt: 00000000_000003a1_810e81a8_05_0000
[ 1173.873665] pkt: 00000000_00000000_810e81ac_05_0000
[ 1173.878531] pkt: 00000000_000003a1_810e81b0_05_0000
[ 1173.883397] pkt: 00000000_00000000_810e81b4_05_0000
[ 1173.888263] pkt: 00000000_000003a1_810e81b8_05_0000
[ 1173.893111] pkt: 00000000_00000000_810e81bc_05_0000
[ 1173.897977] pkt: 00000000_000003a1_810e81c0_05_0000
[ 1173.902843] pkt: 00000000_00000000_810e81c4_05_0000
[ 1173.907690] pkt: 00000000_000003a1_810e81c8_05_0000
[ 1173.912556] pkt: 00000000_00000000_810e81cc_05_0000
[ 1173.917423] pkt: 00000000_000003a1_810e81d0_05_0000
[ 1173.922270] pkt: 00000000_00000000_810e81d4_05_0000
[ 1173.927136] pkt: 00000000_000003a1_810e81d8_05_0000
[ 1173.932002] pkt: 00000000_00000000_810e81dc_05_0000
[ 1173.936868] pkt: 00000000_000003a1_810e81e0_05_0000
[ 1173.941716] pkt: 00000000_00000000_810e81e4_05_0000
[ 1173.946582] pkt: 00000000_000003a1_810e81e8_05_0000
[ 1173.951448] pkt: 00000000_00000000_810e81ec_05_0000
[ 1173.956314] pkt: 00000000_000003a1_810e81f0_05_0000
[ 1173.961162] pkt: 00000000_00000000_810e81f4_05_0000
[ 1173.966028] pkt: 00000000_000003a1_810e81f8_05_0000
[ 1173.970894] pkt: 00000000_00000000_810e81fc_05_0000
[ 1173.975742] pkt: 00000000_000003a1_810e8200_05_0000
[ 1173.980608] pkt: 00000000_00000000_810e8204_05_0000
[ 1173.985474] pkt: 00000000_000003a1_810e8208_05_0000
[ 1173.990322] pkt: 00000000_00000000_810e820c_05_0000
[ 1173.995188] pkt: 00000000_000003a1_810e8210_05_0000
[ 1174.000054] pkt: 00000000_00000000_810e8214_05_0000
[ 1174.004920] pkt: 00000000_000003a1_810e8218_05_0000
[ 1174.009768] pkt: 00000000_00000000_810e821c_05_0000
[ 1174.014634] pkt: 00000000_000003a1_810e8220_05_0000
[ 1174.019481] pkt: 00000000_00000000_810e8224_05_0000
[ 1174.024366] pkt: 00000000_000003a1_810e8228_05_0000
[ 1174.029214] pkt: 00000000_00000000_810e822c_05_0000
[ 1174.034080] pkt: 00000000_000003a1_810e8230_05_0000
[ 1174.038946] pkt: 00000000_00000000_810e8234_05_0000
[ 1174.043793] pkt: 00000000_000003a1_810e8238_05_0000
[ 1174.048660] pkt: 00000000_00000000_810e823c_05_0000
[ 1174.053526] pkt: 00000000_000003a1_810e8240_05_0000
[ 1174.058373] pkt: 00000000_00000000_810e8244_05_0000
[ 1174.063239] pkt: 00000000_000003a1_810e8248_05_0000
[ 1174.068105] pkt: 00000000_00000000_810e824c_05_0000

Hard freeze

I need to insert nops between writes to the mailbox from Epiphany.

Demonstrated in #84

ERROR: [Place 30-640] Place Check : This design requires more MMCME2_ADV cells than are available in the target device. This design requires 3 of such cell types but only 2 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device.

The above error was seen when attempting to build the hdmi version of the fpga for the xc7z010clg400-1 part. Note I have added in the ADI hdmi modules so this error is unlikely to be seen when building a headless version.

Double checked this issue.. built the ADI adv7511 project and found that it uses 2 MMCME2_ADV cells (clk_wiz, axi_clkgen). built the oh headless version and found that it uses 1 MMCME2_ADV cell (etx_clocks)..

Might try out workround suggested in http://forums.parallella.org/viewtopic.php?f=10&t=1069&start=30

Just a quick reminder to push common/dv/dv_top.v and common/dv/libs.c these files do not appear but are required by elink/dv/build.sh

Antmicro recently released a sphinx domain for Verilog which enables you to write software style documentation for your Verilog / SystemVerilog code. When you combined it with a few other sphinx extension listed below, you get very nice looking docs.

• https://github.com/SymbiFlow/sphinx-verilog-domain
• https://github.com/SymbiFlow/sphinxcontrib-hdl-diagrams
• https://github.com/SymbiFlow/sphinx_symbiflow_theme
• https://github.com/SymbiFlow/symbolator
• https://github.com/SymbiFlow/sphinxcontrib-session
• sphinxcontrib-wavedrom

Some examples are;

• SymbiFlow documentation - https://SymbiFlow.rtfd.io
• SkyWater-pdk documentation - https://SkyWater-pdk.rtfd.io
• Fomu workshop - https://workshop.fomu.im

Are there any plans to publish versioned releases to a library network like OpenCores? I don't know of any other such sites for HDL, but I'm curious because there are some advantages to gathering up "finished product" into a community-vetted index.

We need to support split access to the low/high 32-bits of every register for ARMv7 (and other archs that only supports 32-bit width memory operations).
Make it a compile time option?

Otherwise you need to cache the state somewhere globally

Hi, All

The project in oh/src/parallella/fpga/parallella_accelerator is build fail, and because of missing oh/src/parallella/hdl/parallella_accelerator.v

Transactions seem to work as long as the FIFO in mtx.v are never over-filled.
Adding a long sleep on every 32rd (FIFO depth) access and the problem disappears.

Tested with this commit:
53dea82

At least Epiphany DMA read from ERAM seems broken. Seems to get stuck and then if ARM reads from Epiphany the timeout circuitry kicks in and kills the program with an external abort exception.

not empty flag is never updated

IIRC

[0x8e000000] == 0x87654321

mov r0, 0
movt r0, 0x8e00
ldrh r0,[r0]

This results in r0 now holding 0x8e004321, when it should be 0x00004321.

Will the platform field in ELINK_VERSION be the same as in the old elink?

enum elink_platform {
        E_PLATF_INVAL = 0,
        E_PLATF_E16_7Z020_GPIO,
        E_PLATF_E16_7Z020_NO_GPIO,
        E_PLATF_E16_7Z010_GPIO,
        E_PLATF_E16_7Z010_NO_GPIO,
        E_PLATF_E64_7Z020_GPIO,
        E_PLATF_MAX
};

// Ola

With this commit:
5dc9375

Setup

ELINK_RXCFG = { .mmu_enable = 1, .remap_mode = 2} = 0xA

Memory mappings written to :ELINK_RXMMU table

ERAM [0x8e000000-90000000] --> [0x3e000000-0x40000000]
addr=0x810ec700 val=0x000003e0
addr=0x810ec704 val=0x00000000
addr=0x810ec708 val=0x000003e1
addr=0x810ec70c val=0x00000000
addr=0x810ec710 val=0x000003e2
addr=0x810ec714 val=0x00000000
...
addr=0x810ec7f0 val=0x000003fe
addr=0x810ec7f4 val=0x00000000
addr=0x810ec7f8 val=0x000003ff
addr=0x810ec7fc val=0x00000000

FPGA ELINK (so Epiphany can access mailbox regs)
addr=0x810ec080 val=0x00000810
addr=0x810ec084 val=0x00000000

What happens

Writes from Epiphany to ERAM does not work.
Reads from Epiphany to ERAM returns something (not random, different value from different addresses), but the wrong data.
Writes to ELINK_MAILBOXLO does work, iff the FPGA elink is present in the RX MMU table. In fact if I change the FPGA elink entry in the MMU table to an ERAM address then writes from Epiphany to ELINK_MAILBOXLO.is written to ERAM
NB: Entries in ELINK_RXMMU not cleared on reset via ELINK_RESET (which is fine, can do that in S/W)

In fpga/parallella/headless_e16_z70[12]0/system_bd.tcl:

When upgrading Vivado, you must change the line with:

set scripts_vivado_version 2015.4

to match the Vivado version. Even though this file is re-generated in the
build, it is required by the build to create an initial block design.

Ugh!!!

The root directory contains both COPYING (GPLv3) and a LICENSE file (MIT). It seems all the source files refer to the GPLv3.

1. assert reset bit
2. deassert reset bit
3. write something to RXCFG register
   = write gets lost ( RXCFG = 0)
4. assert reset bit
5. deassert reset bit
6. usleep(1000)
7. write something to RXCFG register
   = write gets through

Isn't there a AXI wait signal (WREADY?) we can use?

Alternatively can we do a read loop in sw over the reset register until the reset bit reads back as de-asserted?

or make it mandatory to sleep after the reset bit has been deasserted

Test case:
fclk0 = 25 MHz clkdiv=254 sclk = 100 kHz

1. Master writes 0xff to SPI_USER0
2. Master reads data from SPI_USER0

As seen in the waveform capture master does not transfer the two most significant bits in the data byte.
This does not seem to happen in simulation.

waveform.vcd.txt