Consecutive writes to rxmmu table results in system freeze about oh HOT 6 OPEN

How are you writing?

-dma, block mem copy, or a slow for loop of e-writes?

What if any is the difference in behaviour between the three cases?

2016-02-16 10:32 GMT-05:00 Ola Jeppsson [email protected]:

Until MMU table entries have a valid bit we must initialize all RXMMU
table entries to point to a safe 1MB region.
When I try to do that, the system freezes.
[0x31a00000-0x31b00000] is a 1MB contiguous reserved region
(dma_alloc_coherent()).
This is reproducible, always gets stuck on the same write.
Inserting sleeps doesn't make any difference.

Log:

[ 1173.299226] epiphany elink0: elink_reset: enter
[ 1173.303742] pkt: 00000000_00000003_810f0200_05_0000
[ 1173.309198] pkt: 00000000_00000000_810f0200_05_0000
[ 1173.314728] pkt: 00000000_ffffffff_810f0208_05_0000
[ 1173.319539] pkt: 00000000_aaaaaaaa_810f0310_05_0000
[ 1173.324386] pkt: 00000000_0000000a_810f0314_05_0000
[ 1173.329271] pkt: 00000000_00000001_810f0210_05_0000
[ 1173.334118] pkt: 00000000_00000000_810f0300_05_0000
[ 1173.338984] epiphany elink0: elink_update_mmu_mappings: enter
[ 1173.344717] epiphany elink0: elink_update_mmu_mappings: Initializing MMU table to safe 1MB region
[ 1173.353564] pkt: 00000000_000003a1_810e8000_05_0000
[ 1173.358430] pkt: 00000000_00000000_810e8004_05_0000
[ 1173.363278] pkt: 00000000_000003a1_810e8008_05_0000
[ 1173.368144] pkt: 00000000_00000000_810e800c_05_0000
[ 1173.373010] pkt: 00000000_000003a1_810e8010_05_0000
[ 1173.377858] pkt: 00000000_00000000_810e8014_05_0000
[ 1173.382724] pkt: 00000000_000003a1_810e8018_05_0000
[ 1173.387590] pkt: 00000000_00000000_810e801c_05_0000
[ 1173.392456] pkt: 00000000_000003a1_810e8020_05_0000
[ 1173.397304] pkt: 00000000_00000000_810e8024_05_0000
[ 1173.402170] pkt: 00000000_000003a1_810e8028_05_0000
[ 1173.407036] pkt: 00000000_00000000_810e802c_05_0000
[ 1173.411884] pkt: 00000000_000003a1_810e8030_05_0000
[ 1173.416750] pkt: 00000000_00000000_810e8034_05_0000
[ 1173.421616] pkt: 00000000_000003a1_810e8038_05_0000
[ 1173.426482] pkt: 00000000_00000000_810e803c_05_0000
[ 1173.431330] pkt: 00000000_000003a1_810e8040_05_0000
[ 1173.436196] pkt: 00000000_00000000_810e8044_05_0000
[ 1173.441062] pkt: 00000000_000003a1_810e8048_05_0000
[ 1173.445909] pkt: 00000000_00000000_810e804c_05_0000
[ 1173.450776] pkt: 00000000_000003a1_810e8050_05_0000
[ 1173.455642] pkt: 00000000_00000000_810e8054_05_0000
[ 1173.460508] pkt: 00000000_000003a1_810e8058_05_0000
[ 1173.465355] pkt: 00000000_00000000_810e805c_05_0000
[ 1173.470221] pkt: 00000000_000003a1_810e8060_05_0000
[ 1173.475088] pkt: 00000000_00000000_810e8064_05_0000
[ 1173.479935] pkt: 00000000_000003a1_810e8068_05_0000
[ 1173.484801] pkt: 00000000_00000000_810e806c_05_0000
[ 1173.489667] pkt: 00000000_000003a1_810e8070_05_0000
[ 1173.494515] pkt: 00000000_00000000_810e8074_05_0000
[ 1173.499381] pkt: 00000000_000003a1_810e8078_05_0000
[ 1173.504247] pkt: 00000000_00000000_810e807c_05_0000
[ 1173.509113] pkt: 00000000_000003a1_810e8080_05_0000
[ 1173.513961] pkt: 00000000_00000000_810e8084_05_0000
[ 1173.518827] pkt: 00000000_000003a1_810e8088_05_0000
[ 1173.523693] pkt: 00000000_00000000_810e808c_05_0000
[ 1173.528541] pkt: 00000000_000003a1_810e8090_05_0000
[ 1173.533407] pkt: 00000000_00000000_810e8094_05_0000
[ 1173.538273] pkt: 00000000_000003a1_810e8098_05_0000
[ 1173.543139] pkt: 00000000_00000000_810e809c_05_0000
[ 1173.547987] pkt: 00000000_000003a1_810e80a0_05_0000
[ 1173.552853] pkt: 00000000_00000000_810e80a4_05_0000
[ 1173.557719] pkt: 00000000_000003a1_810e80a8_05_0000
[ 1173.562585] pkt: 00000000_00000000_810e80ac_05_0000
[ 1173.567433] pkt: 00000000_000003a1_810e80b0_05_0000
[ 1173.572299] pkt: 00000000_00000000_810e80b4_05_0000
[ 1173.577165] pkt: 00000000_000003a1_810e80b8_05_0000
[ 1173.582013] pkt: 00000000_00000000_810e80bc_05_0000
[ 1173.586879] pkt: 00000000_000003a1_810e80c0_05_0000
[ 1173.591745] pkt: 00000000_00000000_810e80c4_05_0000
[ 1173.596592] pkt: 00000000_000003a1_810e80c8_05_0000
[ 1173.601458] pkt: 00000000_00000000_810e80cc_05_0000
[ 1173.606325] pkt: 00000000_000003a1_810e80d0_05_0000
[ 1173.611191] pkt: 00000000_00000000_810e80d4_05_0000
[ 1173.616038] pkt: 00000000_000003a1_810e80d8_05_0000
[ 1173.620904] pkt: 00000000_00000000_810e80dc_05_0000
[ 1173.625770] pkt: 00000000_000003a1_810e80e0_05_0000
[ 1173.630637] pkt: 00000000_00000000_810e80e4_05_0000
[ 1173.635484] pkt: 00000000_000003a1_810e80e8_05_0000
[ 1173.640350] pkt: 00000000_00000000_810e80ec_05_0000
[ 1173.645216] pkt: 00000000_000003a1_810e80f0_05_0000
[ 1173.650064] pkt: 00000000_00000000_810e80f4_05_0000
[ 1173.654930] pkt: 00000000_000003a1_810e80f8_05_0000
[ 1173.659796] pkt: 00000000_00000000_810e80fc_05_0000
[ 1173.664644] pkt: 00000000_000003a1_810e8100_05_0000
[ 1173.669510] pkt: 00000000_00000000_810e8104_05_0000
[ 1173.674376] pkt: 00000000_000003a1_810e8108_05_0000
[ 1173.679242] pkt: 00000000_00000000_810e810c_05_0000
[ 1173.684108] pkt: 00000000_000003a1_810e8110_05_0000
[ 1173.688956] pkt: 00000000_00000000_810e8114_05_0000
[ 1173.693822] pkt: 00000000_000003a1_810e8118_05_0000
[ 1173.698688] pkt: 00000000_00000000_810e811c_05_0000
[ 1173.703536] pkt: 00000000_000003a1_810e8120_05_0000
[ 1173.708402] pkt: 00000000_00000000_810e8124_05_0000
[ 1173.713268] pkt: 00000000_000003a1_810e8128_05_0000
[ 1173.718116] pkt: 00000000_00000000_810e812c_05_0000
[ 1173.722982] pkt: 00000000_000003a1_810e8130_05_0000
[ 1173.727848] pkt: 00000000_00000000_810e8134_05_0000
[ 1173.732714] pkt: 00000000_000003a1_810e8138_05_0000
[ 1173.737562] pkt: 00000000_00000000_810e813c_05_0000
[ 1173.742428] pkt: 00000000_000003a1_810e8140_05_0000
[ 1173.747294] pkt: 00000000_00000000_810e8144_05_0000
[ 1173.752141] pkt: 00000000_000003a1_810e8148_05_0000
[ 1173.757007] pkt: 00000000_00000000_810e814c_05_0000
[ 1173.761874] pkt: 00000000_000003a1_810e8150_05_0000
[ 1173.766740] pkt: 00000000_00000000_810e8154_05_0000
[ 1173.771587] pkt: 00000000_000003a1_810e8158_05_0000
[ 1173.776453] pkt: 00000000_00000000_810e815c_05_0000
[ 1173.781319] pkt: 00000000_000003a1_810e8160_05_0000
[ 1173.786167] pkt: 00000000_00000000_810e8164_05_0000
[ 1173.791033] pkt: 00000000_000003a1_810e8168_05_0000
[ 1173.795899] pkt: 00000000_00000000_810e816c_05_0000
[ 1173.800765] pkt: 00000000_000003a1_810e8170_05_0000
[ 1173.805613] pkt: 00000000_00000000_810e8174_05_0000
[ 1173.810479] pkt: 00000000_000003a1_810e8178_05_0000
[ 1173.815345] pkt: 00000000_00000000_810e817c_05_0000
[ 1173.820211] pkt: 00000000_000003a1_810e8180_05_0000
[ 1173.825059] pkt: 00000000_00000000_810e8184_05_0000
[ 1173.829925] pkt: 00000000_000003a1_810e8188_05_0000
[ 1173.834791] pkt: 00000000_00000000_810e818c_05_0000
[ 1173.839657] pkt: 00000000_000003a1_810e8190_05_0000
[ 1173.844486] pkt: 00000000_00000000_810e8194_05_0000
[ 1173.849371] pkt: 00000000_000003a1_810e8198_05_0000
[ 1173.854219] pkt: 00000000_00000000_810e819c_05_0000
[ 1173.859085] pkt: 00000000_000003a1_810e81a0_05_0000
[ 1173.863951] pkt: 00000000_00000000_810e81a4_05_0000
[ 1173.868817] pkt: 00000000_000003a1_810e81a8_05_0000
[ 1173.873665] pkt: 00000000_00000000_810e81ac_05_0000
[ 1173.878531] pkt: 00000000_000003a1_810e81b0_05_0000
[ 1173.883397] pkt: 00000000_00000000_810e81b4_05_0000
[ 1173.888263] pkt: 00000000_000003a1_810e81b8_05_0000
[ 1173.893111] pkt: 00000000_00000000_810e81bc_05_0000
[ 1173.897977] pkt: 00000000_000003a1_810e81c0_05_0000
[ 1173.902843] pkt: 00000000_00000000_810e81c4_05_0000
[ 1173.907690] pkt: 00000000_000003a1_810e81c8_05_0000
[ 1173.912556] pkt: 00000000_00000000_810e81cc_05_0000
[ 1173.917423] pkt: 00000000_000003a1_810e81d0_05_0000
[ 1173.922270] pkt: 00000000_00000000_810e81d4_05_0000
[ 1173.927136] pkt: 00000000_000003a1_810e81d8_05_0000
[ 1173.932002] pkt: 00000000_00000000_810e81dc_05_0000
[ 1173.936868] pkt: 00000000_000003a1_810e81e0_05_0000
[ 1173.941716] pkt: 00000000_00000000_810e81e4_05_0000
[ 1173.946582] pkt: 00000000_000003a1_810e81e8_05_0000
[ 1173.951448] pkt: 00000000_00000000_810e81ec_05_0000
[ 1173.956314] pkt: 00000000_000003a1_810e81f0_05_0000
[ 1173.961162] pkt: 00000000_00000000_810e81f4_05_0000
[ 1173.966028] pkt: 00000000_000003a1_810e81f8_05_0000
[ 1173.970894] pkt: 00000000_00000000_810e81fc_05_0000
[ 1173.975742] pkt: 00000000_000003a1_810e8200_05_0000
[ 1173.980608] pkt: 00000000_00000000_810e8204_05_0000
[ 1173.985474] pkt: 00000000_000003a1_810e8208_05_0000
[ 1173.990322] pkt: 00000000_00000000_810e820c_05_0000
[ 1173.995188] pkt: 00000000_000003a1_810e8210_05_0000
[ 1174.000054] pkt: 00000000_00000000_810e8214_05_0000
[ 1174.004920] pkt: 00000000_000003a1_810e8218_05_0000
[ 1174.009768] pkt: 00000000_00000000_810e821c_05_0000
[ 1174.014634] pkt: 00000000_000003a1_810e8220_05_0000
[ 1174.019481] pkt: 00000000_00000000_810e8224_05_0000
[ 1174.024366] pkt: 00000000_000003a1_810e8228_05_0000
[ 1174.029214] pkt: 00000000_00000000_810e822c_05_0000
[ 1174.034080] pkt: 00000000_000003a1_810e8230_05_0000
[ 1174.038946] pkt: 00000000_00000000_810e8234_05_0000
[ 1174.043793] pkt: 00000000_000003a1_810e8238_05_0000
[ 1174.048660] pkt: 00000000_00000000_810e823c_05_0000
[ 1174.053526] pkt: 00000000_000003a1_810e8240_05_0000
[ 1174.058373] pkt: 00000000_00000000_810e8244_05_0000
[ 1174.063239] pkt: 00000000_000003a1_810e8248_05_0000
[ 1174.068105] pkt: 00000000_00000000_810e824c_05_0000

Hard freeze

—
Reply to this email directly or view it on GitHub
#50.

Andreas Olofsson, CEO/Founder at Adapteva
Cell: +1 781 325 6688
Twitter: @adapteva
Web: adapteva.com

Linkedin: linkedin.com/in/andreasolofsson

http://www.adapteva.com/

from oh.

How are you writing?

-dma, block mem copy, or a slow for loop of e-writes?

What if any is the difference in behaviour between the three cases?

No dma, I only use dma_alloc_coherent() to allocate the memory region. It doesn't matter which value I write.

It's a for loop around iowrite32()
http://www.makelinux.net/ldd3/chp-9-sect-4#chp-9-sect-4.2
which all of the driver code uses.
No difference if I add sleeps, always freezes at the same place.

from oh.

I stand corrected, not always on same addr ....

from oh.

Or rather, if I don't write to the high 32 bits, I get slightly further

[   52.884891] pkt: 00000000_000003a1_810e8290_05_0000

Always the same address though. No difference if I have no sleep or 100 msec sleep between iterations.

Looks like some queue is filling up?

from oh.

If I interleave with reads (ignoring writing to upper 32 bits)

[   33.477945] epiphany elink0: elink_reset: enter
[   33.482406] pkt: 00000000_00000003_810f0200_05_0000
[   33.497502] pkt: 00000000_00000000_810f0200_05_0000
[   33.512506] pkt: 00000000_ffffffff_810f0208_05_0000
[   33.517335] pkt: 00000000_aaaaaaaa_810f0310_05_0000
[   33.522201] pkt: 00000000_0000000a_810f0314_05_0000
[   33.527049] pkt: 00000000_00000001_810f0210_05_0000
[   33.531915] pkt: 00000000_00000000_810f0300_05_0000
[   33.536781] epiphany elink0: elink_update_mmu_mappings: enter
[   33.542495] epiphany elink0: elink_update_mmu_mappings: Initializing MMU table to safe 1MB region
[   33.551361] pkt: 00000000_000003a1_810e8000_05_0000
[   33.607818] pkt: 00000000_810d0000_810e8000_04_0000
[   33.662820] pkt: 00000000_000003a1_810e8008_05_0000
[   33.717785] pkt: 00000000_810d0000_810e8008_04_0000
[   33.772878] pkt: 00000000_000003a1_810e8010_05_0000
[   33.827788] pkt: 00000000_810d0000_810e8010_04_0000
[   33.882882] pkt: 00000000_000003a1_810e8018_05_0000
[   33.937902] pkt: 00000000_810d0000_810e8018_04_0000
[   33.992811] pkt: 00000000_000003a1_810e8020_05_0000
[   34.047813] pkt: 00000000_810d0000_810e8020_04_0000
...  snip ...
[   40.043518] pkt: 00000000_000003a1_810e81d8_05_0000
[   40.098538] pkt: 00000000_810d0000_810e81d8_04_0000
[   40.153558] pkt: 00000000_000003a1_810e81e0_05_0000
[   40.208523] pkt: 00000000_810d0000_810e81e0_04_0000
[   40.263616] pkt: 00000000_000003a1_810e81e8_05_0000
[   40.318507] pkt: 00000000_810d0000_810e81e8_04_0000
[   40.373601] pkt: 00000000_000003a1_810e81f0_05_0000
[   40.428529] pkt: 00000000_810d0000_810e81f0_04_0000
[   40.483622] pkt: 00000000_000003a1_810e81f8_05_0000
[   40.538587] pkt: 00000000_810d0000_810e81f8_04_0000
[   40.593589] pkt: 00000000_000003a1_810e8200_05_0000
[   40.648553] pkt: 00000000_810d0000_810e8200_04_0000
[   40.654028] Unhandled fault: imprecise external abort (0x1406) at 0xb6f33314
...

from oh.

@aolofsson
Reset sequence
reset.txt

from oh.