antmicro / rowhammer-tester Goto Github PK

Dear developers,

in your README you mention Currently, only the Arty-A7 (xc7a35t) FPGA board is supported (TARGET=arty) but at the same time also have a section about the ZCU104 board. Can I assume that this is a mistake and both boards are indeed supported? If so, could you please describe if there are any differences and which one would you recommend (e.g., is better tested)?

Thank you very much!

Whenever I build the bitstream for the Arty-A7 35T I receive the warnings:

WARNING: [Synth 8-3331] design VexRiscv has unconnected port iBusWishbone_ERR
WARNING: [Synth 8-3331] design VexRiscv has unconnected port dBusWishbone_ERR
WARNING: [Synth 8-3331] design digilent_arty has unconnected port eth_rx_er
WARNING: [Synth 8-3331] design digilent_arty has unconnected port eth_col
WARNING: [Synth 8-3331] design digilent_arty has unconnected port eth_crs
Along with some other bit errors.

The bit stream generates and flashes to the device but the ethernet IP address is unreachable.
make srv does not connect and shows the error:
[CommUDP] ip: 192.168.100.50 / port: 1234 / tcp port: 1234
Traceback (most recent call last):
File "/rowhammer-tester/venv/bin/litex_server", line 33, in
sys.exit(load_entry_point('litex', 'console_scripts', 'litex_server')())
File "/rowhammer-tester/third_party/litex/litex/tools/litex_server.py", line 283, in main
server.open()
File "/rowhammer-tester/third_party/litex/litex/tools/litex_server.py", line 90, in open
self.comm.open()
File "/rowhammer-tester/third_party/litex/litex/tools/remote/comm_udp.py", line 33, in open
self.probe(self.server, self.port)
File "/rowhammer-tester/third_party/litex/litex/tools/remote/comm_udp.py", line 66, in probe
raise Exception(f"Unable to probe Etherbone server at {self.server}.")
Exception: Unable to probe Etherbone server at 192.168.100.50.
make: *** [Makefile:81: srv] Error 1

Vivado Versions Tested: 2019.1, 2020.2, 2022.1

Dear developers,

I'm totally new in this hardware world, so sorry if these questions might look obvious to you. I'm wondering how exactly we can use this DRAM controller implementation in our research. From the diagram in the README I can see that LiteX is running on a FPGA which is connected to a host (via Ethernet) and via PHY to DRAM.

Does PHY mean that the DRAM is plugged into the FPGA, e.g., by adding a DIMM socket to the FPGA where the DRAM can be plugged into? Because I saw that the Arty A7 only support DDR3L and not the regular DDR DRAM.

Thanks a lot for your explanation!

Hi,

I'm looking for a DRAM attacker device (not via PCIe device ) for my testing.

Kindly help me to get any hardware that can attack directly DRAM.

Thanks
Sridha

Consider this command:

python hw_rowhammer.py --nrows 512 --read_count 20e6 --pattern 01_in_row --row-pairs const --const-rows-pair 131 133 --no-refresh

If I run w/ 0.8e6 hammers, I get this:
.
.
.
Bit-flips for row 130: 19
Bit-flips for row 132: 748
Bit-flips for row 134: 25

which is fine. If I increase to 20e6, I get this:
.
.
.
Progress: [========================================] 16777216 / 16777216 (Errors: 386)
Traceback (most recent call last):
File "hw_rowhammer.py", line 128, in
main(row_hammer_cls=HwRowHammer)
File "/home/sqazi/fpga/litex-rowhammer-tester/rowhammer_tester/scripts/rowhammer.py", line 285, in main
row_hammer.run(row_pairs=row_pairs, read_count=args.read_count, pattern_generator=pattern)
File "hw_rowhammer.py", line 117, in run
errors = self.check_errors(row_pattern)
File "hw_rowhammer.py", line 77, in check_errors
base_addr = min(self.addresses_per_row[row])
KeyError: 4240

I have been testing this on the zcu104 and needed to install some additional dependencies to make it work.

For building OpenOCD:

apt install libtool libusb-1.0-0-dev pkg-config

For the rowhammer tester:

pip install ninja

I'm assuming the ninja installation can be folded into the make deps command so I'll submit a pull request if I can figure it out.

As for the OpenOCD requirements, I'm working on a fresh install of Ubuntu 22.04 LTS, so I'm not sure if this affected it at all.
Should I try adding these to the current list of dependencies or appending it as a note in the docs?

Motivation: we want to be able to compile valid payloads for whatever part that is plugged into the board at the given moment from part-agnostic descriptions. So, it would be nice to be able to look up timing parameters for the part (e.g. tREFI, tFAW) from the scripts. Is this possible?

When doing long running experiments that use python scripts from a bash loop (i.e. frequently creating/destroying connections), the server stops working after some number of iterations. Everything goes back to normal if the server is restarted.

I added a fix in my existing PR (#109) but figured I should mark it as a separate issue since it's not related to documentation.

I observed that doing memory initialization on ZCU104 over the Ethernet, the data read from the VexRiscV CPU's UART (CPU in PL) is malformed. This looks like part of the UART bytes are lost.

I think that this issue is due to the fact that a single memory access from PS can result in multiple wishbone reads in PL.

The current way the access is performed is as described here https://github.com/antmicro/litex-rowhammer-tester/blob/7ceb33f3cc57611fc3711546af1f425f80e7f955/rowhammer_tester/targets/zcu104.py#L177. The problem most likely arises in this data width converter AXILite(128) -> AXILite(32). A single access results in 4 32-bit accesses. Due to the way LiteX CSR address decoders work it may be that all these accesses go to the same CSR. UART read CSR works like a FIFO, and an access consumes the current byte, which would explain why bytes are lost.

If this is the case, than it will cause problems with using the reader_error_offset CSR which also works as a FIFO.

Following the instructions on this readthedocs page, I ran the following script and got an error:

python hw_rowhammer.py --all-rows --nrows 5 --row-pair-distance 0 --payload-executor
Using generated target files in: ../../build/ddr4_datacenter_test_board
Board info: Row Hammer Tester SoC on xc7k160tffg676-1, git: c8ffca06f19d426bdbd094d1c9f815d1fb025dea 2022-12-22 09:32:18

Preparing ...
WARNING: only single word patterns supported, using: 0xffffffff

Filling memory with data ...
Progress: [========================================] 33554432 / 33554432

Verifying written memory ...
Progress: [========================================] 33554432 / 33554432 (Errors: 0)
OK

Running Rowhammer attacks ...
  Repeatable unit: 106
  Repetitions: 53
Memory required for payload executor instructions (40268 bytes) exceeds available payload memory (1024 bytes)
The payload memory size can be changed with '--payload-size ' option

I was able to workaround the issue by setting and environment variable called ARGS that is used by the build:

export ARGS="--payload-size 5000"

Would it be possible to increase the default payload size so that the single row attacks using the '--payload-executor' option work without rebuilding the bitstream?

Running python bios_console.py, I ran a memory test after sucessful memory initialization and saw it hang when the memory size was greater that 4 Gbytes:

litex> mem_test 0x40000000 0x100000000
Memtest at 0x40000000 (3.10GiB)...
  Write: 0x40000000-0x80000000 1.0GiB

The memory test above hung and I had to reset the board. The terminal stopped echoing the things I typed.

Chatting with the Antmicro folks, it sounds like the internal CPU is a 32 bit CPU. The DRAM on the DIMM is mapped at 0x40000000 and thus we can't test 4 Gbytes.

It would be useful if the software checked the range being tested and prevented the hang. It would be better if the memory was accessed through a window or a MMU so that we can test any address in memory. This would be especially useful in the case where we see an unexpected error from a rowhammer attack and we want to validate that the memory at the address of the error is working correctly.

The Edit button found on most of the ReadTheDocs pages for this repository doesn't seem to work, it just takes you back to the same page;

I'm on Debian so there might be some differences, but I had to install three more packages. They apparently weren't dependencies of the ones already listed.

https://packages.ubuntu.com/bionic/autoconf
https://packages.ubuntu.com/bionic/python3-wheel
https://packages.ubuntu.com/bionic/libevent-dev

I created an initial version of the Payload Executor module along with an example payload.

The current version supports most of the required features. It implements all the needed DFI commands (ACT, PRE, REF, READ, ZQC). Each these instructions has a timeslice field defining the delay (in cycles) after the command and an address field. There is also a NOOP instruction that uses just the timeslice (timeslice field is wider because it reuses address bits) and a LOOP instruction which has count/jump fields. LOOP is a relative jump back by jump instructions that gets executed count times (details) and in this implementation has an execution time of 1.

The scratchpad memory is also implemented. It stores the data from subsequent READs at subsequent addresses. If an overflow happens it will be signalized in the status register. Both payload an scratchpad memories can be simply accessed from the main bus.

The current design of the ISA has one limitation which is that the Payload Executor works at the controller frequency and we have a 1:4 frequency ratio between the controller and the DRAM I/Os (so we have 4 DFI phases). Currently there is no way to specify the DFI phase and all the commands are executed on phase 0. This means that the the time resolution is 4 x tCK. In practice this may or may not be a problem, because it is often impossible to send commands so often due to DRAM module timing restrictions. It would be fairly easy to add an additional PHASE field to DFI instructions (reducing the width of the TIMESLICE field), which would allow to send the commands on any phase (with 1 x tCK resolution), but still at most 1 command per 4 x tCK.

As for the the bit widths of instruction fields, they are defined here and can be easily adjusted to suit our needs (especially LOOP count vs jump).

Hi, I hope you are well.

I noticed there are files (such as constraints and others) for the ZCU102 board. I wonder if it is possible to modify the ZCU104 scripts to replicate the tests on the ZCU102. Would the LiteDRAM work on it?

How do I compute the offset to pass into hw_memtest if I want to just find errors in a single row?

Hello,
When running example_half_double.json on the ddr4 datacenter test board I get this error:

Memory required for payload executor instructions (526784 bytes) exceeds available payload memory (1024 bytes) The payload memory size can be changed with '--payload-size ' option.

However, when I change the payload size to 1048576, the design doesn't close timing:

WNS(ns)      TNS(ns)  TNS Failing Endpoints  TNS Total Endpoints      WHS(ns)      THS(ns)  THS Failing Endpoints  THS Total Endpoints     WPWS(ns)     TPWS(ns)  TPWS Failing Endpoints  TPWS Total Endpoints  
-------      -------  ---------------------  -------------------      -------      -------  ---------------------  -------------------     --------     --------  ----------------------  --------------------  
 -3.580   -27995.277                  24192                72831        0.037        0.000                      0                72831        0.264        0.000                       0                 25578  

Here is the Vivado version I used:

Vivado v2022.1.2 (64-bit)
SW Build 3605665 on Fri Aug 5 22:52:02 MDT 2022
IP Build 3603185 on Sat Aug 6 04:07:44 MDT 2022

Has this design been able to meet timing to test the example_half_double.json?

Hello, I'm using the lpddr4 test board to try and run the Row Hammer test, but the only commands that I have gotten to run successfully are "make build" and "make upload". After running both commands, I am unable to connect to Etherbone, and I also noticed that the LEDs on the Ethernet port don't light up. I have gotten the Row Hammer test to work on the Arty A7 board, so I believe I should have the flow set up correctly.

I did notice that during synthesis during "make build", there were two critical warnings related to the Ethernet clocks, is this expected/normal?:

---------------------------------------------------------------------------------
Start Applying XDC Timing Constraints
---------------------------------------------------------------------------------
CRITICAL WARNING: [Synth 8-3321] create_clock attempting to set clock on an unknown port/pin for constraint at line 269 of /home/ssrl01/cindy/litex-rowhammer-tester/build/lpddr4_test_board/gateware/antmicro_lpddr4_test_board.xdc. [/home/ssrl01/cindy/litex-rowhammer-tester/build/lpddr4_test_board/gateware/antmicro_lpddr4_test_board.xdc:269]
CRITICAL WARNING: [Synth 8-3321] create_clock attempting to set clock on an unknown port/pin for constraint at line 271 of /home/ssrl01/cindy/litex-rowhammer-tester/build/lpddr4_test_board/gateware/antmicro_lpddr4_test_board.xdc. [/home/ssrl01/cindy/litex-rowhammer-tester/build/lpddr4_test_board/gateware/antmicro_lpddr4_test_board.xdc:271]
WARNING: [Synth 8-3321] set_false_path : Empty through list for constraint at line 284 of /home/ssrl01/cindy/litex-rowhammer-tester/build/lpddr4_test_board/gateware/antmicro_lpddr4_test_board.xdc. [/home/ssrl01/cindy/litex-rowhammer-tester/build/lpddr4_test_board/gateware/antmicro_lpddr4_test_board.xdc:284]

You colleague, @rw1nkler can provide a bunch of advice here.

Was trying to follow the instructions for ZCU104 boot image preparation. The instructions to generate the FSBL are off of Xilinx's site:

• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842462/Build+PMU+Firmware
• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841798/Build+FSBL

However, what is needed is the hardware description file (HDF) which gets generated from Vivado but I get this error:

Vivado% write_hwdef -force -file filename.hdf
INFO: [Vivado_Tcl 4-424] Cannot write hardware definition file as there are no IPI block design hardware handoff files present

It looks like from the generated verilog file, the ARM core is directly instantiated instead of the using the Xilinx Block Design. Here is snipped of code from the generated zcu104.v (line 25385 also attached file)
zcu104.v.zip

PS8 PS8(
	.MAXIGP0ARREADY(main_axi_ps_ar_ready),
	.MAXIGP0AWREADY(main_axi_ps_aw_ready),
	.MAXIGP0BID(main_axi_ps_b_payload_id),
	.MAXIGP0BRESP(main_axi_ps_b_payload_resp),
	.MAXIGP0BVALID(main_axi_ps_b_valid),
	.MAXIGP0RDATA(main_axi_ps_r_payload_data),

A few questions:

1. What version of Vivado were you using to generate the FSBL?
2. How did you generate the FSBL?
3. Do you have prebuilt binaries for ZCU104?

Hi, I hope you are well.

I am trying to test the Rowhammer on the zcu104 board, but "example_half_double.json" playbook needs 119300 payload_mem_size.

#34
This article explains that setting 1MB of Payload_meme_size uses 88.14% of Block RAM.

How should I solve the problem in this case? Thank you in advance.

https://github.com/antmicro/rowhammer-tester/blob/master/rowhammer_tester/scripts/spd_eeprom.py waits until memory training successfully completes before reading the SPD EEPROM.

This is a problem when the training fails as in the example below:

  m15, b05: |00000000000000000001111111111111| delays: 25+-06
  m15, b06: |00000000000000000000000000000000| delays: -
  m15, b07: |00000000000000000000000000000000| delays: -
  best: m15, b04 delays: 09+-07
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB     
   Read: 0x40000000-0x40200000 2.0MiB     
  bus errors:  0/256
  addr errors: 0/8192
  data errors: 6899/524288
Memtest KO
Memory initialization failed

--============= Console ================--

litex> 
litex> sdram_spd 0
Memory dump:
0x00000000  23 12 0c 01 85 29 00 08 00 60 00 03 08 0b 80 00  #....)...`......
0x00000010  00 00 06 0d f8 3f 00 00 6e 6e 6e 11 00 6e f0 0a  .....?..nnn..n..
0x00000020  20 08 00 05 00 60 18 28 2b 00 78 00 14 3c 00 00   ....`.(+.x..<..
0x00000030  00 00 00 00 00 00 00 00 00 00 00 00 16 16 15 16  ................
0x00000040  03 16 03 16 03 16 03 16 0d 16 16 16 16 16 00 00  ................
0x00000050  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x00000060  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x00000070  00 00 00 00 00 ec 9c 00 00 00 00 00 e7 00 24 97  ..............$.
0x00000080  11 11 41 19 00 86 32 a0 01 65 05 00 00 00 00 00  ..A...2..e......
0x00000090  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x000000a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x000000b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x000000c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x000000d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x000000e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0x000000f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 7b e4  ..............{.

The example above will cause spd_eeprom.py to timeout since the training fails. But it also shows that it is still possible to read the SPD data.

The code needs to wait for the training to complete. But it should read the SPD EEPROM regardless of whether the training is successful or not.

The DRAM module page is out of date since the FPGA can now read the DRAM timing dynamically and program the timing into registers without having to recompile the bitstream:

Can you please update the documentation to reflect this change?

In the section @ https://litex-rowhammer-tester.readthedocs.io/en/latest/general.html#simulation it isn't clear that the simulation is for simulating the LiteX gateware.

Things which should be noted;

• This is simulation rather that means you don't need physical hardware.
• The simulation conflicts with real hardware due to using the same IP address.

Is this normal (especially the line at the bottom)?

make upload
make[1]: Entering directory '/home/sqazi/fpga/litex-rowhammer-tester'
python rowhammer_tester/targets/zcu104.py --load --ip-address 192.168.1.10 --mac-address 0x10e2d5000001 --udp-port 1234
INFO:SoC: __ _ __ _ __
INFO:SoC: / / () /___ | |//
INFO:SoC: / /__/ / __/ -)> <
INFO:SoC: ///_/_//|_|
INFO:SoC: Build your hardware, easily!
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Creating SoC... (2020-12-04 19:09:09)
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:FPGA device : xczu7ev-ffvc1156-2-i.
INFO:SoC:System clock: 125.00MHz.
INFO:SoCBusHandler:Creating Bus Handler...
INFO:SoCBusHandler:32-bit wishbone Bus, 4.0GiB Address Space.
INFO:SoCBusHandler:Adding reserved Bus Regions...
INFO:SoCBusHandler:Bus Handler created.
INFO:SoCCSRHandler:Creating CSR Handler...
INFO:SoCCSRHandler:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
INFO:SoCCSRHandler:Adding reserved CSRs...
INFO:SoCCSRHandler:CSR Handler created.
INFO:SoCIRQHandler:Creating IRQ Handler...
INFO:SoCIRQHandler:IRQ Handler (up to 32 Locations).
INFO:SoCIRQHandler:Adding reserved IRQs...
INFO:SoCIRQHandler:IRQ Handler created.
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Initial SoC:
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:32-bit wishbone Bus, 4.0GiB Address Space.
INFO:SoC:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
INFO:SoC:IRQ Handler (up to 32 Locations).
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoCCSRHandler:ctrl CSR allocated at Location 0.
INFO:SoCBusHandler:io0 Region added at Origin: 0x00000000, Size: 0x100000000, Mode: RW, Cached: False Linker: False.
INFO:SoCCSRHandler:identifier_mem CSR allocated at Location 1.
INFO:SoCCSRHandler:uart_phy CSR allocated at Location 2.
INFO:SoCCSRHandler:uart CSR allocated at Location 3.
INFO:SoCCSRHandler:timer0 CSR allocated at Location 4.
INFO:USMMCM:Creating USMMCM, speedgrade -2.
INFO:USMMCM:Registering Differential ClkIn of 125.00MHz.
INFO:USMMCM:Creating ClkOut0 pll4x of 500.00MHz (+-10000.00ppm).
INFO:USMMCM:Creating ClkOut1 idelay of 500.00MHz (+-10000.00ppm).
INFO:USMMCM:Creating ClkOut2 uart of 125.00MHz (+-10000.00ppm).
INFO:SoCCSRHandler:leds CSR allocated at Location 5.
INFO:SoCCSRHandler:ddrphy CSR allocated at Location 6.
INFO:SoCCSRHandler:controller_settings CSR allocated at Location 7.
INFO:SoCCSRHandler:sdram CSR allocated at Location 8.
INFO:SoCBusHandler:main_ram Region added at Origin: 0x40000000, Size: 0x40000000, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:main_ram added as Bus Slave.
INFO:SoCBusHandler:uartbone added as Bus Master.
INFO:SoCCSRHandler:rowhammer CSR allocated at Location 9.
INFO:SoCBusHandler:pattern_data Region added at Origin: 0x20000000, Size: 0x00000400, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:pattern_data added as Bus Slave.
INFO:SoC:RAM pattern_data added Origin: 0x20000000, Size: 0x00000400, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:pattern_addr Region added at Origin: 0x21000000, Size: 0x00000040, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:pattern_addr added as Bus Slave.
INFO:SoC:RAM pattern_addr added Origin: 0x21000000, Size: 0x00000040, Mode: RW, Cached: True Linker: False.
INFO:SoC:BIST pattern: Length: 16, Data Width: 512-bit, Address width: 32-bit
INFO:SoCCSRHandler:writer CSR allocated at Location 10.
INFO:SoCCSRHandler:reader CSR allocated at Location 11.
INFO:SoCBusHandler:payload Region added at Origin: 0x30000000, Size: 0x00000400, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:payload added as Bus Slave.
INFO:SoC:RAM payload added Origin: 0x30000000, Size: 0x00000400, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:scratchpad Region added at Origin: 0x31000000, Size: 0x00000400, Mode: R, Cached: True Linker: False.
INFO:SoCBusHandler:scratchpad added as Bus Slave.
INFO:SoC:RAM scratchpad added Origin: 0x31000000, Size: 0x00000400, Mode: R, Cached: True Linker: False.
INFO:SoC:Instruction payload: Length: 256, Data Width: 32-bit
INFO:SoC:Scratchpad memory: Length: 16, Data Width: 512-bit
INFO:SoCCSRHandler:payload_executor CSR allocated at Location 12.
INFO:USMMCM:Config:
divclk_divide : 1
clkout2_freq : 125.00MHz
clkout2_divide: 8
clkout2_phase : 0.00°
clkout0_freq : 500.00MHz
clkout0_divide: 2
clkout0_phase : 0.00°
clkout1_freq : 500.00MHz
clkout1_divide: 2
clkout1_phase : 0.00°
vco : 1000.00MHz
clkfbout_mult : 8
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Finalized SoC:
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:32-bit wishbone Bus, 4.0GiB Address Space.
IO Regions: (1)
io0 : Origin: 0x00000000, Size: 0x100000000, Mode: RW, Cached: False Linker: False
Bus Regions: (5)
pattern_data : Origin: 0x20000000, Size: 0x00000400, Mode: RW, Cached: True Linker: False
pattern_addr : Origin: 0x21000000, Size: 0x00000040, Mode: RW, Cached: True Linker: False
payload : Origin: 0x30000000, Size: 0x00000400, Mode: RW, Cached: True Linker: False
scratchpad : Origin: 0x31000000, Size: 0x00000400, Mode: R, Cached: True Linker: False
main_ram : Origin: 0x40000000, Size: 0x40000000, Mode: RW, Cached: True Linker: False
Bus Masters: (1)

• uartbone
  Bus Slaves: (5)
• main_ram
• pattern_data
• pattern_addr
• payload
• scratchpad
  INFO:SoC:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
  CSR Locations: (13)
• ctrl : 0
• identifier_mem : 1
• uart_phy : 2
• uart : 3
• timer0 : 4
• leds : 5
• ddrphy : 6
• controller_settings : 7
• sdram : 8
• rowhammer : 9
• writer : 10
• reader : 11
• payload_executor : 12
  INFO:SoC:IRQ Handler (up to 32 Locations).
  INFO:SoC:--------------------------------------------------------------------------------
  INFO:SoCBusHandler:csr Region added at Origin: 0x00000000, Size: 0x00010000, Mode: RW, Cached: False Linker: False.
  INFO:SoCBusHandler:csr added as Bus Slave.
  INFO:SoCCSRHandler:bridge added as CSR Master.
  INFO:SoCBusHandler:Interconnect: InterconnectShared (1 <-> 6).

****** Vivado v2019.2 (64-bit)
**** SW Build 2708876 on Wed Nov 6 21:39:14 MST 2019
**** IP Build 2700528 on Thu Nov 7 00:09:20 MST 2019
** Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.

WARNING: 'open_hw' is deprecated, please use 'open_hw_manager' instead.
open_hw_manager
INFO: [Labtools 27-2285] Connecting to hw_server url TCP:localhost:3121
INFO: [Labtools 27-2222] Launching hw_server...
INFO: [Labtools 27-2221] Launch Output:

****** Xilinx hw_server v2019.2
**** Build date : Nov 6 2019 at 22:13:42
** Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.

INFO: [Labtools 27-3415] Connecting to cs_server url TCP:localhost:3042
INFO: [Labtools 27-3417] Launching cs_server...
INFO: [Labtools 27-2221] Launch Output:

****** Xilinx cs_server v2019.2.0
**** Build date : Nov 06 2019-21:41:48
** Copyright 2017-2019 Xilinx, Inc. All Rights Reserved.

INFO: [Labtoolstcl 44-466] Opening hw_target localhost:3121/xilinx_tcf/Xilinx/25683A
INFO: [Labtools 27-3164] End of startup status: HIGH
program_hw_devices: Time (s): cpu = 00:00:11 ; elapsed = 00:00:11 . Memory (MB): peak = 1469.859 ; gain = 0.000 ; free physical = 5844 ; free virtual = 23135
INFO: [Labtools 27-1434] Device xczu7 (JTAG device index = 0) is programmed with a design that has no supported debug core(s) in it.
INFO: [Common 17-206] Exiting Vivado at Fri Dec 4 19:09:39 2020...

I wasn't able to find a quick way to go from https://litex-rowhammer-tester.readthedocs.io/en/latest/general.html to this repository.

'make flash' fails if the JTAG/SPI jumper is set to JTAG, but the jumper is not documented. Can you please mention setting the jumper in rowhammer-tester/tree/master/doc/ddr4_datacenter_dram_tester.md?

I also saw the jumper affect 'make upload' but this seems unexpected.

Hi, I hope you are well.

I am trying to test the Rowhammer on the Arty A7 100T board, but when I try export IP_ADDRESS=192.168.100.50 && make srv, it returns the following error: Exception: Unable to probe Etherbone server at 192.168.100.50.

I have built the bitstream using the following commands:
export TARGET=arty

export IP_ADDRESS=192.168.100.50

make build TARGET_ARGS="--variant a7-100"

I manually saved the bitstream to the board's flash memory using vivado, because the make flash TARGET_ARGS="--variant a7-100" command did not work.

After that, with both the USB and Ethernet cables connected to the computer, I tried:
export TARGET=arty

export IP_ADDRESS=192.168.100.50

make srv

and the error occurs.

Also, when trying to simulate, I can't create the TUN interface or configure the IP address of the interface (not even with sudo).

Any ideas of what I am doing wrong?

Thank you for your help.

Hi there,

I've been playing around with various DRAM modules on a ZCU104 trying to get the setup to work, but I've been largely unsuccessful up till now. I'm still trying different troubleshooting steps, but as I've dug deeper into how the whole system works, I've noticed that the clock frequency used on the ZCU104 results in a DRAM clock frequency of 500 MHz (tck = 2 ns). This is surprising to me considering that many DRAM chip data sheets specify maximum clock periods of 1.8 or 1.5 ns per period. Some of the DIMMs I'm testing only spec their operation down to 1600 MT/s, which would be an 800 MHz clock.

I've tried synthesizing with a higher clock frequency, but the LiteX RowHammer module uses the ISERDESE3 blocks in the UltraScale+ I/O. These have a minimum clock period of 1.6ns, or 625 MHz. This doesn't even get us to the 666 MHz that is nominally required for 1333 MT/s speed grade. It seems that enabling realistic DDR4 speeds would entail changes to the upstream LiteX PHY before timing errors resulting from the Rowhammer-tester framework, itself, are even relevant.

I'm still troubleshooting my setup, but at this point I cannot rule out the possibility that the setup is actually running too slow for my DIMMs to operate correctly. Does anybody have any comments on this? I know that normally we worry about things running too fast to work properly and that things running too slowly isn't a problem, but the fact that DRAM spec sheets give max tck specifications leads me to believe there may be some reason for these max values. Perhaps there's a clock synchronizer on the other end that only works above certain frequencies? Perhaps it has to do with refresh?

An example of an error I'm seeing, on one of my modules, is intermittent memory training failure: about half the time, a single bit fails in the memtest at the end of memory training. I've checked and double-checked the module configuration I'm using to build, and I've tried both a Python class defined from data sheet timing and building from the parameters in SPD. The single-bit failure in memory training translates to 1-bit errors in ~200 rows that should not experience errors (because they are far away from the rows being hammered) whenever I run the hw_rowhammer.py script with the parameter --experiment-no 1. Conspicuously, these 1-bit errors in distant rows do not seem to occur when I run the software Rowhammer.py script. I haven't looked into why this might be the case, but my hypothesis is that it has to do with the fact that the BIST logic runs the DRAM at its "capacity" while the wishbone Rowhammer reads allow much more slack in timing.

I'd appreciate any thoughts or advice on why I might be seeing errors like this, and whether the fact that tck is above any realistic DDR4 value could be the cause.

Thanks,
Jacob

Currently it's hardwired in various files (e.g., Makefile, arty.py) which leads to merge conflicts.

I prepared a basic setup for running a Row Hammer attack using EtherBone to fill/check the memory and a simple DMA reader module. I did several tests on the Arty A7 board but was not able to induce any errors in the memory.

Testing procedure

The attack procedure is as follows:

1. DRAM initialization, leveling and a memory test (mem.py)
2. For each row, fill the memory corresponding to that row (all columns) with a given pattern (repeated 32-bit word).
3. Read all the written memory and verify that the data is correct.
4. For each rows pair, perform a row hammer attack, i.e. set the addresses to address0=address(row=row0, col=512) and address1=address(row=row1, col=512) and run alternating DMA reads from these addresses.
5. Read all the written memory once again and find if there were any errors induced.

The pattern, the number of rows, the number of row toggles per rows pair and the choice of row pairs were being changed in different test cases.

Test cases

The initial tests were being run on 1024 rows, attacked in pairs (2n, 2n+1), so (0,1), (2,3), (4,5), ... The pattern used for filling the rows was just 0x55555555, so alternating 1s and 0s in a single row. For each pair there were ~10M row toggles.

Later I did some tests with similar number of rows with the following modifications (added one by one):

• Filling the data with the best pattern according to the paper, i.e. with 1s and 0s per row (odd rows filled with all 1s, even rows filled with all 0s).
• Disabling Auto Precharge to have an explicit ACT+READ+PRE. This didn't in fact affect performance (controller was limited by module timings anyway) and from my understanding shouldn't really make any difference at all.
• Changing the sequence of attacked row pairs to (0, n). To my knowledge this shouldn't make a difference as we still perform the toggling on each row, so we should still be able to induce an error.

Finally I ran longer tests on all 16384 rows of the DRAM chip and ~1.5M row toggles per rows pair (took slightly over 1h per run):

1. Filling even/odd rows with all 0s/1s; attacking pairs of rows (0, n).
2. Filling even/odd rows with all 0s/1s; attacking random pairs of rows (rand(), rand()).
3. Filling each row with a single random 32-bit word, repeated for all the memory corresponding to that row. Attacking pairs (rand(), rand()).

For none of the tests I could identify any error.

I also verified that the reading procedure actually identifies errors by flipping a single bit manually.

Verification in simulation

To verify the correctness of our system I have run it with a simulation model of DRAM in Verilator. I was able to see correct command sequences in waveform dumps. Then I configured the model to log each DFI command that is being sent, printing time/bank/row/column/etc. I prepared a wrapper script that parses all the information and calculates the performance of our DMA reads.

In our setup we run at DDR3-800, with a refresh period of tREF=64ms and refresh command interval of tREFI=64ms/8192. The wrapper script counts the number of row toggles (i.e. of ACT commands) between two REF commands. From my tests we have a median of 93 row toggles between two subsequent REF commands. This gives 764k toggles per tREF, so before that particular row is being refreshed. Following, we have 11.9M toggles per second, which is in the range of frequencies that have been used in "4. Real System Demonstration" of the paper (11.6M, 11.7M, 12.3M and 6.1M).

Side notes

We are currently doing the tests on a single bank. The Row Hammer module could be extended to support attacking all banks at once, which should slightly increase the performance.

During the tests I was able to measure the speed at which we can currently write/read the DRAM memory over EtherBone. From my tests reading 1MB takes ~22 seconds, so reading the whole memory (256MB) should take ~1.5h. Writing is several times faster.

As for the time taken by the above tests on 16384 rows, we were using only 1 bank our of 8, so reading should take 0.1875h, which I measured and it did. Then each attack with ~1.5M toggles took ~0.16 seconds (1.5M/11.9M ~= 0.13s, so a bit of overhead there). Multiplying 0.16s*16833 = 0.728h. So adding everything together (with ~1min of writing) we have around 1h 10min.

Are there any obvious mistakes in the testing methodology used? Do we need better performance or different access patterns? Or is it possible that the DRAM chip we use (MT41K128M16) is not vulnerable?

This is what happens at the end:

ERROR: [DRC UTLZ-1] Resource utilization: F7 Muxes over-utilized in Top Level Design (This design requires more F7 Muxes cells than are available in the target device. This design requires 273875 of such cell types but only 115200 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device.)
ERROR: [DRC UTLZ-1] Resource utilization: F8 Muxes over-utilized in Top Level Design (This design requires more F8 Muxes cells than are available in the target device. This design requires 136545 of such cell types but only 57600 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device.)
ERROR: [DRC UTLZ-1] Resource utilization: LUT as Distributed RAM over-utilized in Top Level Design (This design requires more LUT as Distributed RAM cells than are available in the target device. This design requires 525960 of such cell types but only 101760 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device. Please set tcl parameter "drc.disableLUTOverUtilError" to 1 to change this error to warning.)
ERROR: [DRC UTLZ-1] Resource utilization: LUT as Memory over-utilized in Top Level Design (This design requires more LUT as Memory cells than are available in the target device. This design requires 525968 of such cell types but only 101760 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device. Please set tcl parameter "drc.disableLUTOverUtilError" to 1 to change this error to warning.)
ERROR: [DRC UTLZ-1] Resource utilization: MUXF7 over-utilized in Top Level Design (This design requires more MUXF7 cells than are available in the target device. This design requires 273875 of such cell types but only 115200 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device.)
ERROR: [DRC UTLZ-1] Resource utilization: MUXF8 over-utilized in Top Level Design (This design requires more MUXF8 cells than are available in the target device. This design requires 136545 of such cell types but only 57600 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device.)
ERROR: [DRC UTLZ-1] Resource utilization: RAMD64E over-utilized in Top Level Design (This design requires more RAMD64E cells than are available in the target device. This design requires 524288 of such cell types but only 101760 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device.)
ERROR: [DRC UTLZ-1] Resource utilization: Slice LUTs over-utilized in Top Level Design (This design requires more Slice LUTs cells than are available in the target device. This design requires 660461 of such cell types but only 230400 compatible sites are available in the target device. Please analyze your synthesis results and constraints to ensure the design is mapped to Xilinx primitives as expected. If so, please consider targeting a larger device. Please set tcl parameter "drc.disableLUTOverUtilError" to 1 to change this error to warning.)
INFO: [Vivado_Tcl 4-198] DRC finished with 8 Errors
INFO: [Vivado_Tcl 4-199] Please refer to the DRC report (report_drc) for more information.
ERROR: [Vivado_Tcl 4-23] Error(s) found during DRC. Placer not run.
INFO: [Common 17-83] Releasing license: Implementation
8 Infos, 0 Warnings, 0 Critical Warnings and 9 Errors encountered.
place_design failed
place_design: Time (s): cpu = 00:03:53 ; elapsed = 00:02:21 . Memory (MB): peak = 14902.289 ; gain = 0.000 ; free physical = 8299 ; free virtual = 21718
ERROR: [Common 17-39] 'place_design' failed due to earlier errors.

while executing

"place_design -directive default"
(file "zcu104.tcl" line 45)
INFO: [Common 17-206] Exiting Vivado at Sat Jan 16 10:44:45 2021...
Traceback (most recent call last):
File "rowhammer_tester/targets/zcu104.py", line 244, in
main()
File "rowhammer_tester/targets/zcu104.py", line 241, in main
common.run(args, builder, build_kwargs, target_name=target_name)
File "/home/sqazi/fpga/litex-rowhammer-tester/rowhammer_tester/targets/common.py", line 431, in run
builder.build(**build_kwargs, run=args.build)
File "/home/sqazi/fpga/litex-rowhammer-tester/third_party/litex/litex/soc/integration/builder.py", line 214, in build
vns = self.soc.build(build_dir=self.gateware_dir, **kwargs)
File "/home/sqazi/fpga/litex-rowhammer-tester/third_party/litex/litex/soc/integration/soc.py", line 1048, in build
return self.platform.build(self, *args, **kwargs)
File "/home/sqazi/fpga/litex-rowhammer-tester/third_party/litex/litex/build/xilinx/platform.py", line 53, in build
return self.toolchain.build(self, *args, **kwargs)
File "/home/sqazi/fpga/litex-rowhammer-tester/third_party/litex/litex/build/xilinx/vivado.py", line 344, in build
_run_script(script)
File "/home/sqazi/fpga/litex-rowhammer-tester/third_party/litex/litex/build/xilinx/vivado.py", line 93, in _run_script
raise OSError("Subprocess failed")
OSError: Subprocess failed
make[1]: *** [Makefile:31: build] Error 1
make[1]: Leaving directory '/home/sqazi/fpga/litex-rowhammer-tester'

Can we figure out why it happens and if the situation can be improved? An alternative proposal (if this is not possible to improve) is to introduce the concept of small bounded number of nested loops to reduce the payload size.

Doing the memory checking on the host requires the entire memory to be read over the wishbone bridge. Doing it on the FPGA using the built in self test functionality means it can run at the speed of the memory.

We can use yapf for it. Also the CI should check the style

I have been running this tester on the arty-a7 board. When I started using it, when first following the instructions, the example command

'python hw_rowhammer.py --nrows 512 --read_count 10e6 --pattern 01_in_row --row-pairs const --const-rows-pair 54 133 --no-refresh'

didn't initially produce any errors. I tried a previous commit of the tester from April 20th, and after using the exact same command the tester produced rowhammer errors as expected on rows 53, 55, 132, and 134. I experimented with the current version and found that using the 'const' attack with row numbers defined between 512-1024, 1536-2048, and higher rows between multiples of 512 and skipping multiples of 512 produces expected rowhammer errors (for example, using '--const-rows-pair 550 600' or '--const-rows-pair 1600 1700' both produce rowhammer errors)

Previous commit : a54dc46

Command1.txt
Output of command 1 from current tester
Output of command 1 from older commit

Command2.txt
Output of command 2 from current tester

This is not necessarily a problem to be solved by antmicro, but please double check addressing. We will also do some experiments on our end to see if there is a pattern to how the hammering flips bits.

We would expect to see up to 4 rows with flips for a single pair, usually near the rows accessed.

This is an example of what we actually see instead:

$ python rowhammer.py --nrows 512 --read_count 10e6 --pattern 01_in_row --row-pairs const --const-rows-pair 10 12 --no-refresh
Using generated target files in: ../../build/zcu104

Preparing ...

Filling memory with data ...
................................
Disabling refresh ...

Running row hammer attacks ...
Iter 0 / 1 Rows = ( 10, 12), Count = 10.25M / 10.00M

Reenabling refresh ...

Verifying attacked memory ...
................................
row_errors for row= 36: 315
row_errors for row= 44: 505
row_errors for row= 52: 239

==================================

The addresses seem consistent across runs, so either the DRAM itself has some fancy mapping or there is a problem with partitioning of the address bits into its components.

Dear developers,
We're totally new in this hardware world, sorry if these questions might look obvious to you.

We've been working with ZCU104. While running mem.py, LiteX fails in memtest step. We got this error:

• Termial 1.

(venv) $ make sim
python rowhammer_tester/targets/zcu104.py --build --sim --ip-address 192.168.100.50 --mac-address 0x10e2d5000001 --udp-port 1234 
INFO:SoC:        __   _ __      _  __  
INFO:SoC:       / /  (_) /____ | |/_/  
INFO:SoC:      / /__/ / __/ -_)>  <    
INFO:SoC:     /____/_/\__/\__/_/|_|  
INFO:SoC:  Build your hardware, easily!
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Creating SoC... (2022-05-13 20:13:23)
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:FPGA device : SIM.
INFO:SoC:System clock: 125.000MHz.
INFO:SoCBusHandler:Creating Bus Handler...
INFO:SoCBusHandler:32-bit wishbone Bus, 4.0GiB Address Space.
INFO:SoCBusHandler:Adding reserved Bus Regions...
INFO:SoCBusHandler:Bus Handler created.
INFO:SoCCSRHandler:Creating CSR Handler...
INFO:SoCCSRHandler:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
INFO:SoCCSRHandler:Adding reserved CSRs...
INFO:SoCCSRHandler:CSR Handler created.
INFO:SoCIRQHandler:Creating IRQ Handler...
INFO:SoCIRQHandler:IRQ Handler (up to 32 Locations).
INFO:SoCIRQHandler:Adding reserved IRQs...
INFO:SoCIRQHandler:IRQ Handler created.
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Initial SoC:
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:32-bit wishbone Bus, 4.0GiB Address Space.
INFO:SoC:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
INFO:SoC:IRQ Handler (up to 32 Locations).
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Controller ctrl added.
INFO:SoC:CPU vexriscv added.
INFO:SoC:CPU vexriscv adding IO Region 0 at 0x80000000 (Size: 0x80000000).
INFO:SoCBusHandler:io0 Region added at Origin: 0x80000000, Size: 0x80000000, Mode: RW, Cached: False Linker: False.
INFO:SoC:CPU vexriscv overriding sram mapping from 0x01000000 to 0x10000000.
INFO:SoC:CPU vexriscv setting reset address to 0x00000000.
INFO:SoC:CPU vexriscv adding Bus Master(s).
INFO:SoCBusHandler:cpu_bus0 added as Bus Master.
INFO:SoCBusHandler:cpu_bus1 added as Bus Master.
INFO:SoC:CPU vexriscv adding Interrupt(s).
INFO:SoC:CPU vexriscv adding SoC components.
INFO:SoCBusHandler:rom Region added at Origin: 0x00000000, Size: 0x00010000, Mode: R, Cached: True Linker: False.
INFO:SoCBusHandler:rom added as Bus Slave.
INFO:SoC:RAM rom added Origin: 0x00000000, Size: 0x00010000, Mode: R, Cached: True Linker: False.
INFO:SoCBusHandler:sram Region added at Origin: 0x10000000, Size: 0x00002000, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:sram added as Bus Slave.
INFO:SoC:RAM sram added Origin: 0x10000000, Size: 0x00002000, Mode: RW, Cached: True Linker: False.
INFO:SoCIRQHandler:uart IRQ allocated at Location 0.
INFO:SoCIRQHandler:timer0 IRQ allocated at Location 1.
INFO:SoCCSRHandler:sim_trace CSR allocated at Location 0.
INFO:SoCCSRHandler:sim_marker CSR allocated at Location 1.
INFO:SoCCSRHandler:sim_finish CSR allocated at Location 2.
INFO:SoCCSRHandler:leds CSR allocated at Location 3.
INFO:SoC:Using DRAM module MTA4ATF51264HZ ratio 1:4
Traceback (most recent call last):
  File "rowhammer_tester/targets/zcu104.py", line 279, in <module>
    main()
  File "rowhammer_tester/targets/zcu104.py", line 269, in main
    soc = SoC(**soc_kwargs)
  File "rowhammer_tester/targets/zcu104.py", line 188, in __init__
    super().__init__(**kwargs)
  File "/home/scalar4/source/zcu104/litex-rowhammer-tester/rowhammer_tester/targets/common.py", line 163, in __init__
    verbosity = 3,
  File "/home/scalar4/source/zcu104/litex-rowhammer-tester/third_party/litedram/litedram/phy/model.py", line 576, in __init__
    verbose      = verbosity > SDRAM_VERBOSE_DBG)
  File "/home/scalar4/source/zcu104/litex-rowhammer-tester/third_party/litedram/litedram/phy/model.py", line 236, in __init__
    self.prepare_timings(timings, refresh_mode, memtype)
  File "/home/scalar4/source/zcu104/litex-rowhammer-tester/third_party/litedram/litedram/phy/model.py", line 211, in prepare_timings
    val = val[refresh_mode]
TypeError: tuple indices must be integers or slices, not str
Makefile:37: recipe for target 'sim' failed
make: *** [sim] Error 1

• Termial 2.

(venv) $ python mem.py
Using generated target files in: ../../build/zcu104
 === Waiting for CPU to initialize DRAM ===
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2022 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on May 13 2022 19:21:50
 BIOS CRC passed (42d2439d)

 Migen git sha1: ac70301
 LiteX git sha1: 80b309d1

--=============== SoC ==================--
CPU:		VexRiscv_Min @ 125MHz
BUS:		WISHBONE 32-bit @ 4GiB
CSR:		32-bit data
ROM:		64KiB
SRAM:		8KiB
L2:		0KiB
SDRAM:		1048576KiB 64-bit @ 1000MT/s (CL-9 CWL-9)

--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB     
   Read: 0x40000000-0x40200000 2.0MiB     
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
  Write speed: 49.5MiB/s
   Read speed: 52 === Initialization succeeded. ===
Proceeding ...

Memtest (basic)
Traceback (most recent call last):
  File "mem.py", line 89, in <module>
    run_memtest('basic', itertools.cycle([0xaaaaaaaa, 0x55555555]))
  File "mem.py", line 79, in run_memtest
    errors = memtest(wb, length=memtest_size, generator=generator, **kwargs)
  File "mem.py", line 23, in memtest
    data = memread(wb, length, base=base, burst=burst)
  File "/home/scalar4/source/zcu104/litex-rowhammer-tester/rowhammer_tester/scripts/utils.py", line 173, in memread
    data += wb.read(base + 4 * i, min(burst, n - i))
  File "/home/scalar4/source/zcu104/litex-rowhammer-tester/third_party/litex/litex/tools/litex_client.py", line 67, in read
    packet.decode()
  File "/home/scalar4/source/zcu104/litex-rowhammer-tester/third_party/litex/litex/tools/remote/etherbone.py", line 306, in decode
    header = list(ba[:etherbone_packet_header.length])
TypeError: 'int' object is not subscriptable

We tried with Samsung M471A5244CB0-CTD(DDR4) and Samsung M471A1G44AB0-CWE(DDR4), but still failed. And also, We maually built boot image. It failed again. Lastly, We checked network setup with Wireshark but cannot found any problems.

We accessed FPGA to check Etherbone, surprisingly, /bin/zcu104_etherbone was stuck even though they sending or receiving packets. This issue also occurred when DEBUG is enabled in firmware/zcu104/etherbone/Makefile.

# /bin/zcu104_etherbone 
Serving on port 1234 ...

Thank you for your explanations.

As disscussed in #30 (comment) we would like an instruction that would terminate payload execution without having to execute whole payload memory.

It seems reasonable to use NOOP with timeslice=0 for this purpose. This way memory filled with all zeros would instantly finish execution. Currently instructions with timeslice=0 take 1 cycle to execute (in general timeslice + 1 cycles). With the proposed change we would need to change interpretation of timeslice to be exactly the number of instruction execution cycles.

This solution introduces a case for instructions other than NOOP with timeslice=0. It would be very counterintuitive to interpret timeslice differently for NOOP than for other instructions. But then we have a case where timeslice=0 is illegal for instructions other than NOOP (because we need at least 1 cycle for execution).

I would propose to always have cycles=timeslice. Non-NOOP with timeslice=0 would be an "illegal instruction". In the implementation it will most likely be silently executed the same as if timeslice=1, but this will be considered "undefined behavior". Would it be ok?

I want to run Row Hammer attack tests after changing the values of Mode Registers for DRAM, not a default mode. That is, I want to set the address and control/command value to the desired value every cycle before Row Hammer attack.

I think this question might be related to 'litedram' module. It seems to be available by modifying the sequence and code in the 'sdram_init.py' and 'utils.py' files you provided, but the delay value is added in 'utils.py' file when 'sdram_software_control' mode is on (e.g. 'time.sleep(0.01 + delay * 1e-5') )

Can I set the address and control/command value every cycle?
If so, can the control (dfii_control) and command (dfii_pi0_command) value be set simultaneously at the same cycle?

Thank you for your help.

The original slower implementation supports a variety of patterns, but the script for the hardware implementation only supports single pattern used across all the rows. At the very least, we think it should support patterns that fill the hammered rows with one pattern and the remaining rows with its inverse. In our experience elsewhere, this helps with the probability of getting bit flips.

In our software tool (https://github.com/google/hammer-kit), the way we specify data patterns might interest you. We specify a divisor and we divide the row number by this divisor and look at the remainder. We check the remainder against a bitmask, and if the corresponding bit is set, then we consider it a "victim row". Victim rows get the inverse pattern from all other rows.

$ git log --oneline |head -4
b76dd14 Change IP address to match setup
9e35e50 Resuse address bits in NOOP instructions for TIMESLICE
36dd7ec Add support for multiple ranks
4ea1ee2 Add payload execution scratchpad memory for data from READ commands

$ make upload
./third_party/xc3sprog/xc3sprog -c nexys4 build/arty/gateware/arty.bit
XC3SPROG (c) 2004-2011 xc3sprog project $Rev: 774 $ OS: Linux
Free software: If you contribute nothing, expect nothing!
Feedback on success/failure/enhancement requests:
http://sourceforge.net/mail/?group_id=170565
Check Sourceforge for updates:
http://sourceforge.net/projects/xc3sprog/develop

Using Libftdi,

(venv) sqazi@sqazi-glaptop:~/fpga/litex-rowhammer-tester/scripts$ python mem.py
SDRAM initialization:
control_cmds: [0, 1]
Release reset (ctrl)
Bring CKE high (ctrl)
Load Mode Register 2, CWL=5 (cmd)
Load Mode Register 3 (cmd)
Load Mode Register 1 (cmd)
Load Mode Register 0, CL=6, BL=8 (cmd)
ZQ Calibration (cmd)

Read leveling:
Module 0
Bitslip 00: |11111111111100000000000000000000|
Bitslip 01: |00000000000000111111111111111000|
Bitslip 02: |00000000000000000000000000000011|
Bitslip 03: |00000000000000000000000000000000|
Bitslip 04: |00000000000000000000000000000000|
Bitslip 05: |00000000000000000000000000000000|
Bitslip 06: |00000000000000000000000000000000|
Bitslip 07: |00000000000000000000000000000000|
Best: bitslip = 1, delay = 21 (+-7)
Module 1
Bitslip 00: |11111111111100000000000000000000|
Bitslip 01: |00000000000000111111111111110000|
Bitslip 02: |00000000000000000000000000000011|
Bitslip 03: |00000000000000000000000000000000|
Bitslip 04: |00000000000000000000000000000000|
Bitslip 05: |00000000000000000000000000000000|
Bitslip 06: |00000000000000000000000000000000|
Bitslip 07: |00000000000000000000000000000000|
Best: bitslip = 1, delay = 21 (+-7)

Memtest (basic):

1. ['0x55555555', '0x55555555', '0xaaaaaaaa', '0xaaaaaaaa']
2. ['0xaaaaaaaa', '0xaaaaaaaa', '0x55555555', '0x55555555']
   Mem ok!

Memtest (random):
OK

(venv) sqazi@sqazi-glaptop:~/fpga/litex-rowhammer-tester/scripts$ python execute_payload.py
Transferring the payload ...
Executing ...
Finished
Scratchpad contents:
0x00000000: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000010: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000020: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000030: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000040: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000050: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000060: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000070: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000080: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000090: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000000a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000000b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000000c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000000d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000000e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000000f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000100: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000110: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000120: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000130: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000140: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000150: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000160: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000170: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000180: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000190: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000001a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000001b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000001c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000001d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000001e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000001f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000200: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000210: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000220: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000230: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000240: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000250: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000260: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00000270: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000280: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000290: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000002a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000002b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000002c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000002d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000002e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000002f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000300: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000310: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000320: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000330: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000340: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000350: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000360: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000370: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000380: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000390: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000003a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000003b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000003c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000003d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000003e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000003f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000400: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000410: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000420: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000430: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000440: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000450: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000460: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000470: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000480: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000490: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000004a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000004b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000004c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000004d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000004e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000004f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000500: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000510: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000520: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000530: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000540: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000550: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000560: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000570: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000580: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000590: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000005a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000005b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000005c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000005d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000005e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000005f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000600: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000610: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000620: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000630: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000640: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000650: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000660: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00000670: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000680: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000690: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000006a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000006b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000006c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000006d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000006e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000006f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000700: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000710: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000720: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000730: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000740: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000750: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000760: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000770: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000780: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000790: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000007a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000007b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000007c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000007d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000007e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000007f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000800: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000810: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000820: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000830: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000840: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000850: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000860: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000870: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000880: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000890: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000008a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000008b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000008c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000008d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000008e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000008f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000900: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000910: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000920: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000930: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000940: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000950: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000960: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000970: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000980: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000990: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000009a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000009b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000009c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000009d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000009e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000009f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a60: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00000a70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000a90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000aa0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ab0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ac0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ad0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ae0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000af0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000b90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ba0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000bb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000bc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000bd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000be0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000bf0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000c90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ca0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000cb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000cc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000cd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ce0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000cf0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000d90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000da0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000db0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000dc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000dd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000de0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000df0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e60: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00000e70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000e90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ea0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000eb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ec0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ed0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ee0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ef0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000f90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000fa0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000fb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000fc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000fd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000fe0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00000ff0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001000: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001010: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001020: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001030: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001040: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001050: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001060: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001070: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001080: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001090: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000010a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000010b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000010c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000010d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000010e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000010f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001100: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001110: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001120: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001130: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001140: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001150: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001160: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001170: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001180: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001190: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000011a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000011b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000011c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000011d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000011e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000011f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001200: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001210: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001220: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001230: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001240: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001250: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001260: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00001270: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001280: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001290: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000012a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000012b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000012c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000012d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000012e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000012f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001300: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001310: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001320: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001330: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001340: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001350: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001360: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001370: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001380: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001390: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000013a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000013b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000013c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000013d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000013e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000013f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001400: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001410: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001420: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001430: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001440: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001450: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001460: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001470: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001480: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001490: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000014a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000014b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000014c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000014d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000014e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000014f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001500: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001510: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001520: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001530: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001540: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001550: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001560: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001570: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001580: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001590: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000015a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000015b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000015c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000015d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000015e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000015f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001600: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001610: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001620: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001630: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001640: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001650: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001660: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00001670: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001680: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001690: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000016a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000016b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000016c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000016d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000016e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000016f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001700: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001710: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001720: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001730: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001740: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001750: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001760: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001770: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001780: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001790: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000017a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000017b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000017c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000017d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000017e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000017f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001800: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001810: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001820: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001830: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001840: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001850: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001860: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001870: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001880: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001890: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000018a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000018b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000018c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000018d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000018e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000018f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001900: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001910: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001920: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001930: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001940: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001950: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001960: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001970: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001980: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001990: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000019a0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000019b0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000019c0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000019d0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000019e0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x000019f0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a60: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00001a70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001a90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001aa0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ab0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ac0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ad0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ae0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001af0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001b90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ba0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001bb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001bc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001bd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001be0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001bf0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001c90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ca0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001cb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001cc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001cd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ce0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001cf0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001d90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001da0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001db0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001dc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001dd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001de0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001df0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e60: fd ed f1 a4 ff ef fd ff fd ff 00 00 00 00 00 00 ................
0x00001e70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001e90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ea0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001eb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ec0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ed0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ee0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ef0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f00: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f10: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f20: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f30: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f40: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f50: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f60: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f70: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f80: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001f90: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001fa0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001fb0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001fc0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001fd0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001fe0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............
0x00001ff0: f7 ff fd 7a ff ef f5 9f f5 9f 00 00 00 00 00 00 ...z............

Let me know if you need any additional information / experiments.

Thanks Jedrzej. One question -- why is EtherBone so slow? 1MB/22s = 45KB/s. I was expecting a bandwidth that's two orders of magnitude larger. If reading out the module takes an hour, that's going to really slow down experiments. Will this get faster for future implementations and/or for higher-speed FPGA boards?

Originally posted by @yoongu in #3 (comment)

Would it be possible to keep track of the number of refresh commands issued, regardless of the reason (i.e. both from the payload executor and normal refresh-enabled operation). I would like to have access to this counter atomically with either the beginning or the end of payload execution. Since we know how many refreshes our payload issues, it should be possible to convert from one to the other. So, whichever way is easier for you is fine with me.

When running the dump_regs.py (after entering python virtual env.) I'm seeing this:

(venv) $ python dump_regs.py 
Using generated target files in: ../../build/zcu104
0x82000000: 0xffffffff ctrl_reset
0x82000004: 0xffffffff ctrl_scratch
0x82000008: 0xffffffff ctrl_bus_errors
0x82002000: 0xffffffff uart_rxtx
0x82002004: 0xffffffff uart_txfull
0x82002008: 0xffffffff uart_rxempty
0x8200200c: 0xffffffff uart_ev_status
...
0x82003800: 0xffffffff ddrphy_rst
0x82003804: 0xffffffff ddrphy_en_vtc
0x82003808: 0xffffffff ddrphy_half_sys8x_taps
0x8200380c: 0xffffffff ddrphy_wlevel_en
0x82003810: 0xffffffff ddrphy_wlevel_strobe
0x82003814: 0xffffffff ddrphy_cdly_rst
0x82003818: 0xffffffff ddrphy_cdly_inc
...
0x82005000: 0xffffffff ddrctrl_init_done
0x82005004: 0xffffffff ddrctrl_init_error
...

Which is a problem because when I go to run:

(venv) $ python mem.py 
Using generated target files in: ../../build/zcu104
 === Waiting for CPU to initialize DRAM ===

It gets stuck in a loop because it receives 0xFFFFFFFF for ddrctrl_init_done. I know the communication with the EtherBone on the ZCU104 is working:

(venv) $ python etherbone_perf.py memwrite 32768 --burst 128
Using generated target files in: ../../build/zcu104
Elapsed = 0.097 sec
Size    = 128.000 KB
Speed   = 1.290 MBps

I can't seem to dump values of the CSR, any ideas on what is going on?

Now when the executor is ready we should have a CI system checking for the regressions. Let's use migen sim for unit testing and Verilator for system tests

Both Arty and LPDDR4 test board targets support permanent configuration of the board (a bitstream can be load to on-board's QSPI flash and the FPGA will be autoconfigured on startup).

We should add a flash target to the main Makefile, so a user can update on-board's flash with simple make flash command

Hi,

I am having some issues with the litex_server connection on the ZCU 104 board. I am using the latest prebuilt image posted here and I was having the same connection issues discussed in #74.

As per previous discussions on issue #74, I have recompiled the Etherbone server with gcc-aarch64-linux-gnu and the debug option on. I can see that now the make srv command works and sends and receive some data. However, as I try the leds.py test, it fails.

The output during the test looks something like below:

Before running the test (only the server running),
host:

make srv
litex_server --udp --udp-ip 192.168.100.50 --udp-port 1234
[CommUDP] ip: 192.168.100.50 / port: 1234 / tcp port: 1234

zcu104:

Received 12 byte packet
4e 6f 11 44 00 00 00 00 
00 00 00 00 
Sending 8 byte response
4e 6f 16 44 00 00 00 00

After running the test:

host (terminal 1):

python leds.py 
Using generated target files in: ../../build/zcu104
Didn't get a reponse from the board. Check connection?

host (terminal 2)

litex_server --udp --udp-ip 192.168.100.50 --udp-port 1234
[CommUDP] ip: 192.168.100.50 / port: 1234 / tcp port: 1234
Connected with 127.0.0.1:58214
Disconnect
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/abc/rowhammer-tester/third_party/litex/litex/tools/litex_server.py", line 153, in _serve_thread
    reads += self.comm.read(addr, length, burst)
  File "/home/abc/rowhammer-tester/third_party/litex/litex/tools/remote/comm_udp.py", line 127, in read
    raise socket.timeout
socket.timeout

ZCU104:

Received 12 byte packet
4e 6f 11 44 00 00 00 00 
00 00 00 00 
Sending 8 byte response
4e 6f 16 44 00 00 00 00 
Received 12 byte packet
4e 6f 11 44 00 00 00 00 
00 00 00 00 
Sending 8 byte response
4e 6f 16 44 00 00 00 00 
Received 20 byte packet
4e 6f 10 44 00 00 00 00 
00 0f 00 01 00 00 00 01 
f0 00 58 00 
0xf0005800 => 0x00000052
Sending 20 byte response
4e 6f 14 44 00 00 00 00 
00 0f 01 00 01 00 00 00 
00 00 00 52 
Received 20 byte packet
4e 6f 10 44 00 00 00 00 
00 0f 00 01 00 00 00 02 
f0 00 58 00 
0xf0005800 => 0x00000052
Sending 20 byte response
4e 6f 14 44 00 00 00 00 
00 0f 01 00 02 00 00 00 
00 00 00 52 
Received 20 byte packet
4e 6f 10 44 00 00 00 00 
00 0f 00 01 00 00 00 03 
f0 00 58 00 
0xf0005800 => 0x00000052
Sending 20 byte response
4e 6f 14 44 00 00 00 00 
00 0f 01 00 03 00 00 00 
00 00 00 52 
Received 20 byte packet
4e 6f 10 44 00 00 00 00 
00 0f 00 01 00 00 00 04 
f0 00 58 00 

....

Any suggestion why this might happen? Is there any updated ZCU 104 image that have the fixed Etherbone server? Thanks 🙏

python rowhammer.py 512 -> python rowhammer.py --nrows 512