RFC: Integrate riscv-tests about semu HOT 9 CLOSED

Hello, I would like to integrate riscv-tests into semu project.
Which is recommended way to implement test into semu?

1. Assign only one test binary to semu. The test command would be

   ./semu riscv-tests ./test/riscv-tests/rv64ui_p_add
   

I think the first approach is reasonably making sense to semu since we don't have to modify the loader and/or parsing logic. Can you show something more? e.g., how to trigger the tests and validate from the results.

from semu.

I think the first approach is reasonably making sense to semu since we don't have to modify the loader and/or parsing logic. Can you show something more? e.g., how to trigger the tests and validate from the results.

In main function, semu compares argv[1] if it is "riscv-tests" to run specific test function (e.g., semu_test_start()):

if(!strncmp(argv[1], "riscv-tests", strlen("riscv-tests")))
    semu_test_start(argc, argv);
else
    semu_start(argc, argv);

Refer to rvemu project, semu_test_start() is different with original CPU execution.
It doesn't need to handle pendding interrupt, and it also uses a counter to limit the execution loop of CPU.

To validate the test result, the riscv-tests sets a0 register to indicate the test result.
If a0 equals zero, it means TVM (Test Virtual Machine) passes the test.

#define RVTEST_PASS                                                     \
        fence;                                                          \
        li TESTNUM, 1;                                                  \
        li a7, 93;                                                      \
        li a0, 0;                                                       \
        ecall

#define TESTNUM gp
#define RVTEST_FAIL                                                     \
        fence;                                                          \
1:      beqz TESTNUM, 1b;                                               \
        sll TESTNUM, TESTNUM, 1;                                        \
        or TESTNUM, TESTNUM, 1;                                         \
        li a7, 93;                                                      \
        addi a0, TESTNUM, 0;                                            \
        ecall

The pseudo-code would be:

Open and read one test binary
Allocate new CPU

while counter < 1000
    count +1
    CPU fetches instruction
    CPU executes instruction
end

Semu checks if CPU register a0 equals zero.

from semu.

You can simply accept only one parameter --test which indicates different entry point for emulation. No other parameter should be passed since we can assume the file names for riscv-tests.

from semu.

Sorry I am not sure your point.

I guess the command would be:

./semu --test

For approach one, semu only executes one test binary at a time.
Assume the binary is ./test/riscv-tests/rv64ui_p_add, shouldn't we pass file path to semu?
There are many binaries, rv64ui_p_add, rv64ui_p_addi, rv64ui_p_addiw, etc.

from semu.

For approach one, semu only executes one test binary at a time. Assume the binary is ./test/riscv-tests/rv64ui_p_add, shouldn't we pass file path to semu? There are many binaries, rv64ui_p_add, rv64ui_p_addi, rv64ui_p_addiw, etc.

You can predefine the directory structure for the option --test, which is eventually acknowledged by semu internally. The reason why a special option --test exists is that semu has to proceed the executions of several binary files generated by riscv-tests in turn.

In particular, we might have two variants for the above:

• semu --test : Run all binary files in predefine directory generated by riscv-tests.
• semu --test rv64ui_p_addi : Run the specific test item explicitly.

Since the test procedure is quite tedious, I think it would be better to regard it as unusual operations without the flexibility to assign individual path and/or file names.

from semu.

The riscv-tests has been merged.
However, there are still some building optimizations to try:

1. Build riscv-tests/isa only
2. Build RV64 tests only
3. Skip autoconf
4. Skip make install

from semu.

I would expect the following usage:

semu --test rv64ui_p_addi

It could run the specific test item explicitly, useful when we plan to re-run certain instruction test(s).

from semu.

The riscv-tests has been merged. However, there are still some building optimizations to try:

1. Build riscv-tests/isa only
2. Build RV64 tests only
3. Skip autoconf
4. Skip make install

Done in patch #5 except option 2.

from semu.

In particular, we might have two variants for the above:

• semu --test : Run all binary files in predefine directory generated by riscv-tests.
• semu --test rv64ui_p_addi : Run the specific test item explicitly.

Since the test procedure is quite tedious, I think it would be better to regard it as unusual operations without the flexibility to assign individual path and/or file names.

Since all the features are finished, I close this issue.

from semu.