calliope-edu / calliopeo_astropi Goto Github PK

When you answer the question "Confirm, Calliope Mini is attached to USB [y]" in testcase 3 too fast, testcases 3 and sometimes 4 can fail. After connecting the CalliopEO, the OS needs some time to properly connect the hardware.

This can be solved by adding a wait loop after the user input to the question.

In most test cases we do not need a perfect md5 hash. It mostly does not matter if there is a line more or less.
Changing the hashing from md5 to some fuzzy hash like SSDEEP or sdhash might help getting less false negative tests.

Keep the status weather the mini is connected or not in a variable will help us minimize the user input during tests.

There some comments in the code we do not need anymore.

I encountered a problem during testing that after processing a corrupted zip file the CalliopEO.py script proceeded too fast. For the next zip the script tried to create a run_YYYYMMDD-HHMMSS with the same date/time stamp.

This can be easily fixed by introducing a wait of at least 1 second.

In a couple of testcases md5sum is used to verify the correct output of the Calliope Mini after flashing with a dedicated hex.

Sometimes, the verification goes wrong with md5sum claiming, the files to verify are not in place:

-=# CalliopEO #=-
Unpacked /home/calliope/src/CalliopEO_AstroPi/30sec-counter.zip


programming: /home/calliope/src/CalliopEO_AstroPi/run_20210721-230028/30sec-counter.hex
done
open serial port
Calliope Mini found on /dev/ttyACM0  ..........
done
reading data

Start @ 2021/07/21-23:00:44; Will stop @ 2021/07/22-02:05:44

Send @START@

Received @START@
*****************
17 lines read
done
############################################################################################################

Check: Return code in all cases is 0 ... PASSED
Check: ZIP archive renamed to .done ... PASSED
Check: Folder run_* created ... PASSED
Check: Created two .data files ... PASSED
Check: Verify content of .data file ... md5sum: 30sec-counter.hex: No such file or directory
md5sum: 30sec-counter.hex.data: No such file or directory
md5sum: WARNING: 2 listed files could not be read
NOT PASSED

Reason currently unknown ...

The sync and sleep 1 in all testcases after executing the Python script seems not to be sufficient.

Please, provide simple/basic test program in a subfolder (e.g. calliope_progs). The test program should follow the "boiler plate".
The program should run on a commercial Calliope Mini as well as the modified hardware for the usage onboard the ISS. It should also provide some feedback e.g. via the LED matrix about the status:

• Waiting for Raspberry Pi to send the initial @start@
• Transmitting data
• Finished transmitting data

Please, consider adding a license for this software. Other projects of calliope-edu are using the Apache 2.0 license.

George from Raspberry Pi Foundation recommended to provide a setup script that automates everything that is described in the README. Essentially, a script that would start from the latest Flight OS (download link provided via email) and leave the environment with everything that would be required for Calliope. He also noted, that payload developers prepare a compressed file that is uploaded to the ISS and is supposed to contain everything required for the update. So system updates will require uploading .deb files and Python updates will require uploading Python wheels.

When executed as calliopeI get the following error:
[Errno 13} could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'

User calliope is part of dialout and dialout exists. /dev/ttyACM0 is owned by root:dialout and has 660 permissions. Chown 666 doesnt change anything.

We have ser.timeout=10 in two places. It might speed up the script, when we decrease the timeout to 1s. Are there any downsides?

The filesize is more a guess than a calculation. This might need some improvement. We can keep this if the filesize limit is not that important and we set it high enough.

This needs to be updated and then included in the readme:

Currently, a newly read line from the Calliope Mini is attached to the list lines (see here). Only after this, the list lines is checked against the threshold max-data-size. This allows, that the size can be slightly above the threshold (maximum: size of one line).

This makes this feature "weak" - an attacker could add arbitrary amount of data to the list by providing a very long last line. In addition, this behavior makes it hard to test this feature, because the actual size of the .data file can be any higher than the threshold.

It is better to allow inserting the current line to the list lines only in the case, that the resulting size meets the threshold.

At the moment the script only handle .hex files who are directly zipped.
If it happens that someone Zips a complete Folder the unpacked run_* folder will contain a subfolder where the .hex files are (*.zip/folder/file.hex). These hex files wont be processed.

Unzip has a -j option to do not create subdirectories: unzip -j /path/to/file.zip. I'm not aware if shutil.unpack_archive() can do something similar.

But having multiple folders like

• *.zip/folder1/file1.hex
• *.zip/folder2/file1.hex

might result in ending with only one hex file instead of two. So It could also be an option to just cancel the process if the zip file contains one (or more than one) subfolder und report an error?

It could also be, that someone

Flashing a corrupted hex file is no deterministic testcase. When flashing a corrupted hex file the Calliope Mini keeps the previously flashed hex file and boots it. Therefore, the output depends from the previously hex file.

For a testcase, it is better to first flash a valid hex file and only after this the corrupted hex file.

Add shebang #!/usr/bin/env python3 and make chmod +x CalliopEO.py. Then, the script can be executed with ./CalliopEO.py.

Currently, the user is hard-coded in the script (see e.g. here). Adapt the script in a way, that it could be executed by a user with different name. This way, the script does not require that a dedicated user will be established.

When adding a User in dialout group I get the following error when running the script:

Traceback (most recent call last):
    File "CalliopEO.py", line 11, in <module>
        blk - blkinfo.BlkDiskInfo()
AttributeError: module 'blkinfo' has no attribute 'BlKDiskInfo'

blkinfo was installed by downloading the source file and run python3 -m pip install <filename> <target> as root.

Therer are some things where the serial connection could be improved:

• Repeating connection until mini is reachable instead of fixed sleep()
• repeat sending start string and shorten answer timeout

Currently, there is no straight forward way to test CalliopEO.py. Unfortunately, the script or functions are not object-oriented which makes the usage of tools like unittest a little tricky.

I would recommend to establish a bash script for testing, which we could name testing.sh. This script itself can source a couple of files in a subdirectory. The very first file to source could be a config file (00_config.sh) to set up any bash variable needed for testing. For every test case, we can establish a dedicated file NN_testcase_name.sh, where NN is a number defining the order in which the test cases are executed and testcase_name can be replaced by a short name for the test case.

The test cases can be:

• Run script without Calliope Mini
• Run script with Calliope Mini attached but with no ZIP archive in folder
• Run script with Calliope Mini attached but with invalid ZIP file
• ...

Each bash file for a test case should contain the following:

• Instructions for the test person to establish a proper set-up for the test (e.g. connect or disconnect the Calliope Mini, copy ZIP file to directory). This instructions are basically echo commands.
• Bash instructions to set-up the rest of the environment
• Bash instructions to execute the CalliopEO.py script
• Bash instructions to test the results
• Bash instructions to create a small report for the test case, most important the result for the particular test case

What are your thoughts?

I wonder if we should combine the test cases.

The actuall aproach is like this:

# 1. Stage: Execute the CalliopEO.py script with nominal hex file
cp ${zipfile1} .
${cmd_calliope_script}
# Save return code
ret_code_1=$?

# 2. Stage: Execute the CalliopEO.py script with corrupted zip file
cp ${zipfile2} .
${cmd_calliope_script}
# Save return code
ret_code_2=$?

# 3. Stage: Execute the CalliopEO.py script again with nominal zip file
cp ${zipfile3} .
${cmd_calliope_script}
# Save return code
ret_code_3=$?

That way the python script runs three times.
In a real world scenario we might have 10 .zip files each containing 5 .hex files uploaded at once. Maybe on of them is corrupted and 9 are fine. Then the python script runs one time and we want to be sure, that the script does not stop after it reaches the corrupted zip file.

So I feel like that might result in more stating tests:

cp ${zipfile1} ./01.zip
cp ${zipfile2} ./02.zip
cp ${zipfile1} ./03.zip
${cmd_calliope_script}
# Save return code
ret_code_1=$?

…

# Return code of script is 0?
echo -n "Check: Return code is 0 ... "
if [[ ${ret_code1} -eq 0 ]]; then
    echo "PASSED"
else
    echo "NOT PASSED"
fi

# Renamed .zip to .zip.done or zip.failed?
zipfile1_main=01.zip
zipfile1_done="${zipfile1_main}.done"
zipfile2_main=02.zip
zipfile2_failed="${zipfile2_main}.failed"
zipfile3_main=03.zip
zipfile3_done="${zipfile3_main}.done"
echo -n "Check: ZIP archive renamed to .done/.failed ... "
if [[ ! -e "${zipfile1_main}" && -e "${zipfile1_done}" && ! -e "${zipfile2_main}" && -e "${zipfile2_failed}" && ! -e "${zipfile3_main}" && -e "${zipfile3_done}" ]]; then
    echo "PASSED"
else
    echo "NOT PASSED"
fi

The same applys to test 05 corrupted hex and 07 transm timeout.

Track the results of each test and providing a summary would be nice.

To be done:

• Close all relevant issues
• Update testresults
• Update changelog
• Release main version
• Release essential version

Should we provide a best practice to prevent script exit caused by ssh timeout or isn't that in our scope?
I just tried some longer scripts and got around with nohup

Suggested by esa: I would therefore recommend that you create software that exercise the operational capabilities on Astro Pi and CalliopEO in a tight controlled loop;

• On the Astro Pi mark II; just have a Python script running that will copy the same test program over to the CalliopEO repeatedly and record serial data being received from the CalliopEO to disk. This is basically the full extent of the Astro Pi involvement.
• The test program (hex file) should run on CalliopEO for maybe 15 seconds and rapidly read from each sensor and test every capability that will be used on orbit and transmit the result over serial to the Astro Pi (with a time stamp). Note the time stamp could be implemented on the Astro Pi side if you want.
• When the Astro Pi receives the finished message over serial from CalliopEO the whole process restarts with the Astro Pi copying the test program over again.

@Amerlander: Please provide the .js files to the folder testfiles for reference.

https://github.com/calliope-mini/microbit-dal/blob/ae1b5379d3bf9477ce95afa9b7e2ab9b96a2f559/source/drivers/MicroBitCompassCalibrator.cpp#L292

Not really a code issue, but in the case you missed my SMS:

I opened the channel ##calliope on libera.chat in the case we like to exchange via good-old IRC ...

You can also use the webchat: https://web.libera.chat/

Make sure there is no

• run_* folder
• Zip File
  present in main folder.

We could insert a timestamp in front of each received serial line.

When executed testing.sh shall create a list of available test cases from the directory testcases/ with trailing number, like:

 1    01_first_testcase
 2    02_second_testcase
 3    03_third_testcase

Then ask the user to specify the sequence of test cases by entering the numbers from the first column (use the bash function readarray).

Option: Expand user inputs like 8 3-6 1 to 8 3 4 5 6 1 or 1 4- to 1 4 5 6 7 8 or -3 to 1 2 3.

Where do we place the python file?

I did it like this:

• Place everything in /var/tmp/CalliopEO/
• Run setup script
• Copy everything to /home/calliope/tmp/CalliopEO/
• Then I had to change user rights of the files to be executeable by calliope, I just did chown -R calliope /home/calliope/tmp/CalliopEO. Should we define where to place the files and handle the file rights in the setup script?

Showing the individual md5 hashes should help to make sure that files are correct and the same.

Reference https://makecode.calliope.cc/_JaXhHVDH3V8e as a basic program.

CalliopEO.py currently needs "line-oriented" data from the Calliope Mini. When Calliope Mini transmits "unstructured" data without newlines, CalliopEO.py will not continue processing the data (until it eventually receives a newline).

This could be improved by reading abitrary chunks of data from the serial input buffer. Later, CalliopEO.py can extract the lines and do further processing like adding the time stamp and so on.

Maybe, we can define this as a feature for a v2.

Add installable library files and introductions how to install offline.

Output says:

Start #10 ./testcases/08a_data-limit.sh
---------------------------------
Test: Handle data limit threshold
---------------------------------

-=# CalliopEO #=-
Unpacked /home/calliope/src/CalliopEO_AstroPi/01.zip


programming: /home/calliope/src/CalliopEO_AstroPi/run_20210810-184855/01.hex
done
open serial port
Calliope Mini found on /dev/ttyACM0  ..........
done
reading data

Start @ 2021/08/10-18:49:11; Will stop @ 2021/08/10-21:54:11

Sending @START@

Sending @START@

Received @START@
*****************************************************************************************************************************************************************************
**************************************
Max file size achieved

done
############################################################################################################

Check 1/6: Return code is 0 ... PASSED
Check 2/6: ZIP archive renamed to .done ... PASSED
Check 3/6: Folder run_* created ... PASSED
Check 4/6: Created two .data files ... PASSED
Check5/6: MD5 checksum in folder ./run_20210810-184855 ... md5sum: WARNING: 1 computed checksum did NOT match
NOT PASSED
Check 6/6: First .data meets the size threshold (8000 bytes) ... PASSED
Testcase failed.
---------------------------------
Finished #10 ./testcases/08a_data-limit.sh

The timeout for waiting for the calliope to answer @start@ could be decreased a little bit.
Also the repeating of 5 times might be too much - the chances are high, that it will always fail if it happens two times in a row. So maybe 3 would be enough?