codecoup / auto-pts Goto Github PK

View Code? Open in Web Editor NEW

This project forked from auto-pts/auto-pts

0.0 0.0 2.0 1.97 MB

The Bluetooth PTS automation framework

License: GNU General Public License v2.0

Python 100.00%

auto-pts's Issues

See if we can refactor more test names in ptsprojects/{project}/gap.py

Maybe we can put set_io_cap(display_only) in pre_conditions and thus refactor more tests from the "custom" list

Add unit test coverage

Add unit tests that will cover as much code as possible so that we can verify that there are no syntax errors at least.

Create a CI pipeline

Evaluate which CI solution to use (e.g. Jenkins)
Set up the CI on a dedicated machine
Integrate auto-pts with the CI to allow scheduling and running jobs
Allow choosing and modifying job configuration via a website (Jenkins)
Add support for triggering jobs from new/updated PRs (waiting for github token from IT Admin :))

Implement a pyserial connection between controller and autoptsclient on Windows

For now, Windows workaround for autoptsclient-zephyr uses socat.exe to connect to the controller. Since socat.exe has no support and it is just quick bypass to make autoptsclient-zephyr work on Windows, appropriate solution with pyserial should be implemented.

Implement in auto-pts/tester all test cases not currently supported by auto-pts/tester

Allow collection of btmon logs from the RTT buffer

Implement the option to write btmon logs to the RTT buffer in Zephyr.
Implement reading and saving btmon logs in AutoPTS.(just Linux for now)

codecoup/zephyr#2
#10

Make git config trully optional

Currently, if the user does not specify the "git" section in config the auto-pts client will crash.

    zephyr_hash = bot.common.update_repos(args['project_path'],
                                          cfg["git"])['zephyr']

Document the setup required in order to run auto-pts with Zephyr

In the official Zephyr documentation, as .rst), in terms of hardware required and the software installation/setup steps necessary.

https://github.com/zephyrproject-rtos/zephyr/tree/master/doc/guides/bluetooth

AutoPTS copies xml files related to a test PASS to a single folder

AutoPTS copies xml files related to a test PASS to a single folder after every automated run or manual run. For example: such collected files could be stored in "pts_logs\MESH\log_xmls".

Add more informative test status status on error

For example: WID_NOT_FOUND
It should be easier to distinguish missing implementation from btp error, autopts error, pts error and so on.

TestCase.db path should be configurable

Making TestCase.db configurable in config.py will allow us to have a dedicated database for each testing configuration and thus have useful information on regressions between each run. Currently, there is one common database and official and unofficial test runs interfere with each other breaking the history of test results.

By default, the configuration database should be disabled.

Evaluate running two auto-pts instances on one machine

Investigate effort to allow running two instances on one virtual machine.

Add option to specify autopts server port or automatically select a free port

IUT ready event timeout

Problem description:

There is possibility that after DUT reset we will not receive iut_ready_event, with socket.timeout error. In theory, this should trigger cleanup sequence and end test. This sequence is indeed triggering, but test execution is frozen and no retries/new tests are run.

Logs illustrating this issue are as follows:

2021-04-28 14:56:29,952 root DEBUG iutctl.py                 238   reset                     : About to reset DUT: 'nrfjprog -f nrf52 -r'
2021-04-28 14:56:30,804 root DEBUG iutctl_common.py          169   read                      : read
2021-04-28 14:56:50,810 root ERROR autoptsclient_common.py   757   run_test_case_thread_entry : 
Traceback (most recent call last):
  File "/home/codecoup/workspace/auto-pts/autoptsclient_common.py", line 739, in run_test_case_thread_entry
    test_case.pre_run()
  File "/home/codecoup/workspace/auto-pts/ptsprojects/testcase.py", line 749, in pre_run
    cmd.start()
  File "/home/codecoup/workspace/auto-pts/ptsprojects/testcase.py", line 256, in start
    self.func(*args, **self.kwds)
  File "/home/codecoup/workspace/auto-pts/ptsprojects/zephyr/iutctl.py", line 154, in wait_iut_ready_event
    tuple_hdr, tuple_data = self.btp_socket.read()
  File "/home/codecoup/workspace/auto-pts/pybtp/iutctl_common.py", line 188, in read
    raise socket.timeout
socket.timeout
2021-04-28 14:56:50,817 root DEBUG autoptsclient_common.py   692   get_error_code            : get_error_code returning error code 'BTP TIMEOUT' for exception timeout()
2021-04-28 14:56:51,817 root DEBUG testcase.py               756   post_run                  : post_run MESH MESH/NODE/FRND/LPN/BV-07-C BTP TIMEOUT
2021-04-28 14:56:51,817 root DEBUG testcase.py               242   start                     : Starting test function: class=<class 'ptsprojects.testcase.TestFuncCleanUp'>, func=<bound method Stack.cleanup of <ptsprojects.stack.Stack object at 0x7fce38209670>> start_wid=None stop_wid=None post_wid=None skip_call=None call_count=1 args=() kwds={}
2021-04-28 14:56:51,818 root DEBUG testcase.py               254   start                     : Test function parameters: args=(), kwds={}
2021-04-28 14:56:51,818 root DEBUG autoptsclient_common.py   290   clear_pending_responses   : CallbackThread.clear_pending_responses
2021-04-28 14:56:51,818 root DEBUG autoptsclient_common.py   290   clear_pending_responses   : CallbackThread.clear_pending_responses
2021-04-28 14:56:51,818 root DEBUG testcase.py               242   start                     : Starting test function: class=<class 'ptsprojects.testcase.TestFuncCleanUp'>, func=<bound method ZephyrCtl.stop of <ptsprojects.zephyr.iutctl.ZephyrCtl object at 0x7fce38209820>> start_wid=None stop_wid=None post_wid=None skip_call=None call_count=1 args=() kwds={}
2021-04-28 14:56:51,818 root DEBUG testcase.py               254   start                     : Test function parameters: args=(), kwds={}
2021-04-28 14:56:51,818 root DEBUG iutctl.py                 170   stop                      : <class 'ptsprojects.zephyr.iutctl.ZephyrCtl'>.stop

To reproduce:

The origin of this problem might be problem in closing socat process from previous test case, which would be hard to reproduce.
However, we can mimic the problem by opening communication with board on same port as socat process, after creating socat process, and before DUT restart, like this:

2021-04-29 08:04:12,432 root DEBUG iutctl.py                 73    start                     : Starting socat process: socat -x -v /dev/ttyACM0,rawer,b115200 UNIX-CONNECT:/tmp/bt-stack-tester
2021-04-29 08:04:12,434 root DEBUG iutctl_common.py          226   accept                    : accept
2021-04-29 08:04:12,436 root DEBUG testcase.py               242   start                     : Starting test function: class=<class 'ptsprojects.testcase.TestFunc'>, func=<bound method MynewtCtl.wait_iut_ready_event of <ptsprojects.mynewt.iutctl.MynewtCtl object at 0x7f5d7a419eb0>> start_wid=None stop_wid=None post_wid=None skip_call=None call_count=1 args=() kwds={}
2021-04-29 08:04:12,436 root DEBUG testcase.py               254   start                     : Test function parameters: args=(), kwds={}
2021-04-29 08:04:12,436 root DEBUG iutctl.py                 101   reset                     : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.reset
2021-04-29 08:04:12,436 root DEBUG iutctl.py                 132   stop                      : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.stop
2021-04-29 08:04:13,440 root DEBUG iutctl.py                 60    start                     : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.start
2021-04-29 08:04:13,441 root DEBUG iutctl.py                 84    flush_serial              : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.flush_serial
2021-04-29 08:04:14,445 root DEBUG iutctl.py                 73    start                     : Starting socat process: socat -x -v /dev/ttyACM0,rawer,b115200 UNIX-CONNECT:/tmp/bt-stack-tester
2021-04-29 08:04:14,451 root DEBUG iutctl_common.py          226   accept                    : accept
2021-04-29 08:04:14,455 root DEBUG iutctl.py                 84    flush_serial              : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.flush_serial

Here we begin communication with DUT using pty-like tool (here was used screen /dev/ttyACM0 115200)

2021-04-29 08:04:15,458 root DEBUG iutctl.py                 186   reset                     : About to reset DUT: 'nrfjprog -f nrf52 -r'
2021-04-29 08:04:16,520 root DEBUG iutctl_common.py          169   read                      : read
2021-04-29 08:04:36,527 root ERROR autoptsclient_common.py   757   run_test_case_thread_entry : 
Traceback (most recent call last):
  File "/home/krzysiek/auto-pts/autoptsclient_common.py", line 739, in run_test_case_thread_entry
    test_case.pre_run()
  File "/home/krzysiek/auto-pts/ptsprojects/testcase.py", line 749, in pre_run
    cmd.start()
  File "/home/krzysiek/auto-pts/ptsprojects/testcase.py", line 256, in start
    self.func(*args, **self.kwds)
  File "/home/krzysiek/auto-pts/ptsprojects/mynewt/iutctl.py", line 115, in wait_iut_ready_event
    tuple_hdr, tuple_data = self.btp_socket.read()
  File "/home/krzysiek/auto-pts/pybtp/iutctl_common.py", line 188, in read
    raise socket.timeout
socket.timeout
2021-04-29 08:04:36,536 root DEBUG autoptsclient_common.py   692   get_error_code            : get_error_code returning error code 'BTP TIMEOUT' for exception timeout()

Now, autopts is frozen

Btmon logger for nrf logs over RTT on Windows

We could use fragment of Bluez btmon-logger tool, make use of some jlink commands and compile it under Windows.

Bring tools/create-workspace.py to life

Test and make sure it works
Add documentation

Verify code formatting with CI

Define coding style and use an automated tool to verify style correctness.

Fix zephyr bot behavior

Zephyr bot does not work currently. Probably issue lies in building and flashing part.

Integrate e2e testing

https://github.com/JuulLabs-OSS/BTPTesterCore was forked from auto-pts

Requires a larger refactor
Investigate using e2e testing with BubbleSim as a physical layer and running it in a CI as a first wave of testing

Split btp.py into multiple files

This file is too large. It should be split at least into GAP/GATT/L2CAP/MESH/MMDL.

Create recovery steps

While autoptsclient_bot is running, we would like it not to exit on every exception or hang in some broken test, but to do some recovery steps and continue with next tests.

Fix error message when board is not found

This should be handled gracefully with an informative error message instead of:

ERROR:root:invalid literal for int() with base 10: 'CM0'
Traceback (most recent call last):
  File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1159, in start
    return self.main(args)
  File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1178, in main
    self.args = self.parse_args(_args)
  File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1223, in parse_args
    self.check_args(args)
  File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1264, in check_args
    not os.path.exists('COM' + str(int(tty_file["/dev/ttyS".__len__():]) + 1)):
ValueError: invalid literal for int() with base 10: 'CM0'

Update the ICS documents in the Zephyr documentation

Refactor setting PIXIT values

See if we can remove set_pixit() method from each profile.py
Update XMLRPC API (pts.update_pixit_param) to return an error to the client if cannot set.
Set all the required PIXIT values in preconditions

Fix error logging when PTS dongle is missing

ptscontrol.py should display an informative error message instead of exception log. This is expected and should be handled gracefully

Refactor AutoPTS

The project could use a large refactor. I have a few ideas:

Get rid code duplication - We don't need to list every test case that is supported. A PTS workspace contains all supported test cases. By default we should have some generic wid handler that is common to all platforms and maybe some preconditions (some requirements for test case are not communicated through wids). If a platform needs some specific behavior then it should be implemented but 90% of the time it would be just default handlers and this removes a lot of code. After this change adding support for new platforms would be much easier (to implement, to test and to review).
Get rid of globals - This makes the dependencies complicated and adding new features is problematic. The design is not clean. As an example: iutctl.
Get rid of ZTestCase - It should be made more generic.
Split btp.py into multiple files - This file is huge and indexing it takes a lot of time.
Merging bot client and regular client - We could have a single client that has all the features of a bot client and the ease of use of the regular one. We could move platform specific stuff to iutctl or some other module that is implemented by each platform. We could add command line options to bot that will allow to override some settings that are specifiec in configuration file. So the default would be a config file but if we want we could specify CLI options and it will behave just like the regular client.

Evaluate currently supported and not supported tests according to all features supported by Zephyr

Set up bi-weekly test runs

Setup and configure a dedicated machine
Any changes to auto-pts required to make it work
Prepare Google Drive log storage

AutoPTSServer: Use ETS Manager API instead of PTSControl API

Upgrade auto-pts to use the improved and latest PTS API - ETS Manager.
See: C:\Program Files (x86)\Bluetooth SIG\Bluetooth PTS\SampleCode

Erase flash before building and flashing device

To avoid reading broken data when persistent storage is enabled.

Use workspace to get a list of all supported test cases

We don't need to list every test case that is supported. A PTS workspace contains all supported test cases. By default we should have some generic wid handler that is common to all platforms and maybe some preconditions (some requirements for test case are not communicated through wids). If a platform needs some specific behavior then it should be implemented but 90% of the time it would be just default handlers and this removes a lot of code. After this change adding support for new platforms would be much easier (to implement, to test and to review).

codecoup / auto-pts Goto Github PK

auto-pts's People

Contributors

Watchers

Forkers

auto-pts's Issues

Problem description:

To reproduce:

Recommend Projects

Recommend Topics

Recommend Org