codecoup / auto-pts Goto Github PK
View Code? Open in Web Editor NEWThis project forked from auto-pts/auto-pts
The Bluetooth PTS automation framework
License: GNU General Public License v2.0
This project forked from auto-pts/auto-pts
The Bluetooth PTS automation framework
License: GNU General Public License v2.0
Maybe we can put set_io_cap(display_only) in pre_conditions and thus refactor more tests from the "custom" list
Add unit tests that will cover as much code as possible so that we can verify that there are no syntax errors at least.
For now, Windows workaround for autoptsclient-zephyr uses socat.exe to connect to the controller. Since socat.exe has no support and it is just quick bypass to make autoptsclient-zephyr work on Windows, appropriate solution with pyserial should be implemented.
Currently, if the user does not specify the "git" section in config the auto-pts client will crash.
zephyr_hash = bot.common.update_repos(args['project_path'],
cfg["git"])['zephyr']
In the official Zephyr documentation, as .rst), in terms of hardware required and the software installation/setup steps necessary.
https://github.com/zephyrproject-rtos/zephyr/tree/master/doc/guides/bluetooth
AutoPTS copies xml files related to a test PASS to a single folder after every automated run or manual run. For example: such collected files could be stored in "pts_logs\MESH\log_xmls".
For example: WID_NOT_FOUND
It should be easier to distinguish missing implementation from btp error, autopts error, pts error and so on.
Making TestCase.db configurable in config.py will allow us to have a dedicated database for each testing configuration and thus have useful information on regressions between each run. Currently, there is one common database and official and unofficial test runs interfere with each other breaking the history of test results.
By default, the configuration database should be disabled.
Investigate effort to allow running two instances on one virtual machine.
There is possibility that after DUT reset we will not receive iut_ready_event, with socket.timeout
error. In theory, this should trigger cleanup sequence and end test. This sequence is indeed triggering, but test execution is frozen and no retries/new tests are run.
Logs illustrating this issue are as follows:
2021-04-28 14:56:29,952 root DEBUG iutctl.py 238 reset : About to reset DUT: 'nrfjprog -f nrf52 -r'
2021-04-28 14:56:30,804 root DEBUG iutctl_common.py 169 read : read
2021-04-28 14:56:50,810 root ERROR autoptsclient_common.py 757 run_test_case_thread_entry :
Traceback (most recent call last):
File "/home/codecoup/workspace/auto-pts/autoptsclient_common.py", line 739, in run_test_case_thread_entry
test_case.pre_run()
File "/home/codecoup/workspace/auto-pts/ptsprojects/testcase.py", line 749, in pre_run
cmd.start()
File "/home/codecoup/workspace/auto-pts/ptsprojects/testcase.py", line 256, in start
self.func(*args, **self.kwds)
File "/home/codecoup/workspace/auto-pts/ptsprojects/zephyr/iutctl.py", line 154, in wait_iut_ready_event
tuple_hdr, tuple_data = self.btp_socket.read()
File "/home/codecoup/workspace/auto-pts/pybtp/iutctl_common.py", line 188, in read
raise socket.timeout
socket.timeout
2021-04-28 14:56:50,817 root DEBUG autoptsclient_common.py 692 get_error_code : get_error_code returning error code 'BTP TIMEOUT' for exception timeout()
2021-04-28 14:56:51,817 root DEBUG testcase.py 756 post_run : post_run MESH MESH/NODE/FRND/LPN/BV-07-C BTP TIMEOUT
2021-04-28 14:56:51,817 root DEBUG testcase.py 242 start : Starting test function: class=<class 'ptsprojects.testcase.TestFuncCleanUp'>, func=<bound method Stack.cleanup of <ptsprojects.stack.Stack object at 0x7fce38209670>> start_wid=None stop_wid=None post_wid=None skip_call=None call_count=1 args=() kwds={}
2021-04-28 14:56:51,818 root DEBUG testcase.py 254 start : Test function parameters: args=(), kwds={}
2021-04-28 14:56:51,818 root DEBUG autoptsclient_common.py 290 clear_pending_responses : CallbackThread.clear_pending_responses
2021-04-28 14:56:51,818 root DEBUG autoptsclient_common.py 290 clear_pending_responses : CallbackThread.clear_pending_responses
2021-04-28 14:56:51,818 root DEBUG testcase.py 242 start : Starting test function: class=<class 'ptsprojects.testcase.TestFuncCleanUp'>, func=<bound method ZephyrCtl.stop of <ptsprojects.zephyr.iutctl.ZephyrCtl object at 0x7fce38209820>> start_wid=None stop_wid=None post_wid=None skip_call=None call_count=1 args=() kwds={}
2021-04-28 14:56:51,818 root DEBUG testcase.py 254 start : Test function parameters: args=(), kwds={}
2021-04-28 14:56:51,818 root DEBUG iutctl.py 170 stop : <class 'ptsprojects.zephyr.iutctl.ZephyrCtl'>.stop
The origin of this problem might be problem in closing socat process from previous test case, which would be hard to reproduce.
However, we can mimic the problem by opening communication with board on same port as socat process, after creating socat process, and before DUT restart, like this:
2021-04-29 08:04:12,432 root DEBUG iutctl.py 73 start : Starting socat process: socat -x -v /dev/ttyACM0,rawer,b115200 UNIX-CONNECT:/tmp/bt-stack-tester
2021-04-29 08:04:12,434 root DEBUG iutctl_common.py 226 accept : accept
2021-04-29 08:04:12,436 root DEBUG testcase.py 242 start : Starting test function: class=<class 'ptsprojects.testcase.TestFunc'>, func=<bound method MynewtCtl.wait_iut_ready_event of <ptsprojects.mynewt.iutctl.MynewtCtl object at 0x7f5d7a419eb0>> start_wid=None stop_wid=None post_wid=None skip_call=None call_count=1 args=() kwds={}
2021-04-29 08:04:12,436 root DEBUG testcase.py 254 start : Test function parameters: args=(), kwds={}
2021-04-29 08:04:12,436 root DEBUG iutctl.py 101 reset : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.reset
2021-04-29 08:04:12,436 root DEBUG iutctl.py 132 stop : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.stop
2021-04-29 08:04:13,440 root DEBUG iutctl.py 60 start : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.start
2021-04-29 08:04:13,441 root DEBUG iutctl.py 84 flush_serial : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.flush_serial
2021-04-29 08:04:14,445 root DEBUG iutctl.py 73 start : Starting socat process: socat -x -v /dev/ttyACM0,rawer,b115200 UNIX-CONNECT:/tmp/bt-stack-tester
2021-04-29 08:04:14,451 root DEBUG iutctl_common.py 226 accept : accept
2021-04-29 08:04:14,455 root DEBUG iutctl.py 84 flush_serial : <class 'ptsprojects.mynewt.iutctl.MynewtCtl'>.flush_serial
Here we begin communication with DUT using pty-like tool (here was used screen /dev/ttyACM0 115200
)
2021-04-29 08:04:15,458 root DEBUG iutctl.py 186 reset : About to reset DUT: 'nrfjprog -f nrf52 -r'
2021-04-29 08:04:16,520 root DEBUG iutctl_common.py 169 read : read
2021-04-29 08:04:36,527 root ERROR autoptsclient_common.py 757 run_test_case_thread_entry :
Traceback (most recent call last):
File "/home/krzysiek/auto-pts/autoptsclient_common.py", line 739, in run_test_case_thread_entry
test_case.pre_run()
File "/home/krzysiek/auto-pts/ptsprojects/testcase.py", line 749, in pre_run
cmd.start()
File "/home/krzysiek/auto-pts/ptsprojects/testcase.py", line 256, in start
self.func(*args, **self.kwds)
File "/home/krzysiek/auto-pts/ptsprojects/mynewt/iutctl.py", line 115, in wait_iut_ready_event
tuple_hdr, tuple_data = self.btp_socket.read()
File "/home/krzysiek/auto-pts/pybtp/iutctl_common.py", line 188, in read
raise socket.timeout
socket.timeout
2021-04-29 08:04:36,536 root DEBUG autoptsclient_common.py 692 get_error_code : get_error_code returning error code 'BTP TIMEOUT' for exception timeout()
Now, autopts is frozen
We could use fragment of Bluez btmon-logger tool, make use of some jlink commands and compile it under Windows.
Define coding style and use an automated tool to verify style correctness.
Zephyr bot does not work currently. Probably issue lies in building and flashing part.
https://github.com/JuulLabs-OSS/BTPTesterCore was forked from auto-pts
This file is too large. It should be split at least into GAP/GATT/L2CAP/MESH/MMDL.
While autoptsclient_bot is running, we would like it not to exit on every exception or hang in some broken test, but to do some recovery steps and continue with next tests.
This should be handled gracefully with an informative error message instead of:
ERROR:root:invalid literal for int() with base 10: 'CM0'
Traceback (most recent call last):
File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1159, in start
return self.main(args)
File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1178, in main
self.args = self.parse_args(_args)
File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1223, in parse_args
self.check_args(args)
File "/home/michaln/Workspace/auto-pts/autoptsclient_common.py", line 1264, in check_args
not os.path.exists('COM' + str(int(tty_file["/dev/ttyS".__len__():]) + 1)):
ValueError: invalid literal for int() with base 10: 'CM0'
profile.py
ptscontrol.py should display an informative error message instead of exception log. This is expected and should be handled gracefully
The project could use a large refactor. I have a few ideas:
Upgrade auto-pts to use the improved and latest PTS API - ETS Manager.
See: C:\Program Files (x86)\Bluetooth SIG\Bluetooth PTS\SampleCode
To avoid reading broken data when persistent storage is enabled.
We don't need to list every test case that is supported. A PTS workspace contains all supported test cases. By default we should have some generic wid handler that is common to all platforms and maybe some preconditions (some requirements for test case are not communicated through wids). If a platform needs some specific behavior then it should be implemented but 90% of the time it would be just default handlers and this removes a lot of code. After this change adding support for new platforms would be much easier (to implement, to test and to review).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.