jerryscript-project / js-remote-test Goto Github PK

I would like to publish my test results for Tizen RT IoT.js here but I see that you overwrite history here at daily basis.

How should I publish my test results here?

Look at the memory size of test_buffer.js,

The memory size is abnormally large only in Artik053. We should measure it correctly.

It would be great if could add some automatic testing for this project too. We could start with a DCO checker at first.

I'd like to build the artik053 device, but it fails.

command
python driver.py --device=artik053 --app=iotjs --buildtype=debug

output

echo "create .version file"; \
tools/version.sh -v 1.0 .version; \
chmod 755 .version
create .version file
LN: include/arch to arch/arm/include
LN: arch/arm/src/board to arch/arm/src/artik053/src
LN: include/apps to ../apps/include
make: *** No rule to make target '/mnt/work/Samsung/js-remote-test/projects/iotjs/build/arm-tizenrt/debug/lib/libhttpparser.a', needed by '../build/output/libraries/libhttpparser.a'.  Stop.
make: *** Waiting for unfinished jobs....
LN: arch/arm/src/chip to arch/arm/src/s5j

When I clone latest version of js-remote-test, and follow the guide in README.md on Ubuntu 14.04 LTS, I got following error:

$ python driver.py --device=artik053 --app=iotjs --buildtype=debug
Traceback (most recent call last):
  File "driver.py", line 17, in <module>
    import API
  File "/home/brian/iotjs/js-remote-test/API/__init__.py", line 16, in <module>
    import device
  File "/home/brian/iotjs/js-remote-test/API/device/__init__.py", line 15, in <module>
    import artik053
  File "/home/brian/iotjs/js-remote-test/API/device/artik053.py", line 16, in <module>
    import connection
  File "/home/brian/iotjs/js-remote-test/API/device/connection/__init__.py", line 15, in <module>
    import serialcom
  File "/home/brian/iotjs/js-remote-test/API/device/connection/serialcom.py", line 15, in <module>
    import serial
ImportError: No module named serial

It seems sudo pip install schedule paramiko pyserial xmodem pyrebase is done well.

So I tried on another Ubuntu 14.04 LTS in VirtualBox. It is closer to clean Ubuntu 14.04 LTS.
It shows different error in installing pip.

$ bash install-deps.sh 
...
Setting up linux-libc-dev-armhf-cross (3.13.0-12.32cross1.104) ...
Setting up libc6-dev-armhf-cross (2.19-0ubuntu2cross1.104) ...
Processing triggers for libc-bin (2.19-0ubuntu6.7) ...
sudo: pip: command not found

$ pip
The program 'pip' is currently not installed. You can install it by typing:
sudo apt-get install python-pip

$ sudo apt-get install python-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python-pip : Depends: python-setuptools (>= 0.6c1) but it is not going to be installed
              Recommends: python-dev-all (>= 2.6) but it is not installable
E: Unable to correct problems, you have held broken packages.

Currently, we have two repositories to store web-page files for IoT.js and JerryScript test results:

• https://github.com/Samsung/iotjs-test-results/tree/gh-pages
• https://github.com/jerryscript-project/jerryscript-test-results/tree/gh-pages

It would be nice if jerryscript-test-results repository could be merged into iotjs-test-results to eliminate code duplications. (In this case it does not need to create Pull Requests for both projects with the same content.)
I also prefer if the https://samsung.github.io/iotjs-test-results/ page could show the JerryScript results as well. (Of course, the layout should be modified.)

What do you think?

I think that the current architecture of the project is a bit complex for the current use case. There are too many components (os, app, device, testrunner) that make unnecessary dependencies. It would be nice if the number of the components could be reduced to eliminate dependencies and make the code clearer. I also prefer if the build and test could be well separated.

The following diagram shows a possible new architecture that logically divides the js-remote-test into a builder and a tester parts:

js-remote-test
  |
  |--- API
  |      +--- common
  |      |       +--- utils.py
  |      |       +--- paths.py
  |      |       +--- console.py
  |      |       +--- reporter.py
  |      |
  |      +--- builder
  |      |       +--- config
  |      |       |       +--- iotjs-nuttx.config
  |      |       |       +--- jerry-nuttx.config
  |      |       |
  |      |       +--- patch
  |      |       |        +--- iotjs-memstat.diff
  |      |       |        +--- libtuv-memstat.diff
  |      |       |        +--- jerry-memstat.diff
  |      |       |        +--- nuttx-7.19.diff
  |      |       |        +--- tizenrt-iotjs.diff
  |      |       |        +--- tizenrt-jerryscript.diff
  |      |       |       
  |      |       +--- targets
  |      |       |       +--- host.py
  |      |       |       +--- rpi2.py
  |      |       |       +--- stm32f4dis.py
  |      |       |       +--- artik053.py
  |      |       |
  |      |       +--- builder.py
  |      |       +--- resources.json
  |      |
  |      +--- testrunner
  |              +--- targets
  |              |        +--- communication
  |              |        |        +--- sshcom.py
  |              |        |        +--- serialcom.sh
  |              |        |
  |              |        +--- host.py
  |              |        +--- rpi2.py
  |              |        +--- stm32f4dis.py
  |              |        +--- artik053.py
  |              |
  |              +--- testrunner.py
  |              +--- skiplist.json
  |
  +--- tools
  |      +--- travis_check.py
  |      +--- check_signed_off.sh
  |      +--- check_style.sh
  |
  +--- driver.py

Builder part

The builder part would have a resources.json file that could hold information about git repositories, brach or commit, or a patch list that should be applied before the build:

{
  "nuttx": {
    "repository": "https://bitbucket.org/nuttx/nuttx.git",
    "version": "nuttx-7.19",
    "patches": ["nuttx-7.19.diff"]
  },
  "stlink": {
    "repository": "https://github.com/texane/stlink.git",
    "version": "master"
  },
  "iotjs": {
    "repository": "https://github.com/Samsung/iotjs.git",
    "version": "master",
    "patches": {
      "nuttx": ["iotjs-memstat.diff", "jerry-memstat.diff", "libtuv.diff"],
      "tizenrt": ["tizenrt-iotjs.diff"]
    }
  }
}

The targets of the builder module could build the appropriate softwares (OS, Flasher, Freya, ...) and flash the device (with the appropriate flasher).
Just a note: the config and the patches folders could be moved into the builder module because they are logically belong to that. (Currently, they are in the root folder).

Testrunner

The testrunner module is responsible for the test execution where the targets are responsible for the target specific output processing. This part was already implemented in iotjs.

In general, js-remote-test would work in two modes. The first is a standalone mode when iotjs and jerryscript should be downloaded and used for testing. The second is a submodule mode when the containing application (iotjs, jerryscript) should be used for testing (so it's unnecessary to download iotjs or jerryscript).

@glistening @hs0225 @robertsipka What do you think about this change?

This issue is created by #44 (comment).
I would like to talk in this issue for efficient and clear communication.

I have the data on a firebase database (so write access works). The service responds with available keys, you can check on https://tosyu.github.io/remote-testrunner/ for RPI2. It renders 1 node one the chart (just one result was posted). But there is a problem with rendering the test results (below the chart). I'm guessing there is something wrong with my database configuration:

{
  "rules": {
    ".read": true,
    ".write": "auth != null",
    "$uid": {
      ".write": "$uid === auth.uid"
    }
  }
}

Please help 😢

Since binary size and runtime memory measurement(#45) is almost done, we would like to resume the next priority task. Please automate the code coverage measurement with System I/O on RPI. It is a part of #39 (comment).

Sometimes boards are disconnected from the internet, could we make freya dependancy (for RPI) optional?

Or maybe crosscompile it on linux and send it with the os (like iotjs or jerryscript)?

Binary size in https://samsung.github.io/js-remote-test/# are reported as same for STM32, RPI2, and Artik053 respectively. Comment in https://github.com/Samsung/js-remote-test/blob/master/API/application/iotjs.py#L112 says it is intentional. But we would like to know binary size for each platform.

To support Tizen, we need to work a few things below.

• remote test
• calculate binary size
• calculate memory size for testcase
• publish to https://samsung.github.io/iotjs-test-results

Currently the labels on the charts are duplicated. This means that there are labels at the bottom of the charts (e.g. target-profile binary size, minimal-profile binary size) that are also located on the tooltips (when pointing to a point in the chart).

In case of tooltips, it should be enough just use a color (orange rectangle, blue rectangle) to identify the information, because the charts (at the bottom) already define that.

To make a correct decision (remove labels or not), I've published the idea on my github repository that you can check:

https://samsung.github.io/js-remote-test/ (official)
https://rtakacs.github.io/js-remote-test/ (local developement, removed the labels from tooltips)

Currently, TizenRT is always updated to the latest master before testing. Since TizenRT is continuously changing, we should use a stable (TizenRT) hash in js-remote-test. (In case of NuttX, we use nuttx-7.19 hash tag.)

By @zherczeg's request, I would like to write the requirement for js test framework.

Unifying the test framework

• Both of IoT.js and JerryScript
  ( It is already supported from very initial version )
• Both of remote and local
  ( This unifying idea is raised by @yichoi jerryscript-project/iotjs#1212 )
• Multiple board/os (e.g. RPI/Raspbean, Artik053/TizenRT, ...)
  ( jerryscript-project/iotjs#1073 )

Minimizing the maintenance cost

• Minimize duplicated code to IoT.js and JerryScript repository

System I/O

• Cover system I/O also after related sensors are delivered, this item can be started

Etc

• Remove circular dependencies that exists on current structure
• We would like to choose IoT.js by commit ID or by indicating path to IoT.js/JerryScript