moiri / streamix Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 1.0 174 KB

Root repository for the Streamix project

Makefile 100.00%

streamix's People

Contributors

Watchers

Forkers

simplay

streamix's Issues

Autoconf errors with igraph

On Ubuntu 18.04 LTS, when running autoconf the following error ocured:

sh: 1: cannot open VERSION: No such file

The following patch allowed to run autoconf:

diff --git a/configure.ac b/configure.ac
index acf76cde..9fe50e9f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-AC_INIT(igraph, esyscmd([tr -d '\n' < VERSION]), [email protected])
+AC_INIT(igraph, 0.9.1), [email protected])
 AC_CONFIG_SRCDIR(src/games.c)
 AM_INIT_AUTOMAKE([subdir-objects])
 AC_CONFIG_HEADERS([config.h])

However, in the end I just installed the prepared pakage

sudo apt install libigraph0-dev

Compile errors with zlog

On Ubuntu 18.04 LTS the release 1.2.12 of zlog cannot be compiled.

Unfortunately there is no newer release so I ended up checking out the latest version of the master branch (ccf8c36e918d6f3c7e6fe9e36eedf8583014c5c9 at the time of this writing) which solved the problems.

ZMQ Read and Write Box implementation

In order to allow a Streamix network to interface any application two boxes need to be created:

A ZMQ Reader which allows to read from the ZMQ bus
A ZMQ Writer which allows to write to the ZMQ bus

Test Streamix in combination with ZMQ

A Report on how Streamix performed on ISPW60

This document aims at giving an overview on several performance tests done on the machine ISPW60.
Note that due to the blocking-log issue the RTS was compiled with the -DSMX_LOG_UNSAFE flag.

Hardware

The command sudo lshw -short produces the following output:

H/W path      Device     Class          Description
===================================================
                         system         HP Z240 Tower Workstation (L8T12AV)
/0                       bus            802F
/0/0                     memory         128KiB L1 cache
/0/1                     memory         128KiB L1 cache
/0/2                     memory         1MiB L2 cache
/0/3                     memory         8MiB L3 cache
/0/4                     processor      Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
/0/5                     memory         32GiB System Memory
/0/5/0                   memory         8GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
/0/5/1                   memory         8GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
/0/5/2                   memory         8GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
/0/5/3                   memory         8GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
/0/b                     memory         64KiB BIOS
/0/100                   bridge         Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers
/0/100/1                 bridge         Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16)
/0/100/1/0               display        GM107GL [Quadro K620]
/0/100/1/0.1             multimedia     NVIDIA Corporation
/0/100/14                bus            100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller
/0/100/14/0   usb1       bus            xHCI Host Controller
/0/100/14/1   usb2       bus            xHCI Host Controller
/0/100/14.2              generic        100 Series/C230 Series Chipset Family Thermal Subsystem
/0/100/16                communication  100 Series/C230 Series Chipset Family MEI Controller humdek-unibe-ch/smx-core-rts#1
/0/100/16.3              communication  100 Series/C230 Series Chipset Family KT Redirection
/0/100/17                storage        SATA Controller [RAID mode]
/0/100/1f                bridge         C236 Chipset LPC/eSPI Controller
/0/100/1f.2              memory         Memory controller
/0/100/1f.3              multimedia     100 Series/C230 Series Chipset Family HD Audio Controller
/0/100/1f.4              bus            100 Series/C230 Series Chipset Family SMBus
/0/100/1f.6   eno1       network        Ethernet Connection (2) I219-LM
/0/6          scsi0      storage
/0/6/0.0.0    /dev/sda   disk           1024GB MTFDDAK1T0TBN-1A
/0/6/0.0.0/1  /dev/sda1  volume         511MiB Windows FAT volume
/0/6/0.0.0/2  /dev/sda2  volume         953GiB EXT4 volume
/1                       power          High Efficiency

The command lscpu produces the following output:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               94
Model name:          Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Stepping:            3
CPU MHz:             800.176
CPU max MHz:         4000.0000
CPU min MHz:         800.0000
BogoMIPS:            6816.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            8192K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d

Simple Test Runs

To start things off a simple test run with the following Streamix network was performed:

Src = extern box smx_src_rand( out data )
Snk = box smx_snk_null( in data )

A = Src.Snk

connect A

This program creates two threads, where the first creates random messages and the second destroys them.
The program was run three times where in each run each thread was executed 10'000'000 times.

[2019-09-13 08:49:25.245740] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 296161, wall time: 33.765305)
[2019-09-13 08:49:25.245765] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 296162, wall time: 33.765297)
[2019-09-13 08:49:25.245788] [NOTICE] [main] end main thread (wall time: 33.765709)

[2019-09-13 09:00:45.814958] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 288089, wall time: 34.711493)
[2019-09-13 09:00:45.814981] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 288088, wall time: 34.711521)
[2019-09-13 09:00:45.815004] [NOTICE] [main] end main thread (wall time: 34.711777)

[2019-09-13 09:01:36.458231] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 281976, wall time: 35.463926)
[2019-09-13 09:01:36.458255] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 281976, wall time: 35.463948)
[2019-09-13 09:01:36.458296] [NOTICE] [main] end main thread (wall time: 35.464232)

Next, the number of threads was scaled to half the number of cores (4) with the following program:

Src = extern box smx_src_rand( out data )
Snk = box smx_snk_null( in data )

A = Src.Snk
B = A|A

connect B

This program creates two pairs of two threads, where the first thread of a pair creates random messages and the second destroys them.
The program was run three times where in each run each thread was executed 10'000'000 times.

[2019-09-13 12:40:25.186308] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 304687, wall time: 32.820464)
[2019-09-13 12:40:25.186320] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 304687, wall time: 32.820483)
[2019-09-13 12:40:25.397201] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 302742, wall time: 33.031360)
[2019-09-13 12:40:25.397214] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 302742, wall time: 33.031374)
[2019-09-13 12:40:25.397233] [NOTICE] [main] end main thread (wall time: 33.031630)

[2019-09-13 12:42:35.520930] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 304560, wall time: 32.834195)
[2019-09-13 12:42:35.520943] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 304560, wall time: 32.834218)
[2019-09-13 12:42:35.603027] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 303800, wall time: 32.916297)
[2019-09-13 12:42:35.603043] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 303800, wall time: 32.916317)
[2019-09-13 12:42:35.603091] [NOTICE] [main] end main thread (wall time: 32.916713)

[2019-09-13 12:43:10.210780] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 307393, wall time: 32.531546)
[2019-09-13 12:43:10.210793] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 307394, wall time: 32.531517)
[2019-09-13 12:43:10.843060] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 301533, wall time: 33.163799)
[2019-09-13 12:43:10.843075] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 301533, wall time: 33.163842)
[2019-09-13 12:43:10.843095] [NOTICE] [main] end main thread (wall time: 33.164186)

Next, the number of threads was scaled to the number of cores (8) with the following program:

Src = extern box smx_src_rand( out data )
Snk = box smx_snk_null( in data )

A = Src.Snk
B = A|A|A|A

connect B

This program creates four pairs of two threads, where the first thread of a pair creates random messages and the second destroys them.
The program was run three times where in each run each thread was executed 10'000'000 times.

[2019-09-13 12:07:40.117783] [NOTICE] [net_Src_4] terminate net (loop count: 10000000, loop rate: 226023, wall time: 44.243161)
[2019-09-13 12:07:40.117800] [NOTICE] [net_Snk_5] terminate net (loop count: 10000001, loop rate: 226023, wall time: 44.243162)
[2019-09-13 12:07:40.231047] [NOTICE] [net_Src_6] terminate net (loop count: 10000000, loop rate: 225446, wall time: 44.356404)
[2019-09-13 12:07:40.231066] [NOTICE] [net_Snk_7] terminate net (loop count: 10000001, loop rate: 225446, wall time: 44.356455)
[2019-09-13 12:07:40.444769] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 224365, wall time: 44.570164)
[2019-09-13 12:07:40.444784] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 224365, wall time: 44.570179)
[2019-09-13 12:07:40.501202] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 224081, wall time: 44.626560)
[2019-09-13 12:07:40.501224] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 224081, wall time: 44.626585)
[2019-09-13 12:07:40.501262] [NOTICE] [main] end main thread (wall time: 44.627130)

[2019-09-13 12:08:53.402069] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 224578, wall time: 44.527859)
[2019-09-13 12:08:53.402084] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 224578, wall time: 44.527888)
[2019-09-13 12:08:53.466222] [NOTICE] [net_Src_4] terminate net (loop count: 10000000, loop rate: 224255, wall time: 44.592024)
[2019-09-13 12:08:53.466232] [NOTICE] [net_Snk_5] terminate net (loop count: 10000001, loop rate: 224255, wall time: 44.592013)
[2019-09-13 12:08:53.571265] [NOTICE] [net_Src_6] terminate net (loop count: 10000000, loop rate: 223728, wall time: 44.697065)
[2019-09-13 12:08:53.571281] [NOTICE] [net_Snk_7] terminate net (loop count: 10000001, loop rate: 223728, wall time: 44.697082)
[2019-09-13 12:08:53.572655] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 223721, wall time: 44.698433)
[2019-09-13 12:08:53.572672] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 223721, wall time: 44.698474)
[2019-09-13 12:08:53.572713] [NOTICE] [main] end main thread (wall time: 44.699212)

[2019-09-13 12:10:34.008247] [NOTICE] [net_Src_6] terminate net (loop count: 10000000, loop rate: 226335, wall time: 44.182147)
[2019-09-13 12:10:34.008262] [NOTICE] [net_Snk_7] terminate net (loop count: 10000001, loop rate: 226335, wall time: 44.182178)
[2019-09-13 12:10:34.240421] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 225152, wall time: 44.414342)
[2019-09-13 12:10:34.240432] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 225152, wall time: 44.414327)
[2019-09-13 12:10:34.389968] [NOTICE] [net_Src_4] terminate net (loop count: 10000000, loop rate: 224397, wall time: 44.563845)
[2019-09-13 12:10:34.389984] [NOTICE] [net_Snk_5] terminate net (loop count: 10000001, loop rate: 224397, wall time: 44.563860)
[2019-09-13 12:10:34.578593] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 223451, wall time: 44.752487)
[2019-09-13 12:10:34.578608] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 223451, wall time: 44.752529)
[2019-09-13 12:10:34.578661] [NOTICE] [main] end main thread (wall time: 44.753195)

Next, the number of threads was scaled to the number of supported hw-based threads (16) with the following program:

Src = extern box smx_src_rand( out data )
Snk = box smx_snk_null( in data )

A = Src.Snk
B = A|A|A|A

connect B|B

This program creates eight pairs of two threads, where the first thread of a pair creates random messages and the second destroys them.
The program was run three times where in each run each thread was executed 10'000'000 times.

[2019-09-13 12:15:07.544740] [NOTICE] [net_Snk_7] terminate net (loop count: 10000001, loop rate: 107924, wall time: 92.657120)
[2019-09-13 12:15:07.544805] [NOTICE] [net_Src_6] terminate net (loop count: 10000000, loop rate: 107924, wall time: 92.657074)
[2019-09-13 12:15:07.745435] [NOTICE] [net_Src_8] terminate net (loop count: 10000000, loop rate: 107691, wall time: 92.857811)
[2019-09-13 12:15:07.745448] [NOTICE] [net_Snk_9] terminate net (loop count: 10000001, loop rate: 107691, wall time: 92.857831)
[2019-09-13 12:15:07.779642] [NOTICE] [net_Src_10] terminate net (loop count: 10000000, loop rate: 107651, wall time: 92.891994)
[2019-09-13 12:15:07.779655] [NOTICE] [net_Snk_11] terminate net (loop count: 10000001, loop rate: 107651, wall time: 92.892021)
[2019-09-13 12:15:07.782672] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 107648, wall time: 92.895061)
[2019-09-13 12:15:07.782681] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 107648, wall time: 92.895073)
[2019-09-13 12:15:07.815792] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 107609, wall time: 92.928174)
[2019-09-13 12:15:07.815804] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 107609, wall time: 92.928193)
[2019-09-13 12:15:07.897776] [NOTICE] [net_Src_14] terminate net (loop count: 10000000, loop rate: 107515, wall time: 93.010157)
[2019-09-13 12:15:07.897795] [NOTICE] [net_Snk_15] terminate net (loop count: 10000001, loop rate: 107515, wall time: 93.010158)
[2019-09-13 12:15:08.036174] [NOTICE] [net_Src_12] terminate net (loop count: 10000000, loop rate: 107355, wall time: 93.148537)
[2019-09-13 12:15:08.036187] [NOTICE] [net_Snk_13] terminate net (loop count: 10000001, loop rate: 107355, wall time: 93.148565)
[2019-09-13 12:15:08.063860] [NOTICE] [net_Src_4] terminate net (loop count: 10000000, loop rate: 107323, wall time: 93.176252)
[2019-09-13 12:15:08.063876] [NOTICE] [net_Snk_5] terminate net (loop count: 10000001, loop rate: 107323, wall time: 93.176262)
[2019-09-13 12:15:08.063954] [NOTICE] [main] end main thread (wall time: 93.177301)

2019-09-13 12:19:20.623706] [NOTICE] [net_Src_4] terminate net (loop count: 10000000, loop rate: 108382, wall time: 92.265393)
[2019-09-13 12:19:20.623714] [NOTICE] [net_Snk_5] terminate net (loop count: 10000001, loop rate: 108383, wall time: 92.265378)
[2019-09-13 12:19:20.678100] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 108319, wall time: 92.319809)
[2019-09-13 12:19:20.678113] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 108319, wall time: 92.319826)
[2019-09-13 12:19:20.701408] [NOTICE] [net_Src_6] terminate net (loop count: 10000000, loop rate: 108291, wall time: 92.343117)
[2019-09-13 12:19:20.701425] [NOTICE] [net_Snk_7] terminate net (loop count: 10000001, loop rate: 108291, wall time: 92.343131)
[2019-09-13 12:19:20.775772] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 108204, wall time: 92.417484)
[2019-09-13 12:19:20.775783] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 108204, wall time: 92.417498)
[2019-09-13 12:19:20.827390] [NOTICE] [net_Src_14] terminate net (loop count: 10000000, loop rate: 108144, wall time: 92.469085)
[2019-09-13 12:19:20.827403] [NOTICE] [net_Snk_15] terminate net (loop count: 10000001, loop rate: 108144, wall time: 92.469102)
[2019-09-13 12:19:20.839266] [NOTICE] [net_Src_10] terminate net (loop count: 10000000, loop rate: 108130, wall time: 92.480954)
[2019-09-13 12:19:20.839282] [NOTICE] [net_Snk_11] terminate net (loop count: 10000001, loop rate: 108130, wall time: 92.480878)
[2019-09-13 12:19:20.863187] [NOTICE] [net_Src_8] terminate net (loop count: 10000000, loop rate: 108102, wall time: 92.504849)
[2019-09-13 12:19:20.863200] [NOTICE] [net_Snk_9] terminate net (loop count: 10000001, loop rate: 108102, wall time: 92.504906)
[2019-09-13 12:19:20.895387] [NOTICE] [net_Src_12] terminate net (loop count: 10000000, loop rate: 108064, wall time: 92.537030)
[2019-09-13 12:19:20.895406] [NOTICE] [net_Snk_13] terminate net (loop count: 10000001, loop rate: 108064, wall time: 92.537079)
[2019-09-13 12:19:20.895440] [NOTICE] [main] end main thread (wall time: 92.538273)

[2019-09-13 12:21:00.649555] [NOTICE] [net_Src_10] terminate net (loop count: 10000000, loop rate: 109711, wall time: 91.147777)
[2019-09-13 12:21:00.649582] [NOTICE] [net_Snk_11] terminate net (loop count: 10000001, loop rate: 109711, wall time: 91.147808)
[2019-09-13 12:21:00.694570] [NOTICE] [net_Src_14] terminate net (loop count: 10000000, loop rate: 109657, wall time: 91.192817)
[2019-09-13 12:21:00.694582] [NOTICE] [net_Snk_15] terminate net (loop count: 10000001, loop rate: 109657, wall time: 91.192866)
[2019-09-13 12:21:00.709300] [NOTICE] [net_Src_4] terminate net (loop count: 10000000, loop rate: 109640, wall time: 91.207572)
[2019-09-13 12:21:00.709313] [NOTICE] [net_Snk_5] terminate net (loop count: 10000001, loop rate: 109640, wall time: 91.207588)
[2019-09-13 12:21:00.726226] [NOTICE] [net_Src_6] terminate net (loop count: 10000000, loop rate: 109619, wall time: 91.224496)
[2019-09-13 12:21:00.726241] [NOTICE] [net_Snk_7] terminate net (loop count: 10000001, loop rate: 109619, wall time: 91.224499)
[2019-09-13 12:21:00.778402] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 109557, wall time: 91.276671)
[2019-09-13 12:21:00.778415] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 109557, wall time: 91.276690)
[2019-09-13 12:21:00.807281] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 109522, wall time: 91.305558)
[2019-09-13 12:21:00.807293] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 109522, wall time: 91.305572)
[2019-09-13 12:21:00.813476] [NOTICE] [net_Src_8] terminate net (loop count: 10000000, loop rate: 109514, wall time: 91.311742)
[2019-09-13 12:21:00.813487] [NOTICE] [net_Snk_9] terminate net (loop count: 10000001, loop rate: 109514, wall time: 91.311727)
[2019-09-13 12:21:00.855747] [NOTICE] [net_Src_12] terminate net (loop count: 10000000, loop rate: 109464, wall time: 91.353949)
[2019-09-13 12:21:00.855753] [NOTICE] [net_Snk_13] terminate net (loop count: 10000001, loop rate: 109464, wall time: 91.354019)
[2019-09-13 12:21:00.855814] [NOTICE] [main] end main thread (wall time: 91.355079)

Finally, the number of threads was scaled to the double of the number of supported hw-based threads (32) with the following program:

Src = extern box smx_src_rand( out data )
Snk = box smx_snk_null( in data )

A = Src.Snk
B = A|A|A|A

connect B|B|B|B

This program creates 16 pairs of two threads, where the first thread of a pair creates random messages and the second destroys them.
The program was run once where each thread was executed 10'000'000 times.

[2019-09-13 12:27:02.348223] [NOTICE] [net_Snk_31] terminate net (loop count: 10000001, loop rate: 51723, wall time: 193.334771)
[2019-09-13 12:27:02.348276] [NOTICE] [net_Src_30] terminate net (loop count: 10000000, loop rate: 51723, wall time: 193.334778)
[2019-09-13 12:27:02.645888] [NOTICE] [net_Src_26] terminate net (loop count: 10000000, loop rate: 51644, wall time: 193.632505)
[2019-09-13 12:27:02.645902] [NOTICE] [net_Snk_27] terminate net (loop count: 10000001, loop rate: 51644, wall time: 193.632492)
[2019-09-13 12:27:02.648223] [NOTICE] [net_Src_18] terminate net (loop count: 10000000, loop rate: 51643, wall time: 193.634855)
[2019-09-13 12:27:02.648235] [NOTICE] [net_Snk_19] terminate net (loop count: 10000001, loop rate: 51643, wall time: 193.634908)
[2019-09-13 12:27:02.910606] [NOTICE] [net_Src_2] terminate net (loop count: 10000000, loop rate: 51573, wall time: 193.897280)
[2019-09-13 12:27:02.910614] [NOTICE] [net_Snk_3] terminate net (loop count: 10000001, loop rate: 51573, wall time: 193.897326)
[2019-09-13 12:27:02.973110] [NOTICE] [net_Src_20] terminate net (loop count: 10000000, loop rate: 51557, wall time: 193.959701)
[2019-09-13 12:27:02.973126] [NOTICE] [net_Snk_21] terminate net (loop count: 10000001, loop rate: 51557, wall time: 193.959769)
[2019-09-13 12:27:03.042740] [NOTICE] [net_Snk_25] terminate net (loop count: 10000001, loop rate: 51538, wall time: 194.029327)
[2019-09-13 12:27:03.042769] [NOTICE] [net_Src_24] terminate net (loop count: 10000000, loop rate: 51538, wall time: 194.029375)
[2019-09-13 12:27:03.063629] [NOTICE] [net_Src_8] terminate net (loop count: 10000000, loop rate: 51533, wall time: 194.050312)
[2019-09-13 12:27:03.063638] [NOTICE] [net_Snk_9] terminate net (loop count: 10000001, loop rate: 51533, wall time: 194.050292)
[2019-09-13 12:27:03.108715] [NOTICE] [net_Snk_13] terminate net (loop count: 10000001, loop rate: 51521, wall time: 194.095412)
[2019-09-13 12:27:03.108747] [NOTICE] [net_Src_12] terminate net (loop count: 10000000, loop rate: 51521, wall time: 194.095359)
[2019-09-13 12:27:03.121826] [NOTICE] [net_Src_22] terminate net (loop count: 10000000, loop rate: 51517, wall time: 194.108425)
[2019-09-13 12:27:03.121847] [NOTICE] [net_Snk_23] terminate net (loop count: 10000001, loop rate: 51517, wall time: 194.108496)
[2019-09-13 12:27:03.139808] [NOTICE] [net_Src_10] terminate net (loop count: 10000000, loop rate: 51512, wall time: 194.126496)
[2019-09-13 12:27:03.139826] [NOTICE] [net_Snk_11] terminate net (loop count: 10000001, loop rate: 51512, wall time: 194.126475)
[2019-09-13 12:27:03.172501] [NOTICE] [net_Src_0] terminate net (loop count: 10000000, loop rate: 51504, wall time: 194.159204)
[2019-09-13 12:27:03.172512] [NOTICE] [net_Snk_1] terminate net (loop count: 10000001, loop rate: 51504, wall time: 194.159220)
[2019-09-13 12:27:03.198178] [NOTICE] [net_Src_4] terminate net (loop count: 10000000, loop rate: 51497, wall time: 194.184876)
[2019-09-13 12:27:03.198191] [NOTICE] [net_Snk_5] terminate net (loop count: 10000001, loop rate: 51497, wall time: 194.184892)
[2019-09-13 12:27:03.215053] [NOTICE] [net_Src_28] terminate net (loop count: 10000000, loop rate: 51492, wall time: 194.201621)
[2019-09-13 12:27:03.215067] [NOTICE] [net_Snk_29] terminate net (loop count: 10000001, loop rate: 51492, wall time: 194.201395)
[2019-09-13 12:27:03.234011] [NOTICE] [net_Src_16] terminate net (loop count: 10000000, loop rate: 51487, wall time: 194.220650)
[2019-09-13 12:27:03.234027] [NOTICE] [net_Snk_17] terminate net (loop count: 10000001, loop rate: 51488, wall time: 194.219742)
[2019-09-13 12:27:03.305945] [NOTICE] [net_Src_6] terminate net (loop count: 10000000, loop rate: 51468, wall time: 194.292644)
[2019-09-13 12:27:03.305975] [NOTICE] [net_Snk_7] terminate net (loop count: 10000001, loop rate: 51468, wall time: 194.292627)
[2019-09-13 12:27:03.544753] [NOTICE] [net_Src_14] terminate net (loop count: 10000000, loop rate: 51405, wall time: 194.531398)
[2019-09-13 12:27:03.544768] [NOTICE] [net_Snk_15] terminate net (loop count: 10000001, loop rate: 51405, wall time: 194.531413)
[2019-09-13 12:27:03.544859] [NOTICE] [main] end main thread (wall time: 194.533206)

The following table provides an overview of the results:

	2 TH	4 TH	8 TH	16 TH	32 TH
Walltime [s]	34.6	33.0	44.7	92.4	194.5
Walltime / TH [s]	17.3	8.3	5.6	5.8	6.1
msg rate [kHz]	289.0	606.1	894.9	865.8	822.6
TH rate [kHz]	289.0	303.0	223.7	108.2	51.4

TT

The focus of these tests was to establish a minimal execution period without loosing any messages.
For this purpose the simple network from before was used but wrapped by temporal firewalls:

Src = extern box smx_src_rand( out data )
Snk = box smx_snk_null( in data )

connect tt[10us](Src.Snk)

This program creates three nets, a source, a sink, and a temporal firewall connecting the two.
Each net was executed 1'000'000 times with the execution rate set to different time periods:

10us (10s walltime) < 0.005% DL misses (less than 50)
20us (20s walltime) < 0.001% DL misses (less than 10)
40us (40s walltime) < 0.0005% DL misses (less than 5)
80us (80s walltime) < 0.0001% DL misses (once 1 DL miss)

Next, the number of threads was increased to 7 with the following program:

Src = extern box smx_src_rand( out data )
Snk = box smx_snk_null( in data )

A = tt[10us](Src.Snk)
B = A|A|A

connect B

This program creates seven nets, three pairs of source and sink, and one temporal firewall connecting the pairs.
Each net was executed 1'000'000 times with the execution rate set to different time periods:

10us (10s walltime) < 0.01% DL misses (less than 100)
20us (20s walltime) < 0.002% DL misses (less than 20)
40us (40s walltime) < 0.0005% DL misses (less than 5)
80us (80s walltime) < 0.0001% DL misses (once 1 DL miss)

The results were a little bit worse that in the case of 3 threads.
This might be caused by the poor scalability of temporal firewalls:
TFs with the same execution rate are combined into one to avoid spawning too many threads.
The cost of this, however, is that the load on a TF increases proportionally to the number of connecting nets.
Given that a TF only has to forward messages this should only become a big deal when dealing with hundreds of threads.
The here presented example is rather special because the other nets are extremely simple and don not consume much CPU power.

Zmq

This test series aims at testing ZMQ performance.

The following network was used to determine read and write speed of ZMQ.

Src1 = extern box smx_src_rand( out data )
Snk1 = extern box smx_snk_zmq( in data, in topic open )
Src2 = extern box smx_src_zmq( out data, out topic )
Snk2 = box smx_snk_dump( in data, in topic )

connect Src1.Snk1|Src2.Snk2

This program produces random data and feeds it to a ZMQ source which produces the data to a port on localhost.
At the same time a ZMQ sink is reading data from the same port on localhost.
Everything is executed as fast as possible.

In a first test the PUB/SUB protocol of ZMQ was used.
The program produced 10'000'000 data points and successfully fed the data to ZMQ in 30s (i.e. at a data-rate of 331k messages per second).
The consuming net was not able to keep up with this rate and lost 19% of the data.

The results are the same for the following endpoints configuration:

bind: tcp://*:50030, connect: tcp://localhost:50030
bind: ipc:///tmp/feeds/0, connect: ipc:///tmp/feeds/0

In a first test the PIPELINE protocol of ZMQ was used.
In this case the configuration as used above produced different results:

bind: tcp://*:50030, connect: tcp://localhost:50030
The program produced 10'000'000 data points and successfully fed the data to ZMQ in 44s (i.e. at a data-rate of 227k messages per second).
In contrast to the PUB/SUB example, however, no messages were lost and 10'000'000 data points were successfully consumed.
bind: ipc:///tmp/feeds/0, connect: ipc:///tmp/feeds/0
The program produced 10'000'000 data points and successfully fed the data to ZMQ in 47s (i.e. at a data-rate of 213k messages per second).
On most test runs (3 out of 4) less than 500 data points (<0.005%) were lost.

It is interesting to note that when using the ipc transport, some messages are lost and the program is slightly slower compared to the tcp transport (localhost).

Debug Producer and Consumer

In order to do performance tests a general producer and consumer box implementation need to be created.

Internal Messages as BSON instead of JSON

Internal Streamix messages should be BSON binary structures instead of JSON strings (wherever possible). JSON strings should only be used when communicating over ZMQ.

Web-Based Profiler Analysis

In order to provide a performance overview of an application a GUI needs to be created.

In a first step this can be a simple node-js file which reads the profiler data from the database and produces a table with the following information:

total runtime
execution rate of each net
jitter of the net period
number of reads and writes per channel
the maximal number of elements stored in a queue

Integrate Travis

Build the examples via Travis. Doing so helps us to detect regressions.

Packaging of Streamix Libraries

It has to be decided how Streamix libraries are going to be distributed.

A few options come to mind:

deb
docker

The latter might struggle to cope with an RT-patch on the linux kernel. This paper might provide a solution for this.

Dump Data to File Before Sending to DB

Currently, in smx_mongo data is aggregated in memory during a configurable period of time and as soon as this time has elapsed, the data is stored to the database.

If the program crashes at some point, some data is lost (and this might be a lot, depending on the configured period). In order to remedy this an additional net could be introduced in parallel to smx_mongo which dumps the data to a binary file which is used as a circular buffer.

ZMQ Wrapper of a Box implementation

In order to allow a box implementation to be written in any language an implementation wrapper needs to be created.

One Idea is to use ZMQ and make each port of the box interfacable via the ZMQ bus.

Licenses

Rethink the licensing of all Streamix projects.

Currently, GPL-v3.0 is used which is too restrictive (and incompatible with some libraries that are used).
I would like to allow people to distribute their software under whatever license they wish, however, I would like them to be obligated to report back changes to Streamix code. Thus, MPL-v2.0 seems to be a better fit.

However MPL-v2.0 is not compatible with the igraph library which is distributed under GPL-2.0. GPL-v2.0 requires any Streamix project which makes use of igraph to also be published under GPL-v2.0.

Note that GPL-v3.0, under which Streamix is distributed, is not compatible with GPL-v2.0.

Hence the following licenses should be applied

streamix-c: GPL-v2.0
streamix-rtsp: GPL-v2.0
streamix-sia: GPL-v2.0
streamix-sia-lang: GPL-v2.0
streamix-rts: MPL-v2.0

Luckily, to only project that is used as a library (the RTS) does not depend on igraph. Consequently, any program that links to libsmxrts can be released under any license provided the following conditions are met:

zlog is linked dynamically (distributed under LGPL-v2.1)

Linking a dynamic library of a specific version

The question is: How to link a dynamic library of a specific version without linking it to a general so file?

FSM box implementation - Event Names

A box implementation that allows to control a Stremix network with a configurable Finite-State-Machine (FSM).

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.