Git Product home page Git Product logo

iotg_tsn_ref_sw's Introduction

IOTG TSN Reference Software for Linux

Overview

This project is a set of C-applications and scripts used to showcase different Ethernet-TSN features in Linux on specific Intel IOTG platform/software.

These serve as a practical example for those interested in developing TSN-capable software.

This project introduces three applications, each with their own set of examples:

  • TSQ - Time Synchronization Quality Measurement

    • tsq1 showcases the nanosecond-precision time synchronization between platforms using pulse-per-second (PPS) output & auxiliary timestamping (AUX_TS) input pins.
  • TXRX-TSN - AF_PACKET & AF_XDP socket-based application

    • vs1 showcases the bounded low-latency transmission/reception achievable via AF_PACKET & AF_XDP standard linux socket interfaces, while leveraging various device-specific Ethernet-TSN features.
  • OPCUA-SERVER - AF_PACKET & AF_XDP OPCUA-based application

    • opcua-pkt1 showcases the bounded low-latency transmission/reception achievable via AF_PACKET over libopen62541 (an OPC-UA-based library) APIs, while leveraging various device-specific Ethernet-TSN features over a single-ethernet connection. This application uses JSON files as an input.
    • opcua-pkt2 & opcua-xdp2 showcases the bounded low-latency transmission/reception achievable via AF_PACKET & AF_XDP over libopen62541 (an OPC-UA-based library) APIs, while leveraging various device-specific Ethernet-TSN features over two ethernet connection. This application uses JSON files as an input and require platforms with at least 2 Ethernet-TSN ports (e.g. EHL).
    • opcua-pkt3 & opcua-xdp3 is the same as opcua-*2 but with iperf3 background traffic.

Compatibility

Currently supported hardwares are:

	* Intel Atom XXX - (formerly Elkhart Lake) with its integrated Ethernet controller
	* Intel Core XXX - (formerly Tiger Lake UP3) with its integrated Ethernet controller

Currently supported systems are:

	* Yocto
	* Ubuntu

Important

# Execute all the scripts in "root user"/"super user" privilege mode.
# There are two method to execute all the scripts in "root user"/"super user" privilege mode.

Method 1:
# Enter into "root user"/"super user" privilege mode using command below:
	sudo su

Method 2:
# Execute all the scripts using command below:
	sudo ./<script_name>.sh

Dependencies

General

For system clock:

# Please ensure the system clock is set to current date before using the installer script to install the required dependencies.
# Otherwise the installation will failed.
# Set the system clock to current date using command below:
	date -s YY/MM/DD

For system shell:

# Bash are required as system shell in order to compile and run TSN Reference Software.
# Install bash using command below:
	sudo apt-get install bash

Packages Installer:

# All the TSN Ref Sw App required dependencies can be install by using the packages installer below:

IMPORTANCE: Please read and follow the instruction below before using the packages installer!

# In order to ease the installation of generic packages and IOTG customized helper libraries (libbpf-iotg and sopen62541-iotg) needed,
# we have provide a installation script.
# The script will check out a specific version of the libraries from upstream git and apply our patches on top of it.

# Suggestion is to use the installer script (after having kept the original libbpf and libopen62541 in a safe place).
# This will ensure the tsn ref sw will be able to find/use correct libraries.

NOTE: If your kernel support for Intel XDP+TBS, please ensure the '__u64 txtime' descriptor is available in struct xdp_desc() in /usr/include/linux/if_xdp.h as example below:

/* Rx/Tx descriptor */
struct xdp_desc {
        __u64 addr;
        __u32 len;
        __u32 options;
        __u64 txtime;
};

WARNING: You are able to install libbpf & open62541 without the 'txtime' descriptor but you might facing error/failure during the tests.

# The packages installer branch : https://github.com/intel/iotg_tsn_ref_sw
# Branch name : **master**
# Use the commands below to install the packages:

	cd dependencies
	./packages_installer.sh --proxy=<your.proxy.if.necessary.com> --git-proxy=<your.git.proxy.if.necessary.com>

# NOTE: The proxy and git-proxy are optional arguments, provide if necessary.
# NOTE: The packages installer only support the overwrite installation.
# NOTE: Refer to the below

# NOTE: If you are installing the libbpf and libopen62541-iotg manually, there is a chance the related open62541-iotg.pc file is not there.
# Hence, you might have to comment out this line in **configure.ac**.

**AM_CONDITIONAL([WITH_OPCUA], [test "x$no_open62541_iotg_fork" = "x0"] && test "x${HAVE_XDP}" = "xyes")**

# Disclaimer
	* The packages installers only serves to install dependencies for TSN Ref Sw App.
	* This project is not for intended for production use.
	* This project is intended to be used with specific platforms and bsp, other HW/SW combinations YMMV
	* Users are responsible for their own products' functionality and performance.

# FAQ

	If git clone fail, try the solution below:
	* Please configure the proxy according to your proxy setting before git clone
	* Please configure the system date up-to-date before git clone
	* Please reboot your system before git clone

Yocto

For compilation:

# If you are running from a compatible Intel-provided Yocto BSP & hardware, these
# software dependencies would have already been installed.

	* [Custom linux kernel headers](https://github.com/intel/linux-intel-lts/tree/5.*/preempt-rt)
  	- Include support for Intel XDP+TBS implementation
	* [Custom linux-libc-headers](https://github.com/intel/iotg-yocto-ese-bsp/tree/master/recipes-kernel/linux-libc-headers/linux-libc-headers)
  	- Include support for Intel XDP+TBS implementation
	* [Custom libopen62541-iotg](https://github.com/intel/iotg-yocto-ese-main/tree/master/recipes-connectivity/open62541)
  	- Include support for kernel implementation of Intel XDP+TBS (if_xdp.h change required)
	* [Custom libbpf](https://github.com/intel/iotg-yocto-ese-main/tree/master/recipes-connectivity/libbpf)
  	- Include support for kernel implementation of Intel XDP+TBS (if_xdp.h change required)
	* libelf
	* libjson-c

For run-time:

	* [Custom linux kernel](https://github.com/intel/linux-intel-lts/tree/5.*/preempt-rt)
	* shell tools including awk/sed
	* iproute2-ss200127
	* linuxptp [v3.0](https://github.com/richardcochran/linuxptp/releases/tag/v3.0)
	* Python 3.8.2
	* gnuplot 5.2
	* IceWM - Any GUI/window manager can be used, required to display graphs.

IceWM:

# If you're using a compatible intel-developed Yocto BSP, IceWM is its default
# window manager. Here are some keyboard shortcuts, in case a mouse is not available.

# Notation: C is control, M is meta/alt, S is shift, s is super

1. `<C-M-t>` to open xterm
2. `<M-f8>` to resize window (using arrow keys)
3. `<M-S-f10>` to maximise vertical space
4. `<M-f7>` to move window
5. `<C-M-leftarrow>` to move to the left, `rightarrow` for right
6. `<C-M-esc>` to show window list
7. `<M-space>` to show window actions menu

Ubuntu

Ubuntu based dependencies:

# In order to compile under Ubuntu, these packages need to be installed first (using sudo apt-get install ....) :
	* autoconf
	* build-essential
	* cmake
	* gawk
	* gcc
	* gnuplot
	* pkg-config
	* zlib1g
	* zlib1g-dev
	* libelf-dev
	* libjson-c-dev
	* xterm
	* iperf3
	* linuxptp

NOTE: ensure your proxy settings are correct.
NOTE: All the packages can be install by using the packages_installer.sh above.

For run-time:

	* [Real-Time Ubuntu 22.04](https://ubuntu.com/real-time)
	* iproute2 v5.15.0 [with real-time patches](https://github.com/intel/iotg-yocto-ese-main/tree/master/recipes-connectivity/iproute2/iproute2)
	* [ethtool](https://salsa.debian.org/kernel-team/ethtool) [with real-time patches](https://github.com/intel/iotg-yocto-ese-main/tree/master/recipes-extended/ethtool/ethtool)
	* linuxptp [v3.0 or later](https://github.com/richardcochran/linuxptp/releases/tag/v3.0)
	* Python 3.8.2

Build

# To build tsn ref sw, we are currently providing a single script that will build all
# binaries (tsq, txrx-tsn and opcua-server).

    ```
    cd <tsn_ref_sw_directory>
    ./build.sh
    ```

# To explicitly disable Intel specific XDP+TBS support in tsn ref sw

    ```
    cd <tsn_ref_sw_directory>
    ./build.sh -t
    ```

For further configuration details, check out README_config.md and ShellConfig

Run


This project is optimized to run on the supported hardware list and their respective bundled softwares (IFWI, BSP, preempt-rt kernel) which has been optimized for each platform's capabilities.

For details information please refer to the documentation below:

Documentation

Contents

Disclaimer

  • This project only serves to demonstrate TSN functionality and its usage on supported platforms and their environments.

  • This project is not for intended for production use.

  • This project is intended to be used with specific platforms and bsp, other HW/SW combinations YMMV

  • Users are responsible for their own products' functionality and performance.

Report Issues

If you see an issue, include these details in your issue submission:

  • Hardware setup (Platform, Ethernet controller/NIC)
  • Dependency version
  • OS or Linux distribution
  • Linux kernel version
  • Problem statement
  • Steps to reproduce
  • Images/Screenshots

Contribute

Refer to CONTRIBUTING.md

License

Refer to LICENSE.md

FAQ

For tips on how to run tsn ref sw, please refer to README_faq.md It contains example of certain frequently seen run-time error.

iotg_tsn_ref_sw's People

Contributors

abdulari avatar agunasek avatar azuratarmiziintel avatar ganyifang avatar junannlaiintel avatar jyong2 avatar leosartaj avatar locnnil avatar vincentxw avatar ws-intel avatar yongliang2022 avatar yoongsiang2 avatar zulkifl3 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

iotg_tsn_ref_sw's Issues

Compatibility with Intel I210

Hello,

i would like to know if it is possible, to use the recent iotg_tsn_ref_sw on an intel I210 NIC. I do not see any config files for I210, like there is for the i225.
What would the configuration files for a i210 look like? Are there any plans to adapt the scripts to a i210, or could you give me some hints how to modify the configs according to the i210 capabilities?
Thank you very much!

Best Regards
Markus

Problem with Instructions for Building BSP for Apollo Lake

Hello,

I am trying to build the Apollo Lake BSP, and the instructions seem to be wrong. I am referring to iotg_tsn_ref_sw-userguide-20190131/TSN_UG/index.htm, section Step 1: Prepare the Host Machine and Build the BSP. I have successfully setup the host build machine, cloned the iotg-yocto-bsp-public git repo, and ran the setup.sh script.
There seems to be two problems with the instructions in this section:
There is no E3900-MR4-PREEMPT-RT branch/tag to checkout
The line, "When prompted to build a custom image, select core-image-rt-tsn-sdk." We do not get prompted to "build a custom image," and when setup.sh does present targets, "core-image-rt-tsn-sdk" is not included in the list.

Thanks,

-Jack

Where can i find the i210 tsn driver source code?

Hello,
After i followed the TSN_UG_index.htm, run the setup.sh and create a usb flash drive from <yocto-image.hddimg>. I wonder how the i210 tsn driver implements, but i can't find anything on yocto_build/ useful for that. Does the i210 tsn driver implementation Confidential ?

sendmsg failed - 105: No buffer space available

Hi.

I am trying to recreate some the sample-app-taprio setup from demo 3 scenario 3. However, I am experiencing a sendmsg error in the udp_send function: 105: No buffer space available. I am using ubuntu 18.04 with mainline kernel 4.20.17.

When this error occurs the application stops transmitting any other data than priority 5 using tsn_prio5-s1s2s3, tsn_prio3-s1s2s3 for the application configuration.

Do you have any idea why this error occurs? Also let me know if I should provide additional information.

Regards.

Taus

Problem with Instructions for Building BSP for Apollo Lake

Hello,

I am trying to build the Apollo Lake BSP, and the instructions seem to be wrong. I am referring to iotg_tsn_ref_sw-userguide-20190131/TSN_UG/index.htm, section Step 1: Prepare the Host Machine and Build the BSP. I have successfully setup the host build machine, cloned the iotg-yocto-bsp-public git repo, and ran the setup.sh script.

There seems to be two problems with the instructions in this section:

  1. There is no E3900-MR4-PREEMPT-RT branch/tag to checkout
  2. The line, "When prompted to build a custom image, select core-image-rt-tsn-sdk." We do not get prompted to "build a custom image," and when setup.sh does present targets, "core-image-rt-tsn-sdk" is not included in the list.

Thanks,
Chris

Build fails:unknown type name

I have installed all Dependencies, but I still encountered some problems unknown type name and has no member named when I execute the command ./build.sh. The details are as follows

mv -f $depbase.Tpo $depbase.Po
depbase=echo src/opcua-tsn/opcua_datasource.o | sed 's|[^/]*$|.deps/&|;s|\.o$||';
gcc -DHAVE_CONFIG_H -I. -O2 -g -fstack-protector-strong -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Wformat-overflow -Wno-parentheses -Wno-missing-field-initializers -Wextra -Wall -fno-common -I/usr/local/include/open62541-iotg -I/usr/include/json-c -I/usr/local/include -I/usr/local/include -g -O2 -MT src/opcua-tsn/opcua_datasource.o -MD -MP -MF $depbase.Tpo -c -o src/opcua-tsn/opcua_datasource.o src/opcua-tsn/opcua_datasource.c &&
mv -f $depbase.Tpo $depbase.Po
In file included from src/opcua-tsn/multicallback_server.c:58:
src/opcua-tsn/opcua_custom.h:70:5: error: unknown type name ‘UA_ReaderGroupConfig’
70 | UA_ReaderGroupConfig config;
| ^~~~~~~~~~~~~~~~~~~~
src/opcua-tsn/opcua_custom.h:107:5: error: unknown type name ‘UA_PubSubConnectionConfig’
107 | UA_PubSubConnectionConfig *config;
| ^~~~~~~~~~~~~~~~~~~~~~~~~
src/opcua-tsn/opcua_custom.h:109:5: error: unknown type name ‘UA_PubSubChannel’
109 | UA_PubSubChannel *channel;
| ^~~~~~~~~~~~~~~~
src/opcua-tsn/opcua_custom.h:126:5: error: unknown type name ‘UA_WriterGroupConfig’
126 | UA_WriterGroupConfig config;
| ^~~~~~~~~~~~~~~~~~~~
src/opcua-tsn/multicallback_server.c: In function ‘setupOpcuaServer’:
src/opcua-tsn/multicallback_server.c:153:11: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayersSize’
153 | config->pubsubTransportLayersSize = 0;
| ^~
src/opcua-tsn/multicallback_server.c:154:11: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayers’
154 | config->pubsubTransportLayers =
| ^~
src/opcua-tsn/multicallback_server.c:155:35: error: expected expression before ‘)’ token
155 | (UA_PubSubTransportLayer *) UA_calloc(2, sizeof(UA_PubSubTransportLayer));
| ^
In file included from src/opcua-tsn/opcua_common.h:44,
from src/opcua-tsn/multicallback_server.c:55:
src/opcua-tsn/multicallback_server.c:156:21: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayers’
156 | catch_err(config->pubsubTransportLayers == NULL, "Out of memory");
| ^~
src/opcua-tsn/opcua_utils.h:50:13: note: in definition of macro ‘catch_err’
50 | if (A) {
| ^
src/opcua-tsn/multicallback_server.c:159:15: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayers’
159 | config->pubsubTransportLayers[0] = UA_PubSubTransportLayerEthernetXDP();
| ^~
src/opcua-tsn/multicallback_server.c:161:15: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayers’
161 | config->pubsubTransportLayers[0] = UA_PubSubTransportLayerEthernetETF();
| ^~
src/opcua-tsn/multicallback_server.c:163:11: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayersSize’
163 | config->pubsubTransportLayersSize++;
| ^~
src/opcua-tsn/multicallback_server.c:187:19: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayers’
187 | config->pubsubTransportLayers[1] = UA_PubSubTransportLayerEthernetXDP();
| ^~
src/opcua-tsn/multicallback_server.c:188:19: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayersSize’
188 | config->pubsubTransportLayersSize++;
| ^~
src/opcua-tsn/multicallback_server.c:192:19: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayers’
192 | config->pubsubTransportLayers[1] = UA_PubSubTransportLayerEthernetETF();
| ^~
src/opcua-tsn/multicallback_server.c:193:19: error: ‘UA_ServerConfig’ {aka ‘struct UA_ServerConfig’} has no member named ‘pubsubTransportLayersSize’
193 | config->pubsubTransportLayersSize++;
| ^~
make[1]: *** [Makefile:496:src/opcua-tsn/multicallback_server.o] 错误 1
make[1]: *** 正在等待未完成的任务....
In file included from src/opcua-tsn/opcua_custom.c:46:
src/opcua-tsn/opcua_custom.h:70:5: error: unknown type name ‘UA_ReaderGroupConfig’
70 | UA_ReaderGroupConfig config;
| ^~~~~~~~~~~~~~~~~~~~
src/opcua-tsn/opcua_custom.h:107:5: error: unknown type name ‘UA_PubSubConnectionConfig’
107 | UA_PubSubConnectionConfig *config;
| ^~~~~~~~~~~~~~~~~~~~~~~~~
src/opcua-tsn/opcua_custom.h:109:5: error: unknown type name ‘UA_PubSubChannel’
109 | UA_PubSubChannel *channel;
| ^~~~~~~~~~~~~~~~
src/opcua-tsn/opcua_custom.h:126:5: error: unknown type name ‘UA_WriterGroupConfig’
126 | UA_WriterGroupConfig config;
| ^~~~~~~~~~~~~~~~~~~~
src/opcua-tsn/opcua_custom.c: In function ‘pub_thread’:
src/opcua-tsn/opcua_custom.c:179:31: error: request for member ‘transportSettings’ in something not a structure or union
179 | currentWriterGroup->config.transportSettings = transportSettings;
| ^
make[1]: *** [Makefile:496:src/opcua-tsn/opcua_custom.o] 错误 1
make[1]: 离开目录“/home/lab640/gmj/iotg_tsn_ref_sw”
make: *** [Makefile:327:all] 错误 2

Changes in Linux-5.4

Hi,
I have noticed some changes in kernel/net/sched/sch_taprio.c from Kernel Linux-5.4. I wonder if there is any possibility that we could configure taprio in software mode without I210 or other dedicated network adaptor.
Best regards!

Permission Issues while enabling extts and pps

CPU: Intel Core i3-2120
System: Ubuntu 18.04

I run the setup command for ehl with the following output

ptp4l[8661.727]: timed out while polling for tx timestamp
ptp4l[8661.727]: increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
ptp4l[8661.728]: port 1: send peer delay request failed
ptp4l[8661.728]: port 1: MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[8661.863]: port 1: link down
ptp4l[8661.863]: selected local clock aa00aa.fffe.00aa00 as best master
ptp4l[8661.863]: port 1: assuming the grand master role
ptp4l[8662.073]: selected local clock aa00aa.fffe.00aa00 as best master
ptp4l[8662.074]: port 1: assuming the grand master role
ptp4l[8665.126]: selected local clock aa00aa.fffe.00aa00 as best master
ptp4l[8665.126]: port 1: assuming the grand master role
Mapping socket priority N to VLAN priority N for enp0s25
Turning off vlan stripping
ptp4l[8665.154]: selected local clock aa00aa.fffe.00aa00 as best master
ptp4l[8665.154]: port 1: assuming the grand master role
Setting IRQ affinity based on /usr/share/iotg_tsn_ref_sw/shell/../common/irq_affinity_4c_8tx_8rx.map
Echo-ing 0x01 > /proc/irq/27/smp_affinity --> enp0s25:tx-0 
Echo-ing 0x01 > /proc/irq/27/smp_affinity --> enp0s25:rx-0 
Echo-ing 0x02 > /proc/irq/27/smp_affinity --> enp0s25:tx-1 
Echo-ing 0x02 > /proc/irq/27/smp_affinity --> enp0s25:rx-1 
Echo-ing 0x04 > /proc/irq/27/smp_affinity --> enp0s25:tx-6 
Echo-ing 0x08 > /proc/irq/27/smp_affinity --> enp0s25:rx-2 
Running PTP4L & PHC2SYS
Using gPTP_RGMII-MV1510-1G.cfg
ptp4l[8665.191]: selected /dev/ptp0 as PTP clock
ptp4l[8665.243]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[8665.243]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[8665.243]: port 1: link down
ptp4l[8665.243]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[8665.290]: selected local clock aa00aa.fffe.00aa00 as best master
ptp4l[8665.290]: port 1: assuming the grand master role
ptp4l[8665.540]: port 1: link up
ptp4l[8665.587]: port 1: FAULTY to LISTENING on INIT_COMPLETE
sending: SET GRANDMASTER_SETTINGS_NP
ptp4l[8667.192]: selected local clock aa00aa.fffe.00aa00 as best master
        aa00aa.fffe.00aa00-0 seq 0 RESPONSE MANAGEMENT GRANDMASTER_SETTINGS_NP 
                clockClass              248
                clockAccuracy           0xfe
                offsetScaledLogVariance 0xffff
                currentUtcOffset        37
                leap61                  0
                leap59                  0
                currentUtcOffsetValid   1
                ptpTimescale            1
                timeTraceable           1
                frequencyTraceable      0
                timeSource              0xa0
ptp4l[8669.394]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[8669.394]: selected local clock aa00aa.fffe.00aa00 as best master
ptp4l[8669.394]: port 1: assuming the grand master role
phc2sys[8670.194]: config item (null).clock_servo is 0
phc2sys[8670.194]: config item (null).kernel_leap is 1
phc2sys[8670.194]: config item (null).sanity_freq_limit is 200000000
phc2sys[8670.195]: config item (null).pi_proportional_const is 0.700000
phc2sys[8670.195]: config item (null).pi_integral_const is 0.300000
phc2sys[8670.195]: config item (null).pi_proportional_scale is 0.000000
phc2sys[8670.195]: config item (null).pi_proportional_exponent is -0.300000
phc2sys[8670.195]: config item (null).pi_proportional_norm_max is 0.700000
phc2sys[8670.195]: config item (null).pi_integral_scale is 0.000000
phc2sys[8670.195]: config item (null).pi_integral_exponent is 0.400000
phc2sys[8670.195]: config item (null).pi_integral_norm_max is 0.300000
phc2sys[8670.195]: config item (null).step_threshold is 1.000000
phc2sys[8670.195]: config item (null).first_step_threshold is 0.000020
phc2sys[8670.195]: config item (null).max_frequency is 900000000
phc2sys[8670.195]: config item (null).servo_offset_threshold is 0
phc2sys[8670.195]: config item (null).servo_num_offset_values is 10
phc2sys[8670.195]: PI servo: sync interval 1.000 kp 0.700 ki 0.300000
phc2sys[8670.195]: ioctl PTP_SYS_OFFSET_PRECISE: Operation not supported
phc2sys[8670.195]: config item (null).pi_proportional_const is 0.700000
phc2sys[8670.195]: config item (null).pi_integral_const is 0.300000
phc2sys[8670.195]: config item (null).pi_proportional_scale is 0.000000
phc2sys[8670.195]: config item (null).pi_proportional_exponent is -0.300000
phc2sys[8670.195]: config item (null).pi_proportional_norm_max is 0.700000
phc2sys[8670.195]: config item (null).pi_integral_scale is 0.000000
phc2sys[8670.195]: config item (null).pi_integral_exponent is 0.400000
phc2sys[8670.195]: config item (null).pi_integral_norm_max is 0.300000
phc2sys[8670.195]: config item (null).step_threshold is 1.000000
phc2sys[8670.195]: config item (null).first_step_threshold is 0.000020
phc2sys[8670.195]: config item (null).max_frequency is 900000000
phc2sys[8670.195]: config item (null).servo_offset_threshold is 0
phc2sys[8670.195]: config item (null).servo_num_offset_values is 10
phc2sys[8670.195]: PI servo: sync interval 1.000 kp 0.700 ki 0.300000
phc2sys[8670.195]: config item (null).transportSpecific is 1
phc2sys[8670.195]: config item (null).domainNumber is 0
phc2sys[8670.195]: config item (null).uds_address is '/var/run/ptp4l'
phc2sys[8671.196]: CLOCK_REALTIME phc offset -18801284 s0 freq +100000 delay    753
phc2sys[8672.197]: CLOCK_REALTIME phc offset -19508090 s1 freq -606616 delay    826
phc2sys[8673.197]: CLOCK_REALTIME phc offset   -882985 s2 freq -1489601 delay    826
phc2sys[8674.198]: CLOCK_REALTIME phc offset  -6500819 s2 freq -7372331 delay    798
phc2sys[8675.198]: CLOCK_REALTIME phc offset  -5323636 s2 freq -8145394 delay    829
phc2sys[8676.198]: CLOCK_REALTIME phc offset  -2604336 s2 freq -7023184 delay    834
phc2sys[8677.198]: CLOCK_REALTIME phc offset   -330147 s2 freq -5530296 delay    744
phc2sys[8678.198]: CLOCK_REALTIME phc offset   1049529 s2 freq -4249664 delay    828
phc2sys[8679.199]: CLOCK_REALTIME phc offset   1668363 s2 freq -3315972 delay    767
phc2sys[8680.199]: CLOCK_REALTIME phc offset   1805708 s2 freq -2678118 delay    832
phc2sys[8681.199]: CLOCK_REALTIME phc offset   1703775 s2 freq -2238338 delay    832
phc2sys[8682.199]: CLOCK_REALTIME phc offset   1509491 s2 freq -1921490 delay    799
phc2sys[8683.199]: CLOCK_REALTIME phc offset   1297991 s2 freq -1680143 delay    829
phc2sys[8684.200]: CLOCK_REALTIME phc offset   1112599 s2 freq -1476137 delay    761
phc2sys[8685.200]: CLOCK_REALTIME phc offset    955500 s2 freq -1299457 delay    832
phc2sys[8686.200]: CLOCK_REALTIME phc offset    825625 s2 freq -1142682 delay    832
phc2sys[8687.200]: CLOCK_REALTIME phc offset    716718 s2 freq -1003901 delay    832
phc2sys[8688.200]: CLOCK_REALTIME phc offset    625847 s2 freq -879757 delay    832
phc2sys[8689.201]: CLOCK_REALTIME phc offset    545983 s2 freq -771867 delay    755
phc2sys[8690.201]: CLOCK_REALTIME phc offset    476987 s2 freq -677068 delay    829
phc2sys[8690.206]: CLOCK_REALTIME phc offset    477947 s2 freq -533012 delay    831
Enabling extts on enp0s25 (ptp0)
/usr/share/iotg_tsn_ref_sw/shell/helpers.sh: line 347: /sys/class/ptp/ptp0/extts_enable: Permission denied
Enabling pps on enp0s25 (ptp0)
/usr/share/iotg_tsn_ref_sw/shell/helpers.sh: line 375: /sys/class/ptp/ptp0/period: Permission denied

As detailed in the last two lines, the extts_enable and period files do not exist in ptp directory nor can be created. Am I missing any configuration steps?

AF_XDP Actual data location in payload

I'm using AF_XDP example and it works as expected but so far I'm unable to locate the actual data in payload it sends to receiver. I can see sequence number (payload->seq) , timestamp (payload->tx_timestampA) etc but where can I find the actual data received in payload?
Basically I want to send some custom data over AF_XDP and wants to verify the data received is valid.

Problem with package build "open62541-iotg"

Hardware: intel up2 , cpu N***
System: Ubuntu 20.04

checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... gcc3
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libelf... yes
checking for libbpf... yes
checking for libjson... yes
checking for open62451... no
configure: error: Package requirements (open62541-iotg) were not met:

No package 'open62541-iotg' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables open62451_CFLAGS
and open62451_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
Configure failed

I try to install the open62541, but still failed.

sudo add-apt-repository ppa:open62541-team/ppa
sudo apt-get update
sudo apt-get install libopen62541-1-dev

I want to know how to compile the open62541-iotg package.

./txrx-tsn: error while loading shared libraries: libbpf.so.0: cannot open shar

Hello how are you?

I would like to know if someone can help me with a problem I have been having to run the last two IOTG applications (AF_PACKET & AF_XDP socket-based application and OPCUA-SERVER - AF_PACKET & AF_XDP OPCUA-based application. Specifically, both use the binary . /txrx-tsn, which is causing the problem.

First of all, I would like to indicate that I have installed and resolved all dependencies indicated by you in the README.

Custom linux kernel headers
Custom linux-libc-headers
Custom libopen62541-iotg
Custom libbpf
libelf
libjson-c
For run-time:

Custom linux kernel
shell tools including awk/sed
iproute2-ss200127
linuxptp v3.0
Python 3.8.2
gnuplot 5.2
IceWM - Any GUI/window manager can be used, required to display graphs.

All are up and running.

The problem log.
root@net1:/home/mirc/iotg_tsn_ref_sw# ./run.sh i225 enp2s0 vs1a run
Run.sh selected: vs1a
'./afpkt-txtstamps.txt' -> '/tmp/afpkt-txtstamps.txt'
'./afxdp-txtstamps.txt' -> '/tmp/afxdp-txtstamps.txt'
PHASE 1: AF_PACKET transmit (1010 seconds)
CMD: ./txrx-tsn -i enp2s0 -PtTq 1 -n 500000 -l 64 -y 2000000 -e 700000 -o 20000
./txrx-tsn: error while loading shared libraries: libbpf.so.0: cannot open shar

As you can see above, the txrx-tsn binary cannot find libbpf.so.0. The strange thing that I performed the iotg build, like the ./build command and everything flows fine. Also, the libbpf flgas seem to be correctly indicated (Figure).

image

I did an analysis of the files, MakeFile-->configure-->trxr.c and txrx.h --> in addition to the related sources and found no directory specification.

As the problem seems to be directly related to the libbpf Installation location, in the Figure you can see that the installation location of the necessary files are in the directory converging with the MakeFile.

image

Can anybody help me?

Build fails: open62541-iotg not found

checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... gcc3
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libelf... yes
checking for libbpf... yes
checking for libjson... yes
checking for open62451... no
configure: error: Package requirements (open62541-iotg) were not met:

No package 'open62541-iotg' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables open62451_CFLAGS
and open62451_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
Configure failed

I have installed the open62541 via 'https://github.com/open62541/open62541.git', and it exists in pkg-config --list-all. I want to know how to compile the open62541-iotg package.

> @kcn21 did you fix the issue? I faced the same issue right now, and i am using kernel 4.20 and 5.4. If you can help me, i will be grateful.

@kcn21 did you fix the issue? I faced the same issue right now, and i am using kernel 4.20 and 5.4. If you can help me, i will be grateful.

Hello @liing0228, you have to enable modules for taprio in the kernel, but many kernels didn't have modules for the TAPRIO, so I think that's why you are facing this issue. Please prefer kernel above 5.0 versions if you didn't have dependencies on kernel versions.

Hello @EnfangCui and @vincentxw, I am following tsn.readthdocs.io for AVB examples, and they have written to use the 4.19 kernel for AVB examples. I also tried kernel version 5.12, but it shows an AAF PDU error and abruptly stops the sending. So I have to use Linux Kernel 4.19. Still, in Linux Kernel 4.19, there are no options to enable the TAPRIO module in the kernel under Networking support --> Networking options --> QOS and /or a fair queueing section.
Can you please show a way to add a qdisc module in the kernel? I have updated the iproute2 also, but as there is no module for taprio in the kernel, there is no advantage of iproute2 degradation. Your guidance in this situation is highly appreciated

Thanks,
Kunj Naik

Originally posted by @kcn21 in #5 (comment)

packet loss in Qbv Demo 3 Scenario 3

Hi,
I run the demo in Qbv Demo 3 Scenario 3, and the plotted graph image is perfect. But when reading the generated file latencies.dat which was generated by plot.sh, I found some latencies with 999975 or 1000002, or even bigger, such as 16499976. I think it may be caused by packet loss, and is it normal?

Thanks!

RTNETLINK answers: Operation not supported

Hi,
I installed the latest version of iproute2 solved the problem “Unknown qdisc taprio”. When I run the "tc qdisc replace dev eth0 parent root handle 100 taprio ", it gives a new error :"RTNETLINK answers: Operation not supported."

I have looked on google to try to find a solution. It seems that my kernel configuration is missing something? However, it still gives me the error after adding all these configs as follows.

CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CT_NETLINK=y
CONFIG_SCSI_NETLINK=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_NET_SCHED=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_NETFILTER_XT_TARGET_MARK=y

I wonder is there a way to fix this problem? Could you help me? Look forward to your kind reply. Thanks!

Unknown qdisc "taprio", hence option is unparsable

I used the ubuntu 18.04 with Linux kernel 5.0.1 including "taprio" drivers at Networking support --> Networking options --> QOS and /or fair queueing. But when I run the "tc qdisc replace dev eth0 parent root handle 100 taprio " command, it gives error: "Unknown qdisc "taprio", hence option is unparsable". I wonder how to make the Linux support taprio. Can you give me a more detailed kernel configuration? Look forward to your kind reply. Thanks!

What is the matching open62541 commit to match with master

Hello, What is the matching 'https://github.com/open62541/open62541.git' commit to use?
I have generated amalgamated open62541.h from open62541/master branch
(commit 6427010002a44a927dd2ba22b51acf8d31659329)

while building master branch here, I see this error ->

git/iotg_tsn_ref_sw/sample-app-opcua-pubsub/tutorial_pubsub_publish_sotxtime.c:137:21: error: \u2018UA_PubSubConnectionConfig {aka struct }\u2019 has no member named \u2018sockPrio\u2019
connectionConfig.sockPrio = SOCKET_PRIORITY;

The sample code sample-app-1 make always fail

Hi all,

Follow the manual 605583_TSN_UserGuide, the sample code cannot make. The below error happens.

sample-app-1.c:64:10: fatal error: open62541.h: No such file or directory
#include "open62541.h"
^~~~~~~~~~~~~
compilation terminated.

Is anything I missed

Best regard,

Kunyang

Problem about hardwares

Hello,

I'm a student learning about TSN. I want to run the project on my raspberry pi (4B+). I have succesfully built it on Ubuntu-20.10- Server. However, the pi don't support hardware timestamp so i can't run it. The supported hardwares mentioned in the Compatibility are cpus and NIC. Is there any platform like raspberry pi for beginners to test the project?

Thanks!

double free detected in tcache 2

I am trying to run the taprio example with 5.4.0-25-generic kernel on the recent Ubuntu 20.04 LTS.

After minimal adaption of sample-app-taprio.c to include sock_txtime and txtime_flags from more recent /usr/include/linux/net_tstamp.h, I am facing the double free error when starting the receiver:

➜  sample-app-taprio git:(apollolake-i) ✗ sudo ./sample-app-taprio -i ens2 -x 2 -q "5 3" -y 3
free(): double free detected in tcache 2

Sender seems to start fine.

Are you planning to maintain the repository in future and evaluate against updated linux headers?

ETA for driving patches upstream

What is the ETA for driving the patches for external libraries upstream? Having to patch external librareis is problematic, and prevents this software from being packaged and incorporated into the Linux Distributions (or our internal systems that follow the same rules).

Intel nuc stacked with vs1a setup

Hi all,
I'm trying to test the code with a system based on 2 Nucs with i225 cards. When I launch

./run.sh i225 enp88s0 vs1a setup

I got:
Run.sh selected: vs1a
/var/log folder for logging exists.
combined unmodified, ignoring
no channel parameters changed.
current values: rx 0 tx 0 other 1 combined 4
Mapping socket priority N to VLAN priority N for enp88s0
Setting IRQ affinity based on /home/nuc01/iotg_tsn_ref_sw/shell/../common/irq_affinity_4c_4TxRx.map
Echo-ing 0x01 > /proc/irq/157/smp_affinity --> enp88s0:TxRx-0
Echo-ing 0x04 > /proc/irq/159/smp_affinity --> enp88s0:TxRx-2
Echo-ing 0x08 > /proc/irq/160/smp_affinity --> enp88s0:TxRx-3
Running PTP4L & PHC2SYS
Using gPTP_i225-1G.cfg
Run: tc qdisc replace dev enp88s0 parent root handle 100 taprio num_tc 4 map 3 1 0 2 3 3 3 3 3 3 3 3 3 3 3 3 queues 1@0 1@1 1@2 1@3 base-time 1681459807475613954 sched-entry S 0F 1000000 flags 0x2
Run: tc qdisc replace dev enp88s0 parent 100:3 etf clockid CLOCK_TAI delta 700000 offload
/home/nuc01/iotg_tsn_ref_sw/shell/helpers.sh: line 299: 1694 Killed $CMD

And then the machine cannot be used and it needs a hardware reset.
The station B, an identical nuc, proceeds correctly until the end of the setup script.

Thank you in advance
Claudio Zunino

How to config XDP+TBS availability in kernel?

Hello, in my testing device(Vecow ECX-3000-4G, CPU Intel Core I9-13900TE with three I226-IT NICs), I've installed Ubuntu22.04 in two different way:
(1. ECI image, core jammy https://eci.intel.com/docs/3.0.2/getstarted/building/core-jammy.html# ;
2. Ubuntu Pro realtime-kernel ) ,
but neither do checking, the result is no --- "Checking for XDP+TBS availability in kernel... no".

So my questions is how to enable XDP+TBS in Ubuntu? Beyond Ubuntu, how to configure yocto-based system enabling this kernel feature?

Build fails: libelf not found

Build fails on Debian 10 with

checking dependency style of gcc... gcc3
./configure: line 3410: syntax error near unexpected token `libelf,'
./configure: line 3410: `PKG_CHECK_MODULES(libelf, libelf)'
Configure failed

System:

$ lsb_release -d
Description:    Debian GNU/Linux 10 (buster)

libelf is installed:

~/iotg_tsn_ref_sw$ sudo apt list --installed | grep libelf

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libelf++0/stable,now 0.3-1 amd64  [installiert]
libelf-dev/stable,now 0.176-1.1 amd64  [installiert]
libelf1/stable,now 0.176-1.1 amd64  [installiert]
libelfin-dev/stable,now 0.3-1 amd64  [installiert]

Why after Priority 7 setting etf , ptp4l works abnormally?

I've got a switch with PTP clock enable as master, and connected it with two I210 devices as slave. I want to run "IEEE 802.1Qbv Demo 3 Scenario 3 Time-Aware Traffic Scheduling and LaunchTime Enabled".
I changed file "setup_generic.sh" line 31 without "egress=qos-map 7:7", because i want to run ptp4l without VLAN, so this setup is redundant. I also changed Step 3 [Board A] "$ ./setup_sync.sh -i enp1s0.3 -b boardA" to "$ ./setup_sync.sh -i enp1s0 -b boardB", so both I210 device work as PTP slave. Before Step 10, the ptp4l on BoardA and BoardB works good after Step 9 "python scheduler.py -i enp1s0 -q queue-s3-s4.cfg -e 120 -g gate-s2s3.sched", and I got the correct test for rest steps.

Strange things happened when i wanted to do the test second time. But I changed the file "queue-s3s4.cfg" like below:
5 0 etf 5000000 ----------------> 7 0 etf 5000000
3 1 etf 5000000 ----------------> 6 1 etf 5000000
7 2 ----------------> 5 2

Before step 9 , ptp4l on boardA worked well, but after this step, ptp4l worked abnormally, so i could not do following steps. It happened like this:

ptp4l[1487.416]: rms 19 max 37 freq -10141 +/- 17 delay 198 +/- 0
ptp4l[1488.416]: rms 25 max 62 frep -10167 +/- 33
ptp4l[1489.338]: port 1: link down
ptp4l[1489.338]: port 1: SLAVE to FAULTY on FAULT_DETECTED(FT_UNSPECIFIED)
ptp4l[1489.362]: select local clock a0369f.fffe.628d62 as best master
ptp4l[1489.362]: assuming the grand master role
ptp4l[1489.362]: port 1: master state recommended in slave only mode
ptp4l[1489.362]: port 1: defaultDs.priority1 probably misconfigured
ptp4l[1491.056]: port 1: link up
ptp4l[1491.078]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[1491.162]: port 1: new foreign master 010203.0405.060708-9
ptp4l[1491.411]: select best master clock 010203.0405.060708
ptp4l[1491.411]: updating UTC offset to 0
ptp4l[1491.411]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[1492.028]: send failed: 105 No buffer space available
ptp4l[1492.028]: port1: send delay request failed
ptp4l[1492.028]: port1: UNCALIBRATED to FAULTY on FAULT_DETECTED(FT_UNSPECIFIED)
ptp4l[1508.076]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[1508.161]: port 1: new foreign master 010203.0405.060708-9
ptp4l[1508.411]: select best master clock 010203.0405.060708
ptp4l[1508.411]: updating UTC offset to 0
ptp4l[1508.411]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[1508.910]: send failed: 105 No buffer space available
ptp4l[1508.910]: port1: send delay request failed
ptp4l[1508.910]: port1: UNCALIBRATED to FAULTY on FAULT_DETECTED(FT_UNSPECIFIED)
...
I tried several times later, and find that when "ETF" was set on priority 7, ptp4l can't work normaly. I run such tests from USB flashs which were made from core-image-rt-tsn-sdk-intel-corei7-64-20200630234202.hddimg.

Problem with how to contribute

Hello,

I am trying to submit a pull request, but there is no branch called "development".
How do I resolve this issue?

Add a parameter to specify the socket priority to txrx-tsn

The vlan-prio command line option is used to specify the PCP value and the socket priority:

	{"vlan-prio",	'q',	"NUM",	0, "packet vlan priority, also socket priority\n"
					   "	Def: 0 | Min: 0 | Max: 7"},

The value supplied, between 0 and 7, is then used to:

  • Set the socket prio
  • Set the XDP queue to be used
  • Set the PCP value
	case 'q':
		len = strlen(arg);
		res = strtol((const char *)arg, &str_end, 10);
		if (errno || res < 0 || res >= 7 || str_end != &arg[len])
			exit_with_error("Invalid queue number/socket priority. Check --help");
		opt->socket_prio = (uint32_t)res;
#ifdef WITH_XDP
		opt->x_opt.queue = opt->socket_prio;
#endif
		opt->vlan_prio = opt->socket_prio * 32;
		break;

Implementation suggestion

The implementation would require to parse a new option for 'socket-prio'. The behavior would be as follows:

  • AF_PACKET (socket mode afpkt parameter)

    • Only vlan-prio is provided: same behavior as of now, to keep backwards compatibility.
    • Only socket-prio is provided
      • Check that the value is in the 0 to 15 range
      • Initialize opt->socket_prio to the value provided for the parameter
      • Leave opt->vlan_prio with the default value
    • Both are provided: vlan-prio is configured first as when only vlan-prio is provided. Then, on top of it, apply the socket-prio configuration as above: check that the value is in the 0-15 range, and then initialize opt->socket_prio to the value provided for the parameter. opt->vlan_prio remains with the value provided for the vlan-prio parameter.
  • AF_XDP (socket mode afxdp parameter)

    • Only vlan-prio is provided: same behavior as of now, to keep backwards compatibility.
    • Only socket-prio is provided: raise an error and inform that AF_XDP sockets do not implement a socket priority
    • Both are provided: raise an error and inform that AF_XDP sockets do not implement a socket priority

@AzuraTarmiziIntel could you please provide some guidance about how to proceed with this usage of socket_prio?

Qav simple-talker-cmsg transmits too many packets.

Hi,

I'm trying to follow the demos provided in the documentation, more specifically "Qav Demo 2 Scenario 2 ", but when the simple-talker-cmsg is activated, it transmits 900,000 packets/s instead of 8000 packets/s as in the example.

What settings determine the transfer rate of the simple-talker-cmsg?
What do you believe causes this problem?

Regards.

Taus

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.