netsec-ethz / scion-apps Goto Github PK

Currently, the health check runs as a single block. When future scripts run over a few seconds, users will wonder what's happening without better feedback. This would add:

• Max 60s timeout for any single bash script.
• UI label "In Progress" in amber for in progress scripts.
• UI label "Pending" in grey for scripts prior to execution.
• Health check landing page will show all scripts as "Pending" on page load.
• Background tasks to run scripts and update page.

Following up on netsec-ethz/scion#61, as SCION apps are (will be) installed into /usr/bin thus globally available on the end users' systems, they should

• have meaningful names
• do not clash with already released products

E.g. netcat or ssh is something that has to be renamed as we cannot blindly install a binary named netcat into any system which we expect to be still functional afterwards :)

Currently the sensorfetcher app assumes existence of the $SC env variable pointing to the directory containing /gen/ia. This means it must always be run from the context of scion user, what in the scope of automated infrastructure monitoring is not desired.

If access to /gen/ia is required, the app should be able to consume it e.g. as a command line parameter, but not assume the $SC env variable will be present.

As @matzf suggested in #15 [edit], we should provide some educational notes in context on webapp visualization features. Some users who use the webapp tool may have educated themselves and are eager to view the transparent elements of SCION. Others, may have a more opaque knowledge of its operation, and this would be a chance to educate and draw the users to deeper resources like tutorials, experiments, and the book.

Some notes that would help on each page:

• Health: Why must these checks succeed? What system requirements are at risk and why?
• Apps: We can point to tutorials here in many cases.
• As Topology: What are these services: br.., bs..., cs..., ps..? What do they do?
• ISD TRC: What are TRCs? How are they used? Where do they come from? How are cached used? What do the fields mean?
• AS Certificate: Similar questions as TRCs.

This should move to https://github.com/netsec-ethz/scion-apps for future maintenance: https://github.com/netsec-ethz/scion-viz/tree/master/python/appengine.

Also, webapp/config/servers_default.json could move to the GAE tool in some form to allow for more flexible updates in the future (suggested in #44).

Currently it seems the bwserver is not handling properly more than ~5 clients trying to perform the test at the same time.

Please note this is a very draft bug report without much reliable data. It is based only on the experiment of trying to run 50 instances of bwclient which failed very fast.

If someone decides to jump on it, more specific data can be regenerated and provided.

When RAINS is ready, make the Dial() method resolve hostnames using RAINS istead of using the custom DNS field in the Transport.

Setup continuous integration (circleci, to keep things simple) system for this repository.

This should:

• build
• run formatting / static analysis checks ( go fmt, golint, go vet, ...)
• run unit tests

Similar as in scionproto, place at the beginning of the file:

// Copyright 2019 ETH Zurich
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.package main

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting camerapp.

Currently, the SCION dispatcher requires clients to explicitly set their local SCION address. This is unconvenient.
As soon as dispatcher allows for nil local addresses get rid of mandatory LAddr field in RoundTripper and make it optional instead.
As an option still allow clients to specify the address (or port only).

The server crashes when it receives an SCMP revocation (for example because a client disconnects). This should be handled properly.

related issue in bwtester: #6

Currently the only flag in https://github.com/netsec-ethz/scion-apps/blob/master/deps.sh#L2 is set -x. That means if during runtime of the script something fails, we do not fail and just continue to run.

After a quick glance of the logic it seems we would strongly benefit from failing if something goes wrong in the process. Please have a look at the following run on the environment without Go installed

root@596d8e10ada6:/scion-apps# ./deps.sh 
+ printf '\n### Getting govendor ###\n'

### Getting govendor ###
+ govendor -version
+ grep v1.0.8
+ GOVENDORDIR=/root/go/src/github.com/kardianos/govendor
+ '[' -d /root/go/src/github.com/kardianos/govendor ']'
+ git clone https://github.com/kardianos/govendor /root/go/src/github.com/kardianos/govendor
Cloning into '/root/go/src/github.com/kardianos/govendor'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3421 (delta 0), reused 0 (delta 0), pack-reused 3418
Receiving objects: 100% (3421/3421), 2.31 MiB | 3.30 MiB/s, done.
Resolving deltas: 100% (2235/2235), done.
Checking connectivity... done.
+ cd /root/go/src/github.com/kardianos/govendor
+ git checkout fbbc78e8d1b533dfcf81c2a4be2cec2617a926f7
Note: checking out 'fbbc78e8d1b533dfcf81c2a4be2cec2617a926f7'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at fbbc78e... Update help/text.go
+ go install -v
./deps.sh: line 8: go: command not found
+ printf '\n### Getting vendor libraries\n'

### Getting vendor libraries
+ govendor sync -v
./deps.sh: line 12: govendor: command not found
+ printf '\n### Getting dependencies\n'

### Getting dependencies
+ sudo apt-get install -y capnproto libpam0g-dev
./deps.sh: line 15: sudo: command not found
+ command -v capnpc-go
+ cd vendor/zombiezen.com/go/capnproto2/capnpc-go
./deps.sh: line 17: cd: vendor/zombiezen.com/go/capnproto2/capnpc-go: No such file or directory
+ printf '\n### Compiling capnp ###\n'

### Compiling capnp ###
+ cp vendor/zombiezen.com/go/capnproto2/std/go.capnp vendor/github.com/scionproto/scion/proto/go.capnp
cp: cannot stat 'vendor/zombiezen.com/go/capnproto2/std/go.capnp': No such file or directory
+ cd vendor/github.com/scionproto/scion/go/proto
./deps.sh: line 22: cd: vendor/github.com/scionproto/scion/go/proto: No such file or directory

In this scenario we should have failed during the Getting vendor libraries step, as without govendor the next steps will not succeed anyway.

We should investigate whether simply adding set -e is safe and does not brake the flow.

Following the instructions in the Readme leads to the following errors.

alessio@alessio-System-Product-Name:~/go/src/github.com/netsec-ethz/scion-apps/bat$ govendor sync
alessio@alessio-System-Product-Name:~/go/src/github.com/netsec-ethz/scion-apps/bat$ govendor add +e
alessio@alessio-System-Product-Name:~/go/src/github.com/netsec-ethz/scion-apps/bat$ go install
vendor/github.com/lucas-clemente/quic-go/h2quic/client.go:18:2: use of internal package not allowed
vendor/github.com/lucas-clemente/quic-go/h2quic/client.go:19:2: use of internal package not allowed

For this we need the following commit from scionproto/scion: scionproto/scion@3f6ddc1

Library functions are going to be used by many SCION apps. We need to add tests to make sure they work as intended.

It seems that bwtestclient and server crash because some SCMP messages. Avoid that.
Original driver issue is here: netsec-ethz/scion#18

Currently, bat relies on a remote address / URL path combination. As soon as RAINS is available get rid of the -r flag and use a full URL instead. This first requires shttp library to change. #9

SCION Apps are not available now.

1. mkdir $GOPATH/src/github.com/netsec-ethz does not exist.
2. ./deps.sh govendor sync -v error.

Currently, dispatcher doesn't allow nil local addresses. This is very inconvenient for a tool like bat that exists for quickly sending requests to a web server. As soon as dispatcher allows nil addresses get rid of the '-l' flag and let dispatcher choose address/port. This requires shttp to change as well. #8

In the meantime integrate a mechanism to infer the local address.

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting helloworld.

Create a make rule install so that the binaries are also copied to ~/go/bin .

ubuntu@ubuntu-xenial:~/go/src/github.com/netsec-ethz/scion-apps/webapp$ webapp -a 0.0.0.0 -p 8080 -r .
Output: webapp: command not found

Instead of accepting new connections, I get this in the log of the listener:

2019-06-13 10:00:21.911122+0200 [INFO] New QUIC connection
2019-06-13 10:00:21.911199+0200 [DBUG] Received extra byte connection="&{sess:0xc42034a200 stream:0xc420222a80}" extraByte=[88]
2019-06-13 10:00:21.911276+0200 [INFO] Closing new connection as there's already a connection conn="&{sess:0xc42034a200 stream:0xc420222a80}"

Listener was started with netcat -vv -b -k -l 12345, client with netcat -b 17-ffaa:1:a,[127.0.0.1] 12345

@FR4NK-W mentioned awhile back it would be useful to alter the continuous run of bwtester to automatically switch known paths to obtain and graph bandwidth results over all paths. This would require some visual clue on the results to note which path was used for any given point on the graph. This might simply be done with a number noting the path number and hops, rather than the current mbps. So path 2, 4 hops might show "2 (4)" over the data point.

This would provide a quick way to know how paths in general are performing. In the tooltip over the data point, the list of interfaces IA text could be colored to match the ISD in the Paths tab. Better, if there is time, a simple linear version of the Paths graph could be drawn in the tooltip in place of text.

As a developer I'd like to be able to easily trace owner/author/responsible for different parts of the codebase. In order to achieve this, I'd like CODEOWNERS file to exists in the root directory of the repo following the standard defined in

• https://docs.gitlab.com/ee/user/project/code_owners.html
• https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners

A very easy to digest example is available at https://tech.people-doc.com/using-github-codeowners-file.html

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting sensorapp.

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting bat. It depends on issue #29

When I switched to Local topology, make/make install giving following error:

cd sensorapp/sensorserver/ && go build
# github.com/netsec-ethz/scion-apps/scion-apps/sensorapp/sensorserver
./sensorserver.go:118:15: cannot assign *"github.com/netsec-ethz/scion-apps/vendor/github.com/scionproto/scion/go/lib/snet".Addr to server (type *"github.com/netsec-ethz/scion-apps/scion-apps/vendor/github.com/scionproto/scion/go/lib/snet".Addr) in multiple assignment
Makefile:24: recipe for target 'sensorapp/sensorserver/sensorserver' failed
make: *** [sensorapp/sensorserver/sensorserver] Error 2

I've not investigated yet, but this is the bash result:

Test duration: 20852ms.
IA found: 18-ffaa_1_120
Log: ~/go/src/github.com/scionproto/scion/logs/bs18-ffaa_1_120-1.DEBUG
Seeking regex: Successfully verified PCB
Timeout: No PCBs verified in the last 10 seconds.
date: invalid date ‘Binary file /home/ubuntu/go/src/’

In fact the PCB verification are being written to that log correctly and the test is issuing a false negative.

Noted in #81:

When apps are missing (it seems we don't install them automatically anymore) there is no error shown on the frontend when a continuous test is started. For a single run, there is an error message.

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting all libraries in this repository to the next version of SCIONLab

• pathutil
• scionutil
• shttp

SSH cannot be run on a local topology since it does not take a local address as argument.
Users should be able to provide the bind address using a -c flag like in other apps.

Please connect this repo at https://reviewable.io/repositories so that all PRs get linked to reviews. Thanks!

Currently the imagefetcher app assumes existence of the $SC env variable pointing to the directory containing /gen/ia. This means it must always be run from the context of scion user, what in the scope of automated infrastructure monitoring is not desired.

If access to /gen/ia is required, the app should be able to consume it e.g. as a command line parameter, but not assume the $SC env variable will be present.

A few more things should be done to allow webapp to run as a long-running web server:

• Add export of logging to log file for later debugging.
• Replace panic and fatal logging code to allow the web server to keep running.

Currently https://github.com/netsec-ethz/scion-apps/blob/master/deps.sh#L15 contains an explicit call to sudo which makes it impossible to execute the script in a sudoless environments, i.e. inside a docker.

This is a similar issue to scionproto/scion#3065, i.e. this should be either explicitly stated in the docs if the environment has to be very specific (OS version, non-root user, whatever else) or the scripts should try to cover as generic use as possible.

Currently neither this nor scionproto docs say it's required to be non-root in a non-dockerized setup what is nowadays a more and more popular configuration.

Currently SCIONLab users who are trying to resolve issues or attempt to educate themselves on the operation of SCION may hunt through logs for information. The Health Check and troubleshooting guide proactively provide insight to common configuration issues, however it would be great to provide users a way to navigate unexpected errors and generally monitor healthy operation.

The webapp could provide a Logs tool to graphically and textually show logs from all services, routers, daemons, and dispatcher to:

• highlight warnings/errors/critical errors and stack traces
• allow users to select a highlighted issue and show the location in the logfile for greater context
• provide an index of running services
• monitor general operation of the AS
• display errors graphically by time as well as linearly.

Questions to answer:

• Are there other open source tools that do the above, or should be write it?
• Does this tool already exist for SCIONLab users in the form of logdog?

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting webapp.

When fetching and saving a file from a web server using a console I/O redirection bat must make sure to disable all SCION logging. Currently, SCION output ends up in the saved file.

Maybe we can disable logging of SCION altogether for bat?

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting roughtime.

We initially intended to create integration tests for netcat, however due to issues with CI I've postponed it for now. My current (not working) code can be found here

Currently the imagefetcher app will save the output image in the directory where it's run from with the name returned by the server. This means from the user perspective the output filename is never deterministic.

In the scope of automated infrastructure monitoring it must be strictly possible to set an arbitrary output filename where the image will be saved, similarly to e.g. wget -O

Setup Description

We have a server and a client as in the examples with the only difference that the client will regularly (e.g. every minute) do requests to the server instead of just once.

Problem description

If the request period is higher than 29-30 seconds, then the server will close the connection because QUIC default idle timeout is 30 seconds. But on the client side this won't be noticed and will thus result in the client attempting to reuse the closed connection and then returning NetworkIdleTimeout: No recent network activity.

Suggestions

I see two possibilities:

1. Allow specifying a different IdleTimeout in the server (https://godoc.org/github.com/lucas-clemente/quic-go)
2. Improve connection management in the client so that if the server closes the connection a new one will be established

when using scion-bwtestclient without -c option, bwtestclient tries to access the scion config from the old location:

CRIT[10-27|14:56:47] Fatal error. Exiting. err="open /home/scion/go/src/github.com/scionproto/scion/gen/ia: no such file or directory"

Bwtesterclient will output paths in the console. Webapp translates it's GUI paths to the same console format to choose a path in console interactive mode. The colorizing of paths in bwtesterclient is being interpreted by webapp to provide 0 path matches.

We need to modify webapp to handle this.

When running govendor list +o on current master, it returns some missing libraries:

  m github.com/jtolds/gls                            
  m github.com/smartystreets/assertions              
  m golang.org/x/image/font                          
  m golang.org/x/image/font/basicfont                
  m golang.org/x/image/math/fixed                    
  m roughtime.googlesource.com/go/client/monotime    
  m roughtime.googlesource.com/go/config             
  m roughtime.googlesource.com/go/protocol

Some of them come from our roughtime app, but I don't know if all of them do.
Find out why we have these missing libraries and modify consequently our vendor.json

The SCIONLab update is coming. We need to port all existing SCION applications so they build and run with the next version.
Typically you will need to register and get a user AS from the testing coordinator at
http://http://scionlabtestbed.inf.ethz.ch:8080
And use it to run and test the application you are trying to port.
Another possibility is, of course, to install the next version in your local machine. Just check out the scionlab_nextversion branch from our repository, and build it following the standard procedure.

This issue deals with porting bwtester.

The standard Golang http client resolves IP addresses in the request's host field (with the form host:port), it would be nice if shttp could support this as well. E.g. it should be possible to make a request to https://17-ffaa:0:1102,[192.33.93.166]:33333/foo/bar.
Then it would be possible resolve http requests without using domain names but just the SCION address, partially solving #10 before RAINS is introduced and simplifying the implementation of 2SMS, where domain names are not used.