• Create some test VMs on betaweb023
• Install the dockerized TIRA on betaweb023
• Can we use directly all VMs in the dockerized VM?
  • I.e. run the manual tests to verify that the VMs still work inside dockerized TIRA

Revision of installation procedure of the tira-host. Containerization of tira-host facilitates deployment procedure and helps isolate infrastructure dependencies.

1. Webis specific:

• salt state tira-test-env

2. Generic installation:

• install prerequisites
• run tira-host docker container

Right now, I always log into the java-frontend server in kubernetes to run my tira commands.
The java frontend server is a well suited environment, because it can run tira -r out of the box, and also the access and secret key for the disraptor integration are in this environment.

Right now, I have a bash alias, but I would preferr to have a command tira admin-shell that runs a specified command within the java-frontend pod.

We lost a few virtual machines on betaweb042 because the HDD /media/storage1 was broken.
So can we leverage Ceph-RBD to prevent such data loss?

The only purpose of this ticket is to verify, whether:

• Can we mount the disc from RBD into virtualbox: https://docs.ceph.com/en/latest/rbd/libvirt/?
• How would one archive/unarchive this?

In order to be able to switch branches.

I deleted:

ir-lecture-task-1.prototext

taskId:     "ir-lecture-task-1"
taskName:   "IR-Lab Task 1 (WS 19/20)"
taskDescription:   "Retrieval in a focused argument collection to support argumentative conversations."
trainingDataset: "touche-2020-first-subtask-2019-11-23"
virtualMachineId: "ir-lecture-master"
hostId: "ir-lecture"
web: "https://temir.org/teaching/information-retrieval.html"
commandPlaceholder: "mySoftware -i $inputDataset -o $outputDir"
commandDescription: "Commands should take the arguments <code>$inputDataset</code> and <code>$outputDir</code>. The variable <code>$inputDataset</code> points to a directory which contains two files: <code>args-me.json</code> (the corpus), and <code>topics.xml</code> (the topics for which documents should retrieved). Your software should create a standard trec run file in <code>$outputDir/run.txt</code>."

ir-lecture-task-2.prototext

taskId:     "ir-lecture-task-2"
taskName:   "IR-Lab Task 2 (WS 19/20)"
taskDescription:   "Retrieval in a generic web crawl to answer comparative questions with argumentative results."
trainingDataset: "touche-2020-second-subtask-2019-11-23"
virtualMachineId: "ir-lecture-master"
hostId: "ir-lecture"
web: "https://temir.org/teaching/information-retrieval.html"
commandPlaceholder: "mySoftware -i $inputDataset -o $outputDir"
commandDescription: "Commands should take the arguments <code>$inputDataset</code> and <code>$outputDir</code>. The variable <code>$inputDataset</code> points to a directory that contains a <code>topics.xml</code> file that contains the topics for which documents should be retrieved. Your software should create a standard trec run file in <code>$outputDir/run.txt</code>."

The button in the above picture does not work properly right now.
Clicking it once effectively reloads the same website. Clicking it another time then works as expected.

The function of going back is implemented with an onclick event issuing history.back().
Most likely there is an issue with how Disraptor (resp. Ember) loads the website and how history.back() works in combination with that.

The HTML regarding that button looks as follows:

<a href="#" onclick="history.back();" class="btn btn-default pull-left tira-panel-heading-back-button" title="Go back">
    hit<span class="glyphicon glyphicon-chevron-left"></span>
</a>

• Remove dnsmasq service
• Remove dnsmasq commands from the tira scripts
• IPTables: FORWARD from all TIRA interfaces to lo.
• Test this on betaweb111
  • Creation of new VM (and in old VMS)
  • Existing VMs

This worked once, but is currently not working anymore.

Currently there is no dev environment for the Discourse/Disraptor instance which forces to develop in tiny steps to not crash tira.io.
Also there in no documentation on how to set up one yet.

However, there already are scripts and configuration files for that at tira-discourse-deployment/dev.

These need to be changed slightly:

• support selection of namespace
• sort out database maximal size to not create big volumes for every instance
• support namespace-overlapping access to tiras frontend instances (otherwise these would have to be in the same namespace, which is unfeasible)
• document the usage

• Include everything related to discourse into this repository
• Discourse should run inside webisservices
• tira-web-2 should run inside webisservices
• tira-bg should run inside webisservices

• Tira must run in the webisservices namespace of kubernetes
• Tira must use Ingress

Some files/folders in /mnt/ceph/tira were accidentially deleted on 03.12.2020 at 15:30 (we don't know which files).

• Restore Backup
  • The following command is running in a screen session on betaweb023 (in the directory /mnt/ceph):
    rsync --progress -a tira/.snap/daily_20201201_061503/ tira 2>&1|tee /home/webis/tira-backup.log
• Compare the versions again
  • the following command is running in a screen session on betaweb023:
    time rsync --dry-run --progress -a tira/.snap/daily_20201201_061503/ tira
    Differences:

    model/users/users.prototext
    model/virtual-machines/
    model/virtual-machines/virtual-machines.txt
    

    • diff tira/.snap/daily_20201201_061503/model/users/users.prototext tira/model/users/users.prototext looks good
    • diff tira/.snap/daily_20201201_061503/model/virtual-machines/virtual-machines.txt tira/model/virtual-machines/virtual-machines.txt looks good
• Compare older version of the backup (maybe one week before) to be 100% sure we have not lost anything.
  - Command is running on betaweb023: time rsync --dry-run --progress -a tira/.snap/daily_20201129_061507/ tira
  Differences:
  backup/ backup/touche-task1-virtual-machines/ log/errors/ log/virtual-machines/virtual-machine-log.txt model/softwares/celebrity-profiling/ model/users/ model/users/users.prototext model/virtual-machines/ model/virtual-machines/virtual-machines.txt
  - I reviewed all the changes to this previous snapshot manually, everything is fine.

Some TODOs as discussed via chat:

• Move everything into a single table

• Add a new column for host organization

• Add the search box as per the script found here: https://github.com/webis-de/webis-de-assets/blob/master/js/filter.js and as shown here: https://pan.webis.de/shared-tasks.html

• Add a heading: "Shared Tasks" (just like on the aforementioned page.

• Add a column with editions of a shared task, like to what is found on the PAN shared tasks page.
  (This will likely require changing the Proto-Message so as to include an optional list of links for each task, which may be a more intricate change; if it turns out to be more complicated, rather fix the above and make this into a new issue).

We have the postgres in /mnt/ceph/storage/data-in-production/disraptor/discourse/postgres-data.

This is the Meta-Achievement for the website migration.

• Run - review UI, see #87
• Run - create new review, see #88
• Run - run actions (unblind, delete, download)
• VM - VM Info UI, see #89
• VM - ajax (start, stop, shutdown)
• VM - host grpc actions (vm_info, start, stop, shutdown)
• VM - Frontend grpc actions (start, stop, shutdown) #90
• Software - UI
• Software - ajax action (run, evaluate, abort)
• Software - Frontend grpc actions (run, evaluate, abort)
• Software - host grpc actions (run, evaluate, abort)

• I remember that we had to accept some license agreement during the installation of the Virtualbox Extpack plugin.
• Is there private data in the image?

Issue with running full tira setup locally in docker network mode bridge (as well as in host mode - iptables conficts)

• Save versions of the image in cephFS (using an appropriate naming schema; ToDo: what is a good place for that images?).
• Adapt the salt state to load the image from cephFS.

• Update the documentation to the new disraptor frontend: e.g. create a new admin/reviewer user.
• Standards/Schema regarding groups names etc: https://docs.google.com/document/d/1Jh0tyEOpHou1YiN3P1JLOVCmqPOjY-f17_PHJI_CSn8/edit?usp=sharing
• Recipies for Debugging/etc. of vms in the new dockerized host

1. To motivate users to use fewer resources
2. To show potential runtime problems (swapping a lot etc.)

Change the style of the header to correspond to that of the webis.de and pan.webis.de pages.
The logo can stay; the right-hand side menus should be changed to be readable on black background.

Since the tasks page is now the default landing page, there is actually no need to have it as an additional entry in the header, anymore.

Deploy the old version of tira at legacy.tira.io, so that we have an fallback version of tira ready in case there are problems with the new disraptor frontend.

From the tira channel:

A question: Would Slurm (https://slurm.schedmd.com/) be a suitable alternative for TIRA's backend?
Important would be that Slurm still protects us from malicious users in the sense that it has to properly sandbox and compartmentalize third party code.
So, before praising Slums other virtues, the question about security needs to be settled.

Since we're rebooting betaweb a lot lately, I needed a way for a VM user to restart their VM on their own afterwards (so that I don't have to do it myself). Since we don't want to give tira users access to arbitrary operations on the host, I came up with a simple script (see below) that can be used to allow a tira user to restart their own VM, but not do anything else on the host. It is installed by placing the user's SSH public in the tira user's authorized_keys file, along with a command restriction. When the user logs in with the corresponding key, the script will run automatically, and no other command (or full shell) will be allowed.
It might make sense to integrate this as a tira sub-command (if something like this doesn't already exist and I just missed it).

tira-vm-start-stop.sh

#!/bin/bash
#
# Control Script for tira VM users.
# Allows starting and stopping of the VM passed on the command line, and
# nothing else.
# Put this as the command="<path-to-this-script> <vm-name>" in front of a
# user's public key in /home/tira/.ssh/authorized_keys

VM="$1"


echo "Virtual machine '${VM}'"
vboxmanage showvminfo "${VM}" | grep '^State:'

read -t 10 -n1 -p "Select VM Operation: (1) start (2) stop (3) restart (0) nothing: " OP

echo

case "$OP" in
	"1")
	exec tira vm-start "$VM" 
	;;
	"2")
	exec tira vm-stop "$VM"
	;;
	"3")
	exec bash -c "tira vm-stop '$VM'; tira vm-start '$VM'"
	;;
	"0")
	echo abort.
	;;
	*)
	echo "unknown operation: $OP"
	;;
esac

As a basic proof of concept, implement a script discourse-create-vm-group.sh (I would prefer a bash script, because that is more the standard way, and should be really easy, and this way we can use the constants defined in bash-helper) located at tira-host/src.

The script should get the name of a VM as input, and then:

• Create the group. The group should have a nice description. E.g., given information on the VM, it should create a nice description of the group. E.g.:

  Members of this group have access to the virtual machine XYZ on BETAWEBXYZ.
  
  The password for the virtual machine XYZ was sent to the participants in a separate mail.
  The virtual machine can be accessed via SSH (host XYZ, port XYZ) or RDP (host XYZ, port XYZ) to install the software(s) that participate in the shared task.
  
  You can SSH into the virtual machine with: sshpass -p XYZ ssh [email protected] -p XYZ -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no
  

• Create an invite link, and print that invite link to stdout, so that we can directly send this invite link to participants
• Groups should be complete private

This script can assume that the api-key is stored in the filesystem (we mount this via kubernetes), e.g. at /etc/discourse/client-api-key. I.e., the script can retrieve its API-key by something like API_KEY=$(cat /etc/discourse/client-api-key).

• This image must be private (i.e. not dockerhub).
• This is the replacement of the old salt-state that runs the docker build
• Stored in /mnt/ceph/tira/data/tira-dockerized-host-images/tira-io-tira-host-VERSION.tar (append only, never overwrite old images)

Probably a change in the output of tira vm-info.

Currently, we list multiple task instances (i.e. annual editions) under the same task label.

• This way, we have no strict association between datasets that belong together.
• There is not strict association between task and year.

Detangle tasks by creating a new task for each instance and change the dataset associations.

vboxmanage modifyvm [vmname] --graphicscontroller vmsvga

Currently a user is certified to use a VM and access runs etc. via the username.
With Discourse integration this allows users to sign up with a username that is assigned to a machine already, if there is no Discourse user yet.
To have consistent behavior this shall be changed to use Discourse-groups instead.

Idea is to create groups for VMs. With this it's also possible to assign one VM to a group of people. Authentication is used as before but a check for membership of the group, allowed to access the machine, is made instead of username checks.

This will resolve the earlier described issue, since Discourse-groups are assigned by moderators.

At last this may be combined with Discourses feature to have threads only visible to group members.

Collect checks for user permissions and request validity in one module, similarly to old website.
Unify procedure for validity checking within all views, i.e.

• 1. check permissions
• 2. [check validity]
• 3. get model data
• 4. build/return view

Methods that should be accounted for:

• checkPathIsEmpty
• checkPathEndsWithSlash
• checkTaskExists
• checkDisraptorSecretKeyIsCorrect
• checkUserSignedIn
• checkUserOrReviewerSignedIn
• checkReviewerSignedIn
• checkVmRunning
• checkVmNotRunning
• checkVmReady
• checkMasterVmRunning
• checkFormFieldsEmpty
• checkFormFieldsSoftwareSet
• checkFormFieldSoftwareCommandValid - cmd != null && cmd.length() > 0 && !CMD_REGEX.matcher(cmd).find()
• checkFormFieldSoftwareWorkingDirValid - dir != null && dir.length() > 0 && !DIR_REGEX.matcher(dir).find()
• checkFormFieldSoftwareDatasetValid - if dataset exists
• checkFormFieldSoftwareActionValid - action is "add", "save", "delete", "run"
• checkFormFieldsRunSet
• checkSoftwaresExists - check if input software exists
• checkSoftwareExists - check if user owns the requested software
• checkUserIdle - check if the VM is doing something (("STARTING", "RUNNING", "BACKOFF", "STOPPING")
• checkUserBusy - Pretty much the same as above.
• checkRunExists
• checkRunNotDeleted
• checkRunDownloadable
• checkFormFieldsEvaluationSet
• checkFormFieldRunValid
• checkFormFieldRunNotOnTestDataset

Requires an update of the corresponding salt state.

The tira web client pod has been moved to a new namespace; the tira k8s command needs to be updated accordingly.

Remove meta information

• Authors
• To Credits in the readme

A first test with unpoly was promising.

We get again "Oops! That page doesn’t exist or is private." for pages that have long loading times.
Example: https://disraptor.tira.io/task/universal-dependency-learning/dataset/conll17-ud-test-2017-05-09/

Last time, we fixed this by increasing the gateway-timeout of the webis-nginx (i.e.: https://git.webis.de/code-generic/code-saltstack/-/blob/master/src/srv/salt/pillars/nginx/webis-web-proxies/site_disraptor_tira_io.sls).

I changed the communication between disraptor and the Java-backend: they now communicate over the internal kubernetes network, so that the Java-Backend does not have a public-facing interface (since the java backend has no authentication anymore). Maybe there are now new timeouts (kubernetes-service-timeout) that we are not aware of?

Think of ways to filter and optionally delete (or move) old run files based on the filtering criteria.

e.g.: delete all runs that were reviewed with errors.

From the discord chat:

Shouldn't we also have a tira-moderator group or some such?
So that not every moderator has to have access to all VM groups individually?
Might also that we consider creating task-specific moderator groups.
tira-moderator-task-clickbait-detection

Build an Admin Panel, so administrative tasks can be done via the Website instead of running scripts.
Routes:

• /tira-admin (GET)
• /tira-admin/*wildcard (GET and POST)

Functions:

• list active VMs and their state
• archive vm -> separate issue
• add task (does not create master vm)
• add evaluator for dataset
• add dataset
• run integration test
• edit Database -> use django admin

Running make run-tira-docker fails, this worked some months back.

The ubuntu20 image uses docker installed in snap.

During sandboxing, /snap/bin is not part of the PATH, and as a result, docker does not work out-of-the box during sandboxing.

Following questions:

Why are the environment variables different during sandboxing?

Sandboxing:

SHELL=/bin/bash
PWD=/home/ir-lab-uh-t1-luke-skywalker/app LOGNAME=ir-lab-uh-t1-luke-skywalker
XDG_SESSION_TYPE=tty _=/usr/bin/env MOTD_SHOWN=pam
HOME=/home/ir-lab-uh-t1-luke-skywalker
LANG=C.UTF-8
SSH_CONNECTION=141.54.132.42 33278 10.0.5.100 22
XDG_SESSION_CLASS=user TERM=xterm
USER=ir-lab-uh-t1-luke-skywalker
SHLVL=1 XDG_SESSION_ID=5
XDG_RUNTIME_DIR=/run/user/1001
SSH_CLIENT=141.54.132.42 33278 22
LC_ALL=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus
SSH_TTY=/dev/pts/0
OLDPWD=/home/ir-lab-uh-t1-luke-skywalker

During login via ssh:

SHELL=/bin/bash
PWD=/home/ir-lab-uh-t1-luke-skywalker
LOGNAME=ir-lab-uh-t1-luke-skywalker
XDG_SESSION_TYPE=tty
MOTD_SHOWN=pam
HOME=/home/ir-lab-uh-t1-luke-skywalker
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SSH_CONNECTION=141.54.132.159 52202 10.0.5.100 22
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=ir-lab-uh-t1-luke-skywalker
SHLVL=1
XDG_SESSION_ID=7
XDG_RUNTIME_DIR=/run/user/1001
SSH_CLIENT=141.54.132.159 52202 22
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/home/ir-lab-uh-t1-luke-skywalker/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus
SSH_TTY=/dev/pts/0
_=/usr/bin/env

Especially the the differences in PATH and LC_ALL are interesting.

Should we install a new ubuntu20 image?

Should we add some mandatory checks to ensure that a virtual machine image is valid? Which software must be installed? How should we test that it works within sandboxing and without sandboxing?

Since many of the users of Tira have educational accounts they may want to use them to SSO.
Shibboleth provides this possibility.

A kind of guide on how to use Shibboleth for SSO is given here:
https://meta.discourse.org/t/shibboleth-sso-with-discourse/142401

Right now, the list of dockerized_tira_hosts is only kept in the salt state (https://git.webis.de/code-generic/code-saltstack/-/blob/master/src/srv/salt/pillars/tira.sls).

It should also be kept in the model/virtual-machine-hosts.txt

Applying the state should sync those files.