"small changes" in b06fc5f disabled the commands to initialize and purge the ElasticSearch index, so the steps

bin/rubberband-ctl create-index
bin/rubberband-ctl populate-index

from the README.md do not work anymore.

What was the reason to remove these (@fschloesser)? And should they be added again?

The search view is restricted to 100 entries. It has now happened to me that I wanted to compare two testruns with different uploader, settings name, test set, and upload date, A and B. I can search and find A easily and select it. However, it is impossible now to search for B without losing the selection A.

I can imagine 3 different ways to cope with this, the easiest of which I would call a "shopping cart".

1. Shopping cart: Like in online shopping, create a feature that saves selected runs permanently during repeated queries. I can select A, add it to my shopping cart, and continue searching for B, select B as well, and finally compare the two.

2. Allow filter disjunction. Extend the filter functionality of the search view. Below every filter, add a plus button that can be clicked to select a second, third, and so on option. I could select "A" as setting, then click the + button to also select another setting "B".

3. Allow tag assignment independently from uploading. I could tag A and B with the same tag, and then search for the tag. This option is, of course, less cool and functional than the two others.

constants.py defines a location for a solu database:

Evaluation will fail if this is not available. The README.md does not mention that this needs to be provided and what the database schema is.

In addition, the location should be customizable via the configuration file (app.cfg).
Ideally, a remote location, or a webrequest, can be specified that Rubberband can query to get the latest version of the database.

A suggestion for a feature: Taking several variants (settings) run on the same testset, generate a performance diagram.

This could be the Dolan and Moré performance diagram, but they depend heavily on the best variant. So I would suggest to generate a "number/percentage of instances solved" (y-axis) vs. "time" (x-axis) diagram, i.e., the y-axis displays for every point in time the number/percentage of instances solved up to this time.

The figures could be generated using own code or gnuplot or tikz/pgfplots. The latter option would allow to include the figures into papers in a nice way, but one would need to also store the results in the file.

Right now, you have to look into the log file to find out the git hash of the SoPlex version used for a run. It would be nice if this would be somewhere at the top of the meta data.

Corresponding ipet issue: GregorCH/ipet#57

Now that we have many queues that we are using, it would be nice if we could filter the search results by the queue.

Currently the search table misses columns showing the solver name and the solver version and the LP solver version. Maybe it makes sense to display this all in one column e.g. "SCIP 4.0.0.2 (SoPlex 3.0.0.1)", but it must work generically, also for other solvers in the future that have empty LP solver column.

Summary -> Details
Statistics -> Summary (displayed first, left most among tabs)

Marc suggested to establish a legend in rubberband that explains the columns of the ipet table, especially, but not exclusively

• dual fail, primal fail, abort
• limit
• unknown
• Q, p
• how do counts work in the initial columns, especially when aggregating
• a description on which evaluation was picked

I am not sure how to best implement this in a generic way. Maybe entries of the ipet xml files can obtain a description tag that is then rendered and displayed?

Probably best to brainstorm offline.

@GregorCH

When sorting the results of a search by one of the columns that represent a date, e.g., "Date added", the sorting is incorrect. It sorts first by day, then month (alphabetically), then year, but this should be the other way around.

Several endpoints remain untested or under-tested. Write some unit tests.

We always improve IPET's output and warning messages. It would be nice to provide a view of IPET's evaluation warning and error output in a dedicated window in the Summary tab. Then, we can more easily spot problems if one wants to be informed if there were any problems such as a nonexisting entry of an instance in the data base, or these annoying warning messages from numpy.

Transfer of issue https://git.zib.de/optimization/rubberband-config/-/issues/48, created by @jakobwitzig:

Sharing a customized evaluation with other developers would nice. I think it would be enough to upload an evalfile for the current evaluation which gets deleted afterward, e.g., after 24h.

Reply from @fschloesser:

The way evalfiles are implemented currently does not allow this easily. There are some things to consider before this can be implemented.

IP_0s_1s and MMM should not be necessary, short should suffice for the tests, right? Maybe even one is enough?

it would be nice that all evalfiles are automatically loaded into the dropdown menu

• all numeric columns display a left-aligned string "--" for missing values which looks weird.
• An infinite gap is not displayed consistently in the Gap column if the infinity is different from 1e+20, probably a hard-coded value somewhere
• sometimes tiny differences in the primal bounds/dual bounds do not have a visible coloring. Maybe use some minimum redness/greenness to visibly distinguish instances where nothing changed from those where it did.

When reimporting a testrun, currently upload time and uploader change to now and the person who performs the reimport. This should not happen: it can make testruns hard to find if you don't know about the reimport.

Remove filename, add meta columns: test set, settings, debug/opt, number of seeds.

This involves extending the database and the upload functionality. The meta file should be optional.

The statistics page should have a bulleted list of different pre-defined ipet evaluations (xml files).

In a second step, we should be able to add a custom ipet evaluation (or a subset of customizations) with a unique URL that can be shared. For this, open a new issue once this one is closed.

This is the quickest proxy to see whether to runs where run on the same hardware.

PS: I like that the search table columns are now automatically displayed in the details of a testrun.

This should remain on the Rubberband side for a predefined set of columns.

It would be awesome to quickly visualize the table data. A simple start would be the following:

The user selects a column of the table by name from a drop down menu. The values of this column are then sorted and displayed as bars in two plots, with and without logarithmic y axis.

I spend some days to get an ElasticSearch 8.15 up and running and to start migrating data from an ElasticSearch 2.x server.
But now I noticed that the README here says that Rubberband only works with ElasticSearch 2.x. (yes, I should have seen that earlier).

I see that there is a branch upgrade-elasticsearch that hasn't been received updates for 6 years. @fschloesser Do you remember what the state of this is? What is the difficulty in using a more recent ElasticSearch?

constants.py defines a number of iPET evaluation files, to be expected in files/static:

Some of these are in staticfiles/static in this repository, but all of them are only found in directory static of ZIB-internal rubberband-config repo.

eval files should not be duplicated and the public repo should not list private eval files.
Preferably, only a directory with eval files is specified in constants.py (or app.cfg), and Rubberband just uses all it finds there (instead of having to give an explicit list).

Three bullet options: aggregated over all seeds; expanded; aggregated plus expanded below

It is currently a tedious process to download the out,err,meta,set-files that a colleague uploaded to rubberband. I suggest to extend the download functionality by a "tbz2" function (next to plaintext, gzip).
If selected, rubberband should

1. group all files that belong to the current comparison, i.e., all out/err/meta/set files for all involved test runs, from elasticsearch.
2. create a single, temporary tbz2 (bzip beats gzip) archive for download
3. delete the archive after the download has been completed.

From https://git.zib.de/optimization/rubberband-config/-/issues/60, created by @jakobwitzig:

I would expect that the choosen default (i.e., the baseline) is printed left whereas the comparison run is printed right. However, this does not seem to be consistent.

In rubberband/rubberband/templates/testset_view.html, the computation of the gap is wrongly simplified.

This would be super useful, only we need to ensure that the groups can be sorted as in the XML schema.

A quick workaround would be to simply number them explicitly 1. all, 2. clean, ... in the XMLs, but maybe there is a better way.

Anyway, let's look at it together on your local development version before it goes online.

From https://git.zib.de/optimization/rubberband-config/-/issues/65, created by @fserra:

It would be nice if one could choose which columns to display in rubberband with a more user friendly method than the toggle. Also for the aggregated table.

Clicking the "view meta file" on

https://rubberband.zib.de/result/AV5DWPc9C3Ahm97cMn_Y

gives a 500 error.

Transfer of issue https://git.zib.de/optimization/rubberband-config/-/issues/47, created by @jakobwitzig:

It would be very nice if some features, e.g., number of (binary / integer / cont) variables or number nonzero objective coefficients, are added to the "long table". This might be related to https://git.zib.de/optimization/rubberband-config/-/issues/46, because those columns could be hidden by default and activated by the user.

From https://git.zib.de/optimization/rubberband-config/-/issues/64, created by @fserra:

In the usual case of detailed view, instances are not aggregated and so the column maxT is redundant. Could one identify when instances are aggregated and remove those columns when this doesn't happen?

If one has specified a GitLab token, but then there is an authentication error (e.g., because the token expired), then this leads to some uncaught exception:

01-08-2024 09:41:46 - 741 ERROR tornado.application Uncaught exception GET / (127.0.0.1)
HTTPServerRequest(protocol='http', host='rubberband.zib.de:443', method='GET', uri='/', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/tornado/web.py", line 1569, in _execute
    result = self.prepare()
  File "/opt/rubberband/package-20170825173100/rubberband/handlers/fe/base.py", line 27, in prepare
    self.current_user = self.get_current_user()
  File "/opt/rubberband/package-20170825173100/rubberband/handlers/fe/base.py", line 84, in get_current_user
    self.access_level = get_user_access_level(email)
  File "/opt/rubberband/package-20170825173100/rubberband/utils/gitlab.py", line 48, in get_user_access_level
    group_users = client.groups.get(group_id).members.list(query=user_mail)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/exceptions.py", line 251, in wrapped_f
    return f(*args, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/mixins.py", line 49, in get
    server_data = self.gitlab.http_get(path, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/__init__.py", line 532, in http_get
    streamed=streamed, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/__init__.py", line 506, in http_request
    response_body=result.content)
gitlab.exceptions.GitlabAuthenticationError: 401: invalid_token
01-08-2024 09:41:46 - 815 ERROR tornado.application Uncaught exception in write_error
Traceback (most recent call last):
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/tornado/web.py", line 1569, in _execute
    result = self.prepare()
  File "/opt/rubberband/package-20170825173100/rubberband/handlers/fe/base.py", line 27, in prepare
    self.current_user = self.get_current_user()
  File "/opt/rubberband/package-20170825173100/rubberband/handlers/fe/base.py", line 84, in get_current_user
    self.access_level = get_user_access_level(email)
  File "/opt/rubberband/package-20170825173100/rubberband/utils/gitlab.py", line 48, in get_user_access_level
    group_users = client.groups.get(group_id).members.list(query=user_mail)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/exceptions.py", line 251, in wrapped_f
    return f(*args, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/mixins.py", line 49, in get
    server_data = self.gitlab.http_get(path, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/__init__.py", line 532, in http_get
    streamed=streamed, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/__init__.py", line 506, in http_request
    response_body=result.content)
gitlab.exceptions.GitlabAuthenticationError: 401: invalid_token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/tornado/web.py", line 1112, in send_error
    self.write_error(status_code, **kwargs)
  File "/opt/rubberband/package-20170825173100/rubberband/handlers/fe/base.py", line 162, in write_error
    self.render("error.html", status_code=status_code, page_title=reason, msg=log_message)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/tornado/web.py", line 766, in render
    html = self.render_string(template_name, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/tornado/web.py", line 905, in render_string
    namespace = self.get_template_namespace()
  File "/opt/rubberband/package-20170825173100/rubberband/handlers/fe/base.py", line 175, in get_template_namespace
    namespace = super(BaseHandler, self).get_template_namespace()
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/tornado/web.py", line 921, in get_template_namespace
    current_user=self.current_user,
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/tornado/web.py", line 1233, in current_user
    self._current_user = self.get_current_user()
  File "/opt/rubberband/package-20170825173100/rubberband/handlers/fe/base.py", line 84, in get_current_user
    self.access_level = get_user_access_level(email)
  File "/opt/rubberband/package-20170825173100/rubberband/utils/gitlab.py", line 48, in get_user_access_level
    group_users = client.groups.get(group_id).members.list(query=user_mail)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/exceptions.py", line 251, in wrapped_f
    return f(*args, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/mixins.py", line 49, in get
    server_data = self.gitlab.http_get(path, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/__init__.py", line 532, in http_get
    streamed=streamed, **kwargs)
  File "/opt/rubberband/production/venv/lib/python3.6/site-packages/gitlab/__init__.py", line 506, in http_request
    response_body=result.content)
gitlab.exceptions.GitlabAuthenticationError: 401: invalid_token
01-08-2024 09:41:46 - 815 ERROR tornado.access  500 GET / (127.0.0.1) 121.41ms

It would be really nice if the exception were caught and Rubberband would display some useful warning or error message instead of a 500 - internal server error.

These are not high priority, but I wanted to create an issue to collect design changes that would enhance the user experience. General:

• A monospaced font: although this sounds very oldschool, I am convinced it improves readability of tables; there are some well-designed monospace fonts out there, e.g. https://fonts.google.com/specimen/Fira+Mono, which could even be good for all text.
• Integrate the menu on the left (Search, ...) into the top bar in order to maximize usage of screen width.
• Remove horizontally separating lines from tables (not 100% sure, let's try).
• Rename the header "Result" to "Evaluate".
• Update the bug report link in FAQ.
• Update or remove the "500" explanation under FAQ.
• Rename FAQ to Help.

In the details table

• Replace the harsh red and green by something "nicer".
• Move the search field to the left.
• Move the "How do I interpret the comparison view?" text as an explanation below the detailed table.

In the Ipet table:

• Right align numbers.
• Remove vertically separating lines within one group.
• Add a background color to columns abort, ..., unkn and the Q columns
• Print instance numbers (in the first columns) as integers, without .0.
• Format time, node, Q values, and p values with less precision; 3 digits should suffice, for time and nodes even 1.
• Remove or shorten the "_shmean".

• Like in the details tab, a search/filter box would be useful.
• Can we create an option (toggle with a check box, or new tab?) such that only the parameters with different values are displayed?
• A similar table design as the other tables would be good.

From @tkralphs

Hey Cristina,

First of all, yay, it's running on my server at Lehigh :). I just looked over the whole thing quickly now and below are the issues I found so far. Keep in mind that we try to do these reviews as "dumb users." I was able to overcome all of these issues, but it would still be nice to improve the documentation if it can be done without much effort.

1. For the section on installing elastic search, the instructions are specific to Ubuntu and probably won't work for most Debian or Fedora/CentOS installations. You don't necessarily need to have explicit instructions for those distros, but it would be good to indicate that what is needed is just to get java 8 installed and that instructions for that can be found by Googling. I was able to get instructions for installing java 8 on Debian pretty easily and that more or less enough to make it through getting elastic search installed.

2. There seems to be a dependency on "click". I had to pip install "click" to get rubberband to run.

3. You can't run "rubberband-ctl" before installing rubberband :), so the

source venv/bin/activate
pip install -e .

has to come before populating the database.

4. The ipet dependency will fail for most users, since it's not on github. Is that necessary? I guess not, since everything seems to be OK without it. Some explanation of what role ipet plays in all of this and what can't be done without it would be good. Of course, ipet will hopfully be available publicly before long.

5. For seeding the database, it might be nice to include some expected output or some indication of what should happen.

6. Same for tests.

After getting past the tests, I really didn't have much idea what to do next, so I just ran

python server.py

and then was actually able to see something at

coral.ie.lehigh.edu:8888

Yay! You won't be able to do the same because the port is only open through our VPN (and I'm going to kill the server shortly). It seems as though there should be a bit more on actually starting the server and maybe some screenshots of what one should see.

More later when I start trying to upload things, etc.

Ted

Improve documentation

requirements.txt and requirements-dev.txt defines explicit versions of Python package dependencies. For iPET, some version from end of 2019 is specified.
With these dependencies, I can get Rubberband to work with Python 3.6 (EOL many years ago), but not a current 3.12.
There are also several PR from dependabot that suggest updating dependencies in order to fix security issues.

Thus, one should see that dependencies get updated. Note, that IPET is very strict on the versions of its dependencies (https://github.com/GregorCH/ipet/blob/master/requirements.txt), with most of them also being a dependency of Rubberband.