Perhaps there should be a “Revert Changes” button in addition to the Save Changes button in case a user messes things up and wants to simply reload the latest model from the server.

Score weights are rendered in General Parameters, we can leave them there or move them to the Weight section.

If the score names that are defined in the configuration note.

To know if the score represents a bid, we could try to search the word 'bid' in the score name. The same happens for recommendation.

I think it is related to the way that we refresh the table. When the note changes the status from 'pending' to 'processing'.

The table always shows 'pending' and after a second it shows 'processing'.

The only way to fix it is refreshing the browser.

I suggest renaming the button to “Save Changes” and only displaying the button when there are changes to the model.

Justification:
As the user edits the draft with veto and lock, it isn’t obvious that these edits are not actually taking place until a button is clicked. The button has a rather ugly tooltip that comes up if you attempt to click it when no changes have been made.

The name of the button (“Update Constraints”) assumes that the user conceives of the vetoing/locking as adding constraints which I believe is somewhat abstruse.

think a little bit about how the preprocessor class is going to work. right now it's incomplete

Only the stats page is showing the breadcrumb menu to go to the conference home page.

Do we want to show it?
The browser is not showing it, we should unify the criteria.

We can leave it and go to 'All assignments' instead.

@zbialecki thoughts?

Given that the schema of the configuration note is more complex, we would need to build an specific UI to enter the following values:

Other option is to try to make it work with the current mkNotePanel that we have now.

• General parameters can be strings.
• Weights can be predefined score fields also saving strings.
• custom maximum loads: could be a list of comma separated values?

We can set the invitation.reply to display these fields.

@zbialecki thoughts?

Currently the location of the green/red user boxes depends on the state of update of the model.

Before “update constraints” button is clicked:
Vetoed primary users are displayed in the alternates list as red.
Locked alternates are displayed in the primary list as green

After the “update constraints” button runs (successfully without a warning)
Vetoed primaries are shown in the primary list
Locked alternates are shown in the alternates list

A couple problems with this:
What’s especially confusing about this is that the UI is actually displaying the REVERSE of what is currently true of the model as it exists in the server side.
As the user edits and clicks “update”, there can be confusing combinations of green and red boxes in the different lists where non-updated ones are in different list than ones that have been persisted.

Would there be a problem in keeping the display of locked/vetoed reviewers as they are in the BEFORE-update-button state regardless of whether update has occurred?

Because of the make configuration button on the page it shows an error if there is no invitation for the config. Make the page do something nicer than an error + button when there is no invitation.

client is: 19 Dec 2018, 09:13:00

server is: Dec 19 2018, 09:13:00

I guess we need to fix the helpers.

I would suggest to create a new configuration note with different assignment notes and DON'T REMOVE any data.

From the UI perspective:

• When the user clicks on 'Run solver' from the matching browser it will appear a popup to give a new name to the configuration note.

in the config invitation, the scores_names and scores_weights fields are set to fixed values (using the "values" key instead of "values-regex"). This results in the bug you see above.

I can't think of any reason why we can't or shouldn't let this be a fixed value.

Program chairs don't have permissions to delete notes from the database.

The matcher should delete the notes using ddate.

It seems the solver is using the max_paper property to assign the papers to the first users and the result is unbalanced.

Is it possible to distribute the load to all the users?

I had a config-note that had values in its content fields that were not compatible with the invitation reply specs (I had created the config-note prior to creating the specs on the invitation). When the matcher tried to set the status of the config-note OR rejected the POST request and the matcher died. I wasn't able to see an indication that the run was in a failed state though because the note wasn't alterred and remained with status initialized.

Given they are displayed in paper forum, we should ensure they have a title.

That means that the matcher is running and the results are not ready to be displayed.

Similar to stats. Apply the constraints to all the visualizations.

Otherwise it removes all the assignments of all the existent configuration notes.

I would like to don't remove data and don't save the assignments below the same invitation. Use another invitation? or create a forum note to represent the config note and then all the assignment notes should point to this forum note.

Posting this on behalf of @marshall62

Would like to be able to sort list of papers by:

• alphabetically by title
• number of assigned reviewers
• min reviewer score
• average reviewer score
• other criteria?

Would like to be able to sort reviewers by:

• alphabetically by user name
• user load (number papers they are assigned to)

It should show a bar with amount of users with no papers assigned:

Left chart should show a bar in the x = 0 and y = # of papers with no assignments.
Right chart should show a bar in the x = 0 and y = # of users with no assignments.

What I mean by permission-constrained browsing: browsing assignment drafts when the user does not necessarily have permission to view all of the paper matching data.

This issue is motivated by a use-case that ICML would like us to support: many ACs using the interface to browse their assignments simultaneously, while preventing information leak and excess collisions. In this scenario:

1. ACs should be able to see information about their assigned papers and to suggest changes on their assignments. They should not be able to see information about other papers, or ideally even see other assignments.
2. ACs should be able to see or infer the current global reviewing load for a particular reviewer, even under the constraints of point 1.
3. Many ACs (i.e. ideally on the order of 100's) should be able to use the system at the same time.

This is what I had in mind:

• Create references for every submitted paper that "masks" most (all?) of the paper note's fields. The mask will be visible by the entire pool of area chairs, but the underlying paper note (the "original") will have its permissions set to include only the assigned AC. (This construct is similar to the way we implement anonymous submissions).
• Modify the paper entries in the assignment browser to show the paper's information if the underlying original note is visible to the user, and if not, show the mask.
  • or alternatively: if the underlying original note is not visible, show a placeholder that doesn't reveal any information about the paper at all -- not even forum IDs.
  • The idea is to prevent any kind of inference about papers that are not assigned to the user -- all they can see is that a reviewer is assigned to some unidentifiable paper that is not their own.
• If the mask is shown, disable the lock/veto buttons on that paper to the user.
• Disallow clicking through masked papers entirely (the idea being that users shouldn't be able to see the assigned reviewers of a masked paper by clicking on the paper)

The above should achieve the three objectives in the following ways:

1. Hides information from users based on permissions of notes
2. Users can estimate reviewer load when suggesting reassignments (by clicking on the reviewer and counting the number of paper entries; some of these entries will have paper information, some will be "masks")
3. Disallowing users from adding suggestions (locks/vetoes) to masked papers should reduce the number of collisions, because constraints are keyed on forum ID in the configuration note's conflicts field (and if each user is assigned a non-overlapping set of papers, there shouldn't be collisions)

@zbialecki @melisabok @marshall62 please try to poke holes in this and/or ask clarifying questions if you have them!

Using the editor to change the status of a config back to Initialized is handy but if the note contained things like conflicts or custom-loads in its content, they may be removed when the dialog writes the config note.

Curious what @melisabok @zbialecki @marshall62 think of changing lock --> upvote up arrow and veto --> downvote down arrow. (Created a new issue label "feature proposal")

Advantages:

• The upvote/downvote mental model is both more general and more common for users than lock/veto Up/down arrows are more general than lock/veto. I would argue that they're more likely to understand that their selection is a suggestion or request, and not necessarily guaranteed, which is useful in a multi-user scenario.
• It still supports the original case in which a handful of program chairs want to apply hard constraints. We can modify the impact of upvotes/downvotes up/down arrows in the matching system itself so that they are treated like hard constraints, when necessary.
• Upvote/downvote is more flexible Up/down arrows open up potential for interesting things in the future (e.g. what if people could upvote/downvote a paper-person pair multiple times? what if they could rank assignments? etc.)

Disadvantages?

EDIT: Sorry if I confused the issue by talking too much about voting. To be clear, what I'm proposing here is making the symbols more general so that it can support a wider variety of uses.

It is not clear what is the value that represent each bar.

It seems that the bar should be in the middle of each tick.

To create this issue:
I worked with Michaels branch that demonstrates how the matcher UI could mask papers from users who don't have read access (which helps reveal part of this issue)

1. I logged in as an AC and locked Dirk W. to a paper and vetoed Sebastian P. from that paper and saved the constraints.

2. Logout and login as a different AC

3. I locked Dirk W to a paper ( a different paper than in step 1) and clicked "Update Constraints"

The dialog is showing the complete list of constraints and says that all of these will be saved.

There's a few issues with this:

The dialog doesn't distinguish those constraints that are unsaved (i.e. the ones I just added) from those that are already saved. The last row is only one that is newly added.

Is it really necessary to show all the constraints in this dialog when I click "update"? In this case, there are constraints saved by other ACs that the current AC can do nothing about because they don't have access to the paper in which those constraints apply. Also, as the list of constraints gets really long, this issue is magnified.

Wouldn't it be better if this dialog only showed the newly added constraints it is about to save?

If it needs to show all of them would it be better to show constraints in two groups: Previously Saved, Newly Added

The https://openreview.net/assignments?venue=ICLR.cc/2019/Workshop/DeepGenStruct
has a configuration where the max number of reviews a reviewer can give is too low so that the supply of reviews is less than the demand. An assert catches this error, but the assert exception is not caught and returned to the main UI with a message (Melisa removed the way I was putting messages into the status). We need to create a second field for the error message so that the UI can give details about why things fail.

Currently it is showing 'Error' when we got this error:

matcher.assignment_graph.AssignmentGraph.AssignmentGraphError: the total supply of reviews (55) must be greater than the total demand (99)

It should be 'No Solution' and give us a clear idea that we need to change the parameters.

Failure is being used only when there is no solution. No Solution is more descriptive and there is no need to save an error_message in the configuration note.

Run matching server as a service of the operating system.

This can be related to this closed issue: #27

See image attached, min_paper is 4 but there is one user with 2 papers assigned.

This is happening with ICML workshop: https://openreview.net/assignments/browse?id=H1gG6BKj34

If Flask is stopped, any config notes that were in a running task will be left with status="Running". This will cause future runs for this config to fail because it checks to see if there is a config with status="Running".
So we want one or both of the following:

1. On flask stop, go clean up config-notes that are left in a running state and set them to some other status.
2. On flask start, clean up config notes similarly.

This will mean that the match server needs to keep a list of config_notes that it is processing. Only when a note is complete, can it remove it from the list.

May need to install pm2 in config file that is read on machine startup.

I think it is related to the range of numbers that bids supports.

ICLR is using a range between -1 to 1.

If there are a lot of papers and reviewers, switching back and forth in the view takes awhile because of the number of elements within the page.

This change has to be done in openreview repo but it is related to the matcher:

In the columns 'Created On' and 'Last Modified' show the time of the date:

throw an exception if there is a score in the metadata that doesn't exist in the configuration note

for example: if two papers are very similar, ensure that they're reviewed by the same user.

another example: ensure that two reviewers are assigned to the same papers

minimum papers per user constraints aren't being honored in all cases. I think I understand what the issue is - right now, assigning fewer than the minimum number of papers is highly "costly" but not strictly impossible.

And move to the top the 'All XXX Conference Assingments'

Render always the configuration table even if there are no configuration notes for that venue.

This will fix the issue where the first note is created it should refresh the table too. Now it is still showing "No assignments have been generated yet for this venue." and it doesn't refresh the table because it doesn't exist in the DOM .

Should we do this every time the constraints are saved?

May be we need to implement something in the matching server that gets the configuration notes and replicates the constrains in the assignment notes.

It should show all the content of the configuration note:

The server is not validating the status of the note to run the solver.

If the status == 'Complete' is running the solver again.

Since the papers in the assignment browser are the anonymous copies, the authors are shown as "Anonymous". It would be nice if they would show the original copy authors when the user has permission.

Within Assignment list add an action "clone" that would add a new configuration row that clones this config (set the label to be the same with -clone appended, set its status to initialized)

When a user is vetoed from the primary list, s/he should be placed into the top of the alternates list. Currently s/he is placed at the bottom of the list.