codegrade / bottlenose Goto Github PK

Currently, the user can input ANYTHING into the meeting time text box under Sections Tab, when creating a class. The user should only be allowed to input the time of section, not other types.

This is the current view:

This is what the Meeting Time should look like: (except the Start/End Date will be Start/End Time)

• Assignments live in buckets (e.g. "Problem Sets", "Exams")
• Each bucket contributes a set weight to the final grade.
• Buckets can optionally be clamped, to allow in-bucket-only extra credit.

Currently, there's a FIFO policy for student submissions, which ordinarily is fair. And with 16 background tasks, the delayed job queue tends to be empty or emptying quickly. But consider two scenarios:

• A student who manages, through some bug of the autograders, to submit a homework that causes the grader to take arbitrarily long. And then, panicking that their homework isn't being graded properly, submits again and again...using up the pool of workers

• A student who manages, through some quirk of browser bugs and or weird computer behavior, to submit their homework 3800 times in one second (as happened last year!), thereby stalling the pipeline for everyone else by a very long time.

We need some more refined notion of fairness: if a student submits repeatedly, newer submissions' autograders should take priority over older ones that haven't run yet, and older submissions' autograders should get "niced", so that if another student submits while that niced autograder is queued, the other students take priority. (If it's a team submission, s/student/team/g wherever appropriate.) This should prevent anyone from spamming the system and blocking out other students from getting timely feedback.

(Note that this is a defense-in-depth measure; there should be other steps taken to ensure that no autograder can run for arbitrarily long without timing out with an error. Currently, the Java style checker is vulnerable to this problem and needs some help.)

ccis: blerner commented on Feb 13

The exam grade submission page doesn't work in Edge. Questions assignments don't work in Safari. Probably there are other JavaScript problems to resolve, too.

I should be able to delete more than half the controller code.

It's not quite reproducible, but some graders reported problems entering line comments on homeworks, only to find that that the comment text vanished. Look into the validation of comments, to see whether they're not getting through for some reason. (Maybe their generated ids, client-side, weren't good; maybe too many comments happened in the same timeout and so some were dropped?)

ccis: blerner commented on Mar 15

From a student:

Hi!

I found you a bug in Bottlenose:

I have my machine's time set to display in UTC (long story) and, although the due dates, etc. on the site show up in the correct time zone; on the submission page itself I am warned that I am past the due date because my local time (21:30ish) is past the local time of the due date (21:00) even though the real time is before the due date (2:00 UTC)

The due dates, etc. are all rendered correctly everywhere else.

ccis: blerner commented 16 days ago

Since we don't want to destructively delete anything, we need to add a flag to assignments marking them as "deleted", so they don't appear in assignment listings or in grade computations or in assignment-weight editing. Then we should have a page allowing us to "undelete" an assignment and clear that flag.

This allows us to correct a mistaken assignment creation, even after some grades have been posted.

Follow-up from #42:

• "Points available:
  This is the percentage of the final grade that this assignment is worth. Default: same as previous assignment. This is the percentage of the final grade that this exam is worth."

• Generally tightening up the configuration options so they take up fewer rows

• The first item in the grader configuration shouldn't have a horizontal line above it

• Existing graders should be rendered more consistently with the newly-created ones

• The "Type: files" line at the top of the edit page should be better styled

We'd like to get Bottlenose to be generic-by-default, so it doesn't depend on being run in CCIS at Northeastern.

• Login (CCIS LDAP)
• Instructor lookup (CCIS LDAP)

ccis: blerner commented on Feb 10

Currently teams can only be dissolved if their end date is nil. The check should be if the end_date is nil or if it is in the future.

As it stands, Bottlenose keeps courses separate. But for things like Fundies accelerated & regular, there needs to be a way to transfer grades from one course to the other, or at least mark the registrations as TRANSFERED so that when we get to final grades, we don't forget that grades are missing!

(esp. the sandbox grader)

(Originally: Prevent self-evals from being submitted past their deadline)

ccis: blerner commented on Nov 22, 2016

Especially if the student is taking a late day... The UI should pop up an alert giving a countdown until the deadline, or something, to make it clear. (We had a student this semester start the self-eval before the late deadline, and finish six minutes after it, which caused the server to think it was two days late, when only a single day was permitted...)

ccis: blerner commented on Feb 10

Partially fixed by 110383f. This may be a sufficient fix; we'll see in practice.

ccis: blerner commented on Dec 18, 2016

And make sure that the grade-so-far computations understand that...

ccis: blerner commented 16 days ago

1. Suppose an instructor creates an exam, with rubric#1. This creates a ExamGrader GraderConfig for the assignment, with the available score being the total weight of the rubric.
2. The instructor modifies the rubric. This currently creates a second GraderConfig for the same assignment, which is certainly wrong. Instead, since no grades have been entered yet, the existing GraderConfig should be modified to have a new total score, and an invariant should be maintained that every grader config is used for at most one assignment.
3. The instructor begins entering grades.
4. The instructor finds a mistake in the rubric and updates it. Now, we can't modify the GraderConfig, because it'll be inconsistent with the grades entered. But we can't replace it, or else the entered grades will no longer be associated with the exam.

Find a consistent semantics in here...

Once we have a tagging system (#34) so that we know which labs students are in, we'd like to enhance the ability to randomly generate teams such that they're only paired off within labs.

Form should be pre-filled with previously entered data.

#24 may solve this.

These are both built in methods in ruby 2.4

We should pull out the custom versions.

Somehow we recently sprouted a user_submission table. That can't be right - it directly conflicts with team_users. The database needs some cleanup to remove redundancy.

ccis: blerner commented on Mar 3

When creating teams, we might want to contact all the unpartnered people and ask if they have preferences for partners. This might be tricky to define, given #10.

When the user is creating a new course, he/she can Add Section under Sections Tab. However, user is currently unable to delete the section if the section is accidentally added.

(lazy extraction is error prone and false savings)

This should apply to reg requests, which means a 2-month default should be safe.

Two months from what? Do we need a start date?

There should be a mechanism to globally adjust points earned within an assignment.

Methods?

ccis: blerner commented on Feb 10

We can't change teams in lab while a homework is still due, or else no one can submit. We need a notion of TeamSets, associated with individual assignments, so that teams can be managed at a finer granularity.

@blerner blerner referenced this issue on Mar 3
Add a link to email all unpartnered people #16

Hidden courses should be visible to admins and users registered for that course.

(copied from ccis)

Not sure if this is actually a good idea in places like tables.

Currently, students see their list of assignments in three places:

• Course list page.
• Course show page.
• Assignments page.

The assignments page should be merged into the course show page.

Perhaps the assignment list should be removed from the course list page.

Restrict the students from seeing the exam grades when the instructor first entered them.
Release all the grades when the button is clicked.

Graders shouldn't see student names as links to their profiles, in the assignment-submissions page. This avoids permission errors, and also graders' confusion about how to regrade assignments, because there are fewer things to click on.

And, there's no way to fix that grade from within BN itself...

Either the Audit log or the standard log, depending.

Unfortunately, checkstyle doesn't come with a particularly friendly API; it's meant entirely as a command-line utility. Wrap the call to checkstyle in a thread with a timeout, same as is done in the tester library for individual tests, so that it can't run "forever".

ccis: blerner commented on Mar 3

Sometimes a submission happens "by a team" but one team member deserves all the credit. Right now it's painful to handle that grading scenario. We should have a button to dissolve the team, create the necessary number of singleton teams, copy the submission and resubmit it on behalf of each student. All grader outputs should be copied over, too.

Implement the Steno grader.

Your change to tap_parser, on line 59, breaks the error handling I'd set up on line 20, and the exception handlers that were anticipating those parse errors (e.g. in grades_controller line 433). This broke the rendering of the checker and junit graders, when they tried to deal with busted student input.

If a TAP file is malformed, then it should not successfully parse, in my opinion. Where do you rely on parsing to always, always succeed, no matter how malformed? And can we come up with a better, more robust solution that works for both of us?

If all the graders for an assignment are autograders, then is there any benefit in not publishing the grades immediately as students submit them?

Follow-up from #42, the default lateness tab should be selected as Course Default, rather than the specific policy itself.

Follow-up from #42, the Remove section button in the course new/edit page could be badly broken if there are already registrations for a section. We need to figure out what to do in that case (probably, disallow removing the section)

We want to have a mechanism for marking students by, e.g., lab section.

This may replace the current course section system, or be added in addition to it.

(ccis: ashesh commented 8 days ago)

When an exam contains one or more extra credit questions, the exported excel spreadsheet includes these questions in the total number of points in the exam. This results in an incorrect calculation of the computed grade for the exam, which is referenced in the Summary worksheet.

Exams with extra credit work correctly on the server. It seems this issue is only when exporting gradesheet in excel.

Follow up from #42, assignments are now missing a validation ensuring that there is at least one grader configured for the assignment.

• Make sure they're not hiding errors that could be reported to the user.
• Make sure they're catching only the right errors.
• Make sure that any errors that get caught get handled / logged.

Currently we have three assignment types:

• Files
• Questions
• Exams

We should split these to separate controllers so they can have separate views.

That probably implies separate models and single-table-inheritance and a subfolder under models.

This will also make it easier to add a couple new types:

• Git
• Just Points

This may require some more thought about type-specific graders, and the grader logic.

redundant with the assignments page that students can already see