No automatic submission after deadlines passes

[malyzajko]

(This issue is potentially specific to the use of Source Academy at Uppsala university.)

The current behaviour is that homeworks are automatically submitted (get status Submitted) once the deadline passes. This also applies to homeworks that have not been attempted, or are not meant to be submitted (e.g. the student has just played around with the code). For graders, this makes it time consuming to see what should and should not be graded.

Ideally, a homework would only have the status 'Submitted' if a student has actually, actively finalized the submission. The deadline should only disable the finalization/submission button.

[martin-henz]

We could make the auto-submit functionality configurable on a per-assessment basis.

Alternatively, or in addition, we could make the auto-submit functionality configurable on a per-course basis.

If we choose to make the auto-submit functionality configurable on both levels, the per-course configuration could set the default and the per-assessment configuration could override the per-course default.

Any views?

[malyzajko]

Either one would work for us, i.e. we would set it on a per-course basis.

For a general solution, having a default and then a per-assessment choice sounds reasonable.

[RichDom2185]

This seems to be more of a backend issue, specifically:

https://github.com/source-academy/backend/blob/470d220e2937d0b73a95a59505e5a8d2b2fb61e4/lib/cadet/jobs/autograder/grading_job.ex#L16-L26

EDIT: This is assuming that we carry forward with this proposal. See my comment below as to whether we should.

[RichDom2185]

Hi, I have some questions to clarify:

This also applies to homeworks that have not been attempted

By "homeworks that have not been attempted", can I clarify an example for this would be a student forgetting to attempt the assignment entirely?

or are not meant to be submitted (e.g. the student has just played around with the code).

Can I clarify, do you mean:

• There is a special type of assignment that are not meant to be submitted to begin with? or
• It is an actual assignment meant to be submitted, but the student did not write anything meaningful inside?
  • If it's this latter, then there is no way to differentiate a student writing gibberish and a student writing partially completed code (just like a normal pen-and-paper test, where it would only be known when a grader sees the submitted paper); see below on why I think we should still submit the attempt in this case

Ideally, a homework would only have the status 'Submitted' if a student has actually, actively finalized the submission.

Personally, I disagree on this. I think that a homework/assignment should have the status 'Submitted' if at least one of the following is true:

• The student has actually, actively finalized the submission.
• The student has attempted at least one question in the assignment, and the deadline of the assignment has passed.

The rationale is that partially completed assignments should still be marked, and it is reasonable for students to sometimes forget to submit an assignment (or perhaps be rushing it last-minute and have it close just before they hit submit).

---

Having said that, I agree with the concerns you raised for graders:

• Opening a submission only to realise it is empty is a waste of time and effort

Here's my counter-proposal:

• We move the "attempted" status to a per-question basis (instead of a per-assignment basis)a
• If a student does not attempt a single question of the assignment, then a submission is created with the status "not attempted" (instead of "submitted") - to still let the grader know of the submission status
• If a student attempts at least one question, retain existing behaviour
• The grader, when browsing through/grading submissions:
  • If that particular question has been attempted: no change in UI
  • If that particular question has not been attempted: Instead of showing the template/boilerplate code, the UI will just let the grader know that this question was not attempted

What do you think?

[RichDom2185]

We could make the auto-submit functionality configurable on a per-assessment basis.

Alternatively, or in addition, we could make the auto-submit functionality configurable on a per-course basis.

If we choose to make the auto-submit functionality configurable on both levels, the per-course configuration could set the default and the per-assessment configuration could override the per-course default.

Any views?

I think this can be a further extension to the solution (to make it configurable), but I don't think this addresses the root cause as for those assignment where auto-submit is set to be on, the problem still exists that if a student only attempts 1 out of the e.g. 14 questions in the assignment, the grader still needs to look through the other 13 questions to verify that the student never actually wrote anything for those 13 questions, which is time-consuming.

[malyzajko]

In our course, the auto-submit has been causing issues for the following two reasons (these may be very specific to how courses are run at Uppsala):

1. Students work in pairs and often individually first play around with an assignment, before 'submitting' one version of the code. The auto-submit has often submitted code that was not meant to be graded, and there was a lot of overhead to figure out which code was to be used for grading for each group.

2. Students have many options for repeating assessments. Hence, they may not want partially solved homeworks to be graded, since once they pass, they cannot improve their grade. But if they submit later and spend more time with the assessment, they may get a better grade.

Ultimately, the issue is that the current auto-submit works differently from other systems that the students are used to, such as Canvas, where they actively have to submit homeworks.
Of course, we can try to explain to students that they need to remove code before the deadline that they don't want submitted. But for our use-case, not having the auto-submit would be a better fit.

That said, any solution that at least help reduce grading overhead of clearly useless or empty solutions would help.

[emptygx]

For the first reason that pair work causes issues, it will be solved by issue #2501, which is tackling team/pair-based assessments and joint submissions to prevent overhead.

Regarding the option of not having the auto-submit for certain assessments, even if a student doesn't attempt any questions, an empty submission is still helpful as it is needed firstly to be able to give a grade to the student, and secondly to give comments if needed.

We also do not want to discount students who forget to click the submit button before the deadline or have attempted some questions but could not finish in time before the deadline. Not having the auto-submit feature might cause more issues as they would lose their work/submissions and have no grade at all. I think most of these points were mentioned before by @RichDom2185.

Instead, I would propose for questions that have not been answered (i.e., still empty or only containing the boilerplate template) to be flagged as "unanswered", reducing time wasted checking if the question has to be graded. Just as @RichDom2185 has mentioned, the proposal would be:

• We move the "attempted" status to a per-question basis (instead of a per-assignment basis)a

• If a student does not attempt a single question of the assignment, then a submission is created with the status "not attempted" (instead of "submitted") - to still let the grader know of the submission status

• If a student attempts at least one question, retain existing behaviour

• The grader, when browsing through/grading submissions:

  • If that particular question has been attempted: no change in UI
  • If that particular question has not been attempted: Instead of showing the template/boilerplate code, the UI will just let the grader know that this question was not attempted

For now, we would focus on having an "unanswered" status for questions not attempted on a per-question basis as well as the UI changes in a separate issue.

@malyzajko would this set of solutions be viable for now?

[malyzajko]

Sorry for my late answer. The proposal will help our use case at least to some extent.
But as I explained before, for the particular workflow at Uppsala university, it would be helpful to be able to disable the auto-submit. That said, I understand that our case may be special.