ddgrader

DDgrader consists of a few tools to parse design documents, with the goal of speeding up their grading, and allowing for quick collection of relevant information. It is broken up into several commands like git so use ddgrader mine, ddgrader setup etc.

mine

ddgrader mine The mine command is responsible for parsing all of the design documents and extracting student and group information. It expects one parameter which is a directory containing the unpacked design documents from Canvas.

###Errors There are several test cases for the parsing/regexes but unfortunately some students just don't follow the format and there is nothing that can be done! Using different logging levels ddgrader -L debug mine unpacked allows the user to discover and hopefully rectify these issues. I've tried to categorize them,

• CRITICAL: the design document is empty or very wrong and may need to be rewritten,
• ERROR: the design document is likely wrong, or lacking some significant information
• WARNING: Student omitted some non-critical information (ranking, slip days)

These should be repaired before running the setup command

report

The report command is responsible for reporting interesting things about a collection of design documents. It is composed of (several) subcommands, listed below.

###report ranking report ranking will generate a report showing students who have been ranked poorly by their group members. This requires the mine command to have been previously successfully executed

see report ranking --help for additional parameters. It is possible to set a different threshold for reporting (--thresh) as well as include all member rankings for poorly ranked students (--all)

###report slip report slip will generate a report showing students who have reported non-zero slip days on their design document. It defaults to 0 if they the slip day line.

--all will optionally show all slip day reports, including 0

setup

The setup command is responsible for taking the information from mine and the students' implementations and linking them together. It creates a directory for each student (by their eid) and symlinks the groups' code, symlinks their design documents, copies the grading template (configuration), and also symlinks other files from their partners who come alphabetically before them (currently)

grade

The grade command will automate the grading process by automatically opening all the relevant files for each student in a loop. These can be configured with the configuration (it optionally takes a eid of a specific student to grade)

collect

The collect command zips up the feedback forms and collects the grades into a form uploadable to canvas. It requires the grading spreadsheet from canvas, and also the index of the column for the grade.

Configuration

ddgrader uses a config file for a lot of parameters. By default it assumes "ddgrader.cfg" in the current working directory, but can optionally specified with --config parameter. Copying the one from the repo is a good place to start.

Walkthrough

1. Install python3 and python3-setuptools (standard on UT linux machines)
2. Clone ddgrader via git clone git@github.com:olemma/ddgrader.git
3. change into ddgrader directory cd ddgrader and install via python3 setup.py develop --user. This will install ddgrader to ~/.local/bin/ddgrader and ~/.local/bin/project_prepare (which you will want to add to your PATH prehaps)
4. Run it via ddgrader --help to see how it works.
5. Make a directory for the project somewhere i.e. projx
6. Download the code submissions from canvas assignments, put the zip in the directory created projx/code_submissions.zip
7. Download the design doc submissions from canvas assignments, extract the zip into some directory projx/raw_docs
8. Copy ddgrader.cfg from the ddgrader folder to projx (this is not necessary but will let you configure it later)
9. Run ddgrader mine raw_docs. This will parse the student design documents into a database-like file (.dds.pkl). Unfortunately, many students don't follow the design document formatting as specified, so a list of errors/warnings will be shown. See mine information above. Typically ERRORs and CRITICAL require editing the offending design documents. Filling out missing partner ids, etc. The parser is very lax, it requires really just a 'Name:' and 'EID:' for each member. It ignores 'Name1', etc. This might take 30min or so.
10. Repeat previous step until no serious errors occur.
11. At this point you can optionally use ddgrader report slip and ddgrader report ranking to get reports on slip days and poorly ranked students.
12. Run project_prepare code_submissions.zip implementations. This will unpack the zipfile into an implementations directory, and rename the folders according to eid1_eid2_eid3 for each group. There may be some errors if the students didn't write their EIDs when they submitted their code, which you can fix by manually renaming the folder.
13. Create a feedback template (see ddgrader.cfg), perhaps called template.txt that has the skeleton of the comment files you will write.
14. Run ddgrader setup implementations. This should create a students/eidxx directory for each student by eid. Inside the folder will be a copy of the template, a copy of their design doc, symlinks to their group's code and symlinks to their partners' feedbacks. There may be some warnings if matching code directory couldn't be found. Usually this is because of invalid EIDs either in their code readme or their design document, and can be fixed by updating those.
15. Grade the design documents, putting your comments/grades in each student's feedback.txt. Optionally use ddgrader grade which loops over all students, opening an editor with the given files. Configure via ddgrader.cfg
16. Once grading is complete, Download the grade spreadsheet as a CSV from canvas (ask Allison)
17. Open the grade.csv in some sort of editor and find the index of the cell corresponding to the design document grade. Set that as grade_column in ddgrader.cfg. (This is a hack).
18. Adjust the regex to ddgrader.cfg to extract the grade from feedback (it will search for this)
19. Run ddgrader collect grades.csv to generate a new csv with the grades and a zipped feedback file.
20. Upload both outputs back to canvas to update the grades and comments.