Rethink how to handle files in the judging process

[lw]

This text has grown longer that I had expected. If you already know the current situation, with its issues, feel free to jump to the Proposal section and spare some time.

Premise

The judging process is the sequence of operations performed by CMS on a submission to obtain a score and all intermediate products (executables and evaluations). This comprises compilation, evaluation and scoring, and usually requires access to data associated to the task and the dataset. The first two steps (i.e. compilation and evaluation) handle files and are managed by the TaskType whereas the third one (i.e. scoring) uses the outcome and text of the evaluation and is managed by the ScoreType.

The files required by this process (we could call them "input", if this name wasn't already in use for something else) are the files submitted by the user, the managers provided by the task/dataset, the input and the (official) output of the testcase. (Note that managers may play a role in both compilation (graders, stubs, etc.) and evaluation (comparators, managers, etc.)). Other files can be generated and the re-used during the process: executables and outputs produced by the user's program (to be compared with the official one).

It would be tempting to think that the only ones provided by the user are the ones of the first kind, but:

• the other ones are provided by admins which can be considered "users" as well;
• we have to support usertests, in which the contestants themselves provide (some) managers and input files.

Problems

At the moment we use the submission_format attribute of tasks, which is basically a list of strings, to specify the names of the files the user has to provide. This is limiting because:

• There's no way, for example, to give a different size limit for each file: the size limit is specified in cms.conf and is the same for all files of all tasks (while inputs of output-only tasks, which are actually outputs, may be much larger that inputs of batch tasks, which are source files).
• All languages need to have the same number of files, with approximately the same structure and names: we only allow to use a %l to insert a language-specific file extension. This worked well so far but I think it won't anymore as soon as we add support for other languages, in particular interpreted scripting languages.
• There's no way to customize the set of files required for usertests. The current solution was quickly written before IOI and, in my opinion, looks like a workaround: have the TaskType list these files via the get_user_managers and get_auto_managers methods. Not flexible and it's again impossible to impose size limits. There's also no way to give a name to the input file (which CWS hardcoded-ly names "input") and to specify its size (except in cms.conf, as before).

Finally, our testcases are designed to hold one input and one output files. This has worked well up to now, but I see cases where tasks may require different settings (see task C, "Garbled Email" from Round 1B of Google Code Jam 2013: I think the dictionary should be represented as an additional input file even if, in that case, due to its being always the same a manager would have worked as well).

For the sake of flexibility I propose to change some of these things.

Proposal

I have not made up my mind about the details yet, but here are some ideas.

The items of submission_format (that is SubmissionFormatElements) should be extended with more information. In particular I suggest to use different elements for files of different languages. Hence filename would contain the filename with the exact extension, no wildcards (e.g. "foo.cpp") and a new field, language, should contain a string defining the language ("c", "cpp", "pas" or None if language-independent, like in output-only tasks). I would then add a description field to contain a short human-readable text describing the file to be shown, for example, in CWS. Additionally I think a codename field is needed: it would contain values like "source", "encoder", "decoder" etc. and it would allow TaskTypes to get the right files they need without knowing the exact filename (which now also depends on the language!). The codename will be hidden from the user and only used internally. And, finally, a max_size field for the maximum file limit (with None meaning infinity?).

An almost identical structure would then be introduced for managers, to be used for usertest submission. Two additional fields would then be added: submission_digest (not nullable) and user_test_digest (nullable). The first tells the digest of the file to be used for this manager when judging submissions, the second tells the one to use in user_test or None to ask the user to provide it. This approach would replace the current manager implementation. This is the part I'm less convinced about: I think it makes sense to split the format definition (that goes in the task) from the actual files (that go in the datasets).

I fear a similar thing has then to be added for input files too (to specify filename, codename and size), and perhaps even for outputs (only to pair filename with codename). This seems an overkill and I'd better think about it another while.

All of this would also require some changes in Job and related classes. I was thinking to have EvaluationJob represent a single testcase instead of a whole list, and to group them together into something like a "JobGroup" (a dict of keywords to Jobs). A single Job would then simply list all submitted files, managers, inputs and outputs indexed by their codename. The attached managers, for example, would be all (and only) the ones for the language of the submission and the language-independent ones.

I was also thinking of having TaskType "suggest" the file formats they need, for example by telling which codenames they understand, whether they're optional, etc. using methods like the current get_user_managers.

What do you think of this proposal? I'd like to read comments and feedback before working better on the details.

[giomasce]

I'm completely in favour of your proposal.

[lw]

Some changes have already been applied (those to Job).

I'm updating this proposal. It now also aims to simplify TaskTypes, by having them refer to all files by codename (instead of mixing filename and list index). The mapping from codenames to actual filenames should be arbitrarily settable by the admin. We therefore need a Schema class for each file type (i.e. File, Manager, Input, Output, UserOutput and perhaps even Executable) and use it as a (codename, filename) pair, with additional information like language and maximum size. (I propose to drop the description, as it's a localization nightmare).

I plan to keep TaskType parameters to specify how certain aspects of the evaluation have to occur. Using them the TaskType should be able to always tell (the codenames of) the files it needs (only "required" files, not "optional" ones). Therefore the AWS interface will be like "Hey, you chose TaskType X with parameters Y=2,Z=3. These settings use 2 input files, codenamed A and B: tell me, for each supported language, their filename and maximum size. A manager, codenamed C, is also used: ...". (The TaskType could also suggest default values).

My idea was to have the description of the schema be given at Task level, whereas the actual files are attached to Submissions (source files), Datasets (managers) and Testcases (inputs and outputs). But the TaskType itself is specified on the Dataset... I still haven't figured out what to do about it.

I'm not sure if we should specify some kind of schema for executables too: we need to somehow determine the -o parameter of the compiler command, but can we do this automatically or do we have to ask the admin?

I'd also like to point out that output files of testcases aren't necessarily equivalent to the data produced by the contestant's program: the former don't need to be a valid answer to the task, they're just some precomputed data (some "notes") that aid and speed up the checker. (Yet, if the checker is a diff, then they need to be a valid answer). Giving them different names (instead of using "output" for both) may help clarify this further.

[lw]

I think that the database objects holding the files submitted by contestants should not have foreign keys referring to the database objects defining the submission format (exactly as we do now, with SubmissionFormatElements). That's because we absolutely want to avoid data loss in case the format gets changed and some of its elements removed (because of a bug or a human error).

Therefore these objects need to be identified by a string. The question is: codename or filename? I prefer the latter, because the codename isn't "stable": it depends on the currently active TaskType which could change during the contest (i.e. from batch+grader to communication to increase security). A change in TaskType may mix up all codenames but we can rely on admins to keep the filenames unchanged (for contestants' benefit too).

This also means that the submission format doesn't need to be stored at Task level. It can be moved at Dataset level, where the TaskType already is. I think this allows for a tremendous simplification. TaskTypes can "suggest" default filenames for the codenames they need. (TaskTypes know exactly which files they need by reading their parameters). We thus simply need a system for admins to "override" these values if they don't like them.

I think this will ease task creation via AWS (the code of importers may benefit too, but admins won't notice it). With the proper helper methods this separation of levels (i.e. default vs override) will be transparent to all other code.

I also propose to rename the outputs of Testcases: "reference" seems a better suited name to me.

Comments?

[giomasce]

I'm not sure I understand all the details of your proposal, but here I write mine (that it resembles yours, at least with respect to objectives and general ideas).

I think we already agree and the supposed life cycle of the files involved here: at the moment, for each task, the administrators have to provide some Managers (at dataset level) and some Inputs and Outputs (one of each category for each test case). With each submission the user provides one or more Files. Compilation is an operation that (in line of principle) takes some Files and some Managers and generates some Executables. Evaluation is an operation that takes some Files, some Executables and some Managers and generates some data that is not part of this mechanism. The whole structure is replicated for user tests, with the exception that there are not test cases and the input is contained directly in the UserTest object.

I think this model is mostly ok, with the most evident problem probably being the ability to have only one input and one output for each test case. The naming choices, which were made at the time of old trivial Batch tasks, also sound a bit inadequate now. Moreover, I find that it would be good to reduce a bit the amount of different classes we use in the process. Plus, of course, we have the need of a more flexible way to specify how the submission is expected to be made.

So, I would like to split this issue in two different problems: first, how the TaskType specifies which files a submission should contain; second, simplifying a bit the handling of all those files.

Submission format interface (aka, fixing SubmissionFormatElement)

I mostly agree with your proposal. Each element of the submission format has to carry a filename (meant to be shown to the user), a codename (meant to be recognized by the TaskType), a max_size (with its obvious meaning) and a language (again, with the obvious meaning). I would also add an optional flags, meaning that the submission can be accepted even if some optional files are missing (useful for output only tasks, for example).

I think it is acceptable to keep the submission format in the dataset, although in this case the administrator must be well aware that they should use consistent submission formats for datasets belonging to the same task. We also have to expect task types to do not rely on the submission being complete at compile and evaluation time: since an administrator error may have happened, they have to ensure themselves that the submission contains all the files they need and handle gently error conditions (which, in this case, are always the administrator's fault and never the user's).

Files life cycle

Once the submission format has became more effective, I think that the whole file handling cycle can be made lighter. For example, instead of having two different file types for each submission (File and Executable) we can have only one, using code names to distinguish between files provided by the user and generated by the system (such as the executables, of course, but perhaps also the output if one wants to retain them). Going further, I would propose that managers, input and output too can be referenced by code name (although in their case the distinction between file name and code name is less important), so that for each file the TaskType can choose whether to use a code name (for language-independent files or files that are handled the same independently from the language) or a file name (for files that need different workflow depending on the language).

I would then rename "managers" to "data set files" and I would merge "input" and "output" in "test case files", with their role specified by their code name (and being TaskType dependent, of course). Of course I expect the TaskType's documentation to specify which code names have to be used and with which meaning; and, of course, to specify them so that there is not clash between files of different "belonging" (to the submission, to the test case or to the dataset).

How to handle user tests

Right now, the user tests implementation carries a whole rewriting of all the file types in the system, since SQLAlchemy cannot store a foreign key to two different tables. My proposal, besides requiring a smaller number of file types, may help fixing this problem, since the Submission class can now store all the information required for either targets. Then we could convert UserTest to a special kind of Submission (which contains all the usual content of the submission and some other files that use the same code names of some inputs or managers and override them from the viewpoint of the TaskType). If we want to use two different classes anyway, we can make a common base class for them (SQLAlchemy is able to manage class inheritance), but I see no clear reason for this.

The mechanism specified by Luca of distinguishing between submission_digest and user_test_digest (which I would rename respectively digest and test_digest) may be used to determine which files are to be provided by the administrators and which from the user requesting a test. If the file locating logic is written so that it prioritize user files over test case files and test case files over dataset files, the TaskType logic probably doesn't event need to be aware of whether it is working on a submission or on a user test (of course care must be put so that this doesn't bring in security risks).

What do you think about this proposal? How much do you feel it close to yours?

(it must be noted that it may be a bit difficult to write contest dump updaters for this set of changes, which is backward compatible, but in a way the depends very much on the TaskTypes involved; yet, this is a price that we'd better pay as soon as possible instead of keep waiting for it)

[lw]

Foundations

I identify three phases in the grading process (actually four if we include scoring).

Step	Consumes (user-provided)	Consumes (admin-provided)	Consumes (autogenerated)	Produces
Compilation	Files[1]	Managers[2]	N/A	Executables
Execution	Files[3]	Managers[4], Inputs	Executables	Outputs
Judging	Files[5]	Managers[6], References	Outputs	outcome

[1] Usually source files for languages that need compilation
[2] Usually headers and "assistant code" (that does I/O and other menial work to alleviate the user)
[3] Usually source files for scripting languages that don't require compilation
[4] Usually assistants for the scripting sources, or executables that interact with the one we're evaluating
[5] Usually non-source files, e.g. for output-only
[6] Usually checkers and comparators

I'm now going to use Assistants, Interactors and Checkers for the three types of Managers (I do not propose to make this classification formal, as there are for sure other types that I missed).

Compilation is clearly distinguished from the others, as it needs to happen once per dataset instead of once per testcase: it's responsible to do any testcase-independent pre-processing of the Files.

The separation between the other two is more faint and blurred. They could also be indistinguishable, as in the Communication task type when the Interactor already includes the Checker and it therefore directly tells the outcome, without producing output files. Hence we merge them and call it just "Evaluation".

Another useful schema, to clarify the setting.

Scope	User-provided	Admin-provided	Generated
Task / Submission	Files	N/A	N/A
Dataset / SubmissionResult	N/A	Managers	Executables[1]
Testcase / Evaluation	N/A[2]	Inputs, References	Outputs

[1] As they also depend on Managers they cannot be at Task level
[2] The fact that output-only tasks have one input per testcase doesn't matter

Aims

One of my aims is to allow admins to arbitrarily set the filename of every file involved in the grading. That accounts for Files as well as for Managers, Inputs, Outputs and also Executables and References, even if it may seems strange and at the moment we don't allow it. That, in turn, means that it's very convenient for TaskTypes to identify and reference all these files by a codename (or by their index in a list, as we do now, but I don't think it's handy).

More than that, I want admins to be able to do little more than changing the filenames. That is, I don't want them to be able to arbitrarily add or remove entries to the submission format. I want the TaskType to provide the submission format, with the codename and optional flag (an idea that you proposed and which I totally approve) already set, and admins are only allowed to set the filename and the max_size, and semi-set the optional flag (i.e. if it's true they can set it to false, but not vice versa).

I also want to remove from TaskTypes the burden to check if a submission complies with the format: by enforcing such a strict format schema it's an operation that CMS (i.e. ES) can do beforehand, and give stronger guarantees to TaskTypes (note: that's a difference with what you said).

I also want TaskTypes to have a single codepath for submissions and user tests: flags like only_execution and get_output on the Job (to selectively activate the features that are exclusive to only one of these operations) seem a good idea to me.

Yet, I'd like to describe an alternative approach to it, just so that we knowingly discard it. We may require TaskTypes to provide one of two possible interfaces: either evaluate() or (execute() and judge()). In the former case testing is not allowed at all, in the latter case it's allowed: user tests are just executed, whereas submissions are executed and judged.

I'll stop here with my comment as I'd like to hear your opinion on this before I go further.