Getting Started

Installing rspecFppGen

0. Make sure the system you would like to generate questions on is a Linux machine (WSL works fine) and has Python3 added to its $PATH
1. Clone our module to the machine you will be writing questions on.
2. Navigate to rspec-questionwriter/rspecFppGen/ and run make install

That's it! You should now have a python package installed called rspecFppGen. If everything works perfectly, your output should look like this:

.../rspec-questionwriter $ cd rspecFppGen/
.../rspec-questionwriter/rspecFppGen $ make install
<output omitted>
Successfully built rspecFppGen
Installing collected packages: rspecFppGen
Successfully installed rspecFppGen-0.1
.../rspec-questionwriter/rspecFppGen $

Here's a paste-able block for your convenience:

git clone https://github.com/ace-lab/rspec-questionwriter
cd rspecFppGen/
make install

YAML file specification

Each question that is shown to the student is specified by exactly one YAML file and an associated System Under Test (SUT). The SUT should be a functioning program excepting the code that is part of the student submission. The YAML file will follow the below specifications.

Every question should have:

1. A prompt to describe the system under test to the student
2. A system to place under test
3. A correct test suite over that system
4. What segments of the solution to blank out for the student
5. A number of mutations to be applied to the system under test

If we have some app in <dir>/ and would like to generate a question using it, we should write our question in a file named <question_name>.yaml. This file should include:

• The prompt of the question (prompt:)
• The file that a student test suite would be written to (submit_to:)
• The reference test suite (solution:)
• And all mutations, heirarchially sorted by variant (mutations:)
  • these mutations must be formatted in diff(1) standard format

This is illustrated by the following code snippet.

prompt: |
  This is where the text shown to the student is specified.
submit_to: app/path/to/submission_file.rb 
solution: 
  pre: |
    This is an optional text field for text that precedes the student's submission box. This will be formatted as code and ruby syntax highlighting will be used.
  lines: | 
    describe functions do
      it "Even function" do
        ?expect(even(1)).to eq False?
        ?expect(even(2)).to eq True?
      end
    end
  post: "This is identical to the `pre` field, but will succeed the student's submission box."
mutations:
  the_first_variant: # the name here will have underscores replaced by spaces and reformatted according to python's <str>.capitalize() when shown to the student
    path/to/file/to/change.rb: |
      19c19
      <     n % 2 == 0
      ---
      >     !(inc(n) % 2 == 0)
    path/to/second/file/to/change.rb: |
      3c3
      <   i = i + 1
      ---
      >   i = i + 2
  the_second_variant: 
    path/to/another/file.rb: |
      5c5,6
      <     queue.append!(request)
      ---
      >     modified = queue.append(request)
      >     new = modified.pop!()

Examples can be seen in ace-lab/rspec-questionwriter/examples/.

Running the generator

We recommend calling the module by the python handle for running modules (python -m). This tool must be called once for each question that has a unique output directory / SUT pair. The command line call follows the structure below:

python3 -m rspecFppGen <output directory> <SUT directory> <input yaml file> [<additional input yaml file1> [<additional input yaml file1> [...]]]

For example, after cloning ace-lab/rspec-questionwriter, to make the leap_year example question in a new examples/questions/ directory on a system with python3 installed:

.../rspec-questionwriter/ $ cd examples
.../rspec-questionwriter/examples $ mkdir questions
.../rspec-questionwriter/examples $ python -m rspecFppGen questions/ leap_year/app/ leap_year/leap_year.yaml
Running FPP generator
Generating from source examples/questions//leap_year.py
- Extracting from source...
- Creating destination directories...
- Copying leap_year.py to examples/questions/leap_year/source.py ...
- Populating examples/questions/leap_year ...
SyntaxError: Could not extract exports from answer
- Populating examples/questions/leap_year/tests ...
Done.
- Overwriting info.json
- Preparing solution
- Loading common files
- Producing mutations
patching file examples/questions//leap_year//tests/suite0/funcs.rb (read from examples/questions//leap_year//tests/common/funcs.rb)
patching file examples/questions//leap_year//tests/suite1/funcs.rb (read from examples/questions//leap_year//tests/common/funcs.rb)
- Writing grader metadata
.../rspec-questionwriter/examples $