Grammar Fuzzer (gfuzzer): Fully Automated Test Generation, Execution, and Evaluation Tool
gfuzzer is a fully automated test generation, execution, and evaluation tool developed for compiler testing of reasoning engines.
gfuzzer is developed at and copyrighted on Institute of Software Technology at Graz University of Technology under GNU General Public License v3.0. See the copyright notice at the end of this document for details.
More information on the details of gfuzzer can be found in our paper, Fully Automated Compiler Testing of a Reasoning Engine via Mutated Grammar Fuzzing. All test inputs generated for this paper can be found here.
- Perl v5.14 or higher
Above is the only requirement of gfuzzer.
- Accepts context-free grammars in BNF format. See
examples/bc.bnf. - Accepts regular expressions as terminals. This feature works only if the grammar is used as a recognizer.
- Comes with a stand alone recognizer and a stand alone tokenizer.
- Can use separate grammars for generating and recognizing.
- If mutation is going to be used, the grammar should be in at least Chomsky Normal Form. Chomsky Reduced Form is recommended.
Usage: perl gFuzzer.pl <GGF> <RGF> <RUT> "<PUT>" "<FM>" [mutate]
GGF : Generator-Grammar-File.
RGF : Recognizer-Grammar-File.
RUT : Rule-Under-Test. Most of the time, the root rule makes sense here.
PUT : Parser-Under-Test. Command to execute your tests. Tests are assumed to be entered from standard input.
FM : Fail Message. The message that Parser-Under-Test shows whenever the test input is not accepted.
mutate : A keyword which triggers grammar mutation. It is optional.
Example: Testing bc
bc is a known precision-calculator utility. Execute the following command from the main directory to generate, execute, and evaluate tests for 10 seconds.
timeout 10 perl gFuzzer.pl examples/bc.bnf examples/bc.bnf start "bc -sql " "(standard_in)"
For using timeout command in Mac OS X, please check Timeout Command on Mac OS X?.
If the command is executed successfully, two files will be generated in the main directory.
- passed.tests
- failed.tests
You can view these files to see the generated test inputs. Each test input is separated by a line.
This allows to count the number of passed tests as follows.
cat passed.tests | grep -e "--" | wc -l
You can also count the number of failed tests as follows.
cat failed.tests | grep -e "--" | wc -l
Depending on your system, gFuzzer may generate more than a hundred tests per second.
During experiments, we used the following parameters for gfuzzer.
GGF : AST19/generator.bnf
RGF : AST19/recognizer.bnf
RUT : start
PUT : "java -jar AST19/ATMS.jar "
FM : "Compile Failed"
mutate : We used this keyword to switch between gfuzzer and mgfuzzer.
Note that we used Java version "1.8.0_112" during experiments.
Please feel free to contact us if you are unable to reproduce the experiments.
gfuzzer: Fully Automated Test Generation, Execution, and Evaluation Tool Copyright (C) 2019 Institute for Software Technology at Graz University of Technology
gfuzzer is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
gfuzzer is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with gfuzzer. If not, see https://www.gnu.org/licenses/.
- mail: [email protected]
- address: Inffeldgasse 16b/II, 8010 Graz/AUSTRIA