Command line interface for the test driver

Command-line interface for the test driver

Authors: Nicholas DeMarco, Sean Murthy

Overview

The following figure shows an example output the test driver generates. The annotations call out the different parts the driver outputs. The command-line options determine which parts are output as well as how those parts are output. They also control where the output is written.

Options

Command-line options may be specified in any order. An option is a combination of a "specifier" and a value, in that order. Every option specifier begins with a hyphen (-). An option specifier and it's value are separated by a space. A space also separates options, if multiple options are specified.

If an option's specifier is used, its value should also be supplied, even if the default value is desired for that option. That is, if the default value is desired for an option, it is better to leave out that option from the command line.

In this section, angle brackets <> are used to denote an option's possible value, the form of option value, or a list of permitted option values. Angle brackets should be left out in actual usage.

• -h <yes, no>: specify if a header should be printed at the start of a test. The default is to print the header. That is, it is not necessary to use the option -h yes.

• -ht <header text>: specify the header text to print, overriding the default text. This option is ignored if header printing is turned off using -h no.

  If the header text contains the "macro" $cmd, that macro is replaced with the name of the executable file, without the filename extension or path information. For example, if the executable's path is bin/stl-test.exe, the macro $cmd is replaced with the text stl-test.

  If the header text contains the macro $suite, that macro is replaced with the name of the test suite being run. If multiple suites are run, $suite is replace in the header text for each suite.

  The default header text is Running $suite (when the -ht option is not used).

  The header text is always followed by a colon and a new line so that any output after the header starts on the the line after the line containing the header.

• -p <detail, indicate, none, auto>: set "pass-report mode", or specify how passed tests are denoted in the output. The value should be one of the following:

  • detail: print the test number, the word Pass, and a "hint" that identifies the actual test. Test number is just a sequence number, and "hint" is the text associated with the test. The hint is always enclosed in parentheses.

  • indicate: print a dot for each passed test. Consecutive dots in the output denote consecutive passing tests. This is the default pass-report mode if -fn, -fo, or -fa option is not used.

  • none: do not output anything for passed tests. This is the default pass-report mode if -fn, -fo, or -fa option is also used.

  • auto: usenone as pass-report mode if -fn, -fo, or -fa option is also used. Otherwise, use indicate as the pass-report mode.

  Note: A failed test is always reported in detail. Unlike passed tests, there is no option to suppress a failed test from being reported.

• -s <yes, no>: denote if a summary of tests should or should not printed when testing ends. The default action is to print a summary.

• -t <# of failed tests to tolerate>: set the number of failed tests the test driver treats as acceptable ("tolerates") before stopping tests. The default threshold is 0. That is, by default, the driver stops when the first test fails.

• -run <list of test suites to run>: semi-colon delimited list of the names of test suites to run. The default (when -run is not used) is to run all test suites defined.

• -fn <filename>: write all output to the specified file, instead of writing it to the standard output device: create the file if it does not exist; print an error message and exit if the file already exists.

  • <filename> may contain the macro $cmd, which is replaced with the executable file's name as described in the -ht option.

  • The default filename extension is .out (used if no extension is included in <filename>).

• -fo <filename>: write all output to the specified file just as in the -fn option, except overwrite the file it already exists.

• -fa <filename>: write all output to the specified file just as in the -fn option, except append output to the file if the file already exists.

Note: A command-line may specify only one option (-fn, -fo, or -fa) to send output to a file.

Examples

This section illustrates the use of the different options the test driver supports. A sample output is included/described with each example.

The dots shown in the results for Examples 1 through 3 are due to -p indicate being the default pass-report mode when results are not written to a file. Examples 4 through 6 illustrate different pass-report modes.

For simplicity, all examples use stl-test.exe as the executable file. In practice, substitute stl-test.exe with the actual name and path of your executable. The examples also assume a test suite named array_test is defined in the executable, and that array_test is the only test suite defined.

1. Turn header off: stl-test.exe -h no

   ..........
   
   Suites run: 1
   Tests completed: 10
   Tests passed: 10
   Tests failed: 0
   

2. Turn header on: stl-test.exe -h yes (recall that -h yes is the default and Running $suite is the default header text)

   Note that the result for Example 1 does not contain the header text seen in the result for this example.

   Running array_test:
   ..........
   
   Suites run: 1
   Tests completed: 10
   Tests passed: 10
   Tests failed: 0
   

3. Set custom header text: stl-test.exe -ht "Results of array_test"

   Results of array_test:
   ..........
   
   Suites run: 1
   Tests completed: 10
   Tests passed: 10
   Tests failed: 0
   

   Alternative command-line for this example: stl-test.exe -ht "Results of $suite"

4. Show details for passed tests: stl-test.exe -p detail

   Running array_test:
   Test# 1: Pass (s.empty())
   Test# 2: Pass (s.size())
   Test# 3: Pass (s.max_size())
   Test# 4: Pass (p.empty())
   Test# 5: Pass (p.size())
   
   Suites run: 1
   Tests completed: 5
   Tests passed: 5
   Tests failed: 0
   

5. Indicate each passed test with a dot: stl-test.exe -p indicate (recall that -p indicate is the default if -fn, -fo, -fa options are not used)

   Running array_test:
   ..........
   
   Suites run: 1
   Tests completed: 10
   Tests passed: 10
   Tests failed: 0
   

6. Do not indicate passed tests: stl-test.exe -p none

   The result from this command-line is the same as that for Example 5, except the dots seen in the result for Example 5 are missing here.

   Running array_test:
   
   Suites run: 1
   Tests completed: 10
   Tests passed: 10
   Tests failed: 0
   

7. Set header text and show details for passed tests:

   stl-test.exe -ht "Results of $suite" -p detail

   This example combines Examples 3 and 4, illustrating the use of multiple options.

   Results of array_test:
   Test# 1: Pass (s.empty())
   Test# 2: Pass (s.size())
   Test# 3: Pass (s.max_size())
   Test# 4: Pass (p.empty())
   Test# 5: Pass (p.size())
   
   Suites run: 1
   Tests completed: 5
   Tests passed: 5
   Tests failed: 0
   

8. Do not print test summary: stl-test.exe -s no

   The result from this command-line is the same as that for Example 5, except the test summary seen in the result for Example 5 is missing.

   Running array_test:
   ..........
   

9. Print test summary: stl-test.exe -s yes (recall that -s yes is the default)

   Running array_test:
   ..........
   
   Suites run: 1
   Tests completed: 10
   Tests passed: 10
   Tests failed: 0
   

10. Tolerate three failed tests: stl-test.exe -t 3

    Running array_test:
    ...
    Test# 4: FAIL (p.size())
    Test# 5: FAIL (s.size())
    ...
    Test# 9: FAIL (p[0])
    Test# 10: FAIL (p[1])
    
    Suites run: 1
    Tests completed: 10
    Tests passed: 6
    Tests failed: 4
    Tests stopped after 4 failure(s)
    

11. Do not tolerate any failed test: stl-test.exe -t 0 (recall that -t 0 is the default)

    Comparing the result from this command-line to that from Example 10, it is seen that, here, testing stops after the first failure.

    Running array_test:
    ...
    Test# 4: FAIL (p.size())
    
    Suites run: 1
    Tests completed: 4
    Tests passed: 3
    Tests failed: 1
    Tests stopped after 1 failure(s)
    

12. Run only some test suites: stl-test.exe -run list_test;string_test

    Just for this example, assume the executable defines many test suites including two with the names list_test and string_test. This command-line causes only the suites named in the -run option to be run.

    Running list_test:
    .....
    
    Tests completed: 5
    Tests passed: 5
    Tests failed: 0
    
    Running string_test:
    .........
    
    Tests completed: 9
    Tests passed: 9
    Tests failed: 0
    
    Suites run: 2
    Tests completed: 14
    Tests passed: 14
    Tests failed: 0
    

13. Write output to a specified file: stl-test.exe -fn results.txt

    Assuming the tests performed are the same as in Example 11, the driver writes the following output to the file results.txt, but only if that file does not already exist. The driver prints an error message and exits if the file already exists.

    The resulting file does not contain any report on passed tests because -p none is the default when -fn, -fo, or -fa option is used.

    Running array_test:
    Test# 4: FAIL (p.size())
    
    Suites run: 1
    Tests completed: 4
    Tests passed: 3
    Tests failed: 1
    Tests stopped after 1 failure(s)
    

14. Construct output filename from exe filename: stl-test.exe -fn $cmd

    The effect is the same as in Example 13 except stl-test.out is used as the filename. The filename extension is .out because no extension is specified on the command-line.

15. Set custom filename extension: stl-test.exe -fn $cmd.txt

    The effect is the same as in Example 13 except stl-test.txt is used as the filename.

16. Omit extension from output filename: stl-test.exe -fn $cmd.

    The effect is the same as in Example 13 except stl-test is used as the filename. The output filename does not have an extension because the filename on the command-line ends with a dot.

17. Indicate pass reports in output file: stl-test.exe -p indicate -fn results.txt

    The effect is the same as in Example 13, except each passed test is denoted with a dot.

    Running array_test:
    ...
    Test# 4: FAIL (p.size())
    
    Suites run: 1
    Tests completed: 4
    Tests passed: 3
    Tests failed: 1
    Tests stopped after 1 failure(s)
    

18. Write/overwrite output to a file: stl-test.exe -fo results.txt

    The effect is the same as in Example 13, except the file is overwritten if it already exists.

19. Write/append output to a file: stl-test.exe -fa results.txt

    The effect is the same as in Example 13, except the file is appended to if it already exists.

20. Omit header, omit pass reports, and omit summary:

    stl-test.exe -h no -p none -s no

    If all tests pass, this command-line does not produce any output, and thus this combination of options is generally not recommended. Nevertheless, this combination can be helpful if the test driver is to be "silent" when all tests pass.

    The following is a possible result from this command-line if a test fails:

    Test# 4: FAIL (p.size())