give.spec/runspecs

Autotesting Specs

&runspecs - autotesting specification follows after this directive until end of file
- Refer to the give manual for further documentation
- Web interface to help you set up the autotesting specfication.
&testmode mode
where mode can be automark, notest
Examples:

Pre-test phase

&compile
rm -f *.c
rm -f a.out

f=grid.c
classify -v $1
dos2unix $f
list $f

options="-O -Wall -Werror"
echo "Compiling with $options ..."
echo
gcc $options $f
compile_status=$?
if [ $compile_status != 0 ]
then
    echo
    echo "** No good. Exiting..."
    exit 1
fi

cp ~/bin/trim.pl .
cp $A/gridTest[1-6] .

&end

&timelim 3
&linelim 120 20
&exec ./a.out

&compile - enter the script to compile submission and to print code listings, terminate with &end
&prep - enter the script to run before each test, terminate with &end
&linelim m n - print the first m output lines and the last n output lines
&timelim s - cpu time limit
&exec command - $exec anywhere in the test suite will be substituted by the command

Test phase

&test 1
&mark 1.5  gridTest1 no valid row col input
&invoke $exec < gridTest1 | ./trim.pl
&match < Student output and > Sample output
diff -iw $stdout $stdres
&showdiff -120w -iWt

&test 2
&mark 1.5  gridTest2 middle cell
&invoke $exec < gridTest2 | ./trim.pl
&match < Student output and > Sample output
diff -iw $stdout $stdres
&showdiff -120w -iWt

&test n
&mark mark description
&invoke command
&match description - put the command to match the student output and the sample output on the following line
&show file
&showdata optional_command
&showdiff arguments
- &showdiff -nw sdiff_arguments - translate into sdiff command and arguments
- &showdiff -w diff_arguments - translate into diff command and arguments
&special mark description - enter the script to define testing that can't be defined using the standard testing directives, terminate with &end
- give.spec/Examples/special

Post-test phase

&performance 10

&subjective 5
Human Marker    5
&end

&totalmark 15

&performance total_testing_mark n
can use &auto instead of &performance
- total_testing_mark is the maximum mark for passing all the tests, each failed test deducts mark from this maximum
- n is either abort_test_number or deferred_test_number depending on the testmode
  - abort_test_number: if testmode is automarked, the autotesting aborts after failing n tests. This is described very well here
  - deferred_test_number: if testmode is deferred, the first n tests are immediate autotesting, thereafter the autotesting is deferred until the human marker is present.
&subjective total_subjective_mark
- description TAB mark
- description TAB mark
- ...
&end
Used by Xmark (classrun -mark). $/!\$ You must insert a TAB character between description and mark in order that Xmark parses the subjective assessment correctly.
&totalmark total_assignment_mark

CategoryGiveSpec