Autotesting Specs

• &runspecs - autotesting specification follows after this directive until end of file

  • Refer to the give manual for further documentation

  • Web interface to help you set up the autotesting specfication.

• &testmode mode

• where mode can be automark, notest

• Examples:

  • 9021-03s1-A

  • 9021-03s1-B

  • 9021-03s1-C

  • 9021-03s1-D

  • give.spec/Examples/notest

Pre-test phase

• &compile
  rm -f *.c
  rm -f a.out
  
  f=grid.c
  classify -v $1
  dos2unix $f
  list $f
  
  options="-O -Wall -Werror"
  echo "Compiling with $options ..."
  echo
  gcc $options $f
  compile_status=$?
  if [ $compile_status != 0 ]
  then
      echo
      echo "** No good. Exiting..."
      exit 1
  fi
  
  cp ~/bin/trim.pl .
  cp $A/gridTest[1-6] .
  
  &end
  
  &timelim 3
  &linelim 120 20
  &exec ./a.out

• &compile - enter the script to compile submission and to print code listings, terminate with &end

• &prep - enter the script to run before each test, terminate with &end

• &linelim m n - print the first m output lines and the last n output lines

• &timelim s - cpu time limit

• &exec command - $exec anywhere in the test suite will be substituted by the command

Test phase

• &test 1
  &mark 1.5  gridTest1 no valid row col input
  &invoke $exec < gridTest1 | ./trim.pl
  &match < Student output and > Sample output
  diff -iw $stdout $stdres
  &showdiff -120w -iWt
  
  &test 2
  &mark 1.5  gridTest2 middle cell
  &invoke $exec < gridTest2 | ./trim.pl
  &match < Student output and > Sample output
  diff -iw $stdout $stdres
  &showdiff -120w -iWt

• &test n

• &mark mark description

• &invoke command

• &match description - put the command to match the student output and the sample output on the following line

• &show file

• &showdata optional_command

• &showdiff arguments

  • &showdiff -nw sdiff_arguments - translate into sdiff command and arguments

  • &showdiff -w diff_arguments - translate into diff command and arguments

• &special mark description - enter the script to define testing that can't be defined using the standard testing directives, terminate with &end

  • give.spec/Examples/special

Post-test phase

• &performance 10
  
  &subjective 5
  Human Marker    5
  &end
  
  &totalmark 15

• &performance total_testing_mark n

  • total_testing_mark is the maximum mark for passing all the tests, each failed test deducts mark from this maximum

  • n is either abort_test_number or deferred_test_number depending on the testmode

    • abort_test_number: if testmode is automarked, the autotesting aborts after failing n tests. This is described very well here

    • deferred_test_number: if testmode is deferred, the first n tests are immediate autotesting, thereafter the autotesting is deferred until the human marker is present.

• &subjective total_subjective_mark

  • description TAB mark

  • description TAB mark

  • ...

  &end

• Used by Xmark (classrun -mark). You must insert a TAB character between description and mark in order that Xmark parses the subjective assessment correctly.

• &totalmark total_assignment_mark