RockTest

Programming Competition Management System

What is RockTest ?

RockTest is a collection of programs and related files to assist in the running and judging of computer programming competitions.

Why RockTest ?

The Computer Science Department at Slippery Rock University runs several computer programming competitions annually, for our own students, for high school students, and, occasionally, for intercollegiate competition. We learned (quickly!) that sneakernet ^? was an ineffective method of program submission. We tried using PC^2, but it didn't meet our needs. Shell scripts can help automate the process, but are ugly and are limited in their ability to create a friendly environment for the contestants.

RockTest v. 2.0 Screenshots

Team's Console	Team's Console Setup Dialog	Judge's Console

Does RockTest solve all the problems of running and administering programming competitions?

No. See this old joke from Boys' Life magazine.

What does RockTest v. 2.0 do?

• For Contestants:
• Names user program and data files.
• Launches the user's choice of editor.
• Launches the appropriate compiler (for compiled languages).
• Runs user programs, redirecting the user's data file into standard input.
• Optional non-graphical programs, such as the pico or vi editors, are opened within windows which remain open until the program is terminated.
• User programs are run at low priority and are automatically killed if they exceed the specified run time limit. Limit may be set (up to 5 minutes) by users.
• Users are prevented from opening several editors on the same file.
• Users can submit programs for judging just by pressing a button.
• Users can view judges decisions as soon as they are made.
• Users can request clarifications of problem statements and instructions.
• Users can view all judges' clarifications as they become available.
• For Judges:
• Determines whether there are programs waiting to be judged.
• Fetches the oldest program file.
• Displays the program file's timestamp, language, team, and problem number.
• Compiles the program file (for compiled languages).
• Runs the user's program against the official data file.
• Displays the output of the user's program.
• Displays the user's program file.
• Displays a listing of the user's directory. This helps figure out what is wrong when contestants insist that a program exists but the judge cannot retrieve it.
• User programs are automatically killed if they exceed the specified run time limit. Limit may be set (up to 5 minutes) by judges.
• Judges can clear contestants' directories of program files before the contest period begins.
• Judges can automatically fetch data for the practice contest and for the official contest.
• Judges submit their rulings at the click of a button.
• Determines whether there are clarification requests pending.
• Clarifications are submitted electronically.
• Both contestants and judges operate from a graphical environment.

What tasks are not automated by RockTest ?

• Registration of contestants and assignment of teams to workstations.
• Starting and stopping of the competition.

What bugs and limitations does RockTest have?

• Rocktest has proven itself highly reliable, at least prior to version 2.0. We used RockTest v. 1.2.x thrice, with numbers of contest teams as high as 40, with no faults that could be attributed to code logic errors. RockTest 2.0 had problems locking other judges out after the first judge's copy locked a critical section and failed to unlock it, but this bug is believed to be fixed in RockTest 2.0.1.
• RockTest is written in Python/TkInter, and runs on Linux or Unix. Although Python/TkInter is platform independent, RockTest invokes various Unix shell commands, making it essentially a Unix application. All parts of RockTest (contestants', judges', and tabulator's consoles) must run on the same computer under timesharing, so it would be difficult to make it run under Windows. However, user workstations may be Windows computers, provided each workstation has an X-server such as Hummingbird Exceed or CygWin. (tm).
• RockTest uses large numbers of concurrent X-sessions, placing considerable demand on the network. Most processing of contestants' programs, compiling, interpreting, and running, takes place on the compute server, requiring adequate memory and (a) fast processor(s). We have dual Pentium-Xeon server with RAID 5, 100 Mbps Ethernet to the desktop, and a gigabit backbone, with the server located across campus from the workstation lab. After version 1.0, this worked well, with excellent response times, with as many as 43 teams concurrently.
• RockTest has no facility for timing a contest. In practice, we start the contest by handing out the problem set. We use the compute server's clock as the official time, and judges do not consider programs unless their timestamps indicate that their last save occurred before the closing time of the competition.
• RockTest runs user programs under "nice" so they run at low priority, and in a separate thread.
• Unresponsive programs are automatically killed when their time is up. Programs with endless loops cannot significantly degrade performance.

What's cool about RockTest?

Programming contestants like RockTest. After its first trial run, several students asked if they couldn't use RockTest as an IDE for coursework (rather than C++ Builder or Visual Studio). Students who had experienced programming contests using PC^2, sneakernet , and scripts found RockTest a much more satisfying experience.

RockTest solves all of the tough problems of running programming contests. RockTest names the students' files, so the contestants won't misname them. It decides where to store them, so the judges' program can find them. All data file access is through I/O redirection, so contestants' data file names don't have to match the judges' data file names. For the judges, programs are easily fetched, compiled, and run by RockTest. Judging turnaround is rapid.

RockTest accommodates multiple programming languages. Currently, it incorporates scripts to handle Bywater BASIC, C (gcc), C++ (g++), FORTRAN (gnu f77), Java (gcj or Sun JDK), PHP, Pascal (fpc), and Python. Contestants can even change programming languages between problems or even between submissions of a single problem.

What are the conditions under which I can use RockTest ?

RockTest is free software, licensed under the General Public License. In brief, this means that you may use RockTest for any purpose whatever, you may make as many copies of it as you wish, you have a right to obtain the source code from the person from whom you have obtained RockTest, you may distribute RockTest to others, but that any modifications you make to RockTest that you distribute to others must be covered by this same license. In addition, since I do not charge for RockTest, there is absolutely no warrantee. (Should there be conflicts between this summary and the actual text of the General Public License, the actual text prevails.)

Downloading RockTest

RockTest consists of several files, available separately or packed into a single archive. None of the files is very large; the three Python programs, rt_team, rt_judge, and rt_tabulate contain a total of about 3100 lines of code, about 115 kilobytes.

The HOWTO tells you how to set RockTest up and start a contest. It contains instructions for the system administrator, contest master, judges, and tabulator.

The 2.0 HOWTO is in Open Document Format, which is readable by OpenOffice.org 2.0 and several other word processors. The submit slip and versions of the HOWTO prior to 2.0 are in OpenOffice 1.0 zipped-XML format. If you don't have OpenOffice.org, click on the link and get it--it's free, but large, and there are versions for Unix, Linux, Windows, and Macintosh. Otherwise, read the .pdf file with something like Kghostview or AcroArch.

Trying Rocktest?

Go to:

SRU High School Computer Programming Competition home page

Professor Conlon's home page

SRU Computer Science Department home page

Slippery Rock University home page

Last modified: March 11, 2006

For more info, contact Michael P. Conlon, Ph.D.