CS180: Writing easy-to-grade labs

Maybe I'm having delusions of grandeur, but I think this should be possible. I'm currently envisioning a method of writing labs that allows a script to perform automatic grading (yes that's right, I've gone mad) for the ease of the lab TA. A little adjusting to how comments in the skeleton are done can go a long way. I'm kind of lazy, so I'll be using my super-trivial code example that I ranted about some time ago, with two TODOs added:

public boolean foo() {
    boolean b = bar();
    if (b == true) {
        /* TODO 1:  Return true */
        /* END TODO 1 */
    }
    /* TODO 2:  Write an else statement that returns false. */
    /* END TODO 2 */
}

Usually the skeleton code will consist of a bunch of blanks you need to fill in, so if we specify and ending line for said blank, then you can tell what the student typed in for the TODO. This is a pretty simple approach, and allows you to swap out TODO implementations (say, a student's implementation for the solution's), which can be done by a grading script. This method would also take away the problem of TODOs that don't compile by default, which is an issue for students who can't finish their labs. You can test one TODO at a time, using the solution for the rest of the program.

Edit:This doesn't really take away the problem of TODOs that come non-compiling, since students can't test their program until it does. Duh. More on this later, maybe. Suggestions welcome.

Theoretically, this idea could be extended to make a script generate the skeleton code as well, since it seems trivial to extract the text between the comments and call it a skeleton:

public boolean foo() {
    boolean b = bar();
    if (b == true) {
        /* TODO 1:  Return true */
        return true;
        /* END TODO 1 */
    }
    /* TODO 2:  Write an else statement that returns false. */
    else {
        return false;
    }
    /* END TODO 2 */
}

I can imagine creating a directory structure for creating labs (kind of like Rails...except not):

solution/
doc/

From this, the lab spec would be written inside doc (how this will be done is yet to be decided), and the solution in the solution folder. Running a script after the solution is finished would:

1. Extract all of the solution text from the TODOs to create the skeleton in a skeleton folder
2. Create a tests/ directory that has a subdirectory for each TODO, where the author can insert Java programs to test that TODO
3. Generate the lab spec in HTML format in a www folder

This entire structure would be mailed out to all of the TAs, while the skeleton and www folders would be what was exposed to the students. When it came time for grading, you could have a generic script (I hope) that would take a test directory and a solution directory and run tests for each TODO on every student's files, generating text that would be mailed to students.

While automation would probably speed up grading by a lot, I still think there's room for grading things by hand. Emphasis on style, such as proper indentation, non-terrible horizontal whitespace) is much more difficult to automatically grade (if someone wants to write a script to do it, be my guest). Granting partial credit and giving advice for writing better code also has to be done manually, and I think it would be good if TAs at least spent time on this once every few labs (perhaps this should be designated; every third lab, maybe?). This process also doesn't help if we create write-from-scratch labs.

I think I've rambled enough for now; comments welcome, etc.