CS180: Debugging

One major point that CS180 seems to be lacking teaching students about debugging--understandably so, because it seems like debugging should be a fairly intuitive process. However, when it comes down to programming projects, plenty of students come in to get help with the same situation: They've programmed their entire project, but it doesn't work. The degree to which their programming is hopeless varies by student, but in general, consulting TAs can get exasperated with bad programming practices and helplessness in regard to fixing bugs, be it at runtime or compile time.

One hopefully helpful way to attack this problem will be to have a lab about debugging, which will be fairly early in the semester. What I'm interested in hearing is what people think should be included in a lab like this. Since the lab is planned to be early on, it's difficult to have students attack complex scenarios and fix bugs in them. Perhaps the only thing we can teach them is the "development cycle," which seems to be: edit -> compile -> fix compiler errors -> compile -> run -> fix runtime bugs -> etc. This is pretty natural to students, and I haven't really seen any students who didn't understand that cycle. The problem really lies in the fact that they have no idea how to fix either type of bug.

In most cases, Java tends to spit out fairly reasonable error messages, which means students should be able to figure them out. Do they even bother to read them before calling a TA over? Sometimes I don't think they do--perhaps it would be in our best interest to have them sit on it for awhile, first. I seriously doubt it's difficult to understand what "missing semicolon" means. Another thing that needs to be pointed out is that compile time errors need to be fixed from the top, not bottom. Since one error (particularly syntax errors) can cause an explosion of false alarms, fixing from the bottom is often unhelpful and incorrect. Even though a compiler may spit out 100 errors, that doesn't mean you have 100 errors to fix!

Teaching students to fix runtime errors is, in my opinion, an order of magnitude more difficult. Part of it lies in the fact that some students will have the wackiest implementations you'll ever see that really make no sense at all, but also somehow work--most of the time. JDB is a potentially helpful tool, but this probably isn't something we want to teach this early on (I think it sucks compared to GDB, anyhow). Other ways of finding errors would be via unit tests, print-and-seek methods, or main methods for testing. The first has not been taught in CS180 at all (or in any CS class, for that matter), so the second is usually what I tell students to do. However, they seem to be unable to insert their own prints in strategic location. This should probably be mentioned in the debugging lab, since it's the easiest to teach. Writing main methods that do testing for you is probably a good idea as well, but this isn't as applicable until students get into projects with multiple classes.

The more and more I talk about this topic, the more I see that there might not be a whole lot that could be put into a debugging lab, besides reading. If the lab is to be early on, then we can only teach them a small amount. However, maybe it would be good to write a series of articles on more "advanced" topics for later projects? Good programming habits aren't really taught in CS180 either, for who knows what reason. For example, programming in incremental stages and testing (at the very least, compiling) at each stage will help reduce the bug fixing stress students experience if they try to program all of a project before compiling. Another thing to point out is that having methods be fairly small/contained is a good idea. It could make finding runtime errors a bit easier, compared to having a main method that's hundreds of lines long.

I also thought that having some sort of debugging exercises later on via CS192 might be helpful, but I have no responsibility in that part of the course planning (and at this stage, I don't really plan to). For now, we'll have to assume that we have to teach them everything they need to know about debugging in one very early lab, and put the reset off for individual reading later. Along that note, someone (Ryan, perhaps) mentioned that having a centralized repository for this kind of documentation would be good--would we have any contributors if this were to be set up?

For the tl;dr (or those too confused by my post that should be edited, but won't be), answer these questions:

• What should be in a debugging lab?
• What shouldn't be in a debugging lab but is still well worth knowing?
• Are you willing to write about these things for a central repository?

CS180: Writing easy-to-grade labs

Maybe I'm having delusions of grandeur, but I think this should be possible. I'm currently envisioning a method of writing labs that allows a script to perform automatic grading (yes that's right, I've gone mad) for the ease of the lab TA. A little adjusting to how comments in the skeleton are done can go a long way. I'm kind of lazy, so I'll be using my super-trivial code example that I ranted about some time ago, with two TODOs added:

public boolean foo() {
    boolean b = bar();
    if (b == true) {
        /* TODO 1:  Return true */
        /* END TODO 1 */
    }
    /* TODO 2:  Write an else statement that returns false. */
    /* END TODO 2 */
}

Usually the skeleton code will consist of a bunch of blanks you need to fill in, so if we specify and ending line for said blank, then you can tell what the student typed in for the TODO. This is a pretty simple approach, and allows you to swap out TODO implementations (say, a student's implementation for the solution's), which can be done by a grading script. This method would also take away the problem of TODOs that don't compile by default, which is an issue for students who can't finish their labs. You can test one TODO at a time, using the solution for the rest of the program.

Edit:This doesn't really take away the problem of TODOs that come non-compiling, since students can't test their program until it does. Duh. More on this later, maybe. Suggestions welcome.

Theoretically, this idea could be extended to make a script generate the skeleton code as well, since it seems trivial to extract the text between the comments and call it a skeleton:

public boolean foo() {
    boolean b = bar();
    if (b == true) {
        /* TODO 1:  Return true */
        return true;
        /* END TODO 1 */
    }
    /* TODO 2:  Write an else statement that returns false. */
    else {
        return false;
    }
    /* END TODO 2 */
}

I can imagine creating a directory structure for creating labs (kind of like Rails...except not):

solution/
doc/

From this, the lab spec would be written inside doc (how this will be done is yet to be decided), and the solution in the solution folder. Running a script after the solution is finished would:

1. Extract all of the solution text from the TODOs to create the skeleton in a skeleton folder
2. Create a tests/ directory that has a subdirectory for each TODO, where the author can insert Java programs to test that TODO
3. Generate the lab spec in HTML format in a www folder

This entire structure would be mailed out to all of the TAs, while the skeleton and www folders would be what was exposed to the students. When it came time for grading, you could have a generic script (I hope) that would take a test directory and a solution directory and run tests for each TODO on every student's files, generating text that would be mailed to students.

While automation would probably speed up grading by a lot, I still think there's room for grading things by hand. Emphasis on style, such as proper indentation, non-terrible horizontal whitespace) is much more difficult to automatically grade (if someone wants to write a script to do it, be my guest). Granting partial credit and giving advice for writing better code also has to be done manually, and I think it would be good if TAs at least spent time on this once every few labs (perhaps this should be designated; every third lab, maybe?). This process also doesn't help if we create write-from-scratch labs.

I think I've rambled enough for now; comments welcome, etc.

CS180: Writing and grading labs

Part of my summer goals for CS180 include improving the way labs in CS180 are done. There seemed to be a lack of standard in handling labs over the semesters that I taught, which actually weren't a huge deal, but I think it's a problem worth addressing. Different TAs tended to write their lab specs in different formats, which could sometimes be confusing; some TAs also wrote labs that were hard to grade, which ended up being annoying for all of the TAs. Specifically, the goals I have are:

• Write a spec writing guide (especially since I may be the only returning lab TA)
• Write a lab spec template and source file template
• Write a few labs based on the specs
• Write a lab "best practices" guideline (slightly unrelated)

In addition, a change that will be made this semester is that the last few labs (or at least some labs) will be written from scratch. A complaint I heard was that CS240's labs were hard to deal with, since many of them were written from scratch. This is a silly problem, and students should be learning how to build a program from the ground up.

Grading is also a worthy problem to address; there is no standard for TAs to grade by, which leads to a natural unfairness in the lab grades. I think expecting the lab author to also create a grading rubric is reasonable, and should help TAs grade more fairly and more quickly. If labs are able to be script-graded, I would expect the TA to write a script for it as well.

Another aspect of grading comes in how the student sees how he or she did, and what went wrong. Traditionally, when us old people took it, TAs would print out labs and hand them back; I tried this last semester, and it was a bit cumbersome to manage all of the paper (and wasteful, in my opinion). CS158/9 had its TAs mail students back with comments, which I thought was a better idea, but the way they had us do it kind of sucked.

For the TAs, I think the improvements for a major part seem to boil down to scripting. I imagine Python scripts could be handy for writing grading scripts, among other things. For one, the best way to generate a uniform HTML page for the lab spec is probably to parse some markup that generates the HTML for you, rather than force everyone to hand-edit a bunch of HTML. Or maybe I'm just deluded. Python could probably make a lot of things easier, but more on that later. Hopefully.

Comments on any of the possible changes are quite welcome, as I'm kind of rambling.