I got my DSO Quad hobbyist class digital scope last week, and I was all set to write up a really scientific post documenting my construction and testing of a shielded AVR cable, then something strange happened. I switched to another 2x5 "unshielded" JTAG cable to reproduce my original problem. When I hooked up my DSO to the JTAG clock line, the programming functionality worked flawlessly at 5 Volts! Curse you, Heisenberg! (Yes, I know that this is not really the meaning of the Uncertainty Principle, but any time I run across a case where the act of measuring something alters the quantity I'm trying to measure, I think it it).
So attaching a scope probe to the clock line made my problems go away. What do I know about oscilloscopes? Not much, but I do know that the lead wires can have capacitance. Interesting. Since I happened to have a bag of 20pF caps handy (really, handy actually, in the bag labeled "C7" in the pic above!), I tried adding a 20pF to the clock line on my JTAG. Wouldn't ya know it, that's all it took to make my JTAG setup happy. No shielded cable needed!
I'm still interested in getting a more scientific analysis of the original problem. Supposedly, 10x oscilloscope probes have lower capacitance than 1x, so that might be a way to attach my DSO Quad and observe the original ill formed clock signal.
Here's a capture I took with the "shielded" cable, when I was still just playing around with the Quad: