I wouldn't place a GeForce 6600/6800 or Radeon X800/850 of any flavor into that unit.
Reason: Your CPU won't be able to process faster.
I'm thinking that maybe, instead of a new GPU (unless there's a problem with it), a better playback software would do the job.
I built a "Media Center" for PVR duties and DVD playback from "scrap" parts lying around. It's a spare AthlonXP 2000, with an nForce2 motherboard. The video card is a GeForce3-Ti200 64Mb. WinDVD Platinum is my playback software. And laughably, the DVD is an Artec DVD-ROM. There is no stuttering, skipping, etc.