I opt for the actual drivers from the chip maker myself (Nvidia or ATI) unless it causes a problem, I generaly don't bother with the drivers, unless they are the same version or pretty close to it. So far, no problems. Yeah, I may be loosing out on some of the extras that the manfacturer's drivers may have, but I've found that some manufacturers don't update their drivers as often as they should or don't always have the most up-to-date stable drivers.
So for me, it's chip manufacturer's and then the card maker's drivers if the chip maker's drivers don't work. Plus, I don't like some of the crapware that some manufacturers make you install when you install your drivers (some of the game trials, or extra utilities that I'll never use).
Just my two cents. It's software, so I doubt that one way or another, you not going to damage your card, unless it screws with your fan speeds and the card overheats, but I've never seen that happen yet.