You'd have to have an absolutely insane traffic load to notice any kind of slowdown, I think. Or, a really, really crappy switch.
Let me put it this way:
My home network. Main switch is a Linksys SD216 (16-port 10/100 switch, most of the house is connected to this.) Second switch is a Linksys SD1008 (8-port Gb switch,) To the Gb switch is connected my main Media Center, 3 HDHomeruns (each a dual HD TV tuner), two Xbox 360s, and the run to my bedroom (where there's another PC used as a Media Center; it's connected to a D-Link DIR-655 running in AP-only mode.) Last port is the connection to the 216.
running (all 3 HDHRs active (read: all 6 tuners in use), both 360s watching TV (Live or recorded, doesn't matter), and my bedroom PC also watching something from the main MC (recorded TV, or a ripped DVD)), I notice no noticeable slowdown on 'other' activity (say, copying a file across the network), nor do I notice any stuttering on anything that's playing.
Mind you, I (and, everyone else in the house) pretty much only watch & record HD content, and my DVD rips are full-bitrate copies (I do a straight rip, and convert to DVR-MS. It's just as big as the main-movie files on the original disc.)
You're overthinking things. You'll be fine.