Is this a Radeon card? Because it might be something to do with their way of idling if the card detects the monitor's been off for a while.
If you're on Windows, also make sure all the PCI-Express related power settings are set to maximum performance.
Yes, R9 270x. As I said I've turned it to never power down monitor now, I always do it myself when I get up anyways. Hopefully this helps.
I have the power profiles on performance already, I dont need any slow down with a desktop, but I've not seen any settings specifically for PCI-E... do you know where to find these settings?