I would say it depends on where your interests lie. If you're interested in the hardware side of computers (circuits, analog/digital, etc.), then choose electrical engineering, or computer engineering (bridges the gap between computer science and electrical engineering). However, if you're primarily interested in the software side of computers, then choose mathematics.
Bottom line, minoring in electrical engineering will make you more of a generalist, while majoring in mathematics will make you more of a specialist (relative to each other). Neither path is bad, and both types of people are needed, it mainly depends on what you're interested in.