Do you guys find that it saves time since you are not reaching for the mouse?
It is one of those tools where you need to spend 15 to 20 minutes going through the built-in tutorial which explains how to use the shortcut keys. Then maybe another half-hour spent reading a tutorial for how to setup vim for use with whatever language you're planning on using for your project in order to become proficient. And yes, after you've become familiar with the keyboard shortcuts, it is often faster to forgo the mouse. (This is pretty much true for all keyboard shortcuts, right?)
Personally, I use Emacs for non-Java development (primarily common lisp and ansi-c) or for churning out a quick text file. When using Emacs, I turn off the mouse pad on my laptop because the time wasted by hitting the pad on accident far outweighs the benefits of having a mouse. Plus, I find the mouse pointer extremely distracting when I'm typing or coding... perhaps that is just me though.
Vim is actually a good tool to know just because you'll often find Unix boxes where either emacs OR vim is installed, not both. Part of the fallout from the war I suppose... =)