Regarding documentation, I think that both of you have made good points. First, there needs to be some documentation. Here are the things that I think documentation can help with...
1) At a minimum, I need to know what the code is supposed to try and accomplish.
2) I would like to get a 10,000 and 50,000 foot view of the system.
3) Having good "javadoc" style documentation is important for libs and utility code.
4) Having some kind of record of what approaches were considered/tried and any current problems is also handy.
5) The user obviously needs a manual or help system.
Of course, these are just goals and getting them right is often hard.
1) This can span from being as vague as a company brochure to a very detailed requirements document. I don't think either extreme is very useful.
2) In a changing system, I don't want to actually "write" this up (unless it is small). Tools need to be able to create this view. The problem is that managers tend to think exclusively in a top-down manner... design and doc first, dev later.... whereas most good devs tend to work in both a top-down and bottom-up manner. Of course, it is always interesting to see people's reactions when the expected and actual design don't match up!
3) The pitfall here is that I've often seen devs create code that should go into a library, but they don't want to do this because then they have to document the code, deal with increased bug reports, etc. Not having a good utility library available is almost always a sure sign of horrible systems/code/devs.
4) People are often reluctant to document their failures. However, knowing what approaches were tried to solve a problem is often a good way for deciding on a new approach w/o reinventing the wheel or giving you some additional information to put you further along. Scientists often state that they build upon each others work... more devs need to do this.
5) I really wish they'd just hire Jipstyle to do this!
Of course, the problems with documentation are just enormous:
1) Projects that spend way too much time up front on documenting everything. The goal is to create a system/program that includes documentation. Often, we get this 180 degrees wrong and basically have a whole bunch of documents and a broken system because people are spending too much time doing things like figuring out how to create an index instead of actually thinking of better ways to solve a problem.
2) Managers that insist on using a fixed process for the documentation. Create this document, then that document, then this document... everyone is stuck in this cookie cutter mental box. Different projects require different types of documentation. Duh!
3) The law of diminishing returns applies to documentation. Too much documentation not only wastes time and provides less utility (if you can't find what you need or aren't willing to read through the tomb, what good is it), but it also makes developers hesitant to change the code in a positive manner. It often sucks up time that could be better spent on prototyping or some other useful work.
We can certainly take a lesson from science. Some wisdom from the very quotable Dr. Feynman:
For example, if you're doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it; other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked — to make sure the other fellow can tell they have been eliminated.
How To Solve Unsolvable Problems
The best lecture I recall started out with Feynmann suggesting that he stop the course, because it wasn't really getting anyplace. Then he decided to talk about what he was doing right then, as an example of real research. He was interested in quantum chromodynamics, and the big frustration at the time was that people had a theory, but it was too difficult to evaluate it and predict numerical results of experiments. He explained that in cases like this, it was hard to know where to start. He wanted to "understand" aspects of the theory, and develop intuition. For example, asymptotic confinement (quarks seem to bind together more tightly as you pull them apart).
He told us that he approaches these issues by rehearsal. 'Think of a simpler problem that seems similar. If you can't solve that, think of a simpler one. If you can solve it, then SOLVE IT. Don't just say you know how to solve it. After that, you might think of a way to attack the harder problem. You might realize something.' Basically, keep poking around and give your intuition a chance to develop and wait for ideas to pop into your head.