That is true. It's all in the hands of the distributor. Actually, that's an interesting thought. Wouldn't it be great to create a system where software is accepted by a series of people, and provided mathematical proof that each of these people okayed the changes? That would be a safer way to distribute software. You never know what kind of stuff the distributor (or packager) might wish to slip in.
That is called Change Management and it is an integral part of any corporate software production process. The implementation
varies, of course, and that is what determines the quality of the finished product.
What if the md5sum of each file, after being compiled with that specific version of code, was listed, and after comparing the hashes, people report that the binaries are safe? (You'd have to compile it under the same conditions if you were to check the md5sums, of course.)
The only thing that an md5 sum can tell you is that the contents of the file have changed. You're proposing a system that provides the same output (either 'changed' or 'not changed) if a comment in the code has changed or if a new function was added called 'stealCCandBankInfo'. This is very inefficient and wouldn't survive in an environment that produces more than 100 lines of code.
My point is that only free software can be observed for blatantly malicious code.
Google 'black box testing'. You don't have to know HOW something works to determine IF it works and WHAT it does.