Print
There’s common mistake that rookie benchmarkers often make when starting out. Many people think that running one benchmark is enough to tell you everything about one type of component. But all a single benchmark will tell you is how a particular component performs that one benchmark. And of all the parts in a PC, the one that’s the most difficult to judge is the CPU. Even with the GPU encroaching on it, the CPU continues to be the king as the vast majority of apps still rely on it for the heavy lifting. From photo editing, to video editing to 3D modeling, anti-virus scans, and decompressing files, the CPU continues to be the go to part that most applications seek out. How you go about testing the CPU really depends what kind of performance you want to test for. Floating point performance? Integer performance? How fast does it encode video or play certain games?
One other key element to consider before you benchmark your CPU is multi-threading. Just as very few applications exploit all of the threads available in a processor, very few benchmarks do as well. That is changing but you’d be surprised at the number of benchmarks that fail to measure the performance of a modern quad core.
FutureMark’s PC Mark Vantage is one of those benchmarks that has one foot in the real-world and another foot in the synthetic. The test uses workloads derived from the apps that come with Windows and traces of common hard drive loads to discern computer performance. The upside is that it uses test scenarios based on “real apps” and is actually a pretty fair estimate of computer performance. Many of the tests blend single-threaded apps with multi-threaded or run multiple applications to gauge multi-tasking performing.
The downside is that not very many people actually use those freebie apps that Microsoft bundles with its OS. Another downside is the abstracted score which makes it hard for people to want to see a pure “hard drive” or “CPU” performance number. Don’t take that all as a negative though. While Windows Photo Gallery doesn’t necessarily translate into direct performance in Photoshop CS4, but we haven’t seen it come down on the wrong side of a CPU test. Generally, we’ve been pretty pleased with PC Mark Vantage. It produces fairly reliable numbers that seem to jibe with other benchmarks. The fact that it’s multi-faceted also gives you a nice way to quickly gauge your machine’s performance.
One more thing to be aware of: the overall score that PC Mark Vantage produces is based on a set of test criteria that is unique to the PC Mark Vantage tests. Individual tests suites for Memory, TV and Movies, Music, Gaming, Communications and Productivity are actually different tests than the overall PC Mark Vantage Score.
Futuremark has a free version of PC Mark Vantage that lets you run it once on your machine. It’s a bit of a pain since you can only view the results online and you have to request a trial-key to run the utility. For additional runs, you have to pay a modest fee of $6.95 to run the main PC Mark Vantage suite. The other suites will cost you $20 and allow you to change some benchmark settings. For the most part, the main PC Mark Vantage suite score is the one that most people care about. Keep in mind, that there is a 64-bit and 32-bit version and you should compare only 64-bit to 64-bit when trying to compare one machine to another.
Running the test is simple--select the 64-bit or 32-bit icon from your desktop after downloading and installing it, and click run benchmark. You’ll be asked to request a trial key and have to supply an email address. Once you’re done, the results will take you a web page where you can see how fast your machine is and also get a reality check by seeing a score from the fastest machine compared to yours.
One of the heaviest workloads you can put on a CPU today is 3D modeling. In 3D modeling, performance is the difference between getting the project done on time or not at all. There are benchmarks for Autudesk’s 3Ds max, Newtek’s Lightwave and other pricey applications, but you have to have licensed copies of these applications which can run into the thousands of dollars. Fortunately, there’s a cheaper way to gauge how a particular system may perform at 3D rendering. Maxon’s Cinebench R10 is based on the company’s rendering engine used in its Cinema 4D modeler. Most 3D modeling is floating point intensive. Again, keep in mind that Cinebench has a 64-bit mode and a 32-bit mode. The benchmark also lets you test it in single threaded or multi-threaded mode.
To run it, simply install it and launch it. Select Rendering 1 CPU to run a single-threaded run or Rendering X CPU to run a multi-threaded test. Like, PC Mark Vantage, the results are expressed as a numerical score--the higher the better. Cinebench 10 is nicely multi-threaded and takes full advantage of today’s many core CPUs.
POV Ray is another popular 3D rendering benchmark that’s available for free from povray.org. One thing you need to know: make sure you download the 3.7 version from the beta page. This is the only version that is multi-threaded. Again, make sure you download the correction version for your OS: 32-bit for 32-bit and 64-bit for 64-bit. To run it, simply click on the render menu item, select run benchmark all CPUs. The score you’re most interested in is the CPU time which is expressed in seconds in the purplish part of the window.
Comments are closed on this article
Philips
October 25, 2011 at 6:12am
Thank you for the casualty of providing all of these information. This is a lot actually. These are very helpful.
Azrael808
February 16, 2011 at 8:13am
This is a great article; I'm currently using this as a basis to benchmark my home machine so I can see how it stacks up against other rigs, but mainly so I can see how much an upgrade improves the overall system. Some useful comments too! :)
I did have one question: when taking an average of the benchmark runs, why do you use the median, and not the mean score? Is it simply because a mean score will most likely produce a result that wasn't actually obtained by a run of a utility?
jlfrank83
January 27, 2010 at 8:12am
The article states that real-world performance is what you are shooting for when you run your benchmarks, Yet they also want you to disable antivirus, disconnect from your network, manually launch idle process, etc, before benchmarking. Isn't creating a "virtual clean room" tainting your results as much as a synthetic benchmark application or biased video driver would? I don't go through all these preflight steps before playing a game or running Photoshop, so why would I do them before benchmarking and expect realistic, real-world results?
mrvander
January 27, 2010 at 8:40am
... one has to eliminate as many variables as possible in order to produce repeatable results. This is a basic of any scientific testing method - whether it be life sciences or computer sciences. Once you can produce a repeatable result - you have established a control (or a baseline.) THEN add in the other variables to determine the effect it has, preferably one at a time. Turn on your virus scanner. Repeat the testing and compare it to your control. Analyze the data. Add in another variable, say your network. Repeat and rinse.
This is also why they don't recommend using FRAPS and just running through a level of your favorite game. It's not accurately reproducable. It can give you a general idea over many runs, but it can vary wildly.
Of course you can customize this to your heart's content and maybe just a general idea is all you need. The article does mention the idea of "play more games" as opposed to just endlessly benchmarking one's PC and analyzing every last little detail to get that extra tenth of a framerate.
nekollx
January 26, 2010 at 6:06pm
Daz 3d is good real world test since the preview/maniptlation window is GPU bound but renders are CPU bound
------------------------------
Coming soon to Lulu.com --Tokusatsu Heroes--
Five teenagers, one alien ghost, a robot, and the fate of the world.
Modred189
January 26, 2010 at 5:55pm
No furmark?
It's one of the most stressing GPU-only benches I know of.
And I must say, real-world benchmarking your graphics card is the best way to go. Pick a game or two you know well (HL2 works well), and play through a level a few times, measuring your fps with fraps. Throw the data into excel and average the columns. You can even make nice graphs from multiple data sets and compare them to show what scenes your PC is best at and which it has issues with.
jlfrank83
January 27, 2010 at 8:19am
Your columns are going to be radically different if you are controlling the action yourself. It's better to use a game that has in-game, real time rendered cutscenes, like Mass Effect or Assassin's Creed. If you use a playable section of a game, your graphics card will be rendering differently every time, since each playthrough is going to vary in length, FOV, etc.
Modred189
January 27, 2010 at 5:54pm
I have done this a few times before, and if you play through the level a few times first, you can get it relatively reproducible. The problem with in-game benchmarks is that they disable a lot of things like AI, which is a big CPU drain.
bikerbub
January 27, 2010 at 10:39am
GTA IV also has a benchmark option. it's not very long, but it is an actively rendered test with AI and lots of gunfire. also somewhat humorous, as you are a clown (if i remember correctly) riding on a moped, through the streets, weaving in between the Liberty City skyscrapers.
