Sport & Auto
- About Future
- Digital Future
- Cookies Policy
- Terms & Conditions
- Investor Relations
- Contact Future
In talks with the hardware press just before showing off Intel’s new Bay Trail part, principal Engineers Ronen Zohar and Francois Piednoel pointed out several “cringe” worthy issues they found in the source code for many benchmarks being used to test tablets today.
The identities of the benchmarks weren’t disclosed but Zohar pointed out several issues with the source of popular benchmarks that don’t actually test what they claim to test. For example, one memory bandwidth benchmark didn’t even stress the tablet’s RAM. Another test used an unrealistic math function that the vast majority of research doesn’t match popular use.
In another test, the developer hoped to create a CPU performance test but going through the source code with the media, Zohar said it was apparent the test didn’t do that. Instead the test only really tested how fast it could update the status bar.
“Your CPU benchmark ‘tests’ how fast you can update the status bar and how fast you can update the clock,” Zohar said he learned from examining the app's source code.
Perhaps worse than inadequate benchmarks is gaming of the tests by vendors. Zohar said in one example he witnessed, running the stock browser on a device and pointing it at a web-based browser resulted in the CPU ratcheting up to higher clocks.
Those who have followed the PC benchmarking scene for ages will feel a sense of déjà vu all over again as the PC went through this too in the early days. Piednoel agrees there are echoes of the early days of the PC when benchmarking was a bit of a wild west.
It’s not just Intel that believes this either.
“I agree with Intel that mobile benchmarking has gotten completely out of hand, and, it does remind me of the 90's,” said analyst Pat Moorhead of Moor Insights & Strategy. A former AMD exec turned analyst, Moorhead said mobile benchmarks have some maturing to do. He also agreed that cheating is rampant but also not quite as black and white as people make it out to be.
AnTuTu is a very popular mobile benchmark
Intel's hands aren’t entirely clean in this either. This summer the company was accused of cheating in the popular AnTuTu benchmark in showdowns with its Clover Trail+ SOC. When run on the popular AnTuTu benchmark, ARM-based tablets would run the full calculation but when the benchmark was run on Clover Trail+ platforms, the benchmark would run the full calculation and then take a shortcut for the rest of the run.
When asked to about the summer dustup over AnTuTu—after having just accused competing ARM vendors of benchmarketing—Zohar chalked it up to legitimate compiler optimization. But, Zohar said, the optimizations had been made many moons ago and made sense. He said Intel had no interaction with the AnTuTu developer except to provide a compiler which was optimized for Intel hardware—not exactly illogical. The optimizations didn’t fabricate numbers he said, the compiler just knew that if the code is asking for the exact same thing it asked for, there’s no reason to waste time and energy since you already know the answer. Why not just give the same answer? When AnTuTu was written to execute the same workload over and over again rather than take the compiler short cut, the Intel CPU actually trailed the ARM chip slightly.
Just days ago AnTuTu Labs, the developer of the AnTuTu benchmark said it has implemented anti-cheating techniques in AnTuTu X.
Moorehead said he doesn’t think Intel’s framing of how it all unrolled doesn’t match what he knows but allegations of “cheating” isn’t exactly unusual.
“I don't know of any relevant hardware company who hasn't been accused of cheating, particularly in CPUs and graphics,” Moorhead said. “The gray area is that one man's cheat is another's optimization. One man's piling on of resources in a benchmark consortium is interpreted as manipulation.”