Cheating in benchmarks is not something new in the tech world, especially in the smartphone industry. We’ve seen big brands such as Samsung and Huawei do this couple years ago.
So how does this “cheating” work? Brands use a higher performance mode in running some apps, mostly targeted to benchmark softwares. This produces misleading results and defeats the purpose of benchmarking apps.
Instead of giving users a more accurate data on how well a smartphone performs in comparison with other devices, it only blows up the numbers as it recognizes benchmarks.
Our limited knowledge also tells us that smartphone companies no longer do such thing as backlash is inevitable. Indeed, the adverse effects outweighed the positive ones.
While we’ve seen smartphone brands abandoning this cunning tactic, an interesting report reveals that a chipset company is guilty of such. According to Anandtech, MediaTek has been “cheating” in its benchmarks.
Anandtech found out the whole “cheating” scenario when he noticed that the older MediaTek Helio P95 in the Europe version of the OPPO Reno 3 Pro performed better than the newer MediaTek Dimensity 1000L of OPPO Reno 3 (Chinese model). At that time, he was already using the tweaked version of PCMark which manufacturers don’t have access to in the Chinese version of OPPO Reno 3.
To confirm, he installed in the OPPO Reno 3 Pro (European version) the said version of PCMark. The results show a huge change in the performance scores. There was a 30% drop in general tests, and as high as 70% difference when it comes to specified workloads. This concludes that there was indeed “cheating” on the matter.
To further test whether OPPO or MediaTek made the “cheating”, he installed the same tweaked PCMark software to another OPPO device, the Reno 3 Pro (Chinese version) which uses a Snapdragon 765G. The results show little difference from the first benchmark points. If OPPO did this, it could have done it with all its devices, the case, however, reveals otherwise.
Anandtech also tested other smartphones running MediaTek Helio P95. The results didn’t come as a surprise as there was indeed a huge gap in points.
Moreover, the results only mean that something is happening with devices running Helio P95 that their performances are higher when using the benchmarking apps.
The logic behind this lies in the software that is specific to benchmark apps. What’s even more intriguing is that this does not only happen to popular consumer-centric apps. Even those used by few people were not spared. The consistency also shows how this is not OPPO acting on its own but rather MediaTek doing the magic.
When asked for their side, MediaTek responded to the whole story explaining the said actions.
The MediaTek statement mainly talks about three points:
- Chipsets use higher performance in some workloads and this accusation is just the same case.
- Phone makers have the choice to deactivate this feature if they want to.
- Other companies are also doing this.
The first point is a faulty analogy. The main goal of benchmark apps is to identify how well a smartphone performs in normal circumstances. This is far from what MediaTek claims that this is only the same with other apps that use high-performance mode.
Unfortunately, the other two claims are well-founded. Years back, Huawei, OPPO, and Samsung were accused of cheating. If smartphone manufacturers wanted to remove it, they could have done so but somehow maintained it as it is.
Lastly, the defense that since everyone’s doing it, it’s fine to do it is nowhere near acceptable. Doing a misleading act cannot be justified because others are also doing it. A wrong act is wrong, so is the act of others like what MediaTek claims.
We had high hopes with MediaTek because it showed us pretty good chipsets in the past three years. This whole “cheating” is not in any way helping. Instead of convincing people to believe in it, this only added the doubts that many already have with its products. And instead of putting efforts on stuff like this one, they should put all their hard work in improving the chipsets. There’s so much more to improve and we all want nothing but better products as this will benefit the consumers.
What do you think of this whole MediaTek “cheating” story? Do you still think benchmarks are reliable?