Maybe It's Time To Stop Letting DxOMark Decide Whose Smartphone Camera Is "Best" (Opinion)

The Samsung Galaxy S7 has the best mobile camera according to image and video quality testing firm DxO Labs, with an overall score of 88 points out of 100. It received a 90 in exposure and contrast, an 83 in color, a 94 in autofocus, 91 in texture, 89 in noise, 79 in artifacts, and 86 in flash. This all sounds very official. And we see DxO scores increasingly-cited and posted as news around the internet because of that absolute, highly-comparable set of values they provide (Android Police has posted such stories - you'll get no argument from me). We did not, however, post an item about DxO's leaderboard-topping score for the S7.

In fact, the Galaxy S7 and S7 edge do have very good cameras. Some of the best (perhaps the best) on any mobile phone out there. I was thoroughly pleased with the vast majority of images the devices captured. Some of the shots really were, at least to my untrained eye, stunning.

And DxO agrees - this is a very good mobile imaging system (again, the best). What, then, is the source of the headline for this editorial post, if we are in general agreement? Quite simply: that DxO has an open conflict of interest (they certainly don't hide it) in judging such things given their financial interest in the whole ordeal, and should probably not be blindly trusted as a source of objective performance data on smartphone cameras.

DxO Labs is first and foremost a consultancy. Their business model is based upon being paid to conduct analysis and optimization of imaging system performance by the companies that make those systems. DxO is not an independent journalistic entity or trade organization - it is a for-profit software consultancy for camera makers. Their product is called DxO Analyzer, and licensees of this suite can opt to purchase "installation, training and consulting services."

This is not to say DxO has any sinister motive. It is to say, though, that DxO's own profitability - nay, its continued existence - is inextricably tied to a model that requires it to convince camera makers that its services and software are necessary for maximizing camera and imaging performance and, in doing so, trusting that DxO understands and objectively assesses what makes a superior imaging system. Except, we can probably assume that isn't really the only thing that drives camera makers to use DxO's services.

The more DxOMark scores are cited in popular media, the more pressure there is upon smartphone and other camera makers to submit devices not only to DxOMark's testing, but to purchase their consultancy and other services to maximize their end score to beat competitors. It almost goes without saying, but DxO makes no promises that the cameras it judges have not been tuned to maximize their scores under DxO's benchmarking tools by DxO themselves. Products that have received versus not received consultancy and tuning from DxO Labs are not identified, and so it is impossible to know which camera has likely been tuned to maximize its score under the test conditions versus which tends to do well without having specifically been adapted to DxO's parameters.

This also gives DxO Labs the power to silently "shame" the companies that choose not to license its software or services. Perhaps some phones never get the benefit of a test at all, given that they are not licensees or do not directly submit their devices to DxO. Perhaps others would not have received such excellent scores without the assistance of DxO themselves in maximizing their performance under the test conditions and values DxO design, implement, and weigh. We do not know that they abuse this power, simply that the power exists, and that the capacity for abuse is clearly present.

Do the Galaxy S7 and S7 edge have great cameras? Absolutely. Maybe even the best of any smartphone to date. And I am not saying there is anything about DxO Labs which should give you reason to dislike them. I am saying that giving DxO awardees headlines for winning tests that DxO writes, evaluates, and offers paid "tutoring" for is something that we should take more seriously going forward, and that we should not to be afraid to question a test whose parameters are not openly published or freely available.

Source: Maybe It's Time To Stop Letting DxOMark Decide Whose Smartphone Camera Is "Best" (Opinion)