Forums > Photography Talk > How much do you trust DXOMark?

Photographer

CE Photog

Posts: 244

Columbus, Ohio, US

I've recently been checking out dxomark to get educated on my next lens purchase. I'm looking at something's that say tamron out performed the Nikon version, which is a hard pill for me to swallow. I've heard the site referred to a lot, but never got to check it out until now. What is your opinion of the site?

Sep 08 14 05:46 am Link

Photographer

Worlds Of Water

Posts: 37732

Rancho Cucamonga, California, US

I pretty much base all my camera purchases on their 'low noise' test results... haven't been let down yet... borat

Sep 08 14 05:53 am Link

Photographer

Jay Leavitt

Posts: 6745

Las Vegas, Nevada, US

Overall for HOW they rate scores, they're pretty terrible (Nikon D600 scores a 94, while the Canon 6D scores an 82 - while nikon d7100 scores an 85, so the 7100 is better than the 6d, obviously)

The specific metrics they rate, though, are pretty spot on. Sharpness of lenses, CA, etc. Aesthetic qualities (attractiveness of OOF areas, contrast transition, etc) don't play into that, so hands on tests in the field usually win for me.

However, the new Tamron gear (the 24-70VC and 70-200VC) and Sigma Art line have been fantastic for the current generation, very much on par with Canikon.

Sep 08 14 08:52 am Link

Photographer

Phil Drinkwater

Posts: 4814

Manchester, England, United Kingdom

If you do some research and are able to understand how they combine their scores you'll realise they're largely garbage.

However the raw numbers and graphs seem to be pretty realistic. I find those very useful.

I wrote about it here http://www.phildweddingphotography.co.u … isleading/

I find it odd that people will believe anything they read without checking what it means..

Sep 08 14 01:31 pm Link

Photographer

Thomas Van Dyke

Posts: 3233

Washington, District of Columbia, US

CE Photog wrote:
I've recently been checking out dxomark to get educated on my next lens purchase. I'm looking at something's that say tamron out performed the Nikon version, which is a hard pill for me to swallow. I've heard the site referred to a lot, but never got to check it out until now. What is your opinion of the site?

CE from your BIO

"Hard work beats talent, when talent doesn't work hard." - Tim Notke

I beg to disagree... talent would not be on Mayhem with this query, instead they would be renting the candidate lens of choice to see if it truly addresses there specific needs...

You are shooting a D7000, if Nikon puts out a firmware update they are certainly not going to verify it works for Tamron's glass...

Case in point: I have a colleague with a D700 and an ultra fast Sigma wide angle... he thought it was the best thing since sliced bread until the lens refused to lock focus in low light... Was shooting right beside him with a lowly D300 and Nikon glass... no problem...

Just saying...

All the best on your journey...

Sep 08 14 01:55 pm Link

Photographer

Robb Mann

Posts: 12327

Baltimore, Maryland, US

They use a well-documented and well-controlled test proceedure for both lenses and cameras, and are one of the least-biased test sites on the web. They do not take sample variation into account, but neither does anyone else. Their tests themselves may or may not be relevant to your specific interests. Its worth reading up on their proceedures.

They are also not reviewing the entire item, just trying to provide empirical data on certain aspects. For example, DXO tests camera sensors, but not ergonomics, focus, or other features like video that might matter greatly to you. Using DXO alone to pick a new camera is like choosing a new girlfriend based solely on height.

Sep 08 14 10:49 pm Link

Photographer

Jerry Nemeth

Posts: 33355

Dearborn, Michigan, US

Their scores are not useful for me.

Sep 08 14 11:41 pm Link

Photographer

Armando D Photography

Posts: 614

Houston, Texas, US

I kind of use it, but I use snapsort to compare cameras based on dynamic range / color bit / and sensor size.

Sep 09 14 12:05 am Link

Photographer

Zack Zoll

Posts: 6895

Glens Falls, New York, US

Phil Drinkwater wrote:
If you do some research and are able to understand how they combine their scores you'll realise they're largely garbage.

However the raw numbers and graphs seem to be pretty realistic. I find those very useful.

I wrote about it here http://www.phildweddingphotography.co.u … isleading/

I find it odd that people will believe anything they read without checking what it means..

I would agree with Phil here. Much like DPReview, if you ignore all their numbers and instead focus on their words, they're generally pretty good.

The problem I have with numerical rating is that something needs to be a benchmark. And as technology improves, that benchmark needs to change. So unless the benchmark is the same as is used by optical engineers(DxO's is not, which is why many of us can understand it), then any numbers are only useful if you're going to go back and re-rate every camera every year.

But if you just read the written reviews, and seek out another couple sources, DxO makes lots of valid points.

The new 'pro' Tamron and Sigma lenses are indeed as good as the CaNikon versions, which has a lot to do with Canon dropping their prices. Personally I still feel that the colour and look of the OEM lenses is more consistent from lens to lens ... But if you're only planning on using zooms or just primes, CaNikon isn't offering you anything you can't get for less from Tamron or Sigma. And if Tamron improves its primes, and Sigma its zooms, then the only advantage of CaNikon will be the name and the resale value.

Sep 09 14 07:27 am Link

Photographer

Kelvin Hammond

Posts: 17397

Billings, Montana, US

-JAY- wrote:
Overall for HOW they rate scores, they're pretty terrible (Nikon D600 scores a 94, while the Canon 6D scores an 82 - while nikon d7100 scores an 85, so the 7100 is better than the 6d, obviously)

The specific metrics they rate, though, are pretty spot on. Sharpness of lenses, CA, etc. Aesthetic qualities (attractiveness of OOF areas, contrast transition, etc) don't play into that, so hands on tests in the field usually win for me.

However, the new Tamron gear (the 24-70VC and 70-200VC) and Sigma Art line have been fantastic for the current generation, very much on par with Canikon.

I own both the 6D and the D610.  At first, I didn't see much difference, but after using both for months now, the D610 does beat the 6D in IQ, especially for noise and low light images.

I did base purchasing both cameras and the lenses I got for them on DxO scores, which did ensure that there were no disappointments. They are pretty good at predicting what a certain camera/lens combo will yield.

That said, I find myself wondering if we used their numbers as a literal guide, and the 6D scores 82, and the D610 scores 94, then it could be said that the D610 has 12% better IQ, which isn't unimaginable.

But they dont score usability....  and that's where the 6D kicks the D610 to the curb.  Canon's "Q" menu is the best nav setup on the market, hands down.  Nikons menu is S L O W and cumbersome. If I have to adjust something on the Nikon during a shoot, their system makes me feel & look like an idiot. Its very frustrating.

Sep 09 14 08:15 am Link

Photographer

American Glamour

Posts: 38813

Detroit, Michigan, US

Phil Drinkwater wrote:
If you do some research and are able to understand how they combine their scores you'll realise they're largely garbage.

However the raw numbers and graphs seem to be pretty realistic. I find those very useful.

I wrote about it here http://www.phildweddingphotography.co.u … isleading/

I find it odd that people will believe anything they read without checking what it means..

I agree here as well.  I find their testing to be quite accurate.   If you look at a specific metric and then compare, based on that metric, you will get useful information.   Those metrics are objective.

The overall rating is "subjective."  It is not that the number is created by a human.  It is that the methodology and weighting to reach that number was created by a human.  It is their own formula which was created subjectively.

So, I use DXOmark all the time, but only for individual tests, not for their composite score.

Sep 09 14 08:24 am Link

Photographer

Daniel Leon

Posts: 1389

Toronto, Ontario, Canada

Those sites are for nerds and bench photographers, i place 0 value in those things.

Sep 09 14 08:26 am Link

Photographer

American Glamour

Posts: 38813

Detroit, Michigan, US

Smedley Whiplash wrote:
But they dont score usability....  and that's where the 6D kicks the D610 to the curb.  Canon's "Q" menu is the best nav setup on the market, hands down.  Nikons menu is S L O W and cumbersome. If I have to adjust something on the Nikon during a shoot, their system makes me feel & look like an idiot. Its very frustrating.

This is slightly off topic but I am going to interject it anyhow.  The thing I find the most interesting is that we seem to have an equal number of people telling us how much better the Canon menus are than Nikon and that the Nikon menus are better than Canon.

At least, in part, that explains why some of us shoot Canon, others Nikon, others Sony, etc.  We really all do have personal preferences.

Sep 09 14 08:26 am Link

Photographer

Zack Zoll

Posts: 6895

Glens Falls, New York, US

Smedley Whiplash wrote:

I own both the 6D and the D610.  At first, I didn't see much difference, but after using both for months now, the D610 does beat the 6D in IQ, especially for noise and low light images.

I did base purchasing both cameras and the lenses I got for them on DxO scores, which did ensure that there were no disappointments. They are pretty good at predicting what a certain camera/lens combo will yield.

That said, I find myself wondering if we used their numbers as a literal guide, and the 6D scores 82, and the D610 scores 94, then it could be said that the D610 has 12% better IQ, which isn't unimaginable.

But they dont score usability....  and that's where the 6D kicks the D610 to the curb.  Canon's "Q" menu is the best nav setup on the market, hands down.  Nikons menu is S L O W and cumbersome. If I have to adjust something on the Nikon during a shoot, their system makes me feel & look like an idiot. Its very frustrating.

Apparently the scores "aren't" out of 100. I use quotes because they were for a while, and now they're not. That 12% difference means nothing, once cameras are regularly getting scored over 100.

The fact that the charts all topped out at 100 is "coincidence".

Sep 09 14 08:28 am Link

Photographer

American Glamour

Posts: 38813

Detroit, Michigan, US

Zack Zoll wrote:
Apparently the scores "aren't" out of 100. I use quotes because they were for a while, and now they're not. That 12% difference means nothing, once cameras are regularly getting scored over 100.

The fact that the charts all topped out at 100 is "coincidence".

I asked that question a couple of years ago.  When the D800 came out, it was becoming clear that sensors would still have to get better and that meant the number would go over 100.  It has and you are correct.  The higher the number goes, the less the difference, as a percentage, between 80 and 90.

Sep 09 14 08:31 am Link

Photographer

Kelvin Hammond

Posts: 17397

Billings, Montana, US

GPS Studio Services wrote:

This is slightly off topic but I am going to interject it anyhow.  The thing I find the most interesting is that we seem to have an equal number of people telling us how much better the Canon menus are than Nikon and that the Nikon menus are better than Canon.

At least, in part, that explains why some of us shoot Canon, others Nikon, others Sony, etc.  We really all do have personal preferences.

I brought it up because the DxO scores are useful, but it's not the only criteria one should use for decision making. Since I own and shoot 4 different brands, it's not a big deal to me, I just match the camera to the job. But my advice to a one-brand shooter would be to get acquainted and familiar with the menu and options as well.

Sep 09 14 12:10 pm Link

Photographer

Lallure Photographic

Posts: 2086

Taylors, South Carolina, US

I believe that you have to stay with what is made for your system, to truly be able to rely on it. Third party items, especially lenses, may not work properly, vs. the manufacturer of the system.

Sep 09 14 01:55 pm Link

Photographer

Phil Drinkwater

Posts: 4814

Manchester, England, United Kingdom

Smedley Whiplash wrote:

I own both the 6D and the D610.  At first, I didn't see much difference, but after using both for months now, the D610 does beat the 6D in IQ, especially for noise and low light images.

I did base purchasing both cameras and the lenses I got for them on DxO scores, which did ensure that there were no disappointments. They are pretty good at predicting what a certain camera/lens combo will yield.

That said, I find myself wondering if we used their numbers as a literal guide, and the 6D scores 82, and the D610 scores 94, then it could be said that the D610 has 12% better IQ, which isn't unimaginable.

But they dont score usability....  and that's where the 6D kicks the D610 to the curb.  Canon's "Q" menu is the best nav setup on the market, hands down.  Nikons menu is S L O W and cumbersome. If I have to adjust something on the Nikon during a shoot, their system makes me feel & look like an idiot. Its very frustrating.

Read my link and it'll explain why you should never use the combined scores. They are totally misleading although at times they might arrive at the correct result. But when they do it's at least partly by accident.

Sep 09 14 03:30 pm Link

Photographer

Carlo P Mk2

Posts: 305

Los Angeles, California, US

Nothing beats renting cameras and doing your own tests under your own terms.

Sep 09 14 03:41 pm Link

Photographer

Kelvin Hammond

Posts: 17397

Billings, Montana, US

Phil Drinkwater wrote:

Read my link and it'll explain why you should never use the combined scores. They are totally misleading although at times they might arrive at the correct result. But when they do it's at least partly by accident.

Looking that over I'd say they rated the D800 higher because it produced better results in many situations, whereas the D1x excelled in DR in low light?  If so, that doesn't seem like an inaccurate representation. One might conclude if you do news or sports, get the D1x, but if you're doing portrait or product or most anything else, the D800.

And that's generally what I see in my area. News always has the D1series, everyone else has Canon5/6/7 or Nikon D700/800.

The combined scores intrigue me. Using DxO I could easily see why my 50D and 7D with the 24-105 and the Tokina 11-16 2.8 were producing less then stellar results. Those combos yielded a combined score of less then 15.   But put a Nikon 12-24 on the D600 and you double the score, and it's noticeably better when pixel peeping. Meanwhile, a 17-40 on the 6D nets you around 19-20, which is in line with what you see on the monitor.

Makes me want to look at the image an Otis/Nikon produces, except that they don't make one in a focal length I'd ever use. Just out of curiosity. But DxO has that rated in the 40's, so it should be stunning.

Sep 09 14 04:07 pm Link