[Andrew Alexander]

If you haven't already, check out Dave's new article on testing for lens sharpness - the Fallibility of Focus!

After you've had a good read, come on back and tell us what you think!

- Andrew

[jamesm007]

Andrew Alexander, on Jun 19 2008, 03:40 PM, said:

If you haven't already, check out Dave's new article on testing for lens sharpness - the Fallibility of Focus!

After you've had a good read, come on back and tell us what you think!

- Andrew

Dave first this is my first post here, although I have read your reviews for some time and honestly have come to respect them as the best, especially with lens. I love the blur plot and how it shows not just some average of corner performance or the worst corner, but the whole frame. I often wonder how other testers rate thier lens in a meaningful way, if you just measure the center performance and average corner or worst corner that's not going to tell you the whole story, that's why I love your interactive charts. I have been pixel peeping a long time (lol) and know most people get it wrong when they have never done it before, not just the testing but the interpreting of whats going on. Lots of lens have been sent back to the manufacturer with user created problems. Anyways...

A real problem and a really engenius solution, I liked the article and hope others who are always doing brick-wall tests and pixel peeping read it. You clearly show how critical focus is to getting an accurate result of what the lens is doing. May I also suggest real world tests, pics of the same, to compare lens. Somehow pics that show what the numbers mean.

[terry]

Hello Dave:
A positive step and thanks for the work. However, a couple of points. This information is valuable in evaluating lens choice. However, most of us use the camera's focus system when we shoot. From your research it would appear manual focus is not going to yield optimal focus. (At my age, even with magnification, I always thought the autofocus worked better). And I've discovered that the half press shutter frequently 'hones' the 'auto' focus.

However, the real issue is how to get critical focus (without resorting to the 'yellow sub'). If I'm not mistaken that involves depth of field and manual focus. Luminous Landscape (among others) has some good tutorials on that subject.

Terry

[C.Inoue]

Thank you for the explanation of your lens testing procedure to determine the point of focus. I have been trying to evaluate some old Nikon 35mm film optics for use on my Nikon D60 to see if they have potential as alternatives to the kit lens 18-55VR. So far results with the kit lens indicate it is sharper using the auto focus vs manual focus for the old Nikon optics without auto focus. However, as your tests indicate, manual focus is unreliable for lens testing and therefore I need a better solution to do my evaluation.

Rather than setting up a similar aperatus (Yellow Submarine) I'm thinking that a single exposure of a group of test objects set up in a linear fashion with a slight offset would also determine the point of focus and then an appropriate lens test chart could be placed at the object distance of best focus to evaluate the lens. This method would use slightly different areas of the lens even if the subjects were located close to the center of the field, so a slight error may be introduced.

The tests results indicate manual focus even under laboratory condtions is hit or miss, so maybe experimenting with these lenses is just an academic exercise and camera users should stick to the new AF lenses.

This post has been edited by C.Inoue: 20 June 2008 - 01:20 AM

[laniciffo]

Hi there,

After reading this very nice and objective article, and as it seems
so difficult to achieve good focus, I started to wonder if reducing
to an extreme level the depth of field could help in this difficult task.

So, what about using macro extension tubes to get a millimeter-long
depth of field ?

I remember playing once with a 85mm 1:1.4 lens and some tubes.
On a screw that was a few inches from the front filter, I could
set the focus precisely on any turn of the thread of the screw.
(sorry if this is bad english...)

As the tube shouldn't impact the lens quality, don't you think it could
be another easy way to find the best focus point ?
There would still be the front or back focus issue, though...


Actually, I thought about this to compensate the poor performances
of the autofocus when doing portraits with very low levels of light.
Unfortunately, my shortest extension tube (12mm) is already too long
to be practical : the max focus distance is less than 1m away.
I would have been glad with a tube that would have reduced the
max focus distance from infinity to 2 or 3m.
It would also be a cheap trick to get shallow DOF even with low-end
lenses.

But this is off-topic, sorry.


Anyway, bravo for your methodology, it really qualifies the lens
and the lens only, and that's a real value.

But, being a reader of "Chasseur d'Images" (french photo magazine),
I know that they would object that the focus problems are part
of the real experience of all photographers.
That's why they don't test lenses alone, but always the association
of a lens + a camera. So they routinely publish more than one test
result per lens: for instance they could test the 70mm Sigma macro
lens on a Canon 40D, then on 450D, 1D, ...on Nikon cameras, ...
They also report when poor sharpness obviously results from bad
focusing, which is also informative about the camera's focus perf.


I think that the two methodologies have both pluses and minuses.
They're somewhat complementary.


This also raises the question of the autofocus test : Chasseur d'Images
uses a well-lit moving target I believe (or an optical simulation of such
a target), but I feel it's not enough, especially as I'm interested in
low-light photography.
The low-light performances of the autofocus should also be characterized,
both in terms of accuracy and speed.


Anyway, thanks for your efforts !

laniciffo from France

[kwai]

Andrew Alexander, on Jun 19 2008, 08:40 PM, said:

If you haven't already, check out Dave's new article on testing for lens sharpness - the Fallibility of Focus!

After you've had a good read, come on back and tell us what you think!

- Andrew

An interesting article, and my hat goes off to Dave and co for the time and effort put into both process and explanation of it.

However, I wonder about the practical implications. Since I don't carry out photography like this in the field, I will inevitably suffer from the reduced focus accuracy of either auto or manual focus. This matters because it means that comparing a blur plot created to give optimal sharpness at the sharpest point may not be the most representative way of understanding the performance I will achieve from the lens.

The demonstration that corner lens performance is critically dependent on ultra-fine tuning of the focus effectively means that the performance I get from the lens will inevtiably be randomly distributed over a range of (potentially quite different) performances. So to understand whether a given lens is "better", what I need to know is not its characteristics when it is perfectly in focus but what its average (in some sense, possibly RMS) characteristics are over different degrees of slight defocussing. i.e. rather than show the results from the analysis selected as showing the best focus, it would arguably be more valuable to show a combination of the results across the range of defocussing that in reality will affect all users.

cheers,
Ben

[yandj]

I hope Olympus will bring back automatic focus bracketing.

Dave's report is absolutely right on.

I had reached the same conclusion. That's why the automatic focus bracketing function available in Olympus's earlier DSLRs is so useful.

I was extremely disappointed to see this feature deleted in newer Olympus models. I would have bought an E3 if it had focus bracketing. I hope the utility of automatic focus bracketing becomes more widely recognized, and Olympus (or any camera makers for that matter) will include it in future models (or firmware revisions) they release.

[Dean]

Nice article. A couple of questions and comments:

Do you refocus at each aperture or do you set and forget the focus? I got the impression that each aperture was refocused. If you refocus, do you report the focus variation versus aperture? If you set the focus only once, is it the best focus wide open?

When you determine the best focus, is the focus determined from the center of the FOV or some average across the FOV? I would assume that you use the center.

It would be very nice to report the variation of focus (both manual and auto) - most people will never focus bracket and they will get results that vary. I know that this brings the camera back into the equation, but some lenses are very sloppy and hard to focus while others are quite good.

My biggest wish is that you would test a lot more lenses! Especially Canon.

[touristguy87]

"This matters because it means that comparing a blur plot created to give optimal sharpness at the sharpest point may not be the most representative way of understanding the performance I will achieve from the lens."

...well, you don't really need "the most representative way" you just need a close approximation to what the lens will give you.

It seems that the blur test results are either "best case" or close to it depending on the quality of the focus. That is something of an idea for another test...how accurate is the focus as a function of focal-length and distance to target and how much do the blur-numbers vary as a function of focus (i.e. if the lens is back-focused by 3 units out of 1000 given the focal-length and/or the focus distance [aieee!], what happens to the blur-numbers). The key take-aways here are that 1) AF is not always accurate (which we all know, those of us who have any experience shooting a digital camera 2) MF is even harder to get right than AF (likewise) and 3) the blur-test results have a focus dependence (somewhat obvious otherwise focus wouldn't be an issue at all). You just need 4) the real-world results will be no better than what you see in the results here.

I'd like to see 5) variation from lens to lens. And 6) which lenses AF well and which don't, and under what conditions (ie what level of contrast and lighting is required to get the focus-error to what value). I know for one, that the Sigma 18-200 DC OS is just horrible in terms of AF accuracy at least on a 400D. Raising 7)...but anyway.

But in any case, in practice, we have to use the AF system, so the most useful thing would be to see how many times you have to hit the shutter button to get the AF error down to what size. What are the statistics for that. But overall the DxO blur results here are valid (at least for this particular lens) if the focus is close and if not the blur results don't really matter anyway.

As an aside, I've tried the split prism approach finding it to be "interesting" but hardly a way to get a good focus for the reason that Dave mentioned (it doesn't take much of a positioning error in the screen to give a large focus error) and also there are errors in reading the split-prism itself. It's not a binary device, it's essentially a magnifying-glass for the focusing-screen.

Still this is very good, for a few reasons, now I'd just automate it

a few x-y and rotational motorized stages should make this a lot easier to do.

...still thinking about this from a real-world approach, overall, to be technically precise, the blur results should have to have an associated error which takes into account the focusing error that one would see in real life, as a real shooter would use the lens. It would make more sense to just repeat the blur test using the AF system and give the standard deviation of the results as a percentage of a given blur value.

To say "these are the blur results +/- 1%, taking into account all the measurement errors and systemic errors in the measurement" is how it should be done. Then if one can focus the lens more precisely that error can be removed from the results. But without going through all this, the results don't really map out into the real-world. They're just a guide. But that's all that you need, is a decent guide. I guess that the concise form of what I'm saying is that he is proving that errors in the AF system swamp the precision of the DxO blur results, and what I'm saying is based on that one simply has to take that AF error into account (because AF is the only practical way to focus the lens for most shooters in most real-world situations)...and by finding the point of exact focus that error is being removed, and subsequently the results are really not "real-word". Close, but not quite. Nonetheless a good guide for purchasing a lens.

One still has to consider the AF error in that decision, though, and the question is does that error vary widely from lens to lens across either type or manufacturer or what. It might be better to get a lens with "less impressive" blur numbers based on "optimal focus" that focuses more accurately and precisely especially in the conditions in which the shooter needs to use it. It is very similar to the concern of how the blur numbers vary as a function of F#, focal-length and distance from center. Diffraction effects are another concern and last but not least if the front- or back focus varies widely as a function of any of the above, the user of that lens is in serious trouble even if it has great blur numbers when properly-focused.

This post has been edited by touristguy87: 20 June 2008 - 05:40 PM

[touristguy87]

...then one really needs to be able to correlate DxO blur#s to IQ for this to make any sense in the end...

it would be nice to see them converted to MTF, say. Which is directly convertible to LPH. One can then set up "groupings" of LPH to suit, and then they can say, "if I use this lens in such and such a way, what are the chances that the IQ will be sufficient for my needs".

Otherwise this is just an exercise in hair-splitting.

At the risk of being burned at the stake for excessive post-length

...what happens if the blur test results are run through the blur corrections in DxO? Does it have focus-correction as well? I do know of at least one application that will make small focus corrections. But shouldn't these results be qualified by how well DxO can correct the lens blur? That would be just as interesting to know as what these blur values are. I would be *very* interested to know if they are not fixed. Certainly the blur-correction modules in DxO will depend on what the measured blur is for a given set of lens settings and lens/camera combination...which raises the possibility that all of these blur results have a much larger error than we know because the focus-distance is a large factor and the lens simply cannot be tested across all focus-distances. The blur data should be determined at least at 3 points along the range of focus distances between the near focus limit and infinity, at *each* focal length.

...in the end you end up just ranking the lenses based on overall performance.

You have to think of Hitler and his General Staff...ok he beat the French and the Poles...but one has to be careful of what conclusions that you come to, from beating the French and the Poles. He shouldn't have assumed that because he easily beat the French and the Poles, that he could easily beat the Russians. When there are several variables in a process and the interplay between them all is not well-known, you don't know for sure what is going to happen until you try for real.

This post has been edited by touristguy87: 20 June 2008 - 06:11 PM

[C.Inoue]

Seems to be a lot of interest in real world tests, or at least test variability, i.e. camera bodies, focal length, auto focus accuracy etc. and how one can use the lens/blur tests as a relative measure of a lens performance on your camera's body. I started from scratch (I didn't know what a "blur unit" was) and read the test methodology primer on the slr gear home page (?) Way too much detail there for a lot of quotes but to summarize a few points as I understand the testing proceedure: The tester runs redundant exposures (f-stop/focal length) through the DxO software, so there is a degree of averaging of the results. This can not be the same variability one will find in the "real world" but should more accurately reflect actual performance, that is -compromised to some degree. Is it a statistically valid compromise ? Frankly I have know idea, and it doesn't really matter, read on.

The primer also says because they have limited lab space, far field tests aren't practical, and ask for owner/user feedback. I don't expect test results from multiple focal lengths would be all that much more helpful anyhow, unless the variability in lens performance at different focal length varies significantly relative to the reference lab performance. I suspect (hope) if this were the case, they'd note it in the review. Sample images and comments from users can be really useful in this regard: far field tests, auto focus accuracy/glitches, durability etc. I bet there are a million test exposures out there for every lens tested here. How'd your shots look?

Really, everyone may want to read the primer before I misquote my way into trouble here. The final point that comes to mind is that the blur test charts are really not absolute references for lens to lens comparisons, even though I thought and hoped they might be. They provide a nice visual reference of the performance of that particular lens on a particular platform and from that one can see how blur units/CA/etc. change with focal length and f-stop. I think it is fair to extrapolate that a particular lens will perform about the same on your camera or what would the point of the testing be. And I think you can draw conclusions about finnicky focus rings, creeping zooms, and noisy focus motors.

I was hoping to find out that a blur unit was an absolute quantity, like some line per mm resolved at distance X, nope. I think it is described with respect to Photoshop's blur function, quite subjective, but apparently valid enough to be quantified.

[Dave Etchells]

C.Inoue, on Jun 20 2008, 01:29 AM, said:

Rather than setting up a similar aperatus (Yellow Submarine) I'm thinking that a single exposure of a group of test objects set up in a linear fashion with a slight offset would also determine the point of focus and then an appropriate lens test chart could be placed at the object distance of best focus to evaluate the lens. This method would use slightly different areas of the lens even if the subjects were located close to the center of the field, so a slight error may be introduced.

Hi C. - I think you're right: For casual lens testing, where you're not trying to measure sharpness all across the frame in the same shot, having several subjects near the center, but at slightly different distances should do the trick.

C.Inoue, on Jun 20 2008, 01:29 AM, said:

The tests results indicate manual focus even under laboratory condtions is hit or miss, so maybe experimenting with these lenses is just an academic exercise and camera users should stick to the new AF lenses.

Many (most?) SLRs will display a focus indicator as you're adjusting focus manually. - From our experience, that should generally be more accurate than just relying on what your eyes tell you, peering through the viewfinder. You'll typically find that the correct-focus indicator will stay lit over a small range of focus-ring motion, so there's actually a (hopefully narrow) range of focus settings that the camera is telling you are all "in focus". In such situations, I'd suggest trying to split the difference between where the indicator comes on at the near end of the range and where it goes off at the far end of the range. - In other words, try to set the focus in the middle of the range that the camera's focus sensor is saying is correctly focused.

Of course, if you know that the particular lens/body combination is front- or back-focusing, you may want to consistently set focus at one edge or the other of the indicated "good" range.

So, as long as your camera can display a focus indication, I think it's perfectly fine to play with older MF lenses. Many older lenses (particularly primes) are quite capable of delivering excellent photographs if they're used correctly.

- Dave E.

[Dave Etchells]

laniciffo, on Jun 20 2008, 05:15 AM, said:

This also raises the question of the autofocus test : Chasseur d'Images
uses a well-lit moving target I believe (or an optical simulation of such
a target), but I feel it's not enough, especially as I'm interested in
low-light photography.
The low-light performances of the autofocus should also be characterized,
both in terms of accuracy and speed.
laniciffo from France

Hi Ianiciffo -

Yes, characterization of AF performance is a whole other can of worms: To what extent it depends on interaction between a particular camera and particular lens, how it's affected by light levels, how it's affected by contrast levels, how it handles moving objects, etc, etc. (Also, what about having multiple objects in the vicinity of an AF point - how well does a camera handle that very common situation?)

Relative to our lens testing, we very consciously decided that we should as much as possible show only the performance of the lens, rather that of the camera, or of interaction between the lens and camera. I think it's certainly the case that some lenses perform differently (as a result of AF performance) on different bodies, and information about that would certainly be useful to real-world photographers owning the particular bodies in question, but I don't think that's a road we can really go down. (Speaking just for us here at SLRgear, at least.) The thing is, most lens designs remain available in the market for many years, while bodies are updated at intervals ranging from 2-3 years for pro models to sometimes less than a year for consumer models. So, even if we tested each lens on every possible body on the market today, all that information would be obsolete in at most only a couple of years. It would be an enormous amount of work with a relatively short "shelf life." This may be less of an issue for a monthly magazine, but our material basically stays on the 'web forever, so I'm always thinking about what's most efficient and appropriate in that context.

I'd like to do something to look at the general issue of body/lens interaction (and even just the matter of focus accuracy of various bodies in general), but that's going to have to be a project for a bit further down the road for us. We have found some pretty significant variations in focus accuracy, as part of the IS (image stabilization) testing we're doing, but we need to get our IS testing fully worked out and in production before we could consider taking on another major product like that. (BTW, stay tuned for our IS testing - We've been working on a methodology for that for almost a year now, and finally have it to the point that we feel it's ready for public publication. I'm hoping to roll it out sometime in the next month or so, so keep checking back from time to time to see when we can finally get the first results posted.)

- Dave E.

[Dave Etchells]

kwai, on Jun 20 2008, 04:58 PM, said:

The demonstration that corner lens performance is critically dependent on ultra-fine tuning of the focus effectively means that the performance I get from the lens will inevtiably be randomly distributed over a range of (potentially quite different) performances. So to understand whether a given lens is "better", what I need to know is not its characteristics when it is perfectly in focus but what its average (in some sense, possibly RMS) characteristics are over different degrees of slight defocussing. i.e. rather than show the results from the analysis selected as showing the best focus, it would arguably be more valuable to show a combination of the results across the range of defocussing that in reality will affect all users.

Hi Ben -

That's an excellent point, and one that's been percolating in the back of my mind for a while now. It would certainly be interesting, and very do-able for us to show lens behavior as a lens goes in/out of focus: Most of the time, we're collecting that data anyway, every time Jim does a "rail test" on a lens as part of finding the optimal focus point. Two thoughts:

1) We may in fact at some point get to doing this. It'd require a bit of software work to package that data together, and display it in an interactive viewer as with the main blur plots themselves. It'd certainly be worthwhile information, because there are definitely differences between lenses in this respect. I'll talk with the team, discuss the possibility of doing this at some point, maybe sooner than later.

2) On the other hand, just knowing whether a lens gets soft in the corners or not tells you a lot of what you're probably interested in knowing. From what I've seen, lenses that have very flat sharpness characteristics seem to be much less susceptible to the critical-focus issue than those that show markedly soft corners in the first place. Some lenses are certainly better than others in this respect, but I think a lot of the value of our blur plots is just in discerning the overall characteristics and tendencies of a lens, rather than making microscopic comparisons. That is, I think the typical lens buyer will be trading off sharpness, flatness, and maximum aperture against price, and just knowing that lens "A" is quite flat but costs $300 more than lens "B" which has soft corners is a significant part of what they need to assist their decision. I suspect (but wouldn't try to say for sure) that the matter of corner sharpness sensitivity to focus accuracy will be a secondary consideration relative to the broader parameters just mentioned.

All that said though, I do think this is something for us to look into doing: We'll give it some consideration. (No promises, but we'll definitely look at it.)

- Dave E.

[Dave Etchells]

yandj, on Jun 20 2008, 05:44 PM, said:

I hope Olympus will bring back automatic focus bracketing.
(...)
I was extremely disappointed to see this feature deleted in newer Olympus models. I would have bought an E3 if it had focus bracketing. I hope the utility of automatic focus bracketing becomes more widely recognized, and Olympus (or any camera makers for that matter) will include it in future models (or firmware revisions) they release.

FWIW, I agree completely: This is a feature I'd like to see on all SLRs. I wouldn't think it'd be hard to do, but there may be complexities I'm unaware of. (Eg, it might be hard to make it work reliably with non-manufacturer lenses, etc.) Still, I'd sure like to see something done in this direction.

[Dave Etchells]

Dean, on Jun 20 2008, 05:48 PM, said:

Do you refocus at each aperture or do you set and forget the focus? I got the impression that each aperture was refocused. If you refocus, do you report the focus variation versus aperture? If you set the focus only once, is it the best focus wide open?

Hi Dean -
I'd have to check with Jim (the main tech who does the lab work for the lens tests), but I think the answer is that it depends - But usually he focuses once, at maximum aperture. AFAIK, most of the time, it's sufficient to find the best focus wide open (from the center of the FOV, answering your second question), and then shoot all the other apertures at that focal distance. In some cases though, you may need to find focus at each aperture: When a lens has a lot of spherical aberration, the focal point can change as you adjust the aperture. I offhand recall Jim showing me this on a particular tele zoom lens (sorry, I forget which) - the effect was noticeable, but I'm not sure I'd call it severe. Of course, as I write this, I'm realizing that the proper approach here might be to just leave the AF set at the point that was best with the aperture wide open: AFAIK, all cameras these days focus with the lens wide open, then stop down for the exposure itself.

Quote

My biggest wish is that you would test a lot more lenses! Especially Canon.

Yeah, I think it's time for us to get back to doing some more Canon lenses. We took a fair bit of time to test lenses from some of the other platforms (Olympus, Sony, Pentax, etc), as we want to have a well-rounded site. I also really want to increase the number of tests we have posted for Sigma models, as they're a *huge* part of the consumer lens market here in the US, and we don't have very many of their lenses posted yet. But yes, I do think it's time to get a few more Canon lenses in and get some more results posted for that mount.

[Dave Etchells]

touristguy87, on Jun 20 2008, 06:43 PM, said:

...then one really needs to be able to correlate DxO blur#s to IQ for this to make any sense in the end...

Hi TG -
An excellent point, and one that I hope to (finally) get around to doing something about. - This has been nagging at me for ages now, I need to get off my duff and show some examples. What I'm thinking of is to post some sample crops that show "This is what 1.0 BxUs (DxO's name for their blur unit; "Blur Experience Units) look like", "This is what 1.5 BxUs look like", etc, etc. I'm thinking of something fairly straightforward: Probably just images of a USAF target showing the different levels of blur. - Maybe we could also include a crop of a more pictorial image as well, shot from a very highly-detailed print, so there'd be no question of DOF coming into play.

Quote

it would be nice to see them converted to MTF, say. Which is directly convertible to LPH. One can then set up "groupings" of LPH to suit, and then they can say, "if I use this lens in such and such a way, what are the chances that the IQ will be sufficient for my needs".

We could perhaps do that, but I don't know whether that'd really be useful to readers or not. Most people would have as little idea of what any given MTF or LPH number actually looks like as they would BxUs. They've been around longer, but I don't know that I've ever seen side by side examples that show what different MTF numbers actually look like in images. So I don't know if we'd do this, but we should certainly show what different BxU numbers look like.

Quote

...what happens if the blur test results are run through the blur corrections in DxO? Does it have focus-correction as well? I do know of at least one application that will make small focus corrections. But shouldn't these results be qualified by how well DxO can correct the lens blur? That would be just as interesting to know as what these blur values are. I would be *very* interested to know if they are not fixed.

Yeah, I'd be curious myself to see how DxO Optics Pro handles the images, what the end results look like when you run them back through DxO Analyzer after correction. - I'm curious, but (so far, at least) not curious enough to spend the time to do the test. Optics Pro is a great program (IMHO), but given the choice of devoting too-scarce resources to doing the just-mentioned evaluation vs putting up another 2-3 new lens tests, I have to come down on the side of doing more lens tests.

Quote

Certainly the blur-correction modules in DxO will depend on what the measured blur is for a given set of lens settings and lens/camera combination...which raises the possibility that all of these blur results have a much larger error than we know because the focus-distance is a large factor and the lens simply cannot be tested across all focus-distances. The blur data should be determined at least at 3 points along the range of focus distances between the near focus limit and infinity, at *each* focal length.

There are certainly variations in lens performance as a function of focal distance, but AFAIK, they're almost always confined to the "near field" condition, where the subject-camera distance is some small multiple of focal length. I don't know offhand what the focal length multiple is that's typically considered as marking the near-field boundary (Jim looked it up once, but I don't remember the rule-of-thumb number), but we always try to keep our lens-subject distance somewhere in the range of 30x focal length or higher. For instance, a 200mm lens has a focal length of 0.2 meters. 30x that is 6 meters: In our lab, Jim parks the Yellow Submarine at a distance of about 20 feet for testing a 200mm lens. - That's the longest lens we can test with our standard light/target positioning. For longer lenses, he waits till Luke (main camera test tech) goes home in the afternoons, uses the light setup in Luke's studio, and shoots the length of the building: That lets us get out to something around 45 feet or so.

Quote

...in the end you end up just ranking the lenses based on overall performance.

Actually, we don't assign ranks ourselves: The rating numbers displayed on the site are the average of reader ratings for each lens: We're working on a minor layout change that will make that a bit more obvious.

[Dave Etchells]

touristguy87, on Jun 20 2008, 06:10 PM, said:

But in any case, in practice, we have to use the AF system, so the most useful thing would be to see how many times you have to hit the shutter button to get the AF error down to what size. What are the statistics for that. But overall the DxO blur results here are valid (at least for this particular lens) if the focus is close and if not the blur results don't really matter anyway.

Actually, in some of our work figuring out methodology for IS testing, we've found that multiple AF cycles often don't improve focus. (This was a surprise to me.) I think it may depend on the lens, but many times, half-pressing the shutter button doesn't make any change in AF setting. - This is for the case with a flat target and absolutely identical framing between AF cycles. In the real world, it may be that slight shifts in framing will cause the AF system to recalculate, and arrive at a slightly different (more accurate? less accurate? - I don't know) focus point on successive cycles.

Quote

Still this is very good, for a few reasons, now I'd just automate it
a few x-y and rotational motorized stages should make this a lot easier to do.

Ai yi yi - don't get me started! I already spend way too many Sunday afternoons building target supports, light boxes, Yellow Submarines, etc as it is!

Quote

...still thinking about this from a real-world approach, overall, to be technically precise, the blur results should have to have an associated error which takes into account the focusing error that one would see in real life, as a real shooter would use the lens. It would make more sense to just repeat the blur test using the AF system and give the standard deviation of the results as a percentage of a given blur value.

To say "these are the blur results +/- 1%, taking into account all the measurement errors and systemic errors in the measurement" is how it should be done. Then if one can focus the lens more precisely that error can be removed from the results. But without going through all this, the results don't really map out into the real-world. They're just a guide. But that's all that you need, is a decent guide. I guess that the concise form of what I'm saying is that he is proving that errors in the AF system swamp the precision of the DxO blur results, and what I'm saying is based on that one simply has to take that AF error into account (because AF is the only practical way to focus the lens for most shooters in most real-world situations)...and by finding the point of exact focus that error is being removed, and subsequently the results are really not "real-word". Close, but not quite. Nonetheless a good guide for purchasing a lens.

Yeah, I agree with all that. Two issues are (1) how to characterize what typical real-world focus variation might be, and (2) how to present the variability of results visually. For (1), that's a project I'd very much like to get into: Just how bad (from what we're seeing, that's the appropriate phrasing for current AF systems, vs "how good") are typical AF systems: How big is the typical (RMS?) error? For (2), as I mentioned in one of my other replies just now, it might be sufficient to provide an interactive viewer of some sort, that lets people see how things vary with minor focal errors, when the lens is wide open. This sort of thing is always a compromise - We could expand the test matrix to show results from 5 different focal settings (two closer, one spot on, two farther) for every aperture and focal length we test at. - But that'd mean posting one lens review/month vs the 6-8 we're currently aiming for. (I guess this is where the automation you suggested would come in. )

And yes, I also definitely agree with your conclusion that our tests are best read as guides, not some ultimate be-all, end-all characterizations of every minute parameter. I think that dovetails nicely with the interests of most of our readers: They don't so much want to read the ultimate treatise on the behavior of Lens X, as they want to be given some reasonably concise info that can help them choose between Lens X and Lens Y.

[Darth AkSarBen]

Andrew Alexander, on Jun 19 2008, 03:40 PM, said:

If you haven't already, check out Dave's new article on testing for lens sharpness - the Fallibility of Focus!

After you've had a good read, come on back and tell us what you think!

- Andrew

On the Carl Zeiss Distagon 28mm f/2 at f/2.0 graphing (3D) It appeared to me that the slight out of focus was better on the edges than the optimal focus? Is that right?

Future cameras should incorporate XFine focus for stationary object that one would like more exacting focus. Internally, the sensor could send image data to the camera's computer for analysis at greatly enlarged detail, perhaps up to 1,000 times to determine pixel to pixel point of focus. This would be incorporated with a matter of holding down 1/2 way for absolute fine tune focus. Idea huh?

[beols069]

Good article!

I have some questions though.

1. I am somewhat surprised about you using a not-HSM Sigma or not using a ring-type USM Canon lens.
AFAIK all "normal geared" lenses have notable/measurable backlash.
2. Canon states an AF-error of "within one CoC" for consumer cameras and "within one third CoC" for the pro cameras.
Which blurr factor is comparable with "one CoC"?

[Erik Magnuson]

beols069, on Jun 22 2008, 05:20 AM, said:

I have some questions though.

1. I am somewhat surprised about you using a not-HSM Sigma or not using a ring-type USM Canon lens.
AFAIK all "normal geared" lenses have notable/measurable backlash.
2. Canon states an AF-error of "within one CoC" for consumer cameras and "within one third CoC" for the pro cameras.
Which blurr factor is comparable with "one CoC"?

1. The lens is supposed to keep track of this, e.g. measure the expected rotation vs. the desired rotation and correct for it with a second movement.

2. Current Canon "consumer" DSLRs have also a high-precision focus segment available for the center point. This segment is activated for lenses with an aperture of f/2.8 or less. But there are two exceptions to this noted in the 40D manual p79): the EF28-80mm f/2.8-4L and the EF50mm f/2.5 Compact Macro.

This test used the Sigma 70mm macro which is an f/2.8 lens. So did it use the high-precision segment or not? I think the answer is probably not. Why? Because the Sigma lens identifies itself to the body as "LensSpec: EF50mm f/2.5 Compact Macro +LSC". (The +LSC mean Life Size Converter). According to the 40D manual, this combo would not use the high-precision sensor.

--
Erik

[touristguy87]

In practice I find that I simply have to hit the AF a few times and try to get a good focus. When I get one I then will hold that focus and take the shot.

But you're very right, hitting it 3 or more times is no guarantee that the focus will get better each time. The error is quite large...but still it is way better than the focus that I can get if I try to do it manually off the focusing-screen even if I have a split-prism in the camera.

Overall I think that this is great stuff...it certainly is valid to correct the focus and take the lens measurements...then the shooter can worry about whether their camera gives a good focus, reliably.

But on the other hand, that is clearly an important issue too and it would be nice to see if the AF accuracy varies from lens to lens. It should vary at least with the lens performance at wide-open, and as you show repeatedly the lens is not very sharp away from center when shot wide-open. So having all these focus-points away from center could actually be counterproductive (it definitely is when there are near items and far items for the camera to focus on).

Just a couple of concerns. Knowing that the blur increases dramatically if the lens is shot wide-open, one would tend not to shoot the lens wide-open if possible. So, say you're shooting at F8 or something where the blur profile is reasonably flat. How much does it change in the region around the point of true focus? Say...the camera front-focuses by 1% of the focus-distance (focal-length?), how much does the blur change? Is this not a measure of how "out of focus" the lens is? This is very important to know, it speaks to the overall "efficiency" of the lens and camera...maybe if not an issue for slrgear, then for IR...if noise is a parameter of interest, certainly focus-accuracy is...but if you were to put a Sigma 18-200 DC OS on a 400d vs a Canon 70-200F2.8, you'd see not only differences in lens sharpness but in focus accuracy, too.

...still one has to start somewhere and this is a great start.

[beols069]

Erik Magnuson, on Jun 22 2008, 04:22 PM, said:

1. The lens is supposed to keep track of this, e.g. measure the expected rotation vs. the desired rotation and correct for it with a second movement.

AFAIK it's the camera that "orders" the lens
Mechanical backlash could mean that a camera command to focus in or out is ineffective because the desired movement got lost between "teeth".

Another point could be the number of steps of the lenses AF-system.
Found this link: http://www.reduser.net/forum/showpost.php?...mp;postcount=44

Th values given there could give some insight in the focusing accuracy of those lenses lenses.

[Darth AkSarBen]

I remember starting out in photography when I was around 8 back in 1961 and all I had was a Kodak Hawkeye 120 film camera. Fixed focus lens, fixed shutter speed, fixed aperture. Still got some pretty decent memories from that old camera. All we could afford was Black and White then.

So, there was no great absolute focus, you relied pretty heavily on DOF and hyperfocus fields. In shooting view cameras, it was difficult at some times to get just the right focus point, even on ground glass. With the rangefinder on my Graflex Crown Graphic, it was also at the mercy of how good the cam was set up. Older cameras were notorious for getting the focus correct off the add on side rangefinder. Much tweaking with small screwdrivers to adjust the mirrors for best focus overall.

Quite an improvement we have in AF lenses now, HUH??

This post has been edited by Darth AkSarBen: 22 June 2008 - 12:05 PM

[Erik Magnuson]

beols069, on Jun 22 2008, 11:45 AM, said:

AFAIK it's the camera that "orders" the lens
Mechanical backlash could mean that a camera command to focus in or out is ineffective because the desired movement got lost between "teeth".

We are perhaps talking about two different types of backlash. The type I'm talking about is when the camera says "move to position X". Then lens then attempts such a move, but because of backlash and other tolerance issues, it first stops at X+2, notices this and then moves back 2 to exactly X. It doesn't seem likely that a camera would order a lens to an intermediate location: either the camera or the lens would round the value to the nearest known position. Now if a lens motor (or some other part) is out of calibration, then it's possible that when commanded to move to X, it really moves to somewhere else (e.g. X - 0.2X). Then you would have front-focus.

beols069, on Jun 22 2008, 11:45 AM, said:

Another point could be the number of steps of the lenses AF-system.
Found this link: http://www.reduser.net/forum/showpost.php?...mp;postcount=44

Th values given there could give some insight in the focusing accuracy of those lenses lenses.

Thanks, that's a interesting link. Perhaps Dave could buy/borrow one of these mounts to help test this aspect of focus precision.

[Lars Clausen]

Andrew Alexander, on Jun 19 2008, 09:40 PM, said:

If you haven't already, check out Dave's new article on testing for lens sharpness - the Fallibility of Focus!

After you've had a good read, come on back and tell us what you think!

- Andrew

An interesting article, and very nice to see some comparisons of various ways to focus. I am disturbed by how much difference a small amount of misfocus can cause in the corners -- is that a general issue, or is it just that you took a particularly noticeable example? The article does come off a bit like you've come up with a totally new way to manage focus. It's not like macro photographers haven't been using rails for decades, even after AF has been introduced. I was also wondering, since you were using a 40D, why you didn't get a better viewfinder screen for it to do these tests -- the fact that you got really bad results without one would exactly be the reason to get one, which would in turn be a nice opportunity to evaluate the precision of various screens. Or is that a topic for a future article?

-Lars

[Dave Etchells]

Erik Magnuson, on Jun 22 2008, 10:22 AM, said:

1. The lens is supposed to keep track of this, e.g. measure the expected rotation vs. the desired rotation and correct for it with a second movement.

I always thought that cameras should do this: That they'd take a second "look" at the focus after having commanded a lens motion, and then make additional corrections as needed. This doesn't appear to be the case, and in fact a member of Canon's technical staff confirmed to me that their SLRs generally don't. (But I've never gotten what I'd consider a full explanation of how their AF systems work...)

Quote

2. Current Canon "consumer" DSLRs have also a high-precision focus segment available for the center point. This segment is activated for lenses with an aperture of f/2.8 or less. But there are two exceptions to this noted in the 40D manual p79): the EF28-80mm f/2.8-4L and the EF50mm f/2.5 Compact Macro.

This test used the Sigma 70mm macro which is an f/2.8 lens. So did it use the high-precision segment or not? I think the answer is probably not. Why? Because the Sigma lens identifies itself to the body as "LensSpec: EF50mm f/2.5 Compact Macro +LSC". (The +LSC mean Life Size Converter). According to the 40D manual, this combo would not use the high-precision sensor.

That's a good point/question. Whether the high-precision AF is used or not should depend on the maximum aperture, whether the aperture allows a wide enough beam spread for the longer-baseline high-precision sensors to be able to see the focus area properly: I think (but don't know) that the high-precision sensors would simply be shadowed if the aperture was too narrow - And that's how the camera would tell whether it could use those sensors or not. Given that the camera is mis-identifying the lens as one that doesn't support the high-precision sensors, it's possible that they're not being used. But I wouldn't expect the camera to base whether or not to use the high-precision sensors on whether it can recognize the lens in some sort of internal database: That could cause problems down the road, even for Canon-branded lenses that were newer, and therefore not in the camera's database. (Which all begs the question though, of how the 40D is mis-identifying the lens. AFAIK, cameras read the lens identity from the processor in the lens itself, rather than looking anything up. Maybe not though.)

My guess as to what the issue is with the two lenses mentioned in the article is simply that they don't have f/2.8 apertures over some of their operating range. The zoom obviously goes down to f/4 at its tele end, so would shadow the high-precision sensors when used at long focal lengths. I suspect the same thing is the case with the Compact Macro: Some macro lenses actually decrease their maximum aperture as you focus closer and closer with them. - This was the case with the new Nikon 60mm f/2.8 that we just tested recently, I suspect it might be the case with the Canon 50mm f/2.5 as well.

Regardless, it'd be worthwhile to repeat the AF example with a Canon lens that's known not to have problems with the high-precision AF system, and see how it behaves. I'll see if I can get to this. No promises, but I'll put it in the queue...

[beols069]

Dave Etchells, on Jun 23 2008, 08:13 PM, said:

(Which all begs the question though, of how the 40D is mis-identifying the lens. AFAIK, cameras read the lens identity from the processor in the lens itself, rather than looking anything up. Maybe not though.)

Have a look at some of the Canon MakerNotes entries: http://www.sno.phy.queensu.ca/~phil/exifto...n.html#LensType
(a complete list of Canon LensType Values)

[Erik Magnuson]

Dave Etchells, on Jun 23 2008, 01:13 PM, said:

I always thought that cameras should do this: That they'd take a second "look" at the focus after having commanded a lens motion, and then make additional corrections as needed. This doesn't appear to be the case, and in fact a member of Canon's technical staff confirmed to me that their SLRs generally don't. (But I've never gotten what I'd consider a full explanation of how their AF systems work...)

Have you ever read RDKirk's write-up from Fred Miranda's forum? I can only find it in google's cache now:
http://209.85.215.104/search?q=cache:Nb9Wa...241524/chat.php

The main thing this description is missing is why AF would ever hunt. I suspect that there is also a confidence value. If the AF calculation is high-confidence, then it doesn't look again. But a low confidence solution would move to the best estimate and try again.

Quote

I think (but don't know) that the high-precision sensors would simply be shadowed if the aperture was too narrow - And that's how the camera would tell whether it could use those sensors or not.

This cannot be the case or taping the pins on a converter would not work to bypass the f/5.6 limitation. It's not a binary blacked out or not, but a degree of degradation.

Quote

That could cause problems down the road, even for Canon-branded lenses that were newer, and therefore not in the camera's database.

Only if there are new exceptions to the f/2.8 rule. Note that no new Canon lenses have been f/2.8-4.

Quote

(Which all begs the question though, of how the 40D is mis-identifying the lens. AFAIK, cameras read the lens identity from the processor in the lens itself, rather than looking anything up. Maybe not though.)

The lens returns a code; the 40D firmware has the code to name lookup. Check the release notes for the last 30D firmware release and it adds names for the 18-55IS and 55-250IS. Sigma lenses typically return the code for the closest Canon equivalent. In this case, the EF50+LSC is Canon's only 70mm macro combination so Sigma chose that. Perhaps intentionally to avoid using the high-precision sensor if it will not work at closest focus. Oddly it seems that DPP uses the lens name string and not the code for it's vignetting/distortion correction. 20D users can have found that they can enable this function if they add the lens ID string to their raw Exif. (The 20D firmware does not have the lens string translation.)

[Dave Etchells]

beols069, on Jun 22 2008, 12:45 PM, said:

AFAIK it's the camera that "orders" the lens
Mechanical backlash could mean that a camera command to focus in or out is ineffective because the desired movement got lost between "teeth".

As part of our work for IS testing, we have in fact seen this sort of thing. This is another example of something that we'd like to study in greater detail at some point, but that's going to take a fair bit of work to separate out how much is the fault of the lens, how much that of the camera. - But what we've observed is that there'll often be different focus error magnitudes, depending on whether a lens is approaching focus from the "near" side or the "far" side. (Eg, initially focused closer than the subject vs farther than the subject.) At this point, we're just trying to get our IS testing worked out (and still keep getting new blur/CA/etc test results posted), so we haven't had time to look into it more closely. Most of the time, we've been working on the Canon platform, with our 20D and 5D test bodies, but we've seen the effect with a Sony lens as well. It seems clear that the lenses are ending up one one side of optimal focus or the other, depending on where they're coming from. - And that the camera bodies aren't detecting that the final focus point is off. It's clearly not a front/back focus thing either (as would be caused by the focus sensor position being off), because the results change depending on where the lens is coming from. There seems to be a pretty noticeable "dead area" within which the camera's AF systems just say "that's in focus", even though it actually isn't. Definitely something I want to look at in more depth once we get our IS testing worked out, and possibly a couple of other tests added to our standard suite.

Quote

Another point could be the number of steps of the lenses AF-system.
Found this link: http://www.reduser.net/forum/showpost.php?...mp;postcount=44
The values given there could give some insight in the focusing accuracy of those lenses lenses.

That's very interesting! I'd somehow always assumed that AF motors were basically similar to DC motors, in that they'd have a more or less continuous range of possible positions. It appears that that may not be the case, or perhaps that the lens's CPU may be set up to provide only certain fixed positions along what might otherwise be a continuous range of motion. Sigh... Yet another thing for the list of things to check out "someday"...