Illumination artifacts at object edges

[benjamin-heasly]

Issue by benjamin-heasly
Friday Oct 24, 2014 at 14:39 GMT
Originally opened as https://github.com/DavidBrainard/RenderToolboxDevelop/issues/7

---

We may need to clean up some artifacts in our illumination images. For example, see light blue artifacts at the edges of some objects:

These artifacts are probably caused by errors in our object pixel mask code (MakeRecipeObjectMasks). This code is responsible for assigning which pixels are "looking at" which objects. It also assigns which pixels are ambiguous and not looking at any particular object. It seems to be making some wrong assignments, which messes up our illumination computations.

Note that the assignment errors have a "relative" quality. For example, the pixel assignments work out fine for most of the door jamb. But where the door jamb appears next to the ground or next to the roof of the building in the background, there are artifacts.

We might want to revise the object pixel mask code to have an "absolute" quality, so that the assignment for a given pixel is less sensitive to the pixel's spatial context.

[benjamin-heasly]

Comment by benjamin-heasly
Friday Oct 24, 2014 at 20:21 GMT

---

A subtle issue occurred to me:

We have to interpolate to smooth out the gap pixels between objects. We currently do this one spectral plane at a time. This might be the wrong way to do it.

For example, at a given pixel, say the median in spectral band 1 corresponds to the contribution of object A. It might be the case that the median in spectral band 5 corresponds to a different objects, B and C. This might be the case if B and C happen to give the same contribution in band 5, and their union happens to outweigh object A in that band. So the total spectrum that we assign to pixel in question might be a mix of objects A, B, and C, and might look different from A, B, and C.

I am not sure if this is really happening. I'm also not sure if it's a cause for concern. But I think it's not what we intend when we say "take the median illumination of nearby pixels". I think the median should be taken with respect to complete spectra, and not separately for each spectral band.

[benjamin-heasly]

Comment by benjamin-heasly
Friday Dec 19, 2014 at 15:41 GMT

---

We can extract various scene "factoids" using the Mitsuba "field" integrator:
https://github.com/DavidBrainard/RenderToolboxDevelop/blob/master/VirtualScenesToolbox/Utilities/RenderMitsubaFactoids.m

One of the factoids is called "shapeIndex". This appears to give us an identifier for each object in the scene. So it's an independent way to identify objects. This might give us insight into our own object detection code.

[benjamin-heasly]

Comment by benjamin-heasly
Friday May 01, 2015 at 17:05 GMT

---

We've had success using the "shapeIndex" factoid to identify objects as well as object borders.

Here are some thoughts for ongoing discussion:

• We now have two methods for identifying objects and borders: our original "Materials" methods, and the new "Factoid" method.
• We use this information for two distinct purposes.
  • One purpose is to figure out which pixels belong to which object. This is pretty well solved.
  • The other is to perform interpolation/smoothing near object borders, in order to remove artifacts in our illumination analysis. This is more subtle because of dithering performed by the renderer during image reconstruction. This might be solved, or we might need to fuss over it more. It depends on how our next batch of renderings turns out.
• It would be nice to always have "crisp" object borders, say 1 pixel wide all the way around. But this might not actually be necessary.
• For some analyses, we might want to know which object "owns" the border. This might affect asymmetric things like false positives vs. false negatives during identification of a given object. We can probably address this kind of question using either method, Materials or Factoids.