Unflagged bad spaxels

[kbarbary]

... make galaxy fit go haywire. Seems like the weight of these spaxels in the data is not set correctly.

SNF20070712-003 (R): Note spaxels in epochs 19 and 20.

SN2011by (R): Note last two epochs.

SNF20070831-015 (B): Note the cold spaxel in the upper part of epoch 18 data. Creates cool spot in galaxy model in step 2 and final.

[kponder]

@kbarbary has there been any work on this issue? I'm definitely seeing some large effects from poor pixel masking. It is probably the cause of some of the fits taking a long time to converge.
The plot included here is the supernova that I was previously emailing you about with the excessive number of iterations and poor chi-square. I'm not sure yet what has changed between the cubes I generated and the previous cubes, but it has caused some of these pixels to not be masked properly.

(and @kboone for interest)

SNF20080707-012 (B)

[kbarbary]

@kponder No work done on this that I know of. I think I opened this issue with the idea that it should be fixed in upstream processing.

One could try to implement some kind of outlier rejection in cubefit, but I would consider that a last resort, and one fraught with new hazards. It would be good to at least understand the source of the outliers first.

[ycopin]

extract_star does not have any special ways to detect outlying spaxels, it just uses the stored variance which should be enough. Has the conversion to NAXIS=3 FITS cube an issue?

[kponder]

I checked the cube that was generated by me (in the plan_cube_generation step) versus the one generated in 02-00.

I identified this specific spaxel that's causing the bad cubefit result as the one with the largest difference between 02-00 and 02-04 in the cube generation. (At lambda = 4357.78 Ang) I checked this for a single cube for SNF20080707-012: 08_217_068_003_4_022_000.

prod	data	variance
02-00	3.80284	274.919
02-04	-737.236	25.0493

So not only is the value very off, it also has a low variance giving it more weight in the fitting. The problem is occurring in either the preprocess or cube generation phase. If this is fixed, then cubefit should run correctly. However, I have been unable find the difference between 02-00 and 02-04 that could cause this change. I'll keep digging.

[kponder]

@kbarbary Does cubefit do any sigma clipping? For SNF20080707-012 (B) I can see the bad spaxel in the spectrum of the host galaxy subtracted (KTESTSP) lightcurve point at 08_217/Aug 4. However, the non-host galaxy subtracted spectrum (KARATEST) does not show the bad spaxel. This bad spaxel is being successfully accounted for when extracting the spectrum in extract_spec2.c, but cubefit does not account for it during its galaxy fit. I think the original spectrum does a 2.5 sigma clipping.

It would be better not to have these issues make it through to cubefit, but I cannot currently find where the issue is during cube generation. I have confirmed that the problem is with cube generation though.

As a check, I de-weighted the "bad" spaxel points by increasing the variance and cubefit converges to something similar to the previous runs for SNF20080707-012 (B).

[kbarbary]

No, cubefit does not do any sigma clipping (aka "outlier rejection").