What should I do if nan values are important information?

[SamKlier]

Hi folks,

I have data which contains nan values. Of course, this is nothing special. Usually, in the Data Science community nan-values are expected to be faulty, it seems to me. Therefore, they should be eliminated beforehand.
However, in my data the nan values are legit and very important for my distance function.

Do you have any suggestions how to handle nan-values which are important information?
Anyways, UMAP could handle nan-values if it would like to, correct?

Thanks in advance!
Sam

[lmcinnes]

UMAP can support custom distance functions as long as they compile with numba (use the numba.njit decorator on the function); as long as you can write a distance function that can handle NaN in the vectors and output sensible (i.e. non NaN) distances then it should all work fine.

[SamKlier]

Hi and thanks for responding so quickly!
Well, I've tried it and it doesn't work, which is what brought me up here.
A minimal example for you to try:

from umap import UMAP
import numpy as np
from numba import njit

@njit
def nan_dist(a: np.ndarray, b: np.ndarray):
    return 0


d = np.arange(100.0).reshape(25, 4)
d[0][0] = np.nan

print(f'Test distance function:\t{nan_dist(d, d)}')

UMAP(metric=nan_dist).fit_transform(d)

numba and the distance function work fine with the nan values. However, UMAP gives me a stack trace:

ValueError: Input contains NaN.

Based on your answer, I guess this is unexpected behaviour?

[lmcinnes]

Yeah, that's not what I was expecting. I think this may be a matter of validation done on the input array, some of which is handled by sklearn, and some may be internal to UMAP. I don't recall explicitly denying NaN inputs so it isn't immediately obvious exactly where this is getting flagged.

In summary: in principle this should work; practice seems to be a little more complicated. You may be able to work around it by simply turning off some of the input checking ... although it might be failing in other places for other reasons unfortunately.

[SamKlier]

Yes, I already tracked it down to the check_array calls which default force_all_finite=True and I changed it to allow-nan which works fine.
However, I wasn't absolutely sure if I spoil anything by doing so.
Thanks, for replying that nan values are fine to the approach in general.

[lmcinnes]

I'm glad to know it works! I might need to add an option to the class so users can turn off the finiteness checking (in general it is probably better to leave it on as it can cause surprising behaviour if the user is unaware I would imagine). If you wanted to put in a PR to simply add a check_input_finite=True to the class and pass that through to check_array in the appropriate place that would be greatly appreciated.