Repository navigation
int precision problem in H3Index.SetIndexDigit #1
Description
I was using this port and comparing results with the original C version, and I found an error in some results. After some debugging I finally found that, in some cases, the int precision was not enough in this function H3Index.SetIndexDigit.
After changing it to this it works perfectly now:
public void SetIndexDigit(int res, int digit)
{
ulong p0a = ((ulong)Constants.MAX_H3_RES - (ulong)res);
ulong p0b = (p0a * H3_PER_DIGIT_OFFSET);
ulong p1 = ((ulong)H3_DIGIT_MASK << (int)p0b);
ulong p2 = (((ulong)Constants.MAX_H3_RES - (ulong)res) * (ulong)H3_PER_DIGIT_OFFSET);
Value = (Value & (ulong)~p1) | ((ulong)digit << (int)p2);
}
Thanks for your port! Nice work.
Activity