Issue with deletion in long words

[ausi]

If a word in the index is longer than the index length, deletions dot not work anymore (they incorrectly count as 1 deletion + 1 insertion).

As you can see in the test case that currently fails, the word Mustermann is indexed as Muster (index length 6) and the search for Mutermann is then calculated as if we searched for Muterm. So even though Mustermann and Mutermann only have a distance of 1, you only find them with a distance of 2 or higher because Muster and Muterm have a distance of 2.

This is an issue in the original paper itself as far as I can see. But I think it can be fixed.

[Toflar]

Had to re-enable CI as it's been 6 months. Rebasing should help to have it run.

[ausi]

45a893e is an attempt to fix the issue. But it fails the testResultsMatchResearchPaper test and ends up finding way too many states. I think we need to keep track of the number of deletions in $statesStarC to only allow zero-cost substitutions for these states and not all states.

[Toflar]

Yeah I don't think that many states is an acceptable solution because it will return way too many false-positives 🤔

[ausi]

I think we need to keep track of the number of deletions in $statesStarC to only allow zero-cost substitutions for these states and not all states.

I found a simpler way to do that as the numerical range of all the states that result from words that exceed (or match) the index length is known ahead of time. We can therefore easily check if a state falls into that category and apply a zero cost deletion for them.

I am pretty certain that this is correct and that the original research paper missed handling this edge case. I added a comment to the testResultsMatchResearchPaper method that tries to explain why it has to be correct to also find the state 1869 in the given example. And I added the word "Multere" to the example to further clarify it. Are you in contact with the authors? Maybe we should inform them of our findings?

[Toflar]

Wow, this makes sense and looks like a really simple fix to the problem! ❤️

I am pretty certain that this is correct and that the original research paper missed handling this edge case. I added a comment to the testResultsMatchResearchPaper method that tries to explain why it has to be correct to also find the state 1869 in the given example. And I added the word "Multere" to the example to further clarify it. Are you in contact with the authors? Maybe we should inform them of our findings?

No, I found this paper a while ago by chance. I don't know how we could reach them and also it's not "our findings", it's your findings 😉

[Toflar]

Awesome work, thank you @ausi!