add CompletionFilter - #249
Conversation
|
| while (!tokenGenerator.hasNext()) { | ||
| if (!inputStreamConsumed && input.incrementToken()) { | ||
| val m: Morpheme = morphemeAtt.getMorpheme() | ||
| val surface: String = m.surface() |
There was a problem hiding this comment.
I think surface should be taken from termAttr, in case of e.g. normalized_form filter is applied beforehand (we may also need to note that in this case reading and surface may not match due to stemming).
| override fun incrementToken(): Boolean { | ||
| while (!tokenGenerator.hasNext()) { | ||
| if (!inputStreamConsumed && input.incrementToken()) { | ||
| val m: Morpheme = morphemeAtt.getMorpheme() |
There was a problem hiding this comment.
morphemeAttr.getMorpheme may return null and we need to handle that case.
| when { | ||
| !isAllLowercaseAlphabet(pdgSurface.toString()) && | ||
| isAllLowercaseAlphabet(surface.toString()) -> { | ||
| pdgSurface.append(surface) |
There was a problem hiding this comment.
I want to have comments about what kind of pattern is expected in these when-cases (as written in https://github.com/apache/lucene/blob/728ebd8527b10f3a2072ec18cd80aca220ee85b5/lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseCompletionFilter.java#L183).
|
|
||
| if (tokenGenerator.hasNext()) { | ||
| clearAttributes() | ||
| val token: CompletionToken = tokenGenerator.next() |
There was a problem hiding this comment.
We also need to set morpheme attribute here.
I wonder how we should do when we concatenate tokens during query mode... Probably we can omit morpheme attr (set null) in that case, but I'm not fully sure if it's safe.
|
|
||
| private fun romanize(text: String): List<String> { | ||
| val output = mutableListOf<String>() | ||
| val romaji: String = convertFullWidthAlphabetToHalfWidth(Romanizer.romanize(text)) |
There was a problem hiding this comment.
It would be better if we cover many patterns (ヘボン式、訓令式 etc.), but we may postpone that for future work. Please create an issue about this if so.
98ff305 to
dca4243
Compare
|
@hayashi-mas-wap
|
| val m: Morpheme? = morphemeAtt.getMorpheme() | ||
| // If the upstream filter did not set a morpheme, there is nothing we can add to the | ||
| // completion generator, so skip this input token. | ||
| if (m == null) { |
There was a problem hiding this comment.
Skipping null-morpheme token seems dangerous. It may unexpectedly drop intermediate token (although our other non-graph token filters won't produce such case).
We should keep processing with reading = null or "" this case.
|
|
||
| if (token.morpheme != null) { | ||
| morphemeAtt.setMorpheme(token.morpheme) | ||
| morphemeAtt.setOffsets(token.offsetMap) |
There was a problem hiding this comment.
morphemeAtt.offset is NOT equals to offsetAtt (see e.g.
we need to keep it in tokenGenerator with morpheme (maybe we can have data class for them).
2 participants
target branch:
developbackport:
allImplement SudachiCompletionFilter.
The search and test-scripts will be separated into es and os branch.
#155 is supported.