Normaliser does not expand Arabic numeral in subreading

[snomos]

Compare this:

echo 1600 | ./tools/tts/modes/sme-normaliser.mode    
"<1600>"
	"guhttanuppelohčuođi" Num Sg Nom  "guhttanuppelohčuođi"phon
	"duhátguhttačuođi" Num Sg Nom  "duhátguhttačuohti"phon
	"duhátguhttačuođi" Num Sg Nom  "duhátguhttačuođi"phon
:\n

with this:

echo 1600-logu | ./tools/tts/modes/sme-normaliser.mode          
"<1600-logu>"
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0 "1600-logu"phon
		"1600" Num Arab Cmp/Hyph Cmp <W:0.0> #1->0 "1600-logu"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0 "1600-logu"phon
		"1600" Num Cmp/Hyph Cmp <W:0.0> #1->0 "1600-logu"phon
:\n

Excpected output should have been something like duhátguhttačuođi-logu.

@bbqsrc this must probably be fixed in your code as well.

Reported by https://github.com/Yleisradio

[flammie]

Here's a debug print:

$ echo 1600-logu |\
 hfst-tokenise -g '/home/flammie/github/giellalt/lang-sme/tools/tts/tokeniser-tts-cggt-desc.pmhfst' |\
 divvun-blanktag '/home/flammie/github/giellalt/lang-sme/tools/tts/analyser-gt-whitespace.hfst' |\
 vislcg3 -g '/home/flammie/github/giellalt/lang-sme/tools/tts/generated-remove-lexicalised-compounds.bin' |\
 vislcg3 -g '/home/flammie/github/giellalt/lang-sme/tools/tts/valency.bin' |\
 vislcg3 -g '/home/flammie/github/giellalt/lang-sme/tools/tts/mwe-dis.bin' |\
 cg-mwesplit |\
 vislcg3 -g '/home/flammie/github/giellalt/lang-sme/tools/tts/disambiguator.bin' |\
 vislcg3 -g '/home/flammie/github/giellalt/lang-sme/tools/tts/functions.bin' |\
 vislcg3 -g '/home/flammie/github/giellalt/lang-sme/tools/tts/dependency.bin' |\
 ./src/divvun-normaliser \
 -a  '/home/flammie/github/giellalt/lang-sme/tools/tts/analyser-gt-norm.hfstol' \
 -g  '/home/flammie/github/giellalt/lang-sme/tools/tts/generator-tts-gt-norm.hfstol' \
 -n  '/home/flammie/github/giellalt/lang-sme/tools/tts/transcriptor-gt-desc.hfstol' \
 -t ABBR -t ACR -t Arab -t Sem/Year -t Sem/Date -t Sem/Time-clock -t Symbol -v --debug
"<1600-logu>"
Using lemma: lohku
No expansion tags in
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0
Expanding because of Arab
Using lemma: 1600
1. looking up normaliser
2.a Using normalised form: guhttanuppelohčuođi
2.b regenerating lookup: guhttanuppelohčuođi+Num+Cmp
3. Couldn't regenerate, reanalysing lemma: guhttanuppelohčuođi
3.a got: guhttanuppelotčuohti+Num+Sg+Acc
couldn't match  Num Sg Acc and  Num Cmp
3.a got: guhttanuppelotčuohti+Num+Sg+Gen
couldn't match  Num Sg Gen and  Num Cmp
3.a got: guhttanuppelotčuohti+Num+Sg+Ill+Attr
couldn't match  Num Sg Ill Attr and  Num Cmp
3.a got: guhttanuppelotčuohti+Num+Sg+Loc+Attr
couldn't match  Num Sg Loc Attr and  Num Cmp
3.a got: guhttanuppelotčuođi+Num+Sg+Acc
couldn't match  Num Sg Acc and  Num Cmp
3.a got: guhttanuppelotčuođi+Num+Sg+Gen
couldn't match  Num Sg Gen and  Num Cmp
3.a got: guhttanuppelotčuođi+Num+Sg+Ill+Attr
couldn't match  Num Sg Ill Attr and  Num Cmp
3.a got: guhttanuppelotčuođi+Num+Sg+Loc+Attr
couldn't match  Num Sg Loc Attr and  Num Cmp
3.a got: guhttanuppelotčuođi+Num+Sg+Nom
couldn't match  Num Sg Nom and  Num Cmp
2.a Using normalised form: duhátguhttačuođi
2.b regenerating lookup: duhátguhttačuođi+Num+Cmp
3. Couldn't regenerate, reanalysing lemma: duhátguhttačuođi
3.a got: duhátguhttačuođi+Num+Sg+Acc
couldn't match  Num Sg Acc and  Num Cmp
3.a got: duhátguhttačuođi+Num+Sg+Nom
couldn't match  Num Sg Nom and  Num Cmp
2.a Using normalised form: duhátguhttačuođát
2.b regenerating lookup: duhátguhttačuođát+Num+Cmp
3. Couldn't regenerate, reanalysing lemma: duhátguhttačuođát
3.a got: duhátguhttačuođát+A+Ord+Attr
couldn't match  A Ord Attr and  Num Cmp
3.a got: duhátguhttačuođát+A+Ord+Sg+Nom
couldn't match  A Ord Sg Nom and  Num Cmp
no usable results, printing source:
		"1600" Num Arab Cmp/Hyph Cmp <W:0.0> #1->0

I'm sure there is a reason why I wanted to match up tags but maybe Cmp needs to be an exception within exceptions till

[flammie]

try this

[snomos]

Summary of oral discussions:

@flammie introduces a separate CG step that changes tags for these (and in the future other) specific cases ahead of the normalisation step. That is, this:

"<1600-logu>"
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0 "1600-logu"phon
		"1600" Num Arab Cmp/Hyph Cmp <W:0.0> #1->0 "1600-logu"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0 "1600-logu"phon
		"1600" Num Cmp/Hyph Cmp <W:0.0> #1->0 "1600-logu"phon

should be changed to this:

"<1600-logu>"
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0 "1600-logu"phon
		"1600" Num Arab Sg Nom <W:0.0> #1->0 "1600-logu"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0 "1600-logu"phon
		"1600" Num Sg Nom <W:0.0> #1->0 "1600-logu"phon

With that, at least this part:

"<1600-logu>"
	"lohku" N Sem/Amount Sg Gen <W:0.0> @X #1->0 "1600-logu"phon
		"1600" Num Arab Sg Nom <W:0.0> #1->0 "1600-logu"phon

should then be normalised properly using the existing machinery.

[flammie]

the cg piece is in lang-sme tts now, the other part of the bug depends on is that norm and phon both need to handle one reading with its subreadings at a time and probably print out full set for each. I'll possibly won't have a fix for it before holidays but I think a quick hack to copy things around from sub to main readings with python or something could be workable for prototyping

[snomos]

Thanks a lot! So what is left is the same bug as in #89.

I don't think temporary ad hoc solutions using Python is worth it. Maybe @bbqsrc can implement this properly in the runtime code.

[snomos]

@albbas could you have a look at the relevant C++ code to solve this one and #89? This is high priority right now.

[albbas]

I get this with todays nightly (on mac):

echo 1600-logu | modes/sme-normaliser.mode 
"<1600-logu>"
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "1600-logu"phon
		"guhttanuppelotčuođi" Num Sg Acc  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Gen  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Ill Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Loc Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Acc  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Gen  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Ill Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Loc Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Nom  "guhttanuppelotčuođi"phon
		"duhátguhttačuođi" Num Sg Acc  "duhátguhttačuođi"phon
		"duhátguhttačuođi" Num Sg Nom  "duhátguhttačuođi"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "1600-logu"phon
		"1600" Num Cmp/Hyph <W:0.0> #1->0 "1600-logu"phon
:\n

should the output be something like this?

echo 1600-logu | modes/sme-normaliser.mode 
"<1600-logu>"
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "guhttanuppelotčuođi-logu"phon
		"guhttanuppelotčuođi" Num Sg Acc  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Gen  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Ill Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Loc Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Acc  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Gen  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Ill Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Loc Attr  "guhttanuppelotčuođi"phon
		"guhttanuppelotčuođi" Num Sg Nom  "guhttanuppelotčuođi"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "duhátguhttačuođi-logu"phon
		"duhátguhttačuođi" Num Sg Acc  "duhátguhttačuođi"phon
		"duhátguhttačuođi" Num Sg Nom  "duhátguhttačuođi"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "1600-logu"phon
		"1600" Num Cmp/Hyph <W:0.0> #1->0 "1600-logu"phon
:\n

or simply:

echo 1600-logu | modes/sme-normaliser.mode 
"<1600-logu>"
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "guhttanuppelotčuođi-logu"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "duhátguhttačuođi-logu"phon
	"lohku" N Sem/Amount Sg Gen <W:0.0> #1->0 @X "1600-logu"phon
:\n