topic: Bug in analysis of our corpustexts for forthcoming SIKOR update.

problem: \ symbol either misses dependency node or get ghost analysis, where the dependency node is the final #n->m tag of each regel, and the ghost analysis is the analysis of the non-existing (=empty) character "<>".

to repeat: Run the following pipeline for one of two fst's, where the analysis pipeline in both cases is the same (standing in lang-xxx):

echo "ja \ ja" | hfst-tokenise -cg tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst |vislcg3 -g src/cg3/disambiguator.cg3 |vislcg3 -g src/cg3/functions.cg3 | vislcg3 -g src/cg3/dependency.cg3

Option (a): run the command above with \ as lemma, where the file src/fst/generated-files/symbols.lexc has an the following entry for backslash:

\   Noun_symbols_never_inflected       ;

(this is the case today):

The analysis is:

"<ja>"
	"ja" CC <W:0.0> @X #1->0
: 
"<\>"
	"\" N Symbol <W:0.0>
: 
"<ja>"
	"ja" CC <W:0.0> @X #3->0
:\n

What is missing is the dependency node (see the rightmost node #1 and #3 (sic) on the two other words).

Option (b): In the symbols file, set backslash (or whatever) as lemma, \ as stem, the entry is then:

backslash:\  Noun_symbols_never_inflected       ;

Now recompile, run the same command, and the analysis is:

"<ja>"
	"ja" CC <W:0.0> @CVP #1->0
: 
"<\>"
	"backslash" N Symbol <W:0.0> @X #2->0
"<>"
	"'" PUNCT <W:0.0> #3->4
: 
"<ja>"
	"ja" CC <W:0.0> @CVP #4->0
:\n
"<>"
	"'" PUNCT <W:0.0> #5->5

Note that the dipendency node now is in place. Good. But the downside is the ghost analysis of "<>", that we do not want.

Neither option is optimal, but the former one is worst: Here, the dep node #3->0 is missing, and the analysis for our corpus stops. For the latter version, we get the dep node, as can be seen, but here it gets an empty reading of "<>" (with a depnode) as an unwanted passenger.

Now, (b) does not give exactly what we want, but (a) gives us no analysis at all. The Best Solution would be to have dep analysis and no ghost analysis.

When building the mobile speller for divvun.org, the compilation fails

./configure --without-forrest --enable-spellers --disable-hfst-desktop-spellers --prefix=$HOME/.local --enable-hfst-mobile-speller

/usr/bin/hfst-lexc: The file lexicon.tmp.lexc did not compile cleanly.