Digits wrongly processed by divvun-suggest output awk script about giella-core HOT 15 OPEN

ah yeah these are all interesting ambiguities, a colon at the beginning of the line is used both by whitespace stuff in our cg format and suggestions now, the comma in second field is used to separate suggestions (though probably not empty strings) and spaces in lemma make field counting a bit ambiguous.

so, the current work arounds are a bit hacky.

from giella-core.

The problem with compounds seems to be when one analysis renders a ?, while the other renders the expected output:

:\n
"<Gållelibjes>"
	"libjes" N Sem/Dummytag Sg Nom
		"gålle" N Sem/Mat Cmp/SgGen Cmp
gålle+N+Cmp/SgGen+Cmp#libjes+N+Sg+Nom	?
	"libjes" N Sem/Dummytag Sg Nom
		"gålle" N Sem/Mat Cmp/SgNom Cmp
gålle+N+Cmp/SgNom+Cmp#libjes+N+Sg+Nom	Gållelibjes
"<:>"
	":" CLB
:+CLB	:
:

Here's an extreme case of the duplicate compound bug:

Bjørn Olav Megard le låhkåm cand.polit. sosialantropologijja oajvvefágajn Oslo universitehtas.

is turned into:

Bjørn Olav Megard le låhkåm cand.polit.cand.polit socialantropologidjasocialantropologiddjasocialantropologidjasocialantropologiddja oajvvefágajn Oslo universitehtas.

😀

It seems that in this case, none of the generated forms are identical to the input form (because the input is a NO form, and all output forms are SE forms), and the output forms differ among them. In such a case, we just go for the first one, we don't have the time to do anything more advanced.

from giella-core.

Might need more advanced parser or fix on divvun-suggest side for some of these cases where there's multiple different results, before the awk becomes too unwieldy... For quick patch maybe vislcg's -1 option for picking 1-random to tie break is good enough?

from giella-core.

The vislcg solution will only fix things on the analysis side. The problem is that the generation of the new word forms is ambiguous. And I don't know how to fix this quickly. Adding an option to divvun-suggest to unique the output would help, but is not enough. Also a flag on stderr to warn about unresolved ambiguity would help, there are not that many.

Any suggestion welcome 🙂

from giella-core.

Actually, it seems to be only two cases left:

• cand.polit., where the output matches the input, but is probably confused by the final full stop; an adjustment of the existing code should be enough
• sociala..., where none of the outputs match the input; as a last resort just go with what is at hand / is easiest

Since I don't know awk, I have no idea how hard this would be.

from giella-core.

I actually couldn't get cg-proc -1 to work anyways 😊

it's maybe less spaghetti than it might have been but the code tracks now kind of one output per cohort, which is not the best, but actually to control it cg rules might be the way to go.

For cand.polit. the variants are I think with and without a full stop and maybe one with :v at the end if I read correctly?

from giella-core.

With the changes in 63adeef it is almost perfect. What is still left are some cases where the awk script does not choose the word form that is identical to the input form even when there is such a word form. Example input:

Jus muhtema mielas nágin le vajáldahtedum, de dåhkki ájn ienep ulmutjijt libjjáj oajvvadit.

Earlier output (with multiple forms concatenated):

Jus muhtemuhtema mielas nágin le vajáldahtedum, de dåhkki ájn ienep ulmutjijt libjjáj oajvvadit.

New output after the latest fix:

Jus muhte mielas nágin le vajáldahtedum, de dåhkki ájn ienep ulmutjijt libjjáj oajvvadit.

Expected: since the input form is muhtema, and that word form is among the generated word forms, I had expected that one to be selected.

from giella-core.

Another error:

Äládus- ja oasesdepartementaoasesdepartemænnta

The æ in "Æládus" has become ä, but not in last part of compound, "departemænnta"

from giella-core.

Gehtjav makkár {åvddånbuktemvuohke} l gåvån = Gehtjav makkár {} l gåvån

åvddånbuktemvuohke e err/cmp.

from giella-core.

Anne Silja l aj åvdep giese journalisstan barggam {NRK} Sámeradio åvdås= Anne Silja l aj åvdep giese journalisstan barggam {NRK:A} Sámeradio åvdås

from giella-core.

Dán jagásj Bårjjåsin li guokta {vuostasj} artihkkala = Dán jagásj Bårjjåsin li guokta {vuostas} artihkkala

from giella-core.

Ja {gájkka} galggá sámegiellaj dáhpáduvvat = Ja {gájkav} galggá sámegiellaj dáhpáduvvat

from giella-core.

In retrospect, the processing should have been different:

• grab original word form by default
• grab (first) generated word form only when cohort contains the target tag

Ie keep everything as original unless there is reason to do otherwise.

We might still want to do this for future work for other languages and contexts.

from giella-core.

mm, a long term solution should definitely go in a one of the applications that can process whole cohorts and sentences with all information intact. But this prototype will be very useful for design we should collect and categorise the problems to keep in mind.

from giella-core.