Language filtering for NCI? about irish-bert HOT 4 OPEN

Experiments so far indicate that switching off the language filter does not harm and in fact may be improving results. (Statistical significance testing pending.) This probably means that having a small amount of English is no problem, or that filtering removes too much of Irish data. We decided in today's meeting not to filter the NCI.

from irish-bert.

Re-opening as James wants to try at least one more filter threshold.

from irish-bert.

Issue #4 reports: Looking at 3 examples of English sentences in random locations, they seem to unexpectedly occur in bursts after Irish sentences in the same document. Maybe the corpus is a snapshot of ongoing translation with incomplete parts defaulting to the source language or English. If we can confirm this, this information can be used in language classification: When the classifier is not highly confident it should go with the class of the neighbouring sentences.

from irish-bert.

Turns out yesterday's results cannot be used due to the bug identified in issue #39.

from irish-bert.