giellalt / lang-nob Goto Github PK

Finite state and Constraint Grammar based analysers and proofing tools, and language resources for the Norwegian Bokmål language

Home Page: https://giellalt.uit.no

License: GNU General Public License v3.0

Makefile 1.74% Shell 2.46% M4 2.23% Ruby 1.05% Perl 0.27% XSLT 0.21% Regular Expression 0.84% XML 0.15% YAML 4.21% Text 86.85%

nlp language-resources constraint-grammar finite-state-transducers minority-language proofing-tools giellalt-langs maturity-prod geo-nordic langfam-indoeuropean

lang-nob's Introduction

The Norwegian Bokmål morphology and tools

Download nightly / CI/CD installation packages for testing (contains the core zhfst file(s)):

NB!! Note that the nightly / CI/CD installation packages are not tested for language quality, and might contain regressions and errors.

This repository contains finite state source files for the Norwegian Bokmål language, for building morphological analysers, proofing tools and dictionaries. The data and implementation are licenced under GPLv3 licence, also detailed in the LICENSE. The authors named in the AUTHORS file are available to grant other licencing choices.

Install proofing tools and keyboards for the Norwegian Bokmål language by using the Divvun Installer (some languages are only available via the nightly channel).

Download and test speller files

The speller files downloadable at the top of this page (the *.bhfst files) can be used with divvunspell, to test their performance. These files are the exact same ones as installed on users' computers and mobile phones. Desktop and mobile speller files differ from each other in the error model and should be tested separately — thus also two different downloads.

Documentation

Documentation can be found at:

Core dependencies

In order to compile and use Norwegian Bokmål language morphology and dictionaries, you need:

an FST compiler: HFST, Foma or Xerox Xfst
VislCG3 Constraint Grammar tools

To install VislCG3 and HFST, just copy/paste this into your Terminal on macOS:

curl https://apertium.projectjj.com/osx/install-nightly.sh | sudo bash

or terminal on Ubuntu, Debian or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/apt/install-nightly.sh -O - | sudo bash
sudo apt-get install cg3 hfst

or terminal on RedHat, Fedora, CentOS or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/rpm/install-nightly.sh -O - | sudo bash
sudo dnf install cg3 hfst

Alternatively, the Apertium wiki has good instructions on how to install the dependencies for Mac OS X and how to install the dependencies on linux

Further details and dependencies are described on the GiellaLT Getting Started pages.

Downloading

Using Git:

git clone https://github.com/giellalt/lang-nob

Using Subversion:

svn checkout https://github.com/giellalt/lang-nob.git/trunk lang-nob

Building and installation

INSTALL describes the GNU build system in detail, but for most users it is the usual:

./autogen.sh # This will automatically clone or check out other GiellaLT dependencies
./configure
make
(as root) make install

Citing

If you use language data from more than one GiellaLT language, consider citing our LREC 2022 article on whole infra:

Linda Wiechetek, Katri Hiovain-Asikainen, Inga Lill Sigga Mikkelsen, Sjur Moshagen, Flammie Pirinen, Trond Trosterud, and Børre Gaup. 2022. Unmasking the Myth of Effortless Big Data - Making an Open Source Multi-lingual Infrastructure and Building Language Resources from Scratch. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1167–1177, Marseille, France. European Language Resources Association.

If you use bibtex, following is as it is on ACL anthology:

@inproceedings{wiechetek-etal-2022-unmasking,
    title = "Unmasking the Myth of Effortless Big Data - Making an Open Source
    Multi-lingual Infrastructure and Building Language Resources from Scratch",
    author = "Wiechetek, Linda  and
      Hiovain-Asikainen, Katri  and
      Mikkelsen, Inga Lill Sigga  and
      Moshagen, Sjur  and
      Pirinen, Flammie  and
      Trosterud, Trond  and
      Gaup, B{\o}rre",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation
    Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.125",
    pages = "1167--1177"
}

lang-nob's People

Contributors

Stargazers

Watchers

lang-nob's Issues

Use/NG tag i analyse

Jeg får fjellene fjell+N+Neu+Pl+Def+Use/NG der jeg ville tro at jeg burde ha fått fjellene fjell+N+Neu+Pl+Def.

Det ser ut at som at en tag Use/NG har blitt med videre en plass hvor den kanskje skulle ha blitt tatt bort?

https://gtweb.uit.no/cgi-bin/smi/smi.cgi?text=fjellene&action=analyze&lang=nob&plang=eng

No analysis for "†"

The symbol "†" doesn't get any analysis:

echo "†" | hfst-tokenise --print-all --giella-cg --no-weights --unique tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst | vislcg3 --grammar tools/tokenisers/mwe-dis.bin | cg-mwesplit | vislcg3 --grammar src/cg3/disambiguator.bin | vislcg3 --grammar src/cg3/functions.cg3 | vislcg3 --grammar src/cg3/dependency.bin
:†\n

Test procedure does not ignore lexicon

Here is the lexicon in stems/adjectives.lexc:

LEXICON AmountCont ! gives 10-sidet, 27-årig etc.
sidet a4 ;
språklig a2 ;
års a3 ;
åring m1 ;
årig a2 ;

I tried to add the textstreng AmountCont after each entry, but it did not work, and the test does not pass. The same goes for a similar set in propernouns.

Fails to build in docker unless locales are set

TL,DR: When building in Docker, at least from the ubuntu image, one has to set a different locale in order to get the compilation to not crash. I just tested with en_US.UTF-8, which makes compilation succeed. However, if this means that everything is as it should be, I do not know.

(Can a solution be to just document this behaviour in the "getting started" pages, or is there some underlying cause of this error that can be fixed, so that compiling the model does not require setting these environment variables?)

The required lines in the Dockerfile:

FROM ubuntu:20.04
RUN apt-get update
# required to install the locales
RUN apt-get install locales
RUN locale-gen en_US.UTF-8
RUN update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
# some part of the build process seems to be happy when LANG and LC_ALL is set as such:
ENV DEBIAN_FRONTEND="noninteractive" TZ="Europe/Oslo" LANG="en_US" LC_ALL="en_US.UTF-8"

# .... the rest of the Dockerfile

If you however have a Dockerfile without the locale stuff, but only consisting of instructions very similar to those found in https://giellalt.github.io/infra/GettingStarted.html, like this one...:

$ cat Dockerfile.nob.failing
FROM ubuntu:20.04
RUN apt-get update

ENV DEBIAN_FRONTEND="noninteractive" TZ="Europe/Oslo"

RUN apt-get -y install bc curl git autoconf automake cmake libtool wget antiword wv python3-pip python3-bs4 python3-lxml python3-html5lib python3-feedparser python3-yaml python3-tidylib

RUN curl https://apertium.projectjj.com/apt/install-nightly.sh | bash
RUN apt-get -yf install apertium-all-dev cg3 hfst

# giella-core
WORKDIR /giellalt
RUN git clone --depth 1 https://github.com/giellalt/giella-core
WORKDIR /giellalt/giella-core
RUN ./autogen.sh
RUN ./configure
RUN make -j

# shared-mul
WORKDIR /giellalt
RUN git clone --depth 1 https://github.com/giellalt/shared-mul

RUN git clone --depth 1 https://github.com/giellalt/lang-nob
WORKDIR /giellalt/lang-nob
RUN ./autogen.sh
RUN ./configure --enable-fst-hyphenator --enable-spellers --enable-tokenisers --enable-phonetic --enable-tts
RUN make -j

...Then, trying to build the image using this command...:

$ docker build . -f Dockerfile.nob -t lang-nob

...the process fails, with the following error lines printed before the process terminates:

  HRGX2FST spellrestrict-nfd2nfc.hfst
/usr/bin/hfst-regexp2fst: spellrestrict-nfd2nfc.regex: XRE parsing failed in expression #1 separated by semicolons
make[2]: *** [Makefile:885: spellrestrict-nfd2nfc.hfst] Error 1
make[2]: *** Deleting file 'spellrestrict-nfd2nfc.hfst'
make[2]: *** Waiting for unfinished jobs....
/usr/bin/hfst-regexp2fst: spellrelax-nfc2nfd.regex: XRE parsing failed in expression #1 separated by semicolons
make[2]: *** [Makefile:885: spellrelax-nfc2nfd.hfst] Error 1
make[2]: *** Deleting file 'spellrelax-nfc2nfd.hfst'
rm spellrelax-with-flagtags.hfst spellrelax-mobile-keyboard.hfst spellrelax-with-tags.hfst inituppercase.hfst spellrelax-tags.hfst allcaps.hfst downcase-derived_proper-strings.hfst spellrelax.hfst
make[2]: Leaving directory '/giellalt/lang-nob/src/orthography'
make[1]: *** [Makefile:1212: all-recursive] Error 1
make[1]: Leaving directory '/giellalt/lang-nob/src'
make: *** [Makefile:540: all-recursive] Error 1
The command '/bin/sh -c make -j' returned a non-zero code: 2

./configure --enable-tokenisers causes make install to fail

Plain ./configure works

The error is:

make[3]: *** [Makefile:668: install-gielladataDATA] Error 1
make[3]: Leaving directory '/home/boerre/gut/giellalt/lang-nob/src/cg3'
make[2]: *** [Makefile:740: install-am] Error 2
make[2]: Leaving directory '/home/boerre/gut/giellalt/lang-nob/src/cg3'
make[1]: *** [Makefile:1176: install-recursive] Error 1
make[1]: Leaving directory '/home/boerre/gut/giellalt/lang-nob/src'
make: *** [Makefile:538: install-recursive] Error 1

Dates are not parsed correctly (nob) (

This issue was created automatically with bugzilla2github

Bugzilla Bug 2427

Date: 2017-09-27T14:10:44+02:00
From: Børre Gaup <<borre.gaup>>
To: Trond Trosterud <<trond.trosterud>>
CC: ciprian.gerstenberger, lene.antonsen

Blocker for: #2405
Last updated: 2017-10-04T16:44:47+02:00

No analysis for "EØS‑regelverket"

I get the following:

echo "EØS‑regelverket" | hfst-tokenise --print-all --giella-cg --no-weights --unique tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst | vislcg3 --grammar tools/tokenisers/mwe-dis.bin | cg-mwesplit | vislcg3 --grammar src/cg3/disambiguator.bin | vislcg3 --grammar src/cg3/functions.cg3 | vislcg3 --grammar src/cg3/dependency.bin
:EØS‑regelverket\n

Merkeleg analyse av ACR-Prop

Sju akronym i fila nob-propernouns.lexc fører til at lemmagenereringstesten ikkje går gjennom (med kommentar om at dei ikkje skal vera med, men han funkar ikkje):

AKP PROP-org ;  ! not-lemmatest
AMI PROP-org ;  ! not-lemmatest
ANC PROP-org ;  ! not-lemmatest
ANSI PROP-org ; ! not-lemmatest
AOF PROP-org ;  ! not-lemmatest
AP PROP-org ;   ! not-lemmatest
AUF PROP-org ;  ! not-lemmatest

Når eg ser på analysen av dessa akronyma så er det større problem enn at dei ikkje går gjennom lemmatestinga:

echo ANSI | hfst-lookup -q src/analyser-gt-norm.hfstol
ANSI	ANSI+N+Prop+Sem/Org+ACR+N+Prop+ACR+Msc+Sg+Indef	0.000000
ANSI	ANSI+Use/Circ+Use/SpellNoSugg+N+Prop+Sem/Org+ACR+Dyn+Msc+Sg+Indef	0.000000

(den andre analysen er dynamisk, den kan ignorerast).

Analysen stemmer ikkje i det heile med LexC-koden:

LEXICON Root
   ProperNoun      ; !!≈ * @CODE@

LEXICON ProperNoun
 @U.CmpHyph.TRUE@ ProperNoun-nob-nocomp ;

LEXICON ProperNoun-nob-nocomp
AKP PROP-org ;  ! not-lemmatest
AMI PROP-org ;  ! not-lemmatest
ANC PROP-org ;  ! not-lemmatest
ANSI PROP-org ; ! not-lemmatest
AOF PROP-org ;  ! not-lemmatest
AP PROP-org ;   ! not-lemmatest
AUF PROP-org ;  ! not-lemmatest

LEXICON PROP-org
+N+Prop+Sem/Org: K ;

LEXICON K 
          ENDLEX ;

LEXICON ENDLEX
 @D.CmpOnly.FALSE@@D.CmpPref.TRUE@@D.NeedNoun.ON@ # ;

(kode direkte frå kjeldefilene, utan kommentarar og irrelevante stigar.)

Det er INGENTING i denne koden som kan gje den analysen vi faktisk får. Eg forstår ikkje dette, og det er samtidig dette som gjer at testane ikkje går gjennom. Med den venta analysen hadde det ikkje vore problem med lemmagenereringa.

No analysis for ""