Git Product home page Git Product logo

giellalt / lang-crk Goto Github PK

View Code? Open in Web Editor NEW
13.0 26.0 1.0 14.54 MB

Finite state and Constraint Grammar based analysers and proofing tools, and language resources for the Plains Cree language

Home Page: https://giellalt.uit.no

License: Other

Makefile 1.32% Shell 3.86% M4 1.45% Python 0.12% Awk 0.21% Regular Expression 2.28% XML 0.11% YAML 11.99% Text 78.66%
finite-state-transducers constraint-grammar minority-language nlp language-resources proofing-tools indigenous-languages giellalt-langs maturity-beta geo-northamerica

lang-crk's Introduction

The Plains Cree morphology and tools

Maturity Lemma count GitHub issues Build Status License Desktop speller download Mobile speller download

👉🏼 Download the latest FSTs here (plains-cree-fsts*.zip) 👈🏼

👉 Documentation 👈🏼

This repository contains finite state source files for the Plains Cree language, for building morphological analysers, proofing tools and dictionaries. The data and implementation are licenced under LICENSE licence, also detailed in the LICENSE. The authors named in the AUTHORS file are available to grant other licencing choices.

Install proofing tools and keyboards for the Plains Cree language by using the Divvun Installer (some languages are only available via the nightly channel).

Download and test speller files

The speller files downloadable at the top of this page (the *.bhfst files) can be used with divvunspell, to test their performance. These files are the exact same ones as installed on users' computers and mobile phones. Desktop and mobile speller files differ from each other in the error model and should be tested separately — thus also two different downloads.

Documentation

Documentation can be found at:

Core dependencies

In order to compile and use the Plains Cree language morphology and dictionaries, you need:

To install VislCG3 and HFST, just copy/paste this into your Terminal on Mac OS X:

curl https://apertium.projectjj.com/osx/install-nightly.sh | sudo bash

or terminal on Ubuntu, Debian or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/apt/install-nightly.sh -O - | sudo bash
sudo apt-get install cg3 hfst

or terminal on RedHat, Fedora, CentOS or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/rpm/install-nightly.sh -O - | sudo bash
sudo dnf install cg3 hfst

Alternatively, the Apertium wiki has good instructions on how to install the dependencies for Mac OS X and how to install the dependencies on linux

Further details and dependencies are described on the GiellaLT Getting Started pages.

Downloading

Using Git:

git clone https://github.com/giellalt/lang-crk

Using Subversion:

svn checkout https://github.com/giellalt/lang-crk.git/trunk lang-crk

Building and installation

INSTALL describes the GNU build system in detail, but for most users it is the usual:

./autogen.sh # This will automatically clone or check out other GiellaLT dependencies
./configure
make
(as root) make install

Citing

If you use language data from more than one GiellaLT language, consider citing our LREC 2022 article on whole infra:

Linda Wiechetek, Katri Hiovain-Asikainen, Inga Lill Sigga Mikkelsen, Sjur Moshagen, Flammie Pirinen, Trond Trosterud, and Børre Gaup. 2022. Unmasking the Myth of Effortless Big Data - Making an Open Source Multi-lingual Infrastructure and Building Language Resources from Scratch. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1167–1177, Marseille, France. European Language Resources Association.

If you use bibtex, following is as it is on ACL anthology:

@inproceedings{wiechetek-etal-2022-unmasking,
    title = "Unmasking the Myth of Effortless Big Data - Making an Open Source
    Multi-lingual Infrastructure and Building Language Resources from Scratch",
    author = "Wiechetek, Linda  and
      Hiovain-Asikainen, Katri  and
      Mikkelsen, Inga Lill Sigga  and
      Moshagen, Sjur  and
      Pirinen, Flammie  and
      Trosterud, Trond  and
      Gaup, B{\o}rre",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation
    Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.125",
    pages = "1167--1177"
}

lang-crk's People

Contributors

aarppe avatar acl-sigel avatar albbas avatar andrewdotn avatar arnikki avatar atticusha avatar bbqsrc avatar eddieantonio avatar flammie avatar katieschmirler avatar leneantonsen avatar lynnda-hill avatar rtxanson avatar rueter avatar snomos avatar trondtr avatar uldai avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

andrewdotn

lang-crk's Issues

Regression: the descriptive analyzer no longer provides an analysis for "na nipat"

Did something change with the handling of reduplication?

 $ (echo "na nipat"; echo "e nipat") | hfst-lookup -q analyser-gt-desc.hfstol
na nipat	na nipat+?	inf

e nipat	PV/e+nipâw+V+AI+Cnj+3Sg	0.000000

This was a test case in cree-intelligent-dictionary, but no longer!

EDIT: this was the old analysis:

 $ echo "na nipat" | hfst-lookup -q crk-descriptive-analyzer.hfstol
na nipat	RdplW+nipâw+V+AI+Cnj+Prs+3Sg	0.000000

Incorrect stem for patahohkâtêw

patahohkâtêw < a VTA but stem listed as patahohkâtêw rather than patahohkât (t3?)

Issue originating in lexc or from Arok's materials?

More accurate labeling of FSTs?

@eddieantonio I've been pondering about this for a while, namely that the terms descriptive and normative are not really accurately descriptors of the varying FSTs. Rather, the terms strict (or standard) and relaxed might better represent the relationship of the FSTs with normative/standard orthography. Perhaps we could follow this with the FSTs created with quick.mk.

Fix import of certain VII stems

This has been fixed in the source code in a very ad hoc way. Basically, there was an error in the export of the source that has now been corrected. I've gone into src/fst/stems/verb_stems.lexc and changed all the errant VIIn that were classified as VIIw. This should fix these issues for now, but we will have to double check that no -n finals are in VIIw and vice versa when we re-export from source to our FST in the next update.

Originally posted by @atticusha in UAlbertaALTLab/morphodict#573 (comment)

Incorporate transcription of accented <ý> to simple <y> in lemmas on the analysis side

In order to more easily support using the same LEXC source for both crk and cwd (and other Western Cree dialects), we would want to incorporate a simple filter/transcriptor converting the accented <ý> to simple <y> to the analysis/underlying side of all the FSTs. For Woods Cree, we would similarly convert the accented <ý> to simple <th>. While the coding of <ý> is already implemented in the LEXC source on the stem side (for crk), this yet needs to be added to the lemma side of the LEXC code as well.

@snomos The REGEX source for the transcriptor has already been written and added to the repo as: https://github.com/giellalt/lang-crk/blob/main/src/filters/convert-accented-y-to-simple-y.regex - so what needs to be done is to incorporate this to the appropriate place(s) in the various Makefiles.

Possible incorrect definition for "aspi-"

Working on a transcription last night with several fluent speakers and we encountered aspi- in a few places, including this phrase:

ta-apsi-kâkîsimoyahk "to lean on our prayers/faith"

The fluent speakers asserted aspi- means "to lean on", however itwewina is showing that aspi- means "badly".

Is it possible the definition offered on itwêwina is incorrect?

Some particles not analyzing

The particle (pronoun?) êwako is not analyzing properly with the latest FSTs (Release fst-v2021.7.8):

❯ echo "êwako" | hfst-optimized-lookup -q crk-strict-analyzer.hfstol
êwako	êwako	+?

❯ echo "êwako" | hfst-optimized-lookup -q crk-relaxed-analyzer.hfstol
êwako	êwako	+?

I notice it does analyze with a y:

❯ echo "êyako" | hfst-optimized-lookup -q crk-strict-analyzer.hfstol
êyako	IC+RdplW+PV/ako+_+Cnj+Err/Frag

Though that particular analysis doesn't make sense to me 🤔

The README is... quite out of date

It still references the SVN stuff, has plenty of link rot, and is generally not very useful. Somebody (probably me) needs to update it!

Single incorrect VTA-4 form?

make check indicates the following for nakatêw (VTA-4) (the only verb with any genuinely dubious/incorrect form):

Test 0: Verb - nakatêw - ok (Lexical/Generation)
------------------------------------------------
[146/193][FAIL] nakatêw+V+TA+Imp+Imm+2Pl+3PlO => Missing results: nakatihkik
[146/193][FAIL] nakatêw+V+TA+Imp+Imm+2Pl+3PlO => Unexpected results: nakatihkok

Test 0 - Passes: 200, Fails: 1, Total: 201
----------------------------------------------
Test 1: Verb - nakatêw - ok (Surface/Analysis)
----------------------------------------------
[138/175][FAIL] nakatihkik => Missing results: nakatêw+V+TA+Imp+Imm+2Pl+3PlO 

@atticusha Which is correct? The YAML file form nakatihkik or the FST generated form nakatihkok?

Issues with autogen on initial instal

@snomos @aarppe

I am re-setting up the infrastructure on a new computer and when I run ./autogen.sh -l from lang-crk/, I receive the following error:

Looking for giella-core ...
giella-core found in ../

Looking for giella-shared ...
giella-shared found in ../

Initial automake setup of lang-crk

ERROR: could not find a login script to add the variable to!

For reference, I am using a zsh shell on MacOS 12.1 with an M1Pro chip.

Unable to make

Hi all. running the appropriate autopens and configs, I've found this error when trying to do an initial make.

make

*** Compiling the crk language. ***

CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /Users/atticus/lang-crk/build-aux/missing autoconf
Making all in .

*** Compiling the crk language. ***

CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /Users/atticus/lang-crk/build-aux/missing autoconf
make[1]: Nothing to be done for `all-am'.
Making all in src
Making all in fst
  XFST     lexicon.tmp.xfst

*** Warning: Ignoring info strings. ***
10000...

*** ERROR: Sublexicon is mentioned but not defined. (Symbols) ***
make[2]: *** [lexicon.tmp.xfst] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

This error is present on MacOS (10.15.4) and Windows 10. There seems to be a missing autoconf. Any idea on how to proceed at @snomos, @Trondtr ?

`minihkwahastimwêw' is not in the model

minihkwahastimwêw is not in the model, but is in the AW corpus. It is the case that minihkwahatimwêw returns on itwêwina, though this does not seem to have access to the inflectional paradigms morphology. Further, it doesn't seem to exist in my

minihkwahatimwêw does not appear to be analyzed by src/crk-relaxed-analyzer-giellaltbuild.hfstol. Despite this, the stem appears to exist in the verb stems file. Weirdly, it is coded as a VTA, despite returning as a VAI in itwêwina.

Regression in `PV/ka+{{lemma}}+V+AI+Ind+12P`

I mentioned this yesterday, there seems to be a regression in the rendering of the future definite 21P form for VAI verbs:

❯ echo 'PV/ka+atoskêw+V+AI+Ind+12P' | hfst-optimized-lookup --silent ./fsts/crk-normative-generator.hfstol 2>&1
!! Warning: file contains more than one transducer          !!
!! This is currently not handled - using only the first one !!
PV/ka+atoskêw+V+AI+Ind+12P	PV/ka+atoskêw+V+AI+Ind+12P	+?
❯ echo 'PV/ka+nikamow+V+AI+Ind+12P' | hfst-optimized-lookup --silent ./fsts/crk-normative-generator.hfstol 2>&1
!! Warning: file contains more than one transducer          !!
!! This is currently not handled - using only the first one !!
PV/ka+nikamow+V+AI+Ind+12P	PV/ka+nikamow+V+AI+Ind+12P	+?

I would expect those to yield

kika-atoskânaw
kika-atoskânânaw

and

kika-nikamonaw
kika-nikamonânaw

respectively.

This is using the v2020.12.4-post.0 FSTs.

./configure fails when looking for gt-version.sh

On MacOS 12.1, M1Pro chip

When running ./configure the following error is raised:

checking for gt-core.sh... false
checking whether we can set GIELLA_CORE... /Users/atticusharrigan/lang-crk/./../giella-core
checking for gt-version.sh... no
checking the version of the Giella Core... configure: error: gt-version.sh could not be found, installation is incomplete!

I have looked for gt-core.sh and gt-version.sh and found similar files with .in suffixes at the following locations:

/Users/atticusharrigan/giella-core/scripts/gt-version.sh.in
/Users/atticusharrigan/giella-core/scripts/gt-core.sh.in

I renamed these simply as .sh files and reran the configure command but was told that giella-core is too old (despite pulling this all just today).

@snomos @aarppe

Could not open 'analyser-gt-strict.hfst'

#25 was fixed, but new problems occur. Same script https://gist.github.com/TinoDidriksen/bc44df6cddf551454d58adc14b416192 and the nightly build yields this:

"/usr/local/bin/hfst-fst2fst"  -w -i analyser-gt-strict.hfst -o analyser-gt-strict.hfstol
/usr/local/bin/hfst-fst2fst: Could not open 'analyser-gt-strict.hfst'. No such file or directory
make[2]: *** [analyser-gt-strict.hfstol] Error 1

If I try it with -j1 then it stops at:

mv -f analyser-gt-strict.hfst crk-strict-analyzer-giellaltbuild.hfst
Creating dictionary version of crk-strict-analyzer-giellaltbuild.hfst
hfst-compose 1 --harmonize-flags -1 crk-strict-analyzer-giellaltbuild.hfst -2 crk-dict-filter.hfst |\
                hfst-minimize - -o crk-strict-analyzer-for-dictionary-giellaltbuild.hfst
hfst-compose: No more than two transducer files may be given
hfst-minimize: <stdin> is not a valid transducer file
make[2]: *** [crk-strict-analyzer-for-dictionary-giellaltbuild.hfst] Error 1

The macron variant is no longer included in the LO crk spell-checker OXT package

The LibreOffice OXT extension for crk no longer includes the macron variant. While being the less used SRO variant in comparison to the SRO circumflex one, we do have some super-users (e.g. AEW) who'd prefer to use the macron variant.

EDIT: the macron variant is not included in: crk_LO-voikko-5.0.oxt, but can be installed with crk-x-macron_LO-voikko-5.0.oxt. It used to be that one got both circumflex and macron SRO variants with crk_LO-voikko-5.0.oxt, but that appears no longer to be the case. Is that intentional?

miyêw still not working

kimîtin -> kimiyitin
nimîk -> nimiyik

among several others, likely due to underlying short i in suffixes.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.