Git Product home page Git Product logo

giellalt / lang-myv Goto Github PK

View Code? Open in Web Editor NEW
2.0 27.0 0.0 97.6 MB

Finite state and Constraint Grammar based analysers and proofing tools, and language resources for the Erzya language

Home Page: https://giellalt.uit.no

License: GNU Lesser General Public License v3.0

Makefile 0.55% Shell 0.74% M4 0.72% Python 16.92% Perl 0.03% XSLT 0.21% Regular Expression 0.73% XML 0.04% YAML 1.41% Text 78.65%
finite-state-transducers constraint-grammar minority-language nlp language-resources proofing-tools giellalt-langs maturity-beta geo-russia langfam-uralic

lang-myv's Introduction

The Erzya morphology and tools

Maturity Lemma count GitHub issues License Doc Build Status CI/CD Build Status

Download nightly / CI/CD installation packages for testing (contains the core zhfst file(s)):

Windows MacOS Mobile

NB!! Note that the nightly / CI/CD installation packages are not tested for language quality, and might contain regressions and errors.

This repository contains finite state source files for the Erzya language, for building morphological analysers, proofing tools and dictionaries. The data and implementation are licenced under GNU GPL licence, also detailed in the LICENSE. The authors named in the AUTHORS file are available to grant other licencing choices.

Install proofing tools and keyboards for the Erzya language by using the Divvun Installer (some languages are only available via the nightly channel).

Download and test speller files

The speller files downloadable at the top of this page (the *.bhfst files) can be used with divvunspell, to test their performance. These files are the exact same ones as installed on users' computers and mobile phones. Desktop and mobile speller files differ from each other in the error model and should be tested separately — thus also two different downloads.

Documentation

Documentation can be found at:

Core dependencies

In order to compile and use Erzya language morphology and dictionaries, you need:

To install VislCG3 and HFST, just copy/paste this into your Terminal on macOS:

curl https://apertium.projectjj.com/osx/install-nightly.sh | sudo bash

or terminal on Ubuntu, Debian or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/apt/install-nightly.sh -O - | sudo bash
sudo apt-get install cg3 hfst

or terminal on RedHat, Fedora, CentOS or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/rpm/install-nightly.sh -O - | sudo bash
sudo dnf install cg3 hfst

Alternatively, the Apertium wiki has good instructions on how to install the dependencies for Mac OS X and how to install the dependencies on linux

Further details and dependencies are described on the GiellaLT Getting Started pages.

Downloading

Using Git:

git clone https://github.com/giellalt/lang-myv

Using Subversion:

svn checkout https://github.com/giellalt/lang-myv.git/trunk lang-myv

Building and installation

INSTALL describes the GNU build system in detail, but for most users it is the usual:

./autogen.sh # This will automatically clone or check out other GiellaLT dependencies
./configure
make
(as root) make install

Citing

Rueter, J., Hämäläinen, M., & Partanen, N. (2020). Open-Source Morphology for Endangered Mordvinic Languages. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS) (pp. 94–100). The Association for Computational Linguistics.

    @inproceedings{rueter-etal-2020-open,
        title = "Open-Source Morphology for Endangered Mordvinic Languages",
        author = {Rueter, Jack  and
          H{\"a}m{\"a}l{\"a}inen, Mika  and
          Partanen, Niko},
        booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
        month = nov,
        year = "2020",
        address = "Online",
        publisher = "Association for Computational Linguistics",
        url = "https://aclanthology.org/2020.nlposs-1.13",
        doi = "10.18653/v1/2020.nlposs-1.13",
        pages = "94--100"
    }

If you use language data from more than one GiellaLT language, consider citing our LREC 2022 article on whole infra:

Linda Wiechetek, Katri Hiovain-Asikainen, Inga Lill Sigga Mikkelsen, Sjur Moshagen, Flammie Pirinen, Trond Trosterud, and Børre Gaup. 2022. Unmasking the Myth of Effortless Big Data - Making an Open Source Multi-lingual Infrastructure and Building Language Resources from Scratch. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1167–1177, Marseille, France. European Language Resources Association.

If you use bibtex, following is as it is on ACL anthology:

@inproceedings{wiechetek-etal-2022-unmasking,
    title = "Unmasking the Myth of Effortless Big Data - Making an Open Source
    Multi-lingual Infrastructure and Building Language Resources from Scratch",
    author = "Wiechetek, Linda  and
      Hiovain-Asikainen, Katri  and
      Mikkelsen, Inga Lill Sigga  and
      Moshagen, Sjur  and
      Pirinen, Flammie  and
      Trosterud, Trond  and
      Gaup, B{\o}rre",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation
    Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.125",
    pages = "1167--1177"
}

lang-myv's People

Contributors

albbas avatar arnikki avatar bbqsrc avatar ciprian-no avatar flammie avatar ftyers avatar mikahama avatar rtxanson avatar rueter avatar snomos avatar trondtr avatar trondtynnol avatar unhammer avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lang-myv's Issues

Attempt at establishing distinction between -dict-gt-norm- and -gt-norm fails

Four example words have been selected to provide the *e vs *ä distinction found in the manuscript of the monolingual Erzya dictionary by Kuzʹma Abramov.
In the lexc file we have:

пей+N:пӓй
сэдь+N:сӓдь
седей+N:сьӓдей
эрзя+N:ӓрзя

‹ӓ› has been declared in twolc

the filter: ‹remove-diaereses-enhancement.regex› looks like this:

[[ Ь | ь ] -> 0 ||  _ [ ӓ | Ӓ ] ,, ӓ -> е || [ ь | Ь ]  _ ]   
.o.
ӓ -> е || [ в | В | б | Б | г | Г | ж | Ж | к | К | м | М | п | П | ф | Ф | х | Х | ч | Ч | ш | Ш | щ | Щ ] _ 
.o.
ӓ -> э || [ д | Д | з | З | л | Л | н | Н | р | Р | с | С | т | Т | ц | Ц ] _	
.o.
ӓ -> э || [ .#. | %- ] _ ;

So, there are a number of things going on in one place.
Line 1 removes underlying soft sign preceding underlying ӓ and simultaneously replaces underlying ӓ with е. (failure)
Line 2 replaces underlying ‹ӓ› with ‹е›. (partial success)
Line 3 replaces underlying ‹ӓ› with ‹э› following specific consonants. (partial success)
Line 4 replaces underlying ‹ӓ› with ‹э› word-initially. (partial success)

The script remove-diaereses-enhancement.hfst is called in
lang-myv/src/fst/Makefile.am and lang-myv/src/fst/filters/Makefile.am

The desired result for the four words give above would be:
Analysis

lang-myv jackrueter$ hfst-lookup src/fst/analyser-gt-norm.hfstol 
> пей
пей	пей+N+Sg+Nom+Indef	0,000000

> сэдь
сэдь	сэдь+N+Sg+Nom+Indef	0,000000

> седей
седей	седей+N+Sg+Nom+Indef	0,000000

> эрзя
эрзя	эрзя+N+Sg+Nom+Indef	0,000000

Dict-Generation:

lang-myv jackrueter$ hfst-lookup src/fst/generator-dict-gt-norm.hfst 
hfst-lookup: warning: It is not possible to perform fast lookups with foma format automata.
Using HFST basic transducer format and performing slow lookups
> пей+N+Sg+Nom+Indef
пей+N+Sg+Nom+Indef	пӓй	0,000000

> сэдь+N+Sg+Nom+Indef
сэдь+N+Sg+Nom+Indef	сӓдь	0,000000

> седей+N+Sg+Nom+Indef
седей+N+Sg+Nom+Indef	сьӓдей	0,000000

> эрзя+N+Sg+Nom+Indef
эрзя+N+Sg+Nom+Indef	ӓрзя	0,000000

Generation:

lang-myv jackrueter$ hfst-lookup src/fst/generator-gt-norm.hfstol
> пей+N+Sg+Nom+Indef
пей+N+Sg+Nom+Indef	пей	0,000000

> сэдь+N+Sg+Nom+Indef
сэдь+N+Sg+Nom+Indef	сэдь	0,000000

> седей+N+Sg+Nom+Indef
седей+N+Sg+Nom+Indef	седей	0,000000

> эрзя+N+Sg+Nom+Indef
эрзя+N+Sg+Nom+Indef	эрзя	0,000000

Instead, I get:
Analysis

lang-myv jackrueter$ hfst-lookup src/fst/analyser-gt-norm.hfstol 
> пей
пей	пей+?	inf

> сэдь
сэдь	сэдь+?	inf

> седей
седей	седей+?	inf

> эрзя
эрзя	эрзя+?	inf

Dict-Generation:

lang-myv jackrueter$ hfst-lookup src/fst/generator-dict-gt-norm.hfst 
hfst-lookup: warning: It is not possible to perform fast lookups with foma format automata.
Using HFST basic transducer format and performing slow lookups
> пей+N+Sg+Nom+Indef
пей+N+Sg+Nom+Indef	пӓй	0,000000

> сэдь+N+Sg+Nom+Indef
сэдь+N+Sg+Nom+Indef	сӓдь	0,000000

> седей+N+Sg+Nom+Indef
седей+N+Sg+Nom+Indef	седей+N+Sg+Nom+Indef+?	inf

> эрзя+N+Sg+Nom+Indef
эрзя+N+Sg+Nom+Indef	ӓрзя	0,000000

Generation:

lang-myv jackrueter$ hfst-lookup src/fst/generator-gt-norm.hfstol 
> пей+N+Sg+Nom+Indef
пей+N+Sg+Nom+Indef	пей	0,000000

> сэдь+N+Sg+Nom+Indef
сэдь+N+Sg+Nom+Indef	сэдь	0,000000

> седей+N+Sg+Nom+Indef
седей+N+Sg+Nom+Indef	седей+N+Sg+Nom+Indef+?	inf

> эрзя+N+Sg+Nom+Indef
эрзя+N+Sg+Nom+Indef	ӓрзя	0,000000

Speller does not recognize words with touching full stop

The Erzya speller does not recognize words with touching full stop.

This may have something to do with Cyrillic-letter range.

Лов куцятне палыть чеерь толкс...
Аволь вейке монгак правтынь сельведь,
Кода совинь Мавзолеенть потс.

Весе содыть, кие течи кулось,
Ансяк тенень кемемс пек стака.

Here full stops, both ‹.› and ‹...› following a word will cause it to not be recognized.
If a space is inserted, the word form is recognized.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.