Implements lookup for Foma finite state transducers.
Supports Python 3.5 and up.
pip install fst-lookup
Import the library, and load an FST from a file:
Hint: Test this module by downloading the
eatFST!
>>> from fst_lookup import FST
>>> fst = FST.from_file('eat.fomabin')fst_lookup assumes that the lower label corresponds to the surface
form, while the upper label corresponds to the lemma, and linguistic
tags and features: e.g., your LEXC will look something like
this—note what is on each side of the colon (:):
Multichar_Symbols +N +Sg +Pl
Lexicon Root
cow+N+Sg:cow #;
cow+N+Pl:cows #;
goose+N+Sg:goose #;
goose+N+Pl:geese #;
sheep+N+Sg:sheep #;
sheep+N+Pl:sheep #;
If your FST has labels on the opposite sides—e.g., the upper label
corresponds to the surface form and the upper label corresponds to
the lemma and linguistic tags—then instantiate the FST by providing
the labels="invert" keyword argument:
fst = FST.from_file('eat-inverted.fomabin', labels="invert")Hint: FSTs originating from the HFST suite are often inverted, so try to loading the FST inverted first if
.generate()or.analyze()aren't working correctly!
To analyze a form (take a word form, and get its linguistic analyzes)
call the analyze() function:
def analyze(self, surface_form: str) -> Iterator[Analysis]This will yield all possible linguistic analyses produced by the FST.
An analysis is a tuple of strings. The strings are either linguistic tags, or the lemma (base form of the word).
FST.analyze() is a generator, so you must call list() to get a list.
>>> list(sorted(fst.analyze('eats')))
[('eat', '+N', '+Mass'),
('eat', '+V', '+3P', '+Sg')]To generate a form (take a linguistic analysis, and get its concrete
word forms), call the generate() function:
def generate(self, analysis: str) -> Iterator[str]FST.generate() is a Python generator, so you must call list() to get
a list.
>>> list(fst.generate('eat+V+Past')))
['ate']If you plan to contribute code, it is recommended you use Poetry. Fork and clone this repository, then install development dependencies by typing:
poetry install
Then, do all your development within a virtual environment, managed by Poetry:
poetry shell
This project uses mypy to check static types. To invoke it on this
package, type the following:
mypy -p fst_lookup
To run this project's tests, we use py.test:
poetry run pytest
Building the C extension is handled in build.py
To disable building the C extension, add the following line to .env:
export FST_LOOKUP_BUILD_EXT=False(by default, this is True).
To enable debugging flags while working on the C extension, add the
following line to .env:
export FST_LOOKUP_DEBUG=TRUE(by default, this is False).
If you are creating or modifying existing test fixtures (i.e., mostly pre-built FSTs used for testing), you will need the following dependencies:
- GNU
make - Foma
Fixtures are stored in tests/data/. Here, you will use make to
compile all pre-built FSTs from source:
make
Copyright © 2019–2021 National Research Council Canada.
Licensed under the MIT license.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent