Git Product home page Git Product logo

hfst-python's Introduction

HFST for python

CI/CD CI - Test, Build, and Publish
Package PyPI - Version PyPI - Downloads PyPI - Python Version
Meta CI Buildwheel Badge linting - Ruff PyPI - License

Package description

Package hfst contains python bindings for the Helsinki Finite-State Technology (HFST) C++ library. HFST toolkit is intended for processing natural language morphologies. The toolkit is demonstrated by wide-coverage implementations of a number of languages of varying morphological complexity.

Installation

For most users, simply run...

$ python3 -m pip install hfst

On some OSes, you can install hfst python bindings by running install_nightly.sh.

We compile wheels using cibuildwheel which enables us to publish wheels for CPython and PyPy on a large variety of OS/architecture combinations. If wheels for your platform are not available, open an issue! (Windows support coming soon!)

Usage

C++ side functions and classes are wrapped with SWIG under module 'libhfst'. It is possible to use this module directly, but there is a package named 'hfst' which encapsulates the libhfst module in a more user-friendly manner. The structure of the package is

  • hfst
    • hfst.exceptions
    • hfst.sfst_rules
    • hfst.xerox_rules

The module hfst.exceptions contains HfstException and its subclasses. The modules hfst.sfst_rules and hfst.xerox_rules contain functions that create transducers implementing replace and two-level rules. All other functions and classes are in module hfst.

For documentation and examples, see https://hfst.github.io/python/index.html.

Requirements

Compiling hfst from source requires at least C++ compiler (tested with gcc 5.4.0), readline and getline libraries and setuptools package for python (tested with version 28.8.0). Swig is no longer needed as pre-generated files are included in source distribution.

Compiling from scratch

This repository has a submodule with the underlying C++ code. The first time you clone this repository, run $ git submodule init to initialize the submodule. Thereafter, every time that you want to pull in the latest changes from the C++ hfst repository, run $ git submodule update --remote or $ git pull --recurse-submodules. See the manylinux build script for an example of how to compile the underlying C++ library.

Once the library is available, the package can be installed by running...

python3 -m pip install .

...in the root directory of the repository.

Running tests

Tests are contained in the test/ directory. To run tests, you must first install pytest using python3 -m pip install pytest. Then, in the root directory of this repository, run python3 -m pytest.

Documentation

See wiki-based package documentation on our Github pages. In python, you can also use dir and help commands, e.g.:

dir(hfst)

help(hfst.HfstTransducer)

License

HFST is licensed under Gnu GPL version 3.0.

Troubleshooting

Pip starts to compile from source although there is a wheel available

Try upgrading pip with

python3 -m pip install --upgrade pip

Another reason for this can be that the source package on PyPI is newer (i.e. has a higher version number) than the corresponding wheel for the given environment. Report this via our issue tracker so a fresh wheel can be created.

Error message "command ... failed with error code ..."

Try rerunning pip in verbose mode with

python3 -m pip install --verbose [--upgrade] hfst

to get more information.

TypeError: catching classes that do not inherit from BaseException is not allowed

Some version combinations of SWIG and Python make HFST exception classes subclasses of Python's object instead of Exception. Then you will get the error above. If this is the case, run...

sed -i 's/class HfstException(_object):/class HfstException(Exception):/' libhfst.py

...after build/installation to be able to use HfstException and its subclasses in Python.

Links

HFST project main page: more information about the project

Github issue tracker: for comments, feature requests and bug reports

hfst-python's People

Contributors

reynoldsnlp avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

hfst-python's Issues

error: unknown type name '_Bool'

Both the macos and musllinux builds fail in the same way, error: unknown type name '_Bool' in back-ends/foma/fomalib.h while compiling ConvertFomaTransducer.cc.

Here is the log from macos:

2022-02-23T21:59:07.3340180Z depbase=`echo ConvertFomaTransducer.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
2022-02-23T21:59:07.3341840Z 	/bin/sh ../../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../libhfst/src  -I../../../libhfst/src -I../../../back-ends -I/usr/local/include  -I../../../back-ends/foma  -I../../../back-ends/openfst/src/include -I/usr/local/opt/flex/include -I/usr/local/include/unicode -Wno-deprecated -g -g -O2 -std=c++14 -msse -msse2 -mfpmath=sse -MT ConvertFomaTransducer.lo -MD -MP -MF $depbase.Tpo -c -o ConvertFomaTransducer.lo ConvertFomaTransducer.cc &&\
2022-02-23T21:59:07.3342690Z 	mv -f $depbase.Tpo $depbase.Plo
2022-02-23T21:59:07.4675430Z libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../libhfst/src -I../../../libhfst/src -I../../../back-ends -I/usr/local/include -I../../../back-ends/foma -I../../../back-ends/openfst/src/include -I/usr/local/opt/flex/include -I/usr/local/include/unicode -Wno-deprecated -g -g -O2 -std=c++14 -msse -msse2 -mfpmath=sse -MT ConvertFomaTransducer.lo -MD -MP -MF .deps/ConvertFomaTransducer.Tpo -c ConvertFomaTransducer.cc  -fno-common -DPIC -o .libs/ConvertFomaTransducer.o
2022-02-23T21:59:07.5523090Z In file included from ConvertFomaTransducer.cc:23:
2022-02-23T21:59:07.5523880Z ../../../back-ends/foma/fomalib.h:219:8: error: unknown type name '_Bool'
2022-02-23T21:59:07.5524270Z inline _Bool fsm_set_option(unsigned long long option, void *value) {
2022-02-23T21:59:07.5524850Z        ^

Here is the log from musllinux (alpine): (the error is listed before the actual command because of a race condition)

2022-02-24T01:03:44.2367382Z depbase=`echo ConvertFomaTransducer.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
2022-02-24T01:03:44.3744201Z In file included from ConvertFomaTransducer.cc:23:
2022-02-24T01:03:44.3754033Z ../../../back-ends/foma/fomalib.h:219:8: error: '_Bool' does not name a type
2022-02-24T01:03:44.3754755Z   219 | inline _Bool fsm_set_option(unsigned long long option, void *value) {
2022-02-24T01:03:44.3755271Z       |        ^~~~~
2022-02-24T01:03:45.1335705Z make[3]: *** [Makefile:1356: ConvertFomaTransducer.lo] Error 1
2022-02-24T01:03:45.1336571Z make[2]: *** [Makefile:1216: all-recursive] Error 1
2022-02-24T01:03:45.1337020Z make[1]: *** [Makefile:844: all] Error 2
2022-02-24T01:03:45.1337431Z make: *** [Makefile:451: all-recursive] Error 1
2022-02-24T01:03:45.4910120Z /bin/sh ../../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../libhfst/src  -I../../../libhfst/src -I../../../back-ends -I/usr/include  -I../../../back-ends/foma  -I../../../back-ends/openfst/src/include  -Wno-deprecated -g -g -O2 -std=c++14 -msse -msse2 -mfpmath=sse -MT ConvertFomaTransducer.lo -MD -MP -MF $depbase.Tpo -c -o ConvertFomaTransducer.lo ConvertFomaTransducer.cc &&\
2022-02-24T01:03:45.4911654Z mv -f $depbase.Tpo $depbase.Plo
2022-02-24T01:03:45.4913416Z libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../libhfst/src -I../../../libhfst/src -I../../../back-ends -I/usr/include -I../../../back-ends/foma -I../../../back-ends/openfst/src/include -Wno-deprecated -g -g -O2 -std=c++14 -msse -msse2 -mfpmath=sse -MT ConvertFomaTransducer.lo -MD -MP -MF .deps/ConvertFomaTransducer.Tpo -c ConvertFomaTransducer.cc  -fPIC -DPIC -o .libs/ConvertFomaTransducer.o

Here is the output from a SUCCESSFUL compile (manylinux2010):

2022-02-23T21:55:36.6559408Z depbase=`echo ConvertFomaTransducer.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
2022-02-23T21:55:36.6560849Z 	/bin/sh ../../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../libhfst/src  -I../../../libhfst/src -I../../../back-ends -I/usr/local/include  -I../../../back-ends/foma  -I../../../back-ends/openfst/src/include  -Wno-deprecated -g -g -O2 -std=c++14 -msse -msse2 -mfpmath=sse -MT ConvertFomaTransducer.lo -MD -MP -MF $depbase.Tpo -c -o ConvertFomaTransducer.lo ConvertFomaTransducer.cc &&\
2022-02-23T21:55:36.6562447Z 	mv -f $depbase.Tpo $depbase.Plo
2022-02-23T21:55:36.6563740Z libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../libhfst/src -I../../../libhfst/src -I../../../back-ends -I/usr/local/include -I../../../back-ends/foma -I../../../back-ends/openfst/src/include -Wno-deprecated -g -g -O2 -std=c++14 -msse -msse2 -mfpmath=sse -MT ConvertFomaTransducer.lo -MD -MP -MF .deps/ConvertFomaTransducer.Tpo -c ConvertFomaTransducer.cc  -fPIC -DPIC -o .libs/ConvertFomaTransducer.o
2022-02-23T21:55:36.6565690Z libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../libhfst/src -I../../../libhfst/src -I../../../back-ends -I/usr/local/include -I../../../back-ends/foma -I../../../back-ends/openfst/src/include -Wno-deprecated -g -g -O2 -std=c++14 -msse -msse2 -mfpmath=sse -MT ConvertFomaTransducer.lo -MD -MP -MF .deps/ConvertFomaTransducer.Tpo -c ConvertFomaTransducer.cc -o ConvertFomaTransducer.o >/dev/null 2>&1

Windows build is failing on tr1/unordered_map

The command that it fails on is in back-ends/openfstwin/src/lib:

libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src -I ./../include -I ./../../../dlfcn -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include -g -O2 -std=c++11 -MT fst.lo -MD -MP -MF .deps/fst.Tpo -c fst.cc -DDLL_EXPORT -DPIC -o .libs/fst.o

  make[3]: Entering directory '/d/a/hfst-python/hfst-python/libhfst_src/back-ends/openfstwin/src/lib'
  depbase=`echo compat.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
  /bin/sh ../../../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src  -I ./../include -I ./../../../dlfcn  -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include   -g -O2 -std=c++11 -MT compat.lo -MD -MP -MF $depbase.Tpo -c -o compat.lo compat.cc &&\
  mv -f $depbase.Tpo $depbase.Plo
  libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src -I ./../include -I ./../../../dlfcn -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include -g -O2 -std=c++11 -MT compat.lo -MD -MP -MF .deps/compat.Tpo -c compat.cc  -DDLL_EXPORT -DPIC -o .libs/compat.o
  libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src -I ./../include -I ./../../../dlfcn -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include -g -O2 -std=c++11 -MT compat.lo -MD -MP -MF .deps/compat.Tpo -c compat.cc -o compat.o >/dev/null 2>&1
  depbase=`echo flags.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
  /bin/sh ../../../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src  -I ./../include -I ./../../../dlfcn  -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include   -g -O2 -std=c++11 -MT flags.lo -MD -MP -MF $depbase.Tpo -c -o flags.lo flags.cc &&\
  mv -f $depbase.Tpo $depbase.Plo
  libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src -I ./../include -I ./../../../dlfcn -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include -g -O2 -std=c++11 -MT flags.lo -MD -MP -MF .deps/flags.Tpo -c flags.cc  -DDLL_EXPORT -DPIC -o .libs/flags.o
  libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src -I ./../include -I ./../../../dlfcn -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include -g -O2 -std=c++11 -MT flags.lo -MD -MP -MF .deps/flags.Tpo -c flags.cc -o flags.o >/dev/null 2>&1
  depbase=`echo fst.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
  /bin/sh ../../../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src  -I ./../include -I ./../../../dlfcn  -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include   -g -O2 -std=c++11 -MT fst.lo -MD -MP -MF $depbase.Tpo -c -o fst.lo fst.cc &&\
  mv -f $depbase.Tpo $depbase.Plo
  libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../libhfst/src -I ./../include -I ./../../../dlfcn -DMSC_VER -DOPENFSTEXPORT -I/usr/include -I/mingw/include -g -O2 -std=c++11 -MT fst.lo -MD -MP -MF .deps/fst.Tpo -c fst.cc  -DDLL_EXPORT -DPIC -o .libs/fst.o
  In file included from C:/msys64/mingw64/include/c++/10.2.0/tr1/unordered_map:40,
                   from ./../include/fst/unordered_map.h:11,
                   from ./../include/fst/util.h:24,
                   from ./../include/fst/weight.h:82,
                   from ./../include/fst/pair-weight.h:29,
                   from ./../include/fst/expectation-weight.h:36,
                   from ./../include/fst/arc.h:28,
                   from ./../include/fst/fst.h:34,
                   from fst.cc:21:
  C:/msys64/mingw64/include/c++/10.2.0/tr1/functional_hash.h:79:3: error: template-id 'operator()<>' for 'std::size_t std::tr1::hash<long long unsigned int>::operator()(long long unsigned int) const' does not match any template declaration
     79 |   _TR1_hashtable_define_trivial_hash(unsigned long long);
        |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  In file included from ./../include/fst/fst.h:31,
                   from fst.cc:21:
  ./../include/fst/compat.h:154:10: note: candidate is: 'std::size_t std::tr1::hash<long long unsigned int>::operator()(uint64) const'
    154 |   size_t operator()(uint64 x) const { return x; }
        |          ^~~~~~~~
  In file included from ./../include/fst/sparse-power-weight.h:28,
                   from ./../include/fst/arc.h:34,
                   from ./../include/fst/fst.h:34,
                   from fst.cc:21:
  ./../include/fst/sparse-tuple-weight.h:41:14: error: 'unordered_multimap' has not been declared in 'std'
     41 |   using std::unordered_multimap;
        |              ^~~~~~~~~~~~~~~~~~
  In file included from ./../include/fst/register.h:33,
                   from ./../include/fst/fst.h:36,
                   from fst.cc:21:
  ./../include/fst/generic-register.h: In member function 'virtual EntryType fst::GenericRegister<KeyType, EntryType, RegisterType>::LoadEntryFromSharedObject(const KeyType&) const':
  ./../include/fst/generic-register.h:78:48: error: 'RTLD_LAZY' was not declared in this scope
     78 |     void *handle = dlopen(so_filename.c_str(), RTLD_LAZY);
        |                                                ^~~~~~~~~
  ./../include/fst/generic-register.h:78:20: error: there are no arguments to 'dlopen' that depend on a template parameter, so a declaration of 'dlopen' must be available [-fpermissive]
     78 |     void *handle = dlopen(so_filename.c_str(), RTLD_LAZY);
        |                    ^~~~~~
  ./../include/fst/generic-register.h:78:20: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
  ./../include/fst/generic-register.h:80:55: error: there are no arguments to 'dlerror' that depend on a template parameter, so a declaration of 'dlerror' must be available [-fpermissive]
     80 |       LOG(ERROR) << "GenericRegister::GetEntry : " << dlerror();
        |                                                       ^~~~~~~
  In file included from ./../include/fst/accumulator.h:36,
                   from ./../include/fst/label-reachable.h:32,
                   from ./../include/fst/lookahead-matcher.h:28,
                   from ./../include/fst/matcher-fst.h:26,
                   from fst.cc:26:
  ./../include/fst/replace.h: In constructor 'fst::ArcIterator<fst::ReplaceFst<A, T> >::ArcIterator(const fst::ReplaceFst<A, T>&, fst::ArcIterator<fst::ReplaceFst<A, T> >::StateId)':
  ./../include/fst/replace.h:1074:46: error: expected ';' before '::' token
   1074 |       (fst_.GetImpl())->template CacheImpl<A>::InitArcIterator(state_,
        |                                              ^~
        |                                              ;
  In file included from ./../include/fst/register.h:33,
                   from ./../include/fst/fst.h:36,
                   from fst.cc:21:
  ./../include/fst/generic-register.h: In instantiation of 'EntryType fst::GenericRegister<KeyType, EntryType, RegisterType>::LoadEntryFromSharedObject(const KeyType&) const [with KeyType = std::__cxx11::basic_string<char>; EntryType = fst::FstRegisterEntry<fst::ArcTpl<fst::TropicalWeightTpl<float> > >; RegisterType = fst::FstRegister<fst::ArcTpl<fst::TropicalWeightTpl<float> > >]':
  ./../include/fst/generic-register.h:65:14:   required from 'EntryType fst::GenericRegister<KeyType, EntryType, RegisterType>::GetEntry(const KeyType&) const [with KeyType = std::__cxx11::basic_string<char>; EntryType = fst::FstRegisterEntry<fst::ArcTpl<fst::TropicalWeightTpl<float> > >; RegisterType = fst::FstRegister<fst::ArcTpl<fst::TropicalWeightTpl<float> > >]'
  ./../include/fst/register.h:65:26:   required from 'const Reader fst::FstRegister<A>::GetReader(const string&) const [with A = fst::ArcTpl<fst::TropicalWeightTpl<float> >; fst::FstRegister<A>::Reader = fst::Fst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >* (*)(std::basic_istream<char>&, const fst::FstReadOptions&); std::string = std::__cxx11::basic_string<char>]'
  ./../include/fst/fst.h:212:25:   required from 'static fst::Fst<A>* fst::Fst<A>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::TropicalWeightTpl<float> >; std::istream = std::basic_istream<char>]'
  ./../include/fst/edit-fst.h:663:37:   required from 'static fst::EditFstImpl<A, WrappedFstT, MutableFstT>* fst::EditFstImpl<A, WrappedFstT, MutableFstT>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::TropicalWeightTpl<float> >; WrappedFstT = fst::ExpandedFst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >; MutableFstT = fst::VectorFst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >; std::istream = std::basic_istream<char>]'
  ./../include/fst/edit-fst.h:730:28:   required from 'static fst::EditFst<A, W, M>* fst::EditFst<A, W, M>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::TropicalWeightTpl<float> >; WrappedFstT = fst::ExpandedFst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >; MutableFstT = fst::VectorFst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >; std::istream = std::basic_istream<char>]'
  ./../include/fst/register.h:100:10:   required from 'fst::FstRegisterer<F>::Entry fst::FstRegisterer<F>::BuildEntry() [with F = fst::EditFst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >; fst::FstRegisterer<F>::Entry = fst::FstRegisterEntry<fst::ArcTpl<fst::TropicalWeightTpl<float> > >]'
  ./../include/fst/register.h:96:23:   required from 'fst::FstRegisterer<F>::FstRegisterer() [with F = fst::EditFst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >]'
  fst.cc:64:1:   required from here
  ./../include/fst/generic-register.h:78:57: error: 'dlopen' was not declared in this scope; did you mean 'fdopen'?
     78 |     void *handle = dlopen(so_filename.c_str(), RTLD_LAZY);
        |                                                         ^
        |                                                         fdopen
  ./../include/fst/generic-register.h:80:62: error: 'dlerror' was not declared in this scope; did you mean 'ferror'?
     80 |       LOG(ERROR) << "GenericRegister::GetEntry : " << dlerror();
        |                                                       ~~~~~~~^~
        |                                                       ferror
  ./../include/fst/generic-register.h: In instantiation of 'EntryType fst::GenericRegister<KeyType, EntryType, RegisterType>::LoadEntryFromSharedObject(const KeyType&) const [with KeyType = std::__cxx11::basic_string<char>; EntryType = fst::FstRegisterEntry<fst::ArcTpl<fst::LogWeightTpl<float> > >; RegisterType = fst::FstRegister<fst::ArcTpl<fst::LogWeightTpl<float> > >]':
  ./../include/fst/generic-register.h:65:14:   required from 'EntryType fst::GenericRegister<KeyType, EntryType, RegisterType>::GetEntry(const KeyType&) const [with KeyType = std::__cxx11::basic_string<char>; EntryType = fst::FstRegisterEntry<fst::ArcTpl<fst::LogWeightTpl<float> > >; RegisterType = fst::FstRegister<fst::ArcTpl<fst::LogWeightTpl<float> > >]'
  ./../include/fst/register.h:65:26:   required from 'const Reader fst::FstRegister<A>::GetReader(const string&) const [with A = fst::ArcTpl<fst::LogWeightTpl<float> >; fst::FstRegister<A>::Reader = fst::Fst<fst::ArcTpl<fst::LogWeightTpl<float> > >* (*)(std::basic_istream<char>&, const fst::FstReadOptions&); std::string = std::__cxx11::basic_string<char>]'
  ./../include/fst/fst.h:212:25:   required from 'static fst::Fst<A>* fst::Fst<A>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::LogWeightTpl<float> >; std::istream = std::basic_istream<char>]'
  ./../include/fst/edit-fst.h:663:37:   required from 'static fst::EditFstImpl<A, WrappedFstT, MutableFstT>* fst::EditFstImpl<A, WrappedFstT, MutableFstT>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::LogWeightTpl<float> >; WrappedFstT = fst::ExpandedFst<fst::ArcTpl<fst::LogWeightTpl<float> > >; MutableFstT = fst::VectorFst<fst::ArcTpl<fst::LogWeightTpl<float> > >; std::istream = std::basic_istream<char>]'
  ./../include/fst/edit-fst.h:730:28:   required from 'static fst::EditFst<A, W, M>* fst::EditFst<A, W, M>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::LogWeightTpl<float> >; WrappedFstT = fst::ExpandedFst<fst::ArcTpl<fst::LogWeightTpl<float> > >; MutableFstT = fst::VectorFst<fst::ArcTpl<fst::LogWeightTpl<float> > >; std::istream = std::basic_istream<char>]'
  ./../include/fst/register.h:100:10:   required from 'fst::FstRegisterer<F>::Entry fst::FstRegisterer<F>::BuildEntry() [with F = fst::EditFst<fst::ArcTpl<fst::LogWeightTpl<float> > >; fst::FstRegisterer<F>::Entry = fst::FstRegisterEntry<fst::ArcTpl<fst::LogWeightTpl<float> > >]'
  ./../include/fst/register.h:96:23:   required from 'fst::FstRegisterer<F>::FstRegisterer() [with F = fst::EditFst<fst::ArcTpl<fst::LogWeightTpl<float> > >]'
  fst.cc:65:1:   required from here
  ./../include/fst/generic-register.h:78:57: error: 'dlopen' was not declared in this scope; did you mean 'fdopen'?
     78 |     void *handle = dlopen(so_filename.c_str(), RTLD_LAZY);
        |                                                         ^
        |                                                         fdopen
  ./../include/fst/generic-register.h:80:62: error: 'dlerror' was not declared in this scope; did you mean 'ferror'?
     80 |       LOG(ERROR) << "GenericRegister::GetEntry : " << dlerror();
        |                                                       ~~~~~~~^~
        |                                                       ferror
  ./../include/fst/generic-register.h: In instantiation of 'EntryType fst::GenericRegister<KeyType, EntryType, RegisterType>::LoadEntryFromSharedObject(const KeyType&) const [with KeyType = std::__cxx11::basic_string<char>; EntryType = fst::FstRegisterEntry<fst::ArcTpl<fst::LogWeightTpl<double> > >; RegisterType = fst::FstRegister<fst::ArcTpl<fst::LogWeightTpl<double> > >]':
  ./../include/fst/generic-register.h:65:14:   required from 'EntryType fst::GenericRegister<KeyType, EntryType, RegisterType>::GetEntry(const KeyType&) const [with KeyType = std::__cxx11::basic_string<char>; EntryType = fst::FstRegisterEntry<fst::ArcTpl<fst::LogWeightTpl<double> > >; RegisterType = fst::FstRegister<fst::ArcTpl<fst::LogWeightTpl<double> > >]'
  ./../include/fst/register.h:65:26:   required from 'const Reader fst::FstRegister<A>::GetReader(const string&) const [with A = fst::ArcTpl<fst::LogWeightTpl<double> >; fst::FstRegister<A>::Reader = fst::Fst<fst::ArcTpl<fst::LogWeightTpl<double> > >* (*)(std::basic_istream<char>&, const fst::FstReadOptions&); std::string = std::__cxx11::basic_string<char>]'
  ./../include/fst/fst.h:212:25:   required from 'static fst::Fst<A>* fst::Fst<A>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::LogWeightTpl<double> >; std::istream = std::basic_istream<char>]'
  ./../include/fst/edit-fst.h:663:37:   required from 'static fst::EditFstImpl<A, WrappedFstT, MutableFstT>* fst::EditFstImpl<A, WrappedFstT, MutableFstT>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::LogWeightTpl<double> >; WrappedFstT = fst::ExpandedFst<fst::ArcTpl<fst::LogWeightTpl<double> > >; MutableFstT = fst::VectorFst<fst::ArcTpl<fst::LogWeightTpl<double> > >; std::istream = std::basic_istream<char>]'
  ./../include/fst/edit-fst.h:730:28:   required from 'static fst::EditFst<A, W, M>* fst::EditFst<A, W, M>::Read(std::istream&, const fst::FstReadOptions&) [with A = fst::ArcTpl<fst::LogWeightTpl<double> >; WrappedFstT = fst::ExpandedFst<fst::ArcTpl<fst::LogWeightTpl<double> > >; MutableFstT = fst::VectorFst<fst::ArcTpl<fst::LogWeightTpl<double> > >; std::istream = std::basic_istream<char>]'
  ./../include/fst/register.h:100:10:   required from 'fst::FstRegisterer<F>::Entry fst::FstRegisterer<F>::BuildEntry() [with F = fst::EditFst<fst::ArcTpl<fst::LogWeightTpl<double> > >; fst::FstRegisterer<F>::Entry = fst::FstRegisterEntry<fst::ArcTpl<fst::LogWeightTpl<double> > >]'
  ./../include/fst/register.h:96:23:   required from 'fst::FstRegisterer<F>::FstRegisterer() [with F = fst::EditFst<fst::ArcTpl<fst::LogWeightTpl<double> > >]'
  fst.cc:66:1:   required from here
  ./../include/fst/generic-register.h:78:57: error: 'dlopen' was not declared in this scope; did you mean 'fdopen'?
     78 |     void *handle = dlopen(so_filename.c_str(), RTLD_LAZY);
        |                                                         ^
        |                                                         fdopen
  ./../include/fst/generic-register.h:80:62: error: 'dlerror' was not declared in this scope; did you mean 'ferror'?
     80 |       LOG(ERROR) << "GenericRegister::GetEntry : " << dlerror();
        |                                                       ~~~~~~~^~
        |                                                       ferror
  make[3]: *** [Makefile:428: fst.lo] Error 1
  make[3]: Leaving directory '/d/a/hfst-python/hfst-python/libhfst_src/back-ends/openfstwin/src/lib'
  make[2]: *** [Makefile:388: all-recursive] Error 1
  make[2]: Leaving directory '/d/a/hfst-python/hfst-python/libhfst_src/back-ends/openfstwin/src'
  make[1]: *** [Makefile:388: all-recursive] Error 1
  make[1]: Leaving directory '/d/a/hfst-python/hfst-python/libhfst_src/back-ends/openfstwin'
  make: *** [Makefile:387: all-recursive] Error 1
  make: Leaving directory '/d/a/hfst-python/hfst-python/libhfst_src/back-ends'

pip install fails with lex.yy.c(1840): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory

I followed the instructions at https://pypi.org/project/hfst/

I'm running on Win10 with Python 3.11.3, I also installed the standard Visual C++ build tools. Using a minimal venv:

(hfst_venv) D:\hfst_venv>pip list
Package    Version
---------- -------
pip        24.0
setuptools 65.5.0

Running python -m pip install --verbose --upgrade hfst gave this error:

(hfst_venv) D:\hfst_venv\Scripts>python -m pip install --verbose --upgrade hfst
Using pip 24.0 from D:\hfst_venv\Lib\site-packages\pip (python 3.11)
Collecting hfst
  Using cached hfst-3.12.2.4b0.tar.gz (1.5 MB)
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=40.8.0
    Using cached setuptools-69.1.0-py3-none-any.whl.metadata (6.1 kB)
  Using cached setuptools-69.1.0-py3-none-any.whl (819 kB)
  Installing collected packages: setuptools
  Successfully installed setuptools-69.1.0
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  C:\Users\Me\AppData\Local\Temp\pip-build-env-082ggbuy\overlay\Lib\site-packages\setuptools\dist.py:314: InformationOnly: Normalizing '3.12.2.4_beta' to '3.12.2.4b0'
    self.metadata.version = self._normalize_version(self.metadata.version)
  running egg_info
  writing hfst.egg-info\PKG-INFO
  writing dependency_links to hfst.egg-info\dependency_links.txt
  writing top-level names to hfst.egg-info\top_level.txt
  reading manifest file 'hfst.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'COPYING'
  writing manifest file 'hfst.egg-info\SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
  Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.42.0
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)
  C:\Users\Me\AppData\Local\Temp\pip-build-env-082ggbuy\overlay\Lib\site-packages\setuptools\dist.py:314: InformationOnly: Normalizing '3.12.2.4_beta' to '3.12.2.4b0'
    self.metadata.version = self._normalize_version(self.metadata.version)
  running dist_info
  creating C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst.egg-info
  writing C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst.egg-info\PKG-INFO
  writing dependency_links to C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst.egg-info\dependency_links.txt
  writing top-level names to C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst.egg-info\top_level.txt
  writing manifest file 'C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst.egg-info\SOURCES.txt'
  reading manifest file 'C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'COPYING'
  writing manifest file 'C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst.egg-info\SOURCES.txt'
  creating 'C:\Users\Me\AppData\Local\Temp\pip-modern-metadata-kji93_x3\hfst-3.12.2.4b0.dist-info'
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: hfst
  Running command Building wheel for hfst (pyproject.toml)
  C:\Users\Me\AppData\Local\Temp\pip-build-env-082ggbuy\overlay\Lib\site-packages\setuptools\dist.py:314: InformationOnly: Normalizing '3.12.2.4_beta' to '3.12.2.4b0'
    self.metadata.version = self._normalize_version(self.metadata.version)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-cpython-311
  copying libhfst.py -> build\lib.win-amd64-cpython-311
  creating build\lib.win-amd64-cpython-311\hfst
  copying hfst\__init__.py -> build\lib.win-amd64-cpython-311\hfst
  creating build\lib.win-amd64-cpython-311\hfst\exceptions
  copying hfst\exceptions\__init__.py -> build\lib.win-amd64-cpython-311\hfst\exceptions
  creating build\lib.win-amd64-cpython-311\hfst\sfst_rules
  copying hfst\sfst_rules\__init__.py -> build\lib.win-amd64-cpython-311\hfst\sfst_rules
  creating build\lib.win-amd64-cpython-311\hfst\xerox_rules
  copying hfst\xerox_rules\__init__.py -> build\lib.win-amd64-cpython-311\hfst\xerox_rules
  running build_ext
  building '_libhfst' extension
  creating build\temp.win-amd64-cpython-311
  creating build\temp.win-amd64-cpython-311\Release
  creating build\temp.win-amd64-cpython-311\Release\back-ends
  creating build\temp.win-amd64-cpython-311\Release\back-ends\foma
  creating build\temp.win-amd64-cpython-311\Release\back-ends\openfstwin
  creating build\temp.win-amd64-cpython-311\Release\back-ends\openfstwin\src
  creating build\temp.win-amd64-cpython-311\Release\back-ends\openfstwin\src\lib
  creating build\temp.win-amd64-cpython-311\Release\libhfst
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\implementations
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\implementations\compose_intersect
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\implementations\optimized-lookup
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\parsers
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\parsers\alphabet_src
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\parsers\io_src
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\parsers\rule_src
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\parsers\string_src
  creating build\temp.win-amd64-cpython-311\Release\libhfst\src\parsers\variable_src
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/apply.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/apply.obj /EHsc
  apply.cpp
  back-ends/foma/apply.cpp(93): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  back-ends/foma/apply.cpp(480): warning C4018: '>': signed/unsigned mismatch
  back-ends/foma/apply.cpp(503): warning C4018: '>': signed/unsigned mismatch
  back-ends/foma/apply.cpp(996): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  back-ends/foma/apply.cpp(1037): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  back-ends/foma/apply.cpp(1252): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  back-ends/foma/apply.cpp(1260): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  back-ends/foma/apply.cpp(1303): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/coaccessible.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/coaccessible.obj /EHsc
  coaccessible.cpp
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/constructions.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/constructions.obj /EHsc
  constructions.cpp
  back-ends/foma/constructions.cpp(158): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  back-ends/foma/constructions.cpp(299): warning C4018: '<': signed/unsigned mismatch
  back-ends/foma/constructions.cpp(351): warning C4018: '>': signed/unsigned mismatch
  back-ends/foma/constructions.cpp(371): warning C4018: '<': signed/unsigned mismatch
  back-ends/foma/constructions.cpp(374): warning C4018: '<': signed/unsigned mismatch
  back-ends/foma/constructions.cpp(863): warning C4244: '=': conversion from 'short' to 'unsigned char', possible loss of data
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/define.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/define.obj /EHsc
  define.cpp
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/determinize.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/determinize.obj /EHsc
  determinize.cpp
  back-ends/foma/determinize.cpp(267): warning C4018: '<': signed/unsigned mismatch
  back-ends/foma/determinize.cpp(763): warning C4018: '>=': signed/unsigned mismatch
  back-ends/foma/determinize.cpp(764): warning C4018: '>=': signed/unsigned mismatch
  back-ends/foma/determinize.cpp(817): warning C4018: '<': signed/unsigned mismatch
  back-ends/foma/determinize.cpp(853): warning C4018: '<': signed/unsigned mismatch
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/dynarray.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/dynarray.obj /EHsc
  dynarray.cpp
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/extract.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/extract.obj /EHsc
  extract.cpp
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/flags.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/flags.obj /EHsc
  flags.cpp
  back-ends/foma/flags.cpp(399): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  back-ends/foma/flags.cpp(420): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/int_stack.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/int_stack.obj /EHsc
  int_stack.cpp
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/io.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/io.obj /EHsc
  io.cpp
  "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHFSTEXPORT -DOPENFSTEXPORT -DWINDOWS -DWIN32 -D_CRT_SECURE_NO_WARNINGS -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfstwin/src/include -ID:\hfst_venv\include "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpback-ends/foma/lex.yy.cpp /Fobuild\temp.win-amd64-cpython-311\Release\back-ends/foma/lex.yy.obj /EHsc
  lex.yy.cpp
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(49): warning C4005: 'INT8_MIN': macro redefinition
  lex.yy.c(59): note: see previous definition of 'INT8_MIN'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(50): warning C4005: 'INT16_MIN': macro redefinition
  lex.yy.c(62): note: see previous definition of 'INT16_MIN'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(51): warning C4005: 'INT32_MIN': macro redefinition
  lex.yy.c(65): note: see previous definition of 'INT32_MIN'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(53): warning C4005: 'INT8_MAX': macro redefinition
  lex.yy.c(68): note: see previous definition of 'INT8_MAX'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(54): warning C4005: 'INT16_MAX': macro redefinition
  lex.yy.c(71): note: see previous definition of 'INT16_MAX'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(55): warning C4005: 'INT32_MAX': macro redefinition
  lex.yy.c(74): note: see previous definition of 'INT32_MAX'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(57): warning C4005: 'UINT8_MAX': macro redefinition
  lex.yy.c(77): note: see previous definition of 'UINT8_MAX'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(58): warning C4005: 'UINT16_MAX': macro redefinition
  lex.yy.c(80): note: see previous definition of 'UINT16_MAX'
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\stdint.h(59): warning C4005: 'UINT32_MAX': macro redefinition
  lex.yy.c(83): note: see previous definition of 'UINT32_MAX'
  lex.yy.c(1840): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.39.33519\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  error: subprocess-exited-with-error

  × Building wheel for hfst (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: 'D:\hfst_venv\Scripts\python.exe' 'D:\hfst_venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py' build_wheel 'C:\Users\Me\AppData\Local\Temp\tmp1d6vlw4z'
  cwd: C:\Users\Me\AppData\Local\Temp\pip-install-25y3jm74\hfst_7f623af757b14e6192c0f28503cb19a4
  Building wheel for hfst (pyproject.toml) ... error
  ERROR: Failed building wheel for hfst
Failed to build hfst
ERROR: Could not build wheels for hfst, which is required to install pyproject.toml-based projects

linux build fails `test_streams.py`

../home/rob/repos/hfst-python/test/test_streams.py Fatal Python error: Aborted

Current thread 0x00007f1227a4b200 (most recent call first):
  File "/home/rob/.local/lib/python3.7/site-packages/hfst/libhfst.py", line 4738 in hfst_regex
  File "/home/rob/.local/lib/python3.7/site-packages/hfst/__init__.py", line 349 in regex
  File "/home/rob/repos/hfst-python/test/test_streams.py", line 5 in write_tr
  File "/home/rob/repos/hfst-python/test/test_streams.py", line 80 in run_tests
  File "/home/rob/repos/hfst-python/test/test_streams.py", line 90 in test_streams_sfst
Aborted (core dumped)

the python function tat triggers the failure is...

def write_tr(fname):
    tr1 = hfst.regex('föö:bär')
    tr2 = hfst.regex('0')
    tr3 = hfst.regex('0-0')

    ostr = hfst.HfstOutputStream(filename=fname)
    ostr.write(tr1)
    ostr.write(tr2)
    ostr.write(tr3)
    ostr.flush()
    ostr.close()

...and the final python code in libhfst.py that actually causes the failure:

def hfst_regex(comp, regex_string, error_stream):
    return _libhfst.hfst_regex(comp, regex_string, error_stream)

Note that _libhfst is the compiled so module.

Add script to test linux config locally

The cibuildwheel docs say...

To quickly test your config without doing a git push and waiting for your code to build on CI, you can test the Linux build in a Docker container. On Mac or Linux, with Docker running, try cibuildwheel --platform linux. You'll have to bring your config into the current environment first.

Write a script that will collect the environment variables from .github/workflows/build.yml and run this command for local debugging.

C++ headers not found

Both ubuntu-latest and macos-latest builds gives the following:

    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHAVE_READLINE -DHAVE_GETLINE -I/tmp/pip-req-build-2hkysx82/src/hfst -I/tmp/pip-req-build-2hkysx82/src/hfst/lib -I/tmp/pip-req-build-2hkysx82/libhfst_src/back-ends/foma -I/tmp/pip-req-build-2hkysx82/libhfst_src/back-ends -I/tmp/pip-req-build-2hkysx82/libhfst_src/libhfst/src -I/tmp/pip-req-build-2hkysx82/libhfst_src/libhfst/src/parsers -I/tmp/pip-req-build-2hkysx82/libhfst_src/back-ends/openfst/src/include -I/opt/_internal/cpython-3.6.13/include/python3.6m -c libhfst_src/libhfst/src/implementations/ConvertLogWeightTransducer.cc -o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/ConvertLogWeightTransducer.o -Wno-sign-compare```
    libhfst_src/libhfst/src/implementations/ConvertLogWeightTransducer.cc:25:10: fatal error: back-ends/openfst/src/include/fst/fstlib.h: No such file or directory
       25 | #include "back-ends/openfst/src/include/fst/fstlib.h"
          |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.```

on ubuntu: "ld: cannot find -lhfst"

2021-03-22T15:39:49.8492737Z   g++ -pthread -shared build/temp.linux-x86_64-3.6/src/hfst/libhfst_wrap.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HarmonizeUnknownAndIdentitySymbols.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/string-utils.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstLookupFlagDiacritics.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstXeroxRules.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstSymbolDefs.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstInputStream.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstPrintDot.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstStrings2FstTokenizer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstTokenizer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstOutputStream.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstDataTypes.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstExceptionDefs.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/hfst-string-conversions.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstPrintPCKimmo.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstFlagDiacritics.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstRules.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstApply.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/HfstEpsilonHandler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/HfstTropicalTransducerTransitionData.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/ConvertLogWeightTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/ConvertFomaTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/FomaTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/ConvertOlTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/ConvertTropicalWeightTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/HfstOlTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/LogWeightTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/ConvertTransducerFormat.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/HfstBasicTransition.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/TropicalWeightTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/HfstBasicTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/compose_intersect/ComposeIntersectLexicon.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/compose_intersect/ComposeIntersectRulePair.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/compose_intersect/ComposeIntersectFst.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/compose_intersect/ComposeIntersectUtilities.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/compose_intersect/ComposeIntersectRule.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/optimized-lookup/pmatch_tokenize.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/optimized-lookup/convert.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/optimized-lookup/pmatch.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/optimized-lookup/ospell.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/optimized-lookup/find_epsilon_loops.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/implementations/optimized-lookup/transducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/XfstCompiler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/pmatch_parse.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/htwolcpre3-lexer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/PmatchCompiler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/sfst-compiler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/htwolcpre2-lexer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/lexc-parser.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/LexcCompiler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/xre_utils.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/SfstAlphabet.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/xfst-lexer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/xre_lex.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/sfst-scanner.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/pmatch_lex.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/lexc-utils.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/SfstCompiler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/xfst-utils.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/XreCompiler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/TwolcCompiler.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/htwolcpre1-lexer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/xfst-parser.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/SfstBasic.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/pmatch_utils.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/lexc-lexer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/htwolcpre3-parser.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/xre_parse.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/htwolcpre2-parser.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/htwolcpre1-parser.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/xfst_help_message.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/SfstUtf8.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/alphabet_src/Alphabet.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/io_src/InputReader.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/RightArrowRule.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/Rule.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/RightArrowRuleContainer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/LeftArrowRule.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/ConflictResolvingRightArrowRule.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/LeftRestrictionArrowRule.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/ConflictResolvingLeftArrowRule.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/TwolCGrammar.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/LeftArrowRuleContainer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/RuleContainer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/rule_src/OtherSymbolTransducer.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/string_src/string_manipulation.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/variable_src/RuleVariables.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/variable_src/RuleVariablesConstIterator.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/variable_src/VariableValues.o build/temp.linux-x86_64-3.6/libhfst_src/libhfst/src/parsers/variable_src/RuleSymbolVector.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/reverse.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/lex.lexc.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/dynarray.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/determinize.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/apply.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/rewrite.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/stringhash.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/mem.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/sigma.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/structures.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/lexcread.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/utf8.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/coaccessible.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/topsort.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/flags.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/io.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/spelling.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/regex.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/minimize.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/constructions.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/int_stack.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/define.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/extract.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/trie.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/lex.yy.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/openfst/src/lib/properties.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/openfst/src/lib/symbol-table-ops.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/openfst/src/lib/fst.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/openfst/src/lib/symbol-table.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/openfst/src/lib/compat.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/openfst/src/lib/flags.o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/openfst/src/lib/util.o -lhfst -o build/lib.linux-x86_64-3.6/hfst/_libhfst.cpython-36m-x86_64-linux-gnu.so -lreadline
2021-03-22T15:39:50.4505995Z   /opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: cannot find -lhfst
2021-03-22T15:39:50.4506919Z   collect2: error: ld returned 1 exit status
2021-03-22T15:39:50.4515077Z   error: command 'g++' failed with exit status 1
2021-03-22T15:39:50.4767229Z   ERROR: Failed building wheel for hfst
2021-03-22T15:39:50.4775336Z ERROR: Failed to build one or more wheels

Lots of `void*` errors on `foma/cpp-version/lexcread.cc`

2021-03-22T01:24:09.1809468Z   gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHAVE_READLINE -DHAVE_GETLINE -I/tmp/pip-req-build-_h_5smja/libhfst_src -I/tmp/pip-req-build-_h_5smja/libhfst_src/back-ends -I/tmp/pip-req-build-_h_5smja/libhfst_src/libhfst/src -I/tmp/pip-req-build-_h_5smja/libhfst_src/libhfst/src/parsers -I/tmp/pip-req-build-_h_5smja/libhfst_src/back-ends/foma/cpp-version -I/tmp/pip-req-build-_h_5smja/libhfst_src/back-ends/foma -I/tmp/pip-req-build-_h_5smja/libhfst_src/back-ends/openfst/src/include -I/opt/_internal/cpython-3.6.13/include/python3.6m -c libhfst_src/back-ends/foma/cpp-version/lexcread.cc -o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/cpp-version/lexcread.o -Wno-sign-compare -std=c++11
2021-03-22T01:24:09.2253233Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_add_sigma_hash(char*, int)’:
2021-03-22T01:24:09.2269246Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:154:20: error: invalid conversion from ‘void*’ to ‘lexc_hashtable*’ [-fpermissive]
2021-03-22T01:24:09.2270266Z     154 |     hnew = xxmalloc(sizeof(struct lexc_hashtable));
2021-03-22T01:24:09.2275090Z         |            ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2275677Z         |                    |
2021-03-22T01:24:09.2278479Z         |                    void*
2021-03-22T01:24:09.2281717Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_init()’:
2021-03-22T01:24:09.2285709Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:172:25: error: invalid conversion from ‘void*’ to ‘lexc_hashtable*’ [-fpermissive]
2021-03-22T01:24:09.2288887Z     172 |     hashtable = xxcalloc(SIGMA_HASH_TABLESIZE, sizeof(struct lexc_hashtable));
2021-03-22T01:24:09.2289793Z         |                 ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2290300Z         |                         |
2021-03-22T01:24:09.2290619Z         |                         void*
2021-03-22T01:24:09.2291644Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:176:22: error: invalid conversion from ‘void*’ to ‘bool*’ [-fpermissive]
2021-03-22T01:24:09.2292426Z     176 |     mchash = xxcalloc(256*256, sizeof(Boolean));
2021-03-22T01:24:09.2292828Z         |              ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2295724Z         |                      |
2021-03-22T01:24:09.2297040Z         |                      void*
2021-03-22T01:24:09.2306256Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_add_state(states*)’:
2021-03-22T01:24:09.2307711Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:192:18: error: invalid conversion from ‘void*’ to ‘statelist*’ [-fpermissive]
2021-03-22T01:24:09.2310266Z     192 |     sl = xxmalloc(sizeof(struct statelist));
2021-03-22T01:24:09.2310900Z         |          ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2311242Z         |                  |
2021-03-22T01:24:09.2314334Z         |                  void*
2021-03-22T01:24:09.2322873Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_update_unknowns(int)’:
2021-03-22T01:24:09.2403368Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:217:29: error: invalid conversion from ‘void*’ to ‘states::trans*’ [-fpermissive]
2021-03-22T01:24:09.2404356Z     217 |          newtrans = xxmalloc(sizeof(struct states::trans));
2021-03-22T01:24:09.2405075Z         |                     ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2405564Z         |                             |
2021-03-22T01:24:09.2406079Z         |                             void*
2021-03-22T01:24:09.2407738Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_add_network()’:
2021-03-22T01:24:09.2409453Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:246:26: error: invalid conversion from ‘void*’ to ‘int*’ [-fpermissive]
2021-03-22T01:24:09.2410666Z     246 |     sigreplace = xxcalloc(sigma_max(net->sigma)+1,sizeof(int));
2021-03-22T01:24:09.2411214Z         |                  ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2411589Z         |                          |
2021-03-22T01:24:09.2411939Z         |                          void*
2021-03-22T01:24:09.2413266Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:272:23: error: invalid conversion from ‘void*’ to ‘int*’ [-fpermissive]
2021-03-22T01:24:09.2414076Z     272 |         unk = xxcalloc(sigma_max(lexsigma)+2,sizeof(int));
2021-03-22T01:24:09.2414540Z         |               ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2414857Z         |                       |
2021-03-22T01:24:09.2415182Z         |                       void*
2021-03-22T01:24:09.2416239Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:281:21: error: invalid conversion from ‘void*’ to ‘states**’ [-fpermissive]
2021-03-22T01:24:09.2417097Z     281 |     slist = xxcalloc(sizeof(**slist),maxstate+1);
2021-03-22T01:24:09.2417542Z         |             ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2417867Z         |                     |
2021-03-22T01:24:09.2418175Z         |                     void*
2021-03-22T01:24:09.2419125Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:282:22: error: invalid conversion from ‘void*’ to ‘int*’ [-fpermissive]
2021-03-22T01:24:09.2420042Z     282 |     finals = xxcalloc(sizeof(int),maxstate+1);
2021-03-22T01:24:09.2420545Z         |              ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2420841Z         |                      |
2021-03-22T01:24:09.2421136Z         |                      void*
2021-03-22T01:24:09.2422231Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:285:28: error: invalid conversion from ‘void*’ to ‘states*’ [-fpermissive]
2021-03-22T01:24:09.2423068Z     285 |         newstate = xxmalloc(sizeof(struct states));
2021-03-22T01:24:09.2423703Z         |                    ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2424190Z         |                            |
2021-03-22T01:24:09.2424526Z         |                            void*
2021-03-22T01:24:09.2425497Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:294:21: error: invalid conversion from ‘void*’ to ‘statelist*’ [-fpermissive]
2021-03-22T01:24:09.2426338Z     294 |         s = xxmalloc(sizeof(struct statelist));
2021-03-22T01:24:09.2426937Z         |             ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2427271Z         |                     |
2021-03-22T01:24:09.2427590Z         |                     void*
2021-03-22T01:24:09.2429017Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:302:24: error: invalid conversion from ‘void*’ to ‘states::trans*’ [-fpermissive]
2021-03-22T01:24:09.2430088Z     302 |     newtrans = xxmalloc(sizeof(struct states::trans));
2021-03-22T01:24:09.2430576Z         |                ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2431012Z         |                        |
2021-03-22T01:24:09.2431324Z         |                        void*
2021-03-22T01:24:09.2432573Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:312:32: error: invalid conversion from ‘void*’ to ‘states::trans*’ [-fpermissive]
2021-03-22T01:24:09.2433446Z     312 |             newtrans = xxmalloc(sizeof(struct states::trans));
2021-03-22T01:24:09.2433932Z         |                        ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2434256Z         |                                |
2021-03-22T01:24:09.2434590Z         |                                void*
2021-03-22T01:24:09.2435768Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:323:44: error: invalid conversion from ‘void*’ to ‘states::trans*’ [-fpermissive]
2021-03-22T01:24:09.2436848Z     323 |                         newtrans = xxmalloc(sizeof(struct states::trans));
2021-03-22T01:24:09.2437389Z         |                                    ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2437798Z         |                                            |
2021-03-22T01:24:09.2438173Z         |                                            void*
2021-03-22T01:24:09.2439495Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:338:32: error: invalid conversion from ‘void*’ to ‘states::trans*’ [-fpermissive]
2021-03-22T01:24:09.2440777Z     338 |             newtrans = xxmalloc(sizeof(struct states::trans));
2021-03-22T01:24:09.2441228Z         |                        ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2441556Z         |                                |
2021-03-22T01:24:09.2441865Z         |                                void*
2021-03-22T01:24:09.2442730Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_set_current_lexicon(char*, int)’:
2021-03-22T01:24:09.2443967Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:378:17: error: invalid conversion from ‘void*’ to ‘lexstates*’ [-fpermissive]
2021-03-22T01:24:09.2444788Z     378 |     l = xxmalloc(sizeof(struct lexstates));
2021-03-22T01:24:09.2445191Z         |         ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2445498Z         |                 |
2021-03-22T01:24:09.2445789Z         |                 void*
2021-03-22T01:24:09.2446718Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:384:24: error: invalid conversion from ‘void*’ to ‘states*’ [-fpermissive]
2021-03-22T01:24:09.2447517Z     384 |     newstate = xxmalloc(sizeof(struct states));
2021-03-22T01:24:09.2447943Z         |                ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2448258Z         |                        |
2021-03-22T01:24:09.2448557Z         |                        void*
2021-03-22T01:24:09.2449455Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_add_mc(char*)’:
2021-03-22T01:24:09.2450697Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:679:25: error: invalid conversion from ‘void*’ to ‘multichar_symbols*’ [-fpermissive]
2021-03-22T01:24:09.2451584Z     679 |         mcnew = xxmalloc(sizeof(struct multichar_symbols));
2021-03-22T01:24:09.2452038Z         |                 ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2452357Z         |                         |
2021-03-22T01:24:09.2452655Z         |                         void*
2021-03-22T01:24:09.2453431Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_add_word()’:
2021-03-22T01:24:09.2454886Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:752:28: error: invalid conversion from ‘void*’ to ‘states::trans*’ [-fpermissive]
2021-03-22T01:24:09.2455733Z     752 |         newtrans = xxmalloc(sizeof(struct states::trans));
2021-03-22T01:24:09.2456179Z         |                    ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2456485Z         |                            |
2021-03-22T01:24:09.2456921Z         |                            void*
2021-03-22T01:24:09.2457890Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:756:32: error: invalid conversion from ‘void*’ to ‘states*’ [-fpermissive]
2021-03-22T01:24:09.2458859Z     756 |             newstate = xxmalloc(sizeof(struct states));
2021-03-22T01:24:09.2459449Z         |                        ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2459763Z         |                                |
2021-03-22T01:24:09.2460065Z         |                                void*
2021-03-22T01:24:09.2495544Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘void lexc_merge_states()’:
2021-03-22T01:24:09.2497331Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:892:23: error: invalid conversion from ‘void*’ to ‘lexc_merge_states()::lenlist*’ [-fpermissive]
2021-03-22T01:24:09.2498314Z     892 |     lenlist = xxcalloc(maxlen+1,sizeof(struct lenlist));
2021-03-22T01:24:09.2498796Z         |               ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2499290Z         |                       |
2021-03-22T01:24:09.2499593Z         |                       void*
2021-03-22T01:24:09.2501000Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:904:26: error: invalid conversion from ‘void*’ to ‘lexc_merge_states()::hashstates*’ [-fpermissive]
2021-03-22T01:24:09.2501973Z     904 |     hashstates = xxcalloc(tablesize,sizeof(struct hashstates));
2021-03-22T01:24:09.2502518Z         |                  ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2502829Z         |                          |
2021-03-22T01:24:09.2503135Z         |                          void*
2021-03-22T01:24:09.2505291Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:913:32: error: invalid conversion from ‘void*’ to ‘lexc_merge_states()::lenlist*’ [-fpermissive]
2021-03-22T01:24:09.2506174Z     913 |                 newl = xxcalloc(1,sizeof(struct lenlist));
2021-03-22T01:24:09.2506872Z         |                        ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2507698Z         |                                |
2021-03-22T01:24:09.2508171Z         |                                void*
2021-03-22T01:24:09.2509581Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:923:32: error: invalid conversion from ‘void*’ to ‘lexc_merge_states()::hashstates*’ [-fpermissive]
2021-03-22T01:24:09.2510618Z     923 |                 newh = xxcalloc(1,sizeof(struct hashstates));
2021-03-22T01:24:09.2511635Z         |                        ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2511994Z         |                                |
2021-03-22T01:24:09.2512339Z         |                                void*
2021-03-22T01:24:09.2513181Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc: In function ‘fsm* lexc_to_fsm()’:
2021-03-22T01:24:09.2514722Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:1029:22: warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings]
2021-03-22T01:24:09.2515618Z    1029 |     net = fsm_create("");
2021-03-22T01:24:09.2515998Z         |                      ^~
2021-03-22T01:24:09.2517064Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:1045:18: error: invalid conversion from ‘void*’ to ‘statelist*’ [-fpermissive]
2021-03-22T01:24:09.2518648Z    1045 |     sa = xxmalloc(sizeof(struct statelist)*lexc_statecount);
2021-03-22T01:24:09.2519143Z         |          ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2519561Z         |                  |
2021-03-22T01:24:09.2519865Z         |                  void*
2021-03-22T01:24:09.2521236Z   libhfst_src/back-ends/foma/cpp-version/lexcread.cc:1057:19: error: invalid conversion from ‘void*’ to ‘fsm_state*’ [-fpermissive]
2021-03-22T01:24:09.2522759Z    1057 |     fsm = xxmalloc(sizeof(struct fsm_state)*(linecount+1));
2021-03-22T01:24:09.2523304Z         |           ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-22T01:24:09.2523807Z         |                   |
2021-03-22T01:24:09.2524133Z         |                   void*
2021-03-22T01:24:09.2535231Z   error: command 'gcc' failed with exit status 1
2021-03-22T01:24:09.2788183Z   ERROR: Failed building wheel for hfst
2021-03-22T01:24:09.2804035Z ERROR: Failed to build one or more wheels

double free or corruption on manylinux build

On manylinux, the HFST C++ build passes all of its tests, but the python extension immediately throws an error at runtime: double free or corruption. Given that libc.so.6 is the last thing in the backtrace, it seems likely that this is a double free.

2021-03-10T20:59:37.8949033Z     + pip install /tmp/cibuildwheel/repaired_wheel/hfst-3.15.4.0b0-cp36-cp36m-manylinux2014_x86_64.whl
2021-03-10T20:59:42.8777284Z Processing /tmp/cibuildwheel/repaired_wheel/hfst-3.15.4.0b0-cp36-cp36m-manylinux2014_x86_64.whl
2021-03-10T20:59:42.8778441Z Installing collected packages: hfst
2021-03-10T20:59:42.8779428Z Successfully installed hfst-3.15.4.0b0
2021-03-10T20:59:42.8780680Z     + sh -c 'pushd /project/test && ./test.sh --python $(which python) --verbose ; popd'
2021-03-10T20:59:42.9431182Z *** Error in `/tmp/tmp.QL42AhzNFp/venv/bin/python': double free or corruption (!prev): 0x00000000013ee910 ***
2021-03-10T20:59:42.9443459Z ======= Backtrace: =========
2021-03-10T20:59:42.9448329Z /lib64/libc.so.6(+0x81329)[0x7f377aa79329]
2021-03-10T20:59:42.9454281Z /tmp/tmp.QL42AhzNFp/venv/lib/python3.6/site-packages/hfst/_libhfst.cpython-36m-x86_64-linux-gnu.so(_ZN10HandyDequeISsED2Ev+0x6d)[0x7f377a29252d]
2021-03-10T20:59:42.9459813Z /lib64/libc.so.6(+0x39d39)[0x7f377aa31d39]
2021-03-10T20:59:42.9464360Z /lib64/libc.so.6(+0x39d87)[0x7f377aa31d87]
2021-03-10T20:59:42.9468830Z /lib64/libc.so.6(__libc_start_main+0xfc)[0x7f377aa1a55c]
2021-03-10T20:59:42.9473386Z /tmp/tmp.QL42AhzNFp/venv/bin/python[0x41d7d6]

The arguments given to distutils.Extension in setup.py are the following:

{'define_macros': [('HAVE_FOMA', None),
('HAVE_OPENFST', None),
('HAVE_OPENFST_LOG', None),
('HAVE_READLINE', None),
('HAVE_GETLINE', None),
('NO_CPLUSPLUS_11', None),
('USE_TR1_UNORDERED_MAP_AND_SET', None),
('INCLUDE_TR1_UNORDERED_MAP_AND_SET', None)],
'depends': [],
'export_symbols': [],
'extra_compile_args': ['-Wno-sign-compare', '-Wno-strict-prototypes'],
'extra_link_args': ['-lreadline'],
'extra_objects': [],
'include_dirs': ['/project/hfst',
'/project/hfst/lib',
'/project/hfst_src/back-ends/foma',
'/project/hfst_src/back-ends',
'/project/hfst_src/libhfst/src',
'/project/hfst_src/libhfst/src/parsers',
'/project/hfst_src/back-ends/openfst/src/include'],
'language': 'c++',
'libraries': ['hfst'],
'library_dirs': ['/project/hfst_src/libhfst/src/.libs'],
'name': 'hfst._libhfst',
'optional': None,
'py_limited_api': False,
'runtime_library_dirs': [],
'sources': ['hfst/libhfst.i',
'hfst_src/libhfst/src/parsers/pmatch_lex.cc',
'hfst_src/libhfst/src/parsers/sfst-scanner.cc',
'hfst_src/libhfst/src/parsers/pmatch_parse.cc',
'hfst_src/libhfst/src/parsers/htwolcpre2-lexer.cc',
'hfst_src/libhfst/src/parsers/xfst-utils.cc',
'hfst_src/libhfst/src/parsers/xre_lex.cc',
'hfst_src/libhfst/src/parsers/htwolcpre1-lexer.cc',
'hfst_src/libhfst/src/parsers/TwolcCompiler.cc',
'hfst_src/libhfst/src/parsers/xfst_help_message.cc',
'hfst_src/libhfst/src/parsers/lexc-lexer.cc',
'hfst_src/libhfst/src/parsers/PmatchCompiler.cc',
'hfst_src/libhfst/src/parsers/sfst-compiler.cc',
'hfst_src/libhfst/src/parsers/SfstBasic.cc',
'hfst_src/libhfst/src/parsers/xre_utils.cc',
'hfst_src/libhfst/src/parsers/htwolcpre2-parser.cc',
'hfst_src/libhfst/src/parsers/SfstUtf8.cc',
'hfst_src/libhfst/src/parsers/SfstCompiler.cc',
'hfst_src/libhfst/src/parsers/xfst-lexer.cc',
'hfst_src/libhfst/src/parsers/pmatch_utils.cc',
'hfst_src/libhfst/src/parsers/htwolcpre3-parser.cc',
'hfst_src/libhfst/src/parsers/htwolcpre1-parser.cc',
'hfst_src/libhfst/src/parsers/SfstAlphabet.cc',
'hfst_src/libhfst/src/parsers/xre_parse.cc',
'hfst_src/libhfst/src/parsers/LexcCompiler.cc',
'hfst_src/libhfst/src/parsers/xfst-parser.cc',
'hfst_src/libhfst/src/parsers/lexc-utils.cc',
'hfst_src/libhfst/src/parsers/XreCompiler.cc',
'hfst_src/libhfst/src/parsers/lexc-parser.cc',
'hfst_src/libhfst/src/parsers/htwolcpre3-lexer.cc',
'hfst_src/libhfst/src/parsers/XfstCompiler.cc',
'hfst_src/back-ends/openfst/src/lib/compat.cc'],
'swig_opts': ['-c++', '-I/project/hfst_src/libhfst/src', '-Wall'],
'undef_macros': []}

Cannot compile foma back-end

No matter whether I include foma's C or C++ variant, compilation fails. The choice between variants is in setup.py.

When I include the C version, the error is undeclared disclaimer1, SVN_REV, and disclaimer2, all on the same line.

When I include the C++ version, the errors mainly surround incompatibilities of void * (see below).

C errors:

    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHAVE_READLINE -DHAVE_GETLINE -I/tmp/pip-req-build-ifj1cd37/src/hfst -I/tmp/pip-req-build-ifj1cd37/src/hfst/lib -I/tmp/pip-req-build-ifj1cd37/libhfst_src/back-ends/foma -I/tmp/pip-req-build-ifj1cd37/libhfst_src/back-ends -I/tmp/pip-req-build-ifj1cd37/libhfst_src/libhfst/src -I/tmp/pip-req-build-ifj1cd37/libhfst_src/libhfst/src/parsers -I/tmp/pip-req-build-ifj1cd37/libhfst_src/back-ends/openfst/src/include -I/opt/_internal/cpython-3.6.13/include/python3.6m -c libhfst_src/back-ends/foma/foma.c -o build/temp.linux-x86_64-3.6/libhfst_src/back-ends/foma/foma.o -Wno-sign-compare
    libhfst_src/back-ends/foma/foma.c: In function ‘main’:
    libhfst_src/back-ends/foma/foma.c:166:41: error: ‘disclaimer1’ undeclared (first use in this function); did you mean ‘disclaimer’?
      166 |       printf("%s%i.%i.%i%s (svn r%i)%s",disclaimer1,MAJOR_VERSION,MINOR_VERSION,BUILD_VERSION,STATUS_VERSION,SVN_REV,disclaimer2);
          |                                         ^~~~~~~~~~~
          |                                         disclaimer
    libhfst_src/back-ends/foma/foma.c:166:41: note: each undeclared identifier is reported only once for each function it appears in
    libhfst_src/back-ends/foma/foma.c:166:110: error: ‘SVN_REV’ undeclared (first use in this function)
      166 |       printf("%s%i.%i.%i%s (svn r%i)%s",disclaimer1,MAJOR_VERSION,MINOR_VERSION,BUILD_VERSION,STATUS_VERSION,SVN_REV,disclaimer2);
          |                                                                                                              ^~~~~~~
    libhfst_src/back-ends/foma/foma.c:166:118: error: ‘disclaimer2’ undeclared (first use in this function); did you mean ‘disclaimer’?
      166 |       printf("%s%i.%i.%i%s (svn r%i)%s",disclaimer1,MAJOR_VERSION,MINOR_VERSION,BUILD_VERSION,STATUS_VERSION,SVN_REV,disclaimer2);
          |                                                                                                                      ^~~~~~~~~~~
          |                                                                                                                      disclaimer

C++ errors:

    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:154:12: error: assigning to 'struct lexc_hashtable *' from incompatible type 'void *'
        hnew = xxmalloc(sizeof(struct lexc_hashtable));
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:172:17: error: assigning to 'struct lexc_hashtable *' from incompatible type 'void *'
        hashtable = xxcalloc(SIGMA_HASH_TABLESIZE, sizeof(struct lexc_hashtable));
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:176:14: error: assigning to 'bool *' from incompatible type 'void *'
        mchash = xxcalloc(256*256, sizeof(Boolean));
                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:192:10: error: assigning to 'struct statelist *' from incompatible type 'void *'
        sl = xxmalloc(sizeof(struct statelist));
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:217:21: error: assigning to 'struct states::trans *' from incompatible type 'void *'
                    newtrans = xxmalloc(sizeof(struct states::trans));
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:246:18: error: assigning to 'int *' from incompatible type 'void *'
        sigreplace = xxcalloc(sigma_max(net->sigma)+1,sizeof(int));
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:272:15: error: assigning to 'int *' from incompatible type 'void *'
            unk = xxcalloc(sigma_max(lexsigma)+2,sizeof(int));
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:281:13: error: assigning to 'struct states **' from incompatible type 'void *'
        slist = xxcalloc(sizeof(**slist),maxstate+1);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:282:14: error: assigning to 'int *' from incompatible type 'void *'
        finals = xxcalloc(sizeof(int),maxstate+1);
                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:285:20: error: assigning to 'struct states *' from incompatible type 'void *'
            newstate = xxmalloc(sizeof(struct states));
                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:294:13: error: assigning to 'struct statelist *' from incompatible type 'void *'
            s = xxmalloc(sizeof(struct statelist));
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:302:16: error: assigning to 'struct states::trans *' from incompatible type 'void *'
        newtrans = xxmalloc(sizeof(struct states::trans));
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:312:24: error: assigning to 'struct states::trans *' from incompatible type 'void *'
                newtrans = xxmalloc(sizeof(struct states::trans));
                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:323:36: error: assigning to 'struct states::trans *' from incompatible type 'void *'
                            newtrans = xxmalloc(sizeof(struct states::trans));
                                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:338:24: error: assigning to 'struct states::trans *' from incompatible type 'void *'
                newtrans = xxmalloc(sizeof(struct states::trans));
                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:378:9: error: assigning to 'struct lexstates *' from incompatible type 'void *'
        l = xxmalloc(sizeof(struct lexstates));
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:384:16: error: assigning to 'struct states *' from incompatible type 'void *'
        newstate = xxmalloc(sizeof(struct states));
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:679:17: error: assigning to 'struct multichar_symbols *' from incompatible type 'void *'
            mcnew = xxmalloc(sizeof(struct multichar_symbols));
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    libhfst_src/back-ends/foma/cpp-version/lexcread.cc:752:20: error: assigning to 'struct states::trans *' from incompatible type 'void *'
            newtrans = xxmalloc(sizeof(struct states::trans));
                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    fatal error: too many errors emitted, stopping now [-ferror-limit=]
    4 warnings and 20 errors generated.

`bump_version.sh` does not change `__version__` in `src/hfst/__init__.py`

Add a new file version.py with only one line: version = 'X.X.X.X', and then in __init__.py do from .version import version as __version__, or something like that. This makes it so version bumps won't clutter up the git history of __init__.py.

bump_version.sh should update both VERSION and src/hfst/version.py.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.