patch / lingua-stem-unine-pm5 Goto Github PK
View Code? Open in Web Editor NEWUniversity of Neuchâtel stemmers for Bulgarian, Czech, German, and Persian
Home Page: https://metacpan.org/pod/Lingua::Stem::UniNE
License: Other
University of Neuchâtel stemmers for Bulgarian, Czech, German, and Persian
Home Page: https://metacpan.org/pod/Lingua::Stem::UniNE
License: Other
UniNE provides stemming algorithms for Bengali, Hindi, and Marathi. None of them have current stemmers on CPAN, so let's make them a higher priority.
There are lots of failing CPAN tester reports for v0.05:
http://www.cpantesters.org/distro/L/Lingua-Stem-UniNE.html?version=0.05
Use of uninitialized value $language:: in concatenation (.) or string at /home/cpan/pit/rel/conf/perl-5.16.2/.cpanplus/5.16.2/build/Lingua-Stem-UniNE-0.05/blib/lib/Lingua/Stem/UniNE.pm line 51.
# Looks like you planned 30 tests but ran 9.
# Looks like your test exited with 2 just after 9.
t/01-stemmer.t ..
Dubious, test returned 2 (wstat 512, 0x200)
In all tested versions of Perl v5.19 (.5, .6, .7, .8), all tests using non-Latin scripts (Arabic, Cyrillic) are failing. Non-ASCII characters within the Latin script are fine. This may be a Perl bug in these development versions, in which case it needs to be reported.
http://matrix.cpantesters.org/?dist=Lingua-Stem-UniNE+0.06
Example:
# Failed test 'remove article: -и�'
# at t/bg.t line 16.
# got: 'и�тори'
# expected: 'и�тор'
Some of the stemming algorithms will strip specific diacritical marks from the entire word. This type of word normalization in addition to stemming isn't always desired. Let's add an object attribute to optionally disable it.
For example, the to-be-implemented German stemmer replaces ä
with a
, ö
with o
, and ü
with u
.
The only current Arabic stemmer on CPAN is Lingua::AR::Word, which is of questionable quality. Here's a comment from Lingua::AR::Word::Stem.
#let's strip down every prefix and suffix I'm aware of.
The UniNE Arabic stemmer would be a welcome addition to CPAN, along with the Persian stemmer we already include. I'd make this the next priority after the Indo-Aryan languages (issue #3).
A single CPAN tester report is reporting the following bug.
# Lingua::Stem::UniNE v0.06, Moo v1.004002, Perl v5.14.3 (C:\strawberry5143\perl\bin\perl.exe)
t\00-load.t ..... ok
Undefined subroutine utf8::se called at C:\strawberry5143\cpan\build\Lingua-Stem-UniNE-0.06-KyTxMK\blib\lib/Lingua/Stem/UniNE.pm line 3.
Compilation failed in require at t\01-stemmer.t line 6.
BEGIN failed--compilation aborted at t\01-stemmer.t line 6.
t\01-stemmer.t ..
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 30/30 subtests
We've started implementing different aggressive and light stemmers, so we need an attribute in the Lingua::Stem::UniNE object to select aggressive over the default light stemmers.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.