Git Product home page Git Product logo

Comments (1)

bab2min avatar bab2min commented on August 22, 2024

enum Match에 새 bit flag인 joinEp를 추가하고 아래 함수에 결합 로직만 추가하면 된다.

Kiwi/src/Kiwi.cpp

Lines 533 to 599 in 37bfa60

template<class TokenInfoIt>
TokenInfoIt joinAffixTokens(TokenInfoIt first, TokenInfoIt last, Match matchOptions)
{
if (!(matchOptions & (Match::joinNounPrefix | Match::joinNounSuffix | Match::joinVerbSuffix | Match::joinAdjSuffix | Match::joinAdvSuffix))) return last;
if (std::distance(first, last) < 2) return last;
auto next = first;
++next;
while (next != last)
{
TokenInfo& current = *first;
TokenInfo& nextToken = *next;
// XPN + (NN. | SN) => (NN. | SN)
if (!!(matchOptions & Match::joinNounPrefix)
&& current.tag == POSTag::xpn
&& ((POSTag::nng <= nextToken.tag && nextToken.tag <= POSTag::nnb) || nextToken.tag == POSTag::sn)
)
{
concatTokens(current, nextToken, nextToken.tag);
++next;
}
// (NN. | SN) + XSN => (NN. | SN)
else if (!!(matchOptions & Match::joinNounSuffix)
&& nextToken.tag == POSTag::xsn
&& ((POSTag::nng <= current.tag && current.tag <= POSTag::nnb) || current.tag == POSTag::sn)
)
{
concatTokens(current, nextToken, current.tag);
++next;
}
// (NN. | XR) + XSV => VV
else if (!!(matchOptions & Match::joinVerbSuffix)
&& clearIrregular(nextToken.tag) == POSTag::xsv
&& ((POSTag::nng <= current.tag && current.tag <= POSTag::nnb) || current.tag == POSTag::xr)
)
{
concatTokens(current, nextToken, setIrregular(POSTag::vv, isIrregular(nextToken.tag)));
++next;
}
// (NN. | XR) + XSA => VA
else if (!!(matchOptions & Match::joinAdjSuffix)
&& clearIrregular(nextToken.tag) == POSTag::xsa
&& ((POSTag::nng <= current.tag && current.tag <= POSTag::nnb) || current.tag == POSTag::xr)
)
{
concatTokens(current, nextToken, setIrregular(POSTag::va, isIrregular(nextToken.tag)));
++next;
}
// (NN. | XR) + XSM => MAG
else if (!!(matchOptions & Match::joinAdvSuffix)
&& nextToken.tag == POSTag::xsm
&& ((POSTag::nng <= current.tag && current.tag <= POSTag::nnb) || current.tag == POSTag::xr)
)
{
concatTokens(current, nextToken, POSTag::mag);
++next;
}
else
{
++first;
if (first != next) *first = std::move(*next);
++next;
}
}
return ++first;
}

from kiwi.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.