ikegami-yukino / neologdn Goto Github PK

View Code? Open in Web Editor NEW

266.0 7.0 20.0 531 KB

Japanese text normalizer for mecab-neologd

License: Apache License 2.0

Python 38.48% Shell 0.53% Jupyter Notebook 20.28% Cython 40.71%

preprocessing nlp japanese-language mecab-ipadic-neologd text-normalization

neologdn's Introduction

neologdn

neologdn is a Japanese text normalizer for mecab-neologd.

The normalization is based on the neologd's rules: https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja

Contributions are welcome!

NOTE: Installing this module requires C++11 compiler.

Installation

$ pip install neologdn

Usage

import neologdn
neologdn.normalize("ﾊﾝｶｸｶﾅ")
# => 'ハンカクカナ'
neologdn.normalize("全角記号！？＠＃")
# => '全角記号!?@#'
neologdn.normalize("全角記号例外「・」")
# => '全角記号例外「・」'
neologdn.normalize("長音短縮ウェーーーーイ")
# => '長音短縮ウェーイ'
neologdn.normalize("チルダ削除ウェ~∼∾〜〰～イ")
# => 'チルダ削除ウェイ'
neologdn.normalize("いろんなハイフン˗֊‐‑‒–⁃⁻₋−")
# => 'いろんなハイフン-'
neologdn.normalize("　　　ＰＲＭＬ　　副　読　本　　　")
# => 'PRML副読本'
neologdn.normalize(" Natural Language Processing ")
# => 'Natural Language Processing'
neologdn.normalize("かわいいいいいいいいい", repeat=6)
# => 'かわいいいいいい'
neologdn.normalize("無駄無駄無駄無駄ァ", repeat=1)
# => '無駄ァ'
neologdn.normalize("1995〜2001年", tilde="normalize")
# => '1995~2001年'
neologdn.normalize("1995~2001年", tilde="normalize_zenkaku")
# => '1995〜2001年'
neologdn.normalize("1995〜2001年", tilde="ignore")  # Don't convert tilde
# => '1995〜2001年'
neologdn.normalize("1995〜2001年", tilde="remove")
# => '19952001年'
neologdn.normalize("1995〜2001年")  # Default parameter
# => '19952001年'

Benchmark

# Sample code from
# https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja#python-written-by-hideaki-t--overlast
import normalize_neologd

%timeit normalize(normalize_neologd.normalize_neologd)
# => 9.55 s ± 29.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


import neologdn
%timeit normalize(neologdn.normalize)
# => 6.66 s ± 35.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

neologdn is about x1.43 faster than sample code.

details are described as the below notebook: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

License

Apache Software License.

Contribution

Contributions are welcome! See: https://github.com/ikegami-yukino/neologdn/blob/master/.github/CONTRIBUTING.md

Cited by

Book

山本和英. テキスト処理の要素技術. 近代科学者. P.41. 2021.

Blog

【ライブラリ紹介】テキスト正規化ライブラリ neologdn: https://diatonic.codes/blog/neologdn/
日本語テキストの前処理：neologdn、大文字小文字、Unicode正規化 - tuttieee’s blog: https://tuttieee.hatenablog.com/entry/ja-nlp-preprocess
▲本日の関数==neologdn.normalize()== - TPTブログ: https://ds-blog.tbtech.co.jp/entry/2020/05/11/%E2%96%B2%E6%9C%AC%E6%97%A5%E3%81%AE%E9%96%A2%E6%95%B0%3D%3Dneologdn_normalize%28%29%3D%3D
NLPについて学ぶ: https://zenn.dev/panyoriokome/scraps/d67f68ab50c0c1
テキスト正規化用PythonライブラリをMATLABからコール #Python - Qiita: https://qiita.com/aoimidori/items/ab5a4383b5a7bb307bad
自然言語処理の前処理手順をPythonコード付きでご紹介 | AI活用・AI導入事例の紹介 | AI活用・AI導入事例の紹介: https://www.matrixflow.net/case-study/75/
pythonによる日本語前処理備忘録 | DATUM STUDIO株式会社: https://datumstudio.jp/blog/python%E3%81%AB%E3%82%88%E3%82%8B%E6%97%A5%E6%9C%AC%E8%AA%9E%E5%89%8D%E5%87%A6%E7%90%86%E5%82%99%E5%BF%98%E9%8C%B2/
前処理、前処理、そして、前処理（自然言語処理：日本語編）｜narudesu: https://note.com/narudesu/n/na35de30a583a
ショートカットキーでneologd.normalize: https://scrapbox.io/nishio/%E3%82%B7%E3%83%A7%E3%83%BC%E3%83%88%E3%82%AB%E3%83%83%E3%83%88%E3%82%AD%E3%83%BC%E3%81%A7neologd.normalize
Pythonで自然言語処理を行うための環境構築 #Python - Qiita: https://qiita.com/lawyer_alpaca/items/86b0deda984170203467
Python normalize Examples: https://python.hotexamples.com/examples/neologdn/-/normalize/python-normalize-function-examples.html
株式会社ししまろ (ch-4) 潜在的ディリクレ配分(LDA)によるchABSAデータセットの分析: https://shishimaro.co.jp/blog/ai/538
形態素解析前の日本語文書の前処理 (Python) - け日記: https://ohke.hateblo.jp/entry/2019/02/09/141500
人工知能に言語を理解させる！？自然言語処理に重要なデータの前処理をPythonで徹底解説 | AI研究所: https://ai-kenkyujo.com/programming/make-ai-understand-the-language/
最新wikipediaを反映したMeCabユーザー辞書を作る - NEologd拡張 | ぷらこめ: https://purakome.net/mecab/addwiki/
【自然言語処理入門】文に対してストップワードと正規化から処理を施す | マイナビエンジニアブログ: https://engineerblog.mynavi.jp/technology/nlp_stopword/
表記統一 [自然言語処理の餅屋]: https://www.jnlp.org/nlp/%E6%A0%A1%E6%AD%A3/%E8%A1%A8%E8%A8%98%E7%B5%B1%E4%B8%80
Pytorchを使ってテキスト生成モデルのT5を構築〜Transformersでの転移学習による手軽な実践〜 - 見習いデータサイエンティストの隠れ家: https://www.dskomei.com/entry/2021/09/28/110016
象と散歩: Goolge Colabでお手軽テキストマイニング(日本語前処理): https://walking-elephant.blogspot.com/2023/07/text-mining-normalized.html
【Pythonで自然言語処理（NLP）を実装してみよう！】学ぶべき知識についても徹底解説！ - ベトナムオフショア開発の最前線 by Mattock inc.: https://mattock.jp/blog/artificial-intelligence/nlp/lets-implement-nlp-in-python/
tools [Digital Humanities Japan: Resource Wiki]: https://dhjapan.org/wiki/doku.php?id=tools
Pythonで現代の季語を調べてみた | Aidemy | 10秒で始めるAIプログラミング学習サービスAidemy［アイデミー］: https://aidemy.net/magazine/703/

neologdn's People

Contributors

Stargazers

Watchers

Forkers

id774 pombredanne unnonouno kensuke-mitsuzawa r9y9 whitphx mvlvrd stmsy yskn67 clrafaelr yokoshin nemupm laplacekorea elijahahianyo ahmad-abdellatif emmatools polm kamales777

neologdn's Issues

Suppress removal of spaces between Japanese characters

Is it possible to do the [title]? I noticed that it always removes the space for Japanese characters but for my case I need to preserve it. Thank you very much.

Compilation error with mac OSX Mavericks and Mojave

After Mavericks, since the -stdlib=libc++ flag isn't required anymore, so this causes compilation issue. see:
https://stackoverflow.com/questions/19774778/when-is-it-necessary-to-use-use-the-flag-stdlib-libstdc

Error message:

pip install neologdn
Collecting neologdn
  Using cached https://files.pythonhosted.org/packages/18/fd/9e84b382e4f12b73737faabeeb57fd617198dbb29b7084e28604803f7926/neologdn-0.4.tar.gz
Building wheels for collected packages: neologdn
  Running setup.py bdist_wheel for neologdn ... error
  Complete output from command /Users/ariga/src/backspace-analytics/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/cz/x7xkvznd3r3dh899ww_4kkpr0000gq/T/pip-install-z2p4i3fr/neologdn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/cz/x7xkvznd3r3dh899ww_4kkpr0000gq/T/pip-wheel-w6oxd8ib --python-tag cp36:
  running bdist_wheel
  running build
  running build_ext
  building 'neologdn' extension
  creating build
  creating build/temp.macosx-10.9-x86_64-3.6
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c neologdn.cpp -o build/temp.macosx-10.9-x86_64-3.6/neologdn.o -std=c++11 -mmacosx-version-min=10.7 -stdlib=libc++
  gcc: error: unrecognized command line option '-stdlib=libc++'
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for neologdn
  Running setup.py clean for neologdn
Failed to build neologdn
Installing collected packages: neologdn
  Running setup.py install for neologdn ... error
    Complete output from command /Users/ariga/src/backspace-analytics/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/cz/x7xkvznd3r3dh899ww_4kkpr0000gq/T/pip-install-z2p4i3fr/neologdn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/cz/x7xkvznd3r3dh899ww_4kkpr0000gq/T/pip-record-i72osnwq/install-record.txt --single-version-externally-managed --compile --install-headers /Users/ariga/src/backspace-analytics/venv/bin/../include/site/python3.6/neologdn:
    running install
    running build
    running build_ext
    building 'neologdn' extension
    creating build
    creating build/temp.macosx-10.9-x86_64-3.6
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c neologdn.cpp -o build/temp.macosx-10.9-x86_64-3.6/neologdn.o -std=c++11 -mmacosx-version-min=10.7 -stdlib=libc++
    gcc: error: unrecognized command line option '-stdlib=libc++'
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/Users/ariga/src/backspace-analytics/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/cz/x7xkvznd3r3dh899ww_4kkpr0000gq/T/pip-install-z2p4i3fr/neologdn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/cz/x7xkvznd3r3dh899ww_4kkpr0000gq/T/pip-record-i72osnwq/install-record.txt --single-version-externally-managed --compile --install-headers /Users/ariga/src/backspace-analytics/venv/bin/../include/site/python3.6/neologdn" failed with error code 1 in /private/var/folders/cz/x7xkvznd3r3dh899ww_4kkpr0000gq/T/pip-install-z2p4i3fr/neologdn/

After reverting this PR #9 , installation has been succeeded.

Compile error under MacOS

Problem

Related to the issue #4, I have following error when I install neologdn.

(anaconda3-4.0.0) MacBook-Pro-2% pip install neologdn
Collecting neologdn
  Using cached neologdn-0.2.1.tar.gz
Building wheels for collected packages: neologdn
  Running setup.py bdist_wheel for neologdn ... error
  Complete output from command /Users/kensuke-mi/.pyenv/versions/anaconda3-4.0.0/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/sp/z0_0lktj7nn2s31db2dt5md40000gq/T/pip-build-c856_32f/neologdn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/sp/z0_0lktj7nn2s31db2dt5md40000gq/T/tmp1ynkdxqppip-wheel- --python-tag cp35:
  running bdist_wheel
  running build
  running build_ext
  building 'neologdn' extension
  creating build
  creating build/temp.macosx-10.6-x86_64-3.5
  /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/kensuke-mi/.pyenv/versions/anaconda3-4.0.0/include -arch x86_64 -I/Users/kensuke-mi/.pyenv/versions/anaconda3-4.0.0/include/python3.5m -c neologdn.cpp -o build/temp.macosx-10.6-x86_64-3.5/neologdn.o -std=c++11
  neologdn.cpp:255:10: fatal error: 'unordered_map' file not found
  #include <unordered_map>
           ^
  1 error generated.
  error: command '/usr/bin/clang' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for neologdn
  Running setup.py clean for neologdn
Failed to build neologdn
Installing collected packages: neologdn
  Running setup.py install for neologdn ... error
    Complete output from command /Users/kensuke-mi/.pyenv/versions/anaconda3-4.0.0/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/sp/z0_0lktj7nn2s31db2dt5md40000gq/T/pip-build-c856_32f/neologdn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/sp/z0_0lktj7nn2s31db2dt5md40000gq/T/pip-o6sgy7w5-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'neologdn' extension
    creating build
    creating build/temp.macosx-10.6-x86_64-3.5
    /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/kensuke-mi/.pyenv/versions/anaconda3-4.0.0/include -arch x86_64 -I/Users/kensuke-mi/.pyenv/versions/anaconda3-4.0.0/include/python3.5m -c neologdn.cpp -o build/temp.macosx-10.6-x86_64-3.5/neologdn.o -std=c++11
    neologdn.cpp:255:10: fatal error: 'unordered_map' file not found
    #include <unordered_map>
             ^
    1 error generated.
    error: command '/usr/bin/clang' failed with exit status 1
    
    ----------------------------------------
Command "/Users/kensuke-mi/.pyenv/versions/anaconda3-4.0.0/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/sp/z0_0lktj7nn2s31db2dt5md40000gq/T/pip-build-c856_32f/neologdn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/sp/z0_0lktj7nn2s31db2dt5md40000gq/T/pip-o6sgy7w5-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/sp/z0_0lktj7nn2s31db2dt5md40000gq/T/pip-build-c856_32f/neologdn/

My mac is MacBook Pro (Retina 13-inch、Early 2015)
The issue does not happen in Ubuntu. I don't know why

and my gcc is

MacBook-Pro-2% gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc5/5.4.0/libexec/gcc/x86_64-apple-darwin14.5.0/5.4.0/lto-wrapper
Target: x86_64-apple-darwin14.5.0
Configured with: ../configure --build=x86_64-apple-darwin14.5.0 --prefix=/usr/local/Cellar/gcc5/5.4.0 --libdir=/usr/local/Cellar/gcc5/5.4.0/lib/gcc/5 --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-5 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl014 --with-system-zlib --enable-libstdcxx-time=yes --enable-stage1-checking --enable-checking=release --enable-lto --disable-werror --with-pkgversion='Homebrew gcc5 5.4.0' --with-bugurl=https://github.com/Homebrew/homebrew-versions/issues --enable-plugin --disable-nls --enable-multilib
Thread model: posix
gcc version 5.4.0 (Homebrew gcc5 5.4.0)

Solution

The main problem is that unordered_map and unordered_set are not in std, now it's in std::tr1.
So, I tried to re-write neologdn.cpp directly.

https://github.com/ikegami-yukino/neologdn/blob/master/neologdn.cpp#L255-L256

#include <tr1/unordered_map>
#include <tr1/unordered_set>

https://github.com/ikegami-yukino/neologdn/blob/master/neologdn.cpp#L685-L689

static std::tr1::unordered_map<Py_UNICODE,Py_UNICODE>  __pyx_v_8neologdn_conversion_map;
static std::tr1::unordered_map<Py_UNICODE,Py_UNICODE>  __pyx_v_8neologdn_kana_ten_map;
static std::tr1::unordered_map<Py_UNICODE,Py_UNICODE>  __pyx_v_8neologdn_kana_maru_map;
static std::tr1::unordered_set<Py_UNICODE>  __pyx_v_8neologdn_blocks;
static std::tr1::unordered_set<Py_UNICODE>  __pyx_v_8neologdn_basic_latin;

And then, I run python setup.py install, it worked!

Still....

I don't know how to fix this problem via neologdn.pyx.

Fails to build on Python 3.11 - longintrepr.h: No such file or directory

Python 3.11の環境において、インストールを行うと以下のエラーが発生します。

Using version ^0.5.1 for neologdn

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing neologdn (0.5.1): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  running bdist_wheel
  running build
  running build_ext
  building 'neologdn' extension
  creating build
  creating build/temp.linux-aarch64-cpython-311
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/tmpr_sufb23/.venv/include -I/usr/local/include/python3.11 -c neologdn.cpp -o build/temp.linux-aarch64-cpython-311/neologdn.o -std=c++11
  neologdn.cpp:198:12: fatal error: longintrepr.h: No such file or directory
    198 |   #include "longintrepr.h"
        |            ^~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  

  at /usr/local/lib/python3.11/site-packages/poetry/installation/chef.py:147 in _prepare
      143│ 
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│ 
      146│             if error is not None:
    → 147│                 raise error from None
      148│ 
      149│             return path
      150│ 
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with neologdn (0.5.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "neologdn (==0.5.1)"'.

aio-libs/aiohttp#6600 のISSUEにも記載されていますが、3.11起因で発生するエラーのようです（3.10だと発生しません）
具体的なWorkAroundに関してはまだ分かっていませんが、取り急ぎ共有までに。

Need some explanation about the code

It's been a long time since last update of this file.

But recently I found my coworkers used some code from above file, and many tutorials point to this file.

I feel confused about some of the code.

def unicode_normalize(cls, s):
    pt = re.compile('([{}]+)'.format(cls))

    def norm(c):
        return unicodedata.normalize('NFKC', c) if pt.match(c) else c

    s = ''.join(norm(x) for x in re.split(pt, s))
    return s

Why re.split here?

I compared the output using simply unicodedata.normalize with normalize_neologd():

code:

tests = [
    'Ｐｙｔｈｏｎは好きですか',
    '「初めてのTensorFlow」は定価2200円+税です',
    '「初めての　ＴｅｎｓｏｒＦｌｏｗ」は定価２２００円＋税です',
    '｢初めての TensorFlow｣は定価2200円＋税です',
    '㈱ほげほげ',
    '(株)ほげほげ',
    '！”＃＄％＆’（）＊＋，−．／：；＜＝＞？＠［￥］＾＿｀｛｜｝〜。、・「」',
    '˗֊‐‑‒–⁃⁻₋−',
    '﹣－ｰ—―─━ー',
    '~∼∾〜〰～'
]

if __name__ == '__main__':
    for t in tests:
        print('\nsource:', t)
        print('result:', normalize_neologd(t))
        print('result:', unicodedata.normalize('NFKC', t))

result:

source: Ｐｙｔｈｏｎは好きですか
result1: Pythonは好きですか
result2: Pythonは好きですか

source: 「初めてのTensorFlow」は定価2200円+税です
result1: 「初めてのTensorFlow」は定価2200円+税です
result2: 「初めてのTensorFlow」は定価2200円+税です

source: 「初めての　ＴｅｎｓｏｒＦｌｏｗ」は定価２２００円＋税です
result1: 「初めてのTensorFlow」は定価２２00円+税です
result2: 「初めての TensorFlow」は定価2200円+税です

source: ｢初めての TensorFlow｣は定価2200円＋税です
result1: 「初めてのTensorFlow」は定価2200円+税です
result2: 「初めての TensorFlow」は定価2200円+税です

source: ㈱ほげほげ
result1: ㈱ほげほげ
result2: (株)ほげほげ

source: (株)ほげほげ
result1: (株)ほげほげ
result2: (株)ほげほげ

source: ！”＃＄％＆’（）＊＋，−．／：；＜＝＞？＠［￥］＾＿｀｛｜｝〜。、・「」
result1: !”#$%&’()*+,-./:;<＝>?@[¥]^_`{|}。、・「」
result2: !”#$%&’()*+,−./:;<=>?@[¥]^_`{|}〜。、・「」

source: ˗֊‐‑‒–⁃⁻₋−
result1: -
result2: ˗֊‐‐‒–⁃−−−

source: ﹣－ｰ—―─━ー
result1: ー
result2: --ー—―─━ー

source: ~∼∾〜〰～
result1:
result2: ~∼∾〜〰~

Maybe split is required for older python version?

Unable to import under anaconda3 in pyenv

I installed your package via gcc-5 (homebrew; Mac OS X 10.11.4) through following command, because the Apple's compiler is outdated.

$ env CC=/usr/local/bin/gcc-5 pip install neologdn

After that, I tried to import the package, but I faced to en error like this.

In [1]: import neologdn
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-500d73366729> in <module>()
----> 1 import neologdn

ImportError: dlopen(/Users/shuntaroy/.pyenv/versions/anaconda3-4.0.0/lib/python3.5/site-packages/neologdn.cpython-35m-darwin.so, 2): Symbol not found: __ZNKSt8__detail20_Prime_rehash_policy14_M_need_rehashEmmm
  Referenced from: /Users/shuntaroy/.pyenv/versions/anaconda3-4.0.0/lib/python3.5/site-packages/neologdn.cpython-35m-darwin.so
  Expected in: dynamic lookup

My Python version is Python 3.5.1 |Anaconda 4.0.0 (x86_64)| (default, Dec 7 2015, 11:24:55), which is installed in pyenv.

I'm not aware of Clang nor Cython.
I would appreciate if you could give me an advice.

Thank you.

Fails to build on Python 3.12 - error: use of undeclared identifier 'PyUnicode_FromUnicode'

Python 3.12の環境において、インストールを行うと以下のエラーが発生します。

  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_ext
      building 'neologdn' extension
      creating build
      creating build/temp.macosx-11.0-arm64-cpython-312
      clang -fno-strict-overflow -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /Users/my_name/mambaforge/envs/env_py12/include -arch arm64 -fPIC -O2 -isystem /Users/my_name/mambaforge/envs/env_py12/include -arch arm64 -I/Users/my_name/mambaforge/envs/env_py12/include/python3.12 -c neologdn.cpp -o build/temp.macosx-11.0-arm64-cpython-312/neologdn.o -std=c++11 -mmacosx-version-min=10.7 -stdlib=libc++
      neologdn.cpp:6454:15: error: use of undeclared identifier 'PyUnicode_FromUnicode'
        __pyx_t_8 = __Pyx_PyUnicode_FromUnicode(__pyx_v_buf); if (unlikely(!__pyx_t_8)) __PYX_ERR(0, 192, __pyx_L1_error)
                    ^
      neologdn.cpp:1206:46: note: expanded from macro '__Pyx_PyUnicode_FromUnicode'
      #define __Pyx_PyUnicode_FromUnicode(u)       PyUnicode_FromUnicode(u, __Pyx_Py_UNICODE_strlen(u))
                                                   ^
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for neologdn
ERROR: Could not build wheels for neologdn, which is required to install pyproject.toml-based projects

failed

CondaEnvException: Pip failed

環境

macOS Sonoma 14.2.1
CPU = Apple M1
conda系のツールであるmambaを使用しています
- pipを使って他のツールはインストールできています（japanize-matplotlib）
- Python3.11では動作しています

Add the optional argument for removing tildes

This issue is similar to the issue #10 . I assume we don't have the way we make tildes remain in a sentence when using neologdn.normalize(sentence) for now. I would like the optional argument like remove_tildes as the argument of neologdn.normalize. Below is an example of what my suggestion might look like.

>>> neologdn.normalize("応仁の乱は1467〜1487年の11年間に渡って続きました。")
'応仁の乱は14671487年の11年間に渡って続きました。'
>>> neologdn.normalize("応仁の乱は1467〜1487年の11年間に渡って続きました。", remove_tildes=False)
'応仁の乱は1467〜1487年の11年間に渡って続きました。'

Like this example, I would sometimes prefer tildes to remain in a sentence. If there is no problems in terms of speed or something like that, I think this option is useful.