xiliangsong / python-jianfan Goto Github PK

\u9ebd '麽' is not a valid char in big5. In big5 it should always be 
\u9ebc'麼'

Error found when writing the converted result to a file in big5.

In the mapping:
4e48么 -> 9ebd麽

Should be:
4e48么 -> 9ebc麼

Thanks for the module! Unfortunately it does not work with Python3, despite 
claims that it does. I'd like to fix this :)

What steps will reproduce the problem?

1. python3.4 -c "import jianfan; print(jianfan.ftoj('繁體字'))"

or

2. python3.2 -c "import jianfan; print(jianfan.ftoj('繁體字'))"


What is the expected output? What do you see instead?

I expect to see "繁体字", but instead I get this for Python 3.3+:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/goodmami/python-jianfan/jianfan/__init__.py", line 20, in <module>
    from charsets import gbk, big5
ImportError: No module named 'charsets'


For Python 3.0 -- 3.2, there is this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "jianfan/__init__.py", line 44
    return u''.join(chars)
             ^
SyntaxError: invalid syntax


What version of the product are you using? On what operating system?

This is changeset 6:3376915b5a94 (0.0.2, I think)


Please provide any additional information below.

There are two problems here. First, Python3 does not have implicit relative 
imports, so "from charsets import gbk, big5" fails because "charsets" is not in 
Python libraries. From Python2.5, you can do this:

from __future__ import absolute_import
from .charsets import gbk, big5

But then you lose support for Python2.4 (if that is important).

You can also use the explicit absolute import, like this:

from jianfan.charsets import gbk, big5

I think this works for more versions. I attach the __init__.py with this change.


The second problem is that u'...' unicode strings were dropped from Python3.0 
because all strings unicode, but they were later reinstated (as a syntactic 
variant of regular '...' strings, purely for compatibility reasons) from 
Python3.3. That means that all u'...' strings in charsets and __init__.py will 
give SyntaxErrors for Python 3.0, 3.1, and 3.2. You can do this in charsets.py

from __future__ import unicode_literals
gbk = '...'  # instead of u'...'
big5 = '...'

But then you lose support for Python2.5. Instead, you can define a function in 
charsets.py as described here http://stackoverflow.com/a/6633040, and make the 
gbk and big5 variables that way. I adapted the function to convert if the input 
string is a regular '...' string, but to return strings that are already 
unicode unchanged. Fortunately, you can also use this function to convert input 
strings as well, so it gets extra use. I attach the charsets.py with this 
change.


You may want to test to make sure the new versions work for your intended use 
cases, but I think this will work with all previously supported versions plus 
now all Python 3 versions.

Thanks!

python-jianfan currently only supports Python 2. 

Would you be willing to update it to support Python 2 and 3?

I've attached jianfan.py written to support Python 2 and 3.