x1angli / cvt2utf Goto Github PK
View Code? Open in Web Editor NEWThis lightweight tool converts non-UTF-encoded (such as GB2312, GBK, BIG5 encoded) files to UTF-8 encoding.
License: MIT License
This lightweight tool converts non-UTF-encoded (such as GB2312, GBK, BIG5 encoded) files to UTF-8 encoding.
License: MIT License
We suppose it's straightforward to detect the charset of a non-UTF text file -- but it's not. When using 3rd-party tools to guess the original encoding method of a file, we don't have the author to ask, we don't have the context of the text file -- the entire process feels like reverse-engineering... In short, we rely on probability models to detect the charset, without any guarantee
As a result , we have to set a minimum threshold. Only those encoding results returned by chartdet that are above that threshold level would be accepted. Otherwise, if chartdet can't tell the encoding of a file with confidence, we would simple throw a warning message.
Thanks @runhwguo for your valuable proposal.
bytedata.decode 采用 latin_1 的方式,GB2312 编码的文件也能通过。
Describe the bug
A clear and concise description of what the bug is.
I use the tool to convert .cue and .log files.
I discovered problem with some UTF-8 encoded cue files and many UTF-16 encoded log files.
Some UTF-8 encoded files do not transcoding well and still have problem in mapping.
UTF-16 encoded log files are recognized as latin_1 encoded.
Example is attach.
.bak is the original fils.
error.zip
Python 3.5.4, win32
Python>pip install convert2utf
Collecting convert2utf
Using cached convert2utf-1.0.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "\AppData\Local\Temp\pip-build-lhz8yq_d\convert2utf\setup.py", line 12, in
long_description = pypandoc.convert('README.md', 'rst')
File "python\python35-32\lib\site-packages\pypandoc_init_.py", line 66, in convert
raise RuntimeError("Format missing, but need one (identified source as text as no "
RuntimeError: Format missing, but need one (identified source as text as no file with that name was found).
In virsual stduio, cpp,i want it's chatset can be utf-8-sig
and in java ,i want to be utf-8, without bom.
Python 3.6.1
pip install convert2utf
from cvt2utf import Convert2Utf8
from cvt2utf import Convert2Utf8
ModuleNotFoundError: No module named 'cvt2utf'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.