axgle / mahonia Goto Github PK

package main

import (
"fmt"
"github.com/axgle/mahonia"
)

func main() {

str :="你好" 
fmt.Println("UTF-8 to GBK: ",ConvertToString(str,"utf8","gbk"))

// data :=ConvertToString(str,"utf8","gbk")
fmt.Println("GBK to UTF-8: ",ConvertToString(ConvertToString(str,"utf8","gbk"),"gbk","utf8"))

}

func ConvertToString(src string, srcCode string, tagCode string) string {
srcCoder := mahonia.NewDecoder(srcCode)
srcResult := srcCoder.ConvertString(src)
tagCoder := mahonia.NewDecoder(tagCode)
_, cdata, _ := tagCoder.Translate([]byte(srcResult), true)
result := string(cdata)
return result
}

if I do enc := ma.NewEncoder("gb18030") first and then print with enc.ConvertString(file1.Name)
I would get Chinese garbled.

But if I just do ma.NewDecoder("gb18030").ConvertString(file1.Name) it works

does this library support cp949?

In shiftjis decoder , error convert '' to '¥'

···
e.DOM.Find("p").Each(func(i int, s *goquery.Selection) {
text := s.Text()
result := mahonia.NewDecoder("gbk").ConvertString(text)
fmt.Println(result)
})
···
这是一段爬取代码，text里面保存的是gbk编码的字符串。
我发现只要这个text里面有英文的“”双引号，双引号里面的内容都没有被转码。
输出的结果类似于
···
我是正常的中文鈥満焐氖谴竺ā⒙躺氖切
···
后面的乱码就是在英文的双引号中的文字。
但如果我把整个html页面包括div，li标签等都打印出来，就可以转码正常。
代码类似于：
···
c.OnHTML("#ArtContent", func(e *colly.HTMLElement) {
result := mahonia.NewDecoder("gbk").ConvertString(string(e.Response.Body))
fmt.Println(result)
···
在这里result 是完全转换成中文了，没有乱码。

The file "mahoniconv/mahoniconv.go" imports "code.google.com/p/mahonia". As code.google.com is shutting down, this should be changed to github.com/axgle/mahonia.

package main
import "fmt"
import "github.com/axgle/mahonia"
func main(){
enc:=mahonia.NewEncoder("gbk")
_, data, _ := enc.Translate([]byte(otherEncodingString), true)
fmt.Println(data)
}

I am packaging mahonia for Fedora. Adding the license file is not necessary, but the ticket should be opened according to the guidelines [https://fedoraproject.org/wiki/Packaging:LicensingGuidelines#License_Text]. @axgle

https://bugzilla.redhat.com/show_bug.cgi?id=1480957#c1

In this call
mahonia.NewDecoder("shift-jis").ConvertString(string(s))

if string(s) contain char "", it will be converted to "¥"（ full-width）, which may lead to an error when creating file.

The following code can avoid this problem without changing original package
strings.Replace(mahonia.NewDecoder("shift-jis").ConvertString(string(s)), "¥", "", -1)