processone / iconv Goto Github PK

View Code? Open in Web Editor NEW

54.0 13.0 30.0 229 KB

Fast encoding conversion library for Erlang / Elixir

License: Other

Makefile 0.16% C 4.67% Shell 67.23% Erlang 26.56% M4 1.39%

iconv libiconv encoding iconv-library

iconv's Introduction

iconv

Fast encoding conversion library for Erlang / Elixir

This library is a native binding to libiconv library.

Building

iconv library can be build as follow:

./configure && make

iconv is a rebar-compatible OTP application. Alternatively, you can build it with rebar:

rebar compile

Dependencies

iconv library depends on libiconv.

You can use configure option to pass custom path to libiconv library:

--with-libiconv-prefix[=DIR]  search for libiconv in DIR/include and DIR/lib

Usage

You can start iconv with the following command:

$ erl -pa ebin -pa deps/*/ebin
Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Eshell V10.2.1  (abort with ^G)
1> application:start(iconv).
ok
2> FromEncoding = <<"utf-8">>. 
<<"utf-8">>
3> ToEncoding = <<"iso8859-15">>.
<<"iso8859-15">>
4> Text = <<"Hello">>.
<<"Hello">>
5> iconv:convert(FromEncoding, ToEncoding, Text).            
<<"Hello">>

Elixir

You can use iconv with Elixir mix by adding the dependency as follows:

  defp deps do
    [
      {:iconv, "~> 1.0.10"},
    ]
  end

$ mix deps.get
...
$ mix deps.compile
...
$ iex -S mix
Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Interactive Elixir (1.7.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :iconv.convert("utf-8", "iso8859-15", "Hello")
"Hello"

Development

Test

Unit test

You can run eunit test with the command:

$ rebar eunit

iconv's People

Contributors

Stargazers

Watchers

iconv's Issues

Stuck in upgrading dependencies because of hard coded dependency to p1_utils 1.0.13

Current version of p1_utils is 1.0.18.
An update would be nice :-)

Debugging support?

Seems that iconv is not debuggable...

Can we eliminate support for this?

Tried with Erlang 1.9.3.x, Elixir 1.5.0-dev

[error] failed to load NIF 
_build/dev/lib/iconv/priv/lib/iconv: Library module name 'iconv' does not match calling module 'dbg_debugged'

=INFO REPORT==== 23-Jun-2017::22:49:17 ===
    application: logger
    exited: stopped
    type: temporary
*DBG* 'Elixir.IntellijElixir.DebugServer' got {'DOWN',#Ref<0.0.4.173>,
                                               process,<0.532.0>,
                                               {#{'__exception__' => true,
                                                  '__struct__' => 'Elixir.Mix.Error',
                                                  message => <<"Could not start application iconv: :iconv_app.start(:normal, []) returned an error: :bad_lib">>,
                                                  mix => true},
                                                [{'Elixir.Mix',raise,1,
                                                  [{file,"lib/mix.ex"},
                                                   {line,279}]},
                                                 {'Elixir.Enum',
                                                  '-each/2-lists^foreach/1-0-',
                                                  2,
                                                  [{file,"lib/enum.ex"},
                                                   {line,676}]},
                                                 {'Elixir.Enum',each,2,
                                                  [{file,"lib/enum.ex"},
                                                   {line,676}]},
                                                 {'Elixir.Mix.Tasks.App.Start',
                                                  start,2,
                                                  [{file,
                                                    "lib/mix/tasks/app.start.ex"},
                                                   {line,97}]},
                                                 {'Elixir.Mix.Tasks.App.Start',
                                                  run,1,
                                                  [{file,
                                                    "lib/mix/tasks/app.start.ex"},
                                                   {line,78}]},
                                                 {'Elixir.Mix.Task',run_task,
                                                  3,
                                                  [{file,"lib/mix/task.ex"},
                                                   {line,301}]},
                                                 {'Elixir.Mix.Tasks.Run',run,
                                                  5,
                                                  [{file,
                                                    "lib/mix/tasks/run.ex"},
                                                   {line,87}]},
                                                 {'Elixir.Mix.Tasks.Run',run,
                                                  1,
                                                  [{file,
                                                    "lib/mix/tasks/run.ex"},
                                                   {line,54}]},
                                                 {'Elixir.Mix.Task',run_task,
                                                  3,
                                                  [{file,"lib/mix/task.ex"},
                                                   {line,301}]}]}}

=ERROR REPORT==== 23-Jun-2017::22:49:17 ===
Error in process <0.532.0> with exit value:
{#{'__exception__' => true,
   '__struct__' => 'Elixir.Mix.Error',
   message => <<"Could not start application iconv: :iconv_app.start(:normal, []) returned an error: :bad_lib">>,
   mix => true},
 [{'Elixir.Mix',raise,1,[{file,"lib/mix.ex"},{line,279}]},
  {'Elixir.Enum','-each/2-lists^foreach/1-0-',2,
                 [{file,"lib/enum.ex"},{line,676}]},
  {'Elixir.Enum',each,2,[{file,"lib/enum.ex"},{line,676}]},
  {'Elixir.Mix.Tasks.App.Start',start,2,
                                [{file,"lib/mix/tasks/app.start.ex"},
                                 {line,97}]},
  {'Elixir.Mix.Tasks.App.Start',run,1,
                                [{file,"lib/mix/tasks/app.start.ex"},
                                 {line,78}]},
  {'Elixir.Mix.Task',run_task,3,[{file,"lib/mix/task.ex"},{line,301}]},
  {'Elixir.Mix.Tasks.Run',run,5,[{file,"lib/mix/tasks/run.ex"},{line,87}]},
  {'Elixir.Mix.Tasks.Run',run,1,[{file,"lib/mix/tasks/run.ex"},{line,54}]},
  {'Elixir.Mix.Task',run_task,3,[{file,"lib/mix/task.ex"},{line,301}]}]}

=ERROR REPORT==== 23-Jun-2017::22:49:17 ===
** Generic server 'Elixir.IntellijElixir.DebugServer' terminating 
** Last message in was {'DOWN',#Ref<0.0.4.173>,process,<0.532.0>,
                           {#{'__exception__' => true,
                              '__struct__' => 'Elixir.Mix.Error',
                              message => <<"Could not start application iconv: :iconv_app.start(:normal, []) returned an error: :bad_lib">>,
                              mix => true},
                            [{'Elixir.Mix',raise,1,
                                 [{file,"lib/mix.ex"},{line,279}]},
                             {'Elixir.Enum','-each/2-lists^foreach/1-0-',2,
                                 [{file,"lib/enum.ex"},{line,676}]},
                             {'Elixir.Enum',each,2,
                                 [{file,"lib/enum.ex"},{line,676}]},
                             {'Elixir.Mix.Tasks.App.Start',start,2,
                                 [{file,"lib/mix/tasks/app.start.ex"},
                                  {line,97}]},
                             {'Elixir.Mix.Tasks.App.Start',run,1,
                                 [{file,"lib/mix/tasks/app.start.ex"},
                                  {line,78}]},
                             {'Elixir.Mix.Task',run_task,3,
                                 [{file,"lib/mix/task.ex"},{line,301}]},
                             {'Elixir.Mix.Tasks.Run',run,5,
                                 [{file,"lib/mix/tasks/run.ex"},{line,87}]},
                             {'Elixir.Mix.Tasks.Run',run,1,
                                 [{file,"lib/mix/tasks/run.ex"},{line,54}]},
                             {'Elixir.Mix.Task',run_task,3,
                                 [{file,"lib/mix/task.ex"},{line,301}]}]}}
** When Server state == #{ref => #Ref<0.0.4.173>,
                          socket => #Port<0.13957>,
                          task => {<<"phoenix.server">>,[]}}
** Reason for termination == 
** {#{'__exception__' => true,
      '__struct__' => 'Elixir.Mix.Error',
      message => <<"Could not start application iconv: :iconv_app.start(:normal, []) returned an error: :bad_lib">>,
      mix => true},
    [{'Elixir.Mix',raise,1,[{file,"lib/mix.ex"},{line,279}]},
     {'Elixir.Enum','-each/2-lists^foreach/1-0-',2,
                    [{file,"lib/enum.ex"},{line,676}]},
     {'Elixir.Enum',each,2,[{file,"lib/enum.ex"},{line,676}]},
     {'Elixir.Mix.Tasks.App.Start',start,2,
                                   [{file,"lib/mix/tasks/app.start.ex"},
                                    {line,97}]},
     {'Elixir.Mix.Tasks.App.Start',run,1,
                                   [{file,"lib/mix/tasks/app.start.ex"},
                                    {line,78}]},
     {'Elixir.Mix.Task',run_task,3,[{file,"lib/mix/task.ex"},{line,301}]},
     {'Elixir.Mix.Tasks.Run',run,5,[{file,"lib/mix/tasks/run.ex"},{line,87}]},
     {'Elixir.Mix.Tasks.Run',run,1,[{file,"lib/mix/tasks/run.ex"},{line,54}]},
     {'Elixir.Mix.Task',run_task,3,[{file,"lib/mix/task.ex"},{line,301}]}]}
** (EXIT from #PID<0.70.0>) an exception was raised:
    ** (Mix.Error) Could not start application iconv: :iconv_app.start(:normal, []) returned an error: :bad_lib
        (mix) lib/mix.ex:279: Mix.raise/1
        (elixir) lib/enum.ex:676: Enum."-each/2-lists^foreach/1-0-"/2
        (elixir) lib/enum.ex:676: Enum.each/2
        (mix) lib/mix/tasks/app.start.ex:97: Mix.Tasks.App.Start.start/2
        (mix) lib/mix/tasks/app.start.ex:78: Mix.Tasks.App.Start.run/1
        (mix) lib/mix/task.ex:301: Mix.Task.run_task/3
        (mix) lib/mix/tasks/run.ex:87: Mix.Tasks.Run.run/5
        (mix) lib/mix/tasks/run.ex:54: Mix.Tasks.Run.run/1
        (mix) lib/mix/task.ex:301: Mix.Task.run_task/3

Compilation error with elixir 1.17 and otp 27

I'm trying to compile the iconv dependency using elixir 1.17.2 and OTP 27.0.1 and getting the following error:

$ DIAGNOSTIC=1 mix deps.compile iconv --force

...

===> Uncaught error in rebar_core. Run with DIAGNOSTIC=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: undef
===> Stack trace to the error location:
[{pc_port_env,construct,
     [{state_t,
          "/Users/my_user/code/my_elixir_project/deps/iconv",
...

     [{file,
          "/Users/my_user/code/my_elixir_project/deps/iconv/_build/default/plugins/pc/src/pc_port_specs.erl"},
      {line,244}]},
 {pc_port_specs,'-get_port_spec/3-lc$^2/1-2-',2,
     [{file,
          "/Users/my_user/code/my_elixir_project/deps/iconv/_build/default/plugins/pc/src/pc_port_specs.erl"},
      {line,197}]},
 {pc_port_specs,get_port_spec,3,
     [{file,
          "/Users/my_user/code/my_elixir_project/deps/iconv/_build/default/plugins/pc/src/pc_port_specs.erl"},
      {line,197}]},
 {pc_port_specs,'-construct/1-lc$^0/1-1-',2,
     [{file,
          "/Users/my_user/code/my_elixir_project/deps/iconv/_build/default/plugins/pc/src/pc_port_specs.erl"},
      {line,68}]},
 {pc_port_specs,construct,1,
     [{file,
          "/Users/my_user/code/my_elixir_project/deps/iconv/_build/default/plugins/pc/src/pc_port_specs.erl"},
      {line,68}]},
 {pc_prv_compile,'-do/1-lc$^0/1-0-',2,
     [{file,
          "/Users/my_user/code/my_elixir_project/deps/iconv/_build/default/plugins/pc/src/pc_prv_compile.erl"},
      {line,52}]},
 {pc_prv_compile,do,1,
     [{file,
          "/Users/my_user/code/my_elixir_project/deps/iconv/_build/default/plugins/pc/src/pc_prv_compile.erl"},
      {line,54}]}]
===> When submitting a bug report, please include the output of `rebar3 report "your command"`
** (Mix) Could not compile dependency :iconv, "/Users/my_user/.asdf/installs/elixir/1.17.2/.mix/elixir/1-17/rebar3 bare compile --paths /Users/my_user/code/my_elixir_project/_build/dev/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile iconv --force", update it with "mix deps.update iconv" or clean it with "mix deps.clean iconv"

Full log trace is in the following file (1,8K lines, so added as a file to remove noise)

iconv_elixir_1_17_compile_error.txt

1.0.11 release is missing in hex.pm

Hello.

I've tried to install 1.0.11 release to Elixir project:

  defp deps do
    [
      ...
      {:iconv, "~> 1.0.11"}
    ]

but got an error:

** (Mix) No matching version for iconv ~> 1.0.11 (from: mix.exs) in registry

The latest version is: 1.0.10

It looks like there is only 1.0.10 release on hex.pm: https://hex.pm/packages/iconv

Can you please upload latest release here?

Does not work with mix releases in elixir 1.9

Does not work with mix releases in elixir 1.9.
See stacktrace below.

`17:13:31.775 [info] Application iconv exited: exited in: :iconv_app.start(:normal, [])
** (EXIT) an exception was raised:
** (UndefinedFunctionError) function :p1_nif_utils.get_so_path/3 is undefined (module :p1_nif_utils is not available)
:p1_nif_utils.get_so_path(:iconv, [:iconv], 'iconv')
(iconv) /home/deploy/spu_etl/deps/iconv/src/iconv.erl:40: :iconv.load_nif/0
(iconv) /home/deploy/spu_etl/deps/iconv/src/iconv_app.erl:52: :iconv_app.start/2
(kernel) application_master.erl:277: :application_master.start_it_old/4
{"Kernel pid terminated",application_controller,"{application_start_failure,iconv,{bad_return,{{iconv_app,start,[normal,[]]},{'EXIT',{undef,[{p1_nif_utils,get_so_path,[iconv,[iconv],"iconv"],[]},{iconv,load_nif,0,[{file,"/home/deploy/spu_etl/deps/iconv/src/iconv.erl"},{line,40}]},{iconv_app,start,2,[{file,"/home/deploy/spu_etl/deps/iconv/src/iconv_app.erl"},{line,52}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,277}]}]}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,iconv,{bad_return,{{iconv_app,start,[normal,[]]},{'EXIT',{undef,[{p1_nif_utils,get_so_path,[iconv,[iconv],"iconv"],[]},{iconv

Crash dump is being written to: erl_crash.dump...done`

Issue with missing data

We're experiencing an issue with missing data when processing a stream and using :iconv.convert/3 as part of that stream on each chunk.

As an example - using iconv directly against the same file (utf-16le to utf-8) - we will end up with 24 million of 24 million lines in a CSV file. When processing the file as a stream using :iconv - we'll end up with only 14 million lines.

Just wondering if there is any input out there on a possible reason and what we might be able to do to avoid this.

Thanks

Application cannot be started in production

Trying to use iconv with

Elixir 1.5.2 running on
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Start of the application fails with

    application: iconv                                                                                                        
    exited: {bad_return,                                                                                                      
                {{iconv_app,start,[normal,[]]},                                                                               
                 {'EXIT',                                                                                                     
                     {undef,                                                                                                  
                         [{p1_nif_utils,get_so_path,                                                                          
                              [iconv,[iconv],"iconv"],                                                                        
                              []},                                                                                            
                          {iconv,load_nif,0,                                                                                  
                              [{file,                                                                                         
                                   "/home/stefan/phoenix_deployment/pan/deps/iconv/src/iconv.erl"},                           
                               {line,40}]},                                                                                   
                          {iconv_app,start,2,                                                                                 
                              [{file,                                                                                         
                                   "/home/stefan/phoenix_deployment/pan/deps/iconv/src/iconv_app.erl"},                       
                               {line,52}]},                                                                                   
                          {application_master,start_it_old,4,                                                                 
                              [{file,"application_master.erl"},                                                               
                               {line,273}]}]}}}}                                                                              
    type: permanent                                                                                                           
{"Kernel pid terminated",application_controller,"{application_start_failure,iconv,{bad_return,{{iconv_app,start,[normal,[]]}, 
{'EXIT',{undef,[{p1_nif_utils,get_so_path,[iconv,[iconv],\"iconv\"],[]},{iconv,load_nif,0,[{file,\"/home/stefan/phoenix_deplo 
yment/pan/deps/iconv/src/iconv.erl\"},{line,40}]},{iconv_app,start,2,[{file,\"/home/stefan/phoenix_deployment/pan/deps/iconv/ 
src/iconv_app.erl\"},{line,52}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,273}]}]}}}}}"}   
Kernel pid terminated (application_controller) ({application_start_failure,iconv,{bad_return,{{iconv_app,start,[normal,[]]},{ 
'EXIT',{undef,[{p1_nif_utils,get_so_path,[iconv,[iconv],"iconv"],[]},{iconv                                                   
                                                                                                                              
Crash dump is being written to: erl_crash.dump...done

Not compatible with Mix 1.3.0

I don't know if it is appropriate to submit this issue here, but I don't know if there is any other place to go.

I use Elixir, with {:iconv, "~> 1.0.0"} in my deps section.

In the root dir of my project, I run mix deps.get, it works ok, after that, I run mix, it gives following error:

=INFO REPORT==== 21-Jun-2016::14:00:20 ===
    application: logger
    exited: stopped
    type: temporary
** (Mix) Could not start application iconv: :iconv_app.start(:normal, []) returned an error: {:load_failed, 'Failed to load NIF library: \'/home/foo/bar/_build/dev/lib/iconv/priv/lib/iconv.so: cannot open shared object file: No such file or directory\''}

I use the lastest mix:

foo@bar:~/foo/bar$ mix -v
Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false]

Mix 1.3.0-rc.1 (9eb4781)

It works fine yesterday, and this morning I just updated the version of Elixir to 1.3.0-rc. So I downgrade the mix to 1.2.6, and

foo@bar:~/foo/bar$ rm -rf _build
foo@bar:~/foo/bar$ mix

Everything works fine.

I wonder, how can we make this work with mix 1.3.0? I need some feature in Elixir 1.3, it would be very convenient if so.

For now, I use mix 1.2.6 to compile the package, then switch back to mix 1.3.0 (without deleteing _build dir), it works fine, but this is kind of hacky.

Are we sure it is fast?

The README mentions it is fast. Are we sure?
Other languages have benchmarked iconv and found it to be slow.

https://www.npmjs.com/package/jconv

punctuations will be converted to messy code;

24> <<"Iranian atheist begs: ‘Don’t send me back home … I’ll be killed’"::utf8>> 
  "Iranian atheist begs: ‘Don’t send me back home … I’ll be killed’"
25> :iconv.convert("gbk", "utf-8", <<"Iranian atheist begs: ‘Don’t send me back home … I’ll be killed’"::utf8>>)

"Iranian atheist begs: 鈥楧on鈥檛 send me back home 鈥 I鈥檒l be killed鈥"

How to handle undefined conversions?

I was wondering how to replace undefined conversions by a substitute character when they are outside of the destination encoding, e.g. when I try to convert the euro sign (€) to SHIFT JIS encoding.

In Ruby, we can do this:

"xx€xx".encode('SHIFT_JIS', 'UTF-8', undef: :replace)
=> "xx?xx"

And the € which cannot be converted is replaced by a "?" character. This is important when doing text comparison i.e. https://unicode.org/reports/tr36/#Text_Comparison

When converting charsets, never simply omit characters that cannot be converted; at least substitute U+FFFD (when converting to Unicode) or 0x1A (when converting to bytes) to reduce security problems.

Can we do this using iconv library in Elixir/Erlang? Currently the undefined character is omitted. I guess I could do the conversion char by char and check if it returns an empty string but I was hoping if there is anything more elegant possible?

segfault

Hi, folks.

It seems that there is the same issue in C-code. Like this: https://github.com/emqplus/erlang-iconv/issues/2

Can't compile on OTP23

===> Compiling iconv
===> Linking /home/josh/dev/sendle_ex/deps/iconv/priv/lib/iconv.so
** (Mix) Could not compile dependency :iconv, "/home/josh/.asdf/installs/elixir/1.10.3-otp-23/.mix/rebar3 bare compile --paths="/home/josh/dev/sendle_ex/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile iconv", update it with "mix deps.update iconv" or clean it with "mix deps.clean iconv"

Sorry for the lack of information! Is there anyway I can provide more information/stacktrace about this?

** (ErlangError) erlang error: :nif_not_loaded
    :erlang.nif_error(:nif_not_loaded)
    /home/developer/work/project/deps/iconv/src/iconv.erl:54: :iconv.convert/3

I am requiring it in my mix.exs as:

{:iconv, "~> 1.0"}

And using it like so:

converted = :iconv.convert("iso-8859-1", "utf-8", text)

Argument description?

Can you put in the readme what the expected arguments are? The spec just says it takes 3 iolists. Can you put some example calls in the readme?