akihikodaki / cld3-ruby Goto Github PK

View Code? Open in Web Editor NEW

77.0 6.0 22.0 82 KB

cld3-ruby is an interface of Compact Language Detector v3 (CLD3) for Ruby.

License: Apache License 2.0

Ruby 56.15% C++ 43.85%

rubygems ffi chrome

cld3-ruby's Introduction

cld3-ruby

cld3-ruby is an interface of Compact Language Detector v3 (CLD3) for Ruby.

Usage

require 'cld3'

cld3 = CLD3::NNetLanguageIdentifier.new(0, 1000)

cld3.find_language("こんにちは") # => #<struct Struct::Result language=:ja, probability=1.0, reliable?=true, proportion=1.0, byte_ranges=[]>

cld3.find_language("This is a pen.") # => #<struct Struct::Result language=:en, probability=0.9999408721923828, reliable?=true, proportion=1.0, byte_ranges=[]>

cld3.find_language("здравствуйте") # => #<struct Struct::Result language=:ru, probability=0.3140212297439575, reliable?=false, proportion=1.0, byte_ranges=[]>

Installation

Prerequisites

Bundler
C++ compiler
RubyGems

Instructions

I (Akihiko Odaki) recommend to setup this library installing via gem.

You can also build this library by yourself. Rakefile includes a Rake task to put this code into files buildable as a gem. Build a gem with bundle exec rake command.

Platform-specific information

FreeBSD

FreeBSD port is available as rubygem-cld3 in textproc category.

https://svnweb.freebsd.org/ports/head/textproc/rubygem-cld3/

Troubleshooting

gem install cld3 triggers native library building. If it fails, it is likely that some required facilities are missing. Make sure C++ compiler is installed. I recommend GCC as a C++ compiler.

If you cannot identify the cause of your problem, run spec of this library and see whether the problem is reproducible with it or not. Spec is not included in the gem, so clone the source code repository and then run bundle exec rake spec. The source code repository is at https://github.com/akihikodaki/cld3-ruby.

In case you cannot solve your problem by yourself and cannot help abandoning or find an issue in this library, please open an issue at https://github.com/akihikodaki/cld3-ruby/issues.

If you found an issue and managed to fix it and agree to share the fix under Apache-2.0, please open a pull request at https://github.com/akihikodaki/cld3-ruby/pulls. Your contribution would be appreciated by other users and recorded with Git.

Versioning

The version has 3 parts: major, minor, and patch. They are joined with . as delimiters in the order.

The increment of the major version and the minor version indicates it can involve any change.

The increment of the patch version indicates there is no change of the supported languages and no change of the existing APIs.

Contact

To ask questions or report issues please open issues at https://github.com/akihikodaki/cld3-ruby/issues.

Credits

This program was written by Akihiko Odaki and other contributors. CLD3 was written by its own authors.

cld3-ruby's People

Contributors

Stargazers

Watchers

cld3-ruby's Issues

ProtoBuf on GitHub CI

Good day and let me first thank you for this GEM. It works perfectly for my needs!

I develop on macOS and there, using brew install protobuf, I was instantly able to work with the GEM.

However, I am using GitHub CI and there, protobuf is not installed by default. I tried a few ways of installing it on the CI, but nothing worked. It keeps failing to compile the native extensions of the GEM, stating that protobuf cannot be found.

It would help me to know which files and at which location it is expecting. Is it something that you could help me with?

Thanks in advance.

Empty and nonsense strings are detected as being in a language

Hi,

It may be the underlying CLD3 library rather than your wrapper, but:

cld3 = CLD3::NNetLanguageIdentifier.new(0)
> cld3.find_language("")
=> #<struct Struct::Result language=:ja, probability=0.7837570905685425, :reliable?=true, proportion=1.0>
> cld3.find_language("123")
=> #<struct Struct::Result language=:ja, probability=0.7837570905685425, :reliable?=true, proportion=1.0>

You can get rid of this specific error by requiring at least one byte of data, but still:

> cld3 = CLD3::NNetLanguageIdentifier.new(1)
> cld3.find_language("a")
=> #<struct Struct::Result language=:lb, probability=0.9725591540336609, :reliable?=true, proportion=1.0>

Debain fails to resolve _ZTIN6google8protobuf11MessageLiteE

Reported at mastodon/mastodon#2949 (comment).

have error too with debian...

LoadError: Could not open library '/home/mastodon/live/vendor/bundle/ruby/2.4.0/gems/cld3-3.1.0/lib/../ext/cld3/libcld3.so': /home/mastodon/live/vendor/bundle/ruby/2.4.0/gems/cld3-3.1.0/lib/../ext/cld3/libcld3.so: undefined symbol: _ZTIN6google8protobuf11MessageLiteE

Debian 8 doesn't have g++6 (https://packages.debian.org/fr/jessie/g++) only 4.9
so the use of CLD3 is problematic

@akihikodaki already installed

root@d0p1:~# dpkg -l | grep protobuf
ii libprotobuf-dev:amd64 2.6.1-1 amd64 protocol buffers C++ library (development files)
ii libprotobuf-lite9:amd64 2.6.1-1 amd64 protocol buffers C++ library (lite version)
ii libprotobuf9:amd64 2.6.1-1 amd64 protocol buffers C++ library
ii protobuf-compiler 2.6.1-1 amd64 compiler for protocol buffer definition files

It sounds like a problem with pkg-config.

require "mkmf"

# Check pkg-config first to inform the library is missing if so.
pkg_config("protobuf")

cld3-ruby/ext/cld3/extconf.rb

Line 20 in a82866f

pkg_config("protobuf")

JRuby? FFI?

Hello! I just ran into your library while trying to set up Mastodon on JRuby. The extension does not look very big. It would be great if cld3-ruby could work on JRuby!

I was wondering why this library is not simply using FFI? Is it because of C++?

I would like to help get an FFI version (or a JRuby version through some other means) of cld3-ruby!

Optional Clang support

Hi,

it'd be great to have Clang support here in order to be able to build cld3-ruby native extensions on systems with older GCC (like 4.6), but with Clang (e.g. 3.4) available.

For this, only g++ version just needs to be checked, and if it's <4.7, then fallback to clang++ if it's available. For this case extconf.rb can be edited to have CXX config value set conditionally:

RbConfig::MAKEFILE_CONFIG['CXX'] = "clang++"

I checked both 3.1.2 and 3.1.3 build just fine with Clang 3.4+.

Thanks in advance.

Please document how to install Protobuf dependency for Ubuntu

Could you add the following documentation under the platform specific section in the readme?

On Ubuntu/Debian to install the necessary protobuf dependencies run:
  apt install libprotobuf-dev protobuf-compiler

ruby 3.2.0: Could not open library libcld3.bundle'

updating a ruby app to 3.2.0 (from 3.0.5). Gemfile has cld3 locked to 3.5.0

On invoking this library:

Could not open library '/Users/jsharpe/.rvm/gems/ruby-3.2.0@ctm3/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/libcld3.bundle'

Confirmed the file libcld3.bundle is nowhere to be found. It exists in a similiar location for ruby 3.0.5

Support ffi 1.10.0

Is there anything in the way of supporting ffi 1.10.0?

warning: rb_safe_level will be removed in Ruby 3.0

cld3.rb:91: warning: rb_safe_level will be removed in Ruby 3.0 in 3.3.0.

3.5.2 fails to install on amazonlinux:2

compiling nnet_language_identifier_c.cc
nnet_language_identifier_c.cc:110:1: sorry, unimplemented: non-trivial designated initializers not supported
 };
 ^
nnet_language_identifier_c.cc:110:1: sorry, unimplemented: non-trivial designated initializers not supported
nnet_language_identifier_c.cc:110:1: sorry, unimplemented: non-trivial designated initializers not supported
make: *** [nnet_language_identifier_c.o] Error 1

make failed, exit code 2

3.5.1 also fails with the same error
3.5.0 installs and works

My C is rusty. Various googling suggests this is some kind of "using the wrong compiler issue" - but isn't that something the ruby gem should obfuscate away? Here's a SO with that error: https://stackoverflow.com/questions/31215971/non-trivial-designated-initializers-not-supported

Not sure if this is relevant, but I have this gcc:

gcc --version
gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Segmentation fault in cld-3.4.3

I am running on Ubuntu 20.04 with Ruby 2.7.2 / cld3-ruby 3.4.3 on WSL and get an segmentation fault, wenn (by accident) I passed identical values to min and max to the constructor CLD3::NNetLanguageIdentifier.new(min, max).

/home/coezbek/.rvm/gems/ruby-2.7.2/gems/cld3-3.4.3/lib/cld3.rb:79: [BUG] Illegal instruction at 0x00007f6501ff37a0
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0040 p:---- s:0231 e:000230 CFUNC  :new_NNetLanguageIdentifier

libprotobuf.so.17.0.0

FFI 1.15.0 support

Hi,

Some gems from my Gemfile requires ffi >=1.15.0.

Currently cld3 does not include this version:

gem.add_dependency "ffi", [ ">= 1.1.0", "< 1.15.0" ]

Is there anything that prevents us to enhance this gem to support 1.15.0?

https://github.com/ffi/ffi/blob/master/CHANGELOG.md#1150--2021-03-05

.so file not found

Getting this error after installation in Docker container

Could not open library '/usr/local/bundle/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/libcld3.so': /usr/local/bundle/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/libcld3.so: cannot open shared object file: No such file or directory (LoadError)

The .so file is present in /usr/local/bundle/gems/cld3-3.5.0/lib/ but not /usr/local/bundle/gems/cld3-3.5.0/ext/cld3/libcld3.so

Reproduction: https://github.com/debug-poc/cld3
https://github.com/debug-poc/cld3/actions/runs/3781540490/jobs/6428505336

symbol not found in flat namespace '__ZN6google8protobuf8internal16InternalMetadataD1Ev'

Hello :)

thanks for this gem!

After an upgrade to protobuf, rails console raises this error

/Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library '/Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/cld3-3.4.4/lib/cld3/../../ext/cld3/libcld3.bundle': dlopen(/Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/cld3-3.4.4/lib/cld3/../../ext/cld3/libcld3.bundle, 0x0005): symbol not found in flat namespace '__ZN6google8protobuf8internal16InternalMetadataD1Ev' (LoadError)
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `map'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `ffi_lib'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/cld3-3.4.4/lib/cld3/unstable.rb:22:in `<module:Unstable>'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/cld3-3.4.4/lib/cld3/unstable.rb:19:in `<module:CLD3>'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/cld3-3.4.4/lib/cld3/unstable.rb:18:in `<main>'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bootsnap-1.11.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bootsnap-1.11.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6/lib/active_support/dependencies.rb:332:in `block in require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6/lib/active_support/dependencies.rb:299:in `load_dependency'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6/lib/active_support/dependencies.rb:332:in `require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/cld3-3.4.4/lib/cld3.rb:22:in `<main>'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bootsnap-1.11.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bootsnap-1.11.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bundler-2.3.10/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bundler-2.3.10/lib/bundler/runtime.rb:55:in `each'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bundler-2.3.10/lib/bundler/runtime.rb:55:in `block in require'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bundler-2.3.10/lib/bundler/runtime.rb:44:in `each'
	from /Users/faber/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/bundler-2.3.10/lib/bundler/runtime.rb:44:in `require'

I'm using these versions at the moment:

$ protoc --version
libprotoc 3.20.1

$ bundle show cld3
cld3-3.4.4

$ ruby -v
ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin21]

I've tried to reinstall cld3 with bundle pristine, to delete tmp/cache/bootsnap but got the same error.

Any suggestion?

Thank you!

Dependency install fails beacuse of protobuf

I am trying to install cld3 on an Apple Silicon MacBook running macOS 12.1:

I have tried installing protobuf from source, via homebrew (stable) and via homebrew (from git), always the same error when installing.

Google is of no help.

Build log of bundle install:

Installing cld3 3.4.3 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/lerk/.rvm/gems/ruby-3.0.3/gems/cld3-3.4.3/ext/cld3
/Users/lerk/.rvm/rubies/ruby-3.0.3/bin/ruby -I /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0 -r
./siteconf20211221-97073-qu4a2e.rb extconf.rb
creating Makefile

current directory: /Users/lerk/.rvm/gems/ruby-3.0.3/gems/cld3-3.4.3/ext/cld3
make DESTDIR\= clean

current directory: /Users/lerk/.rvm/gems/ruby-3.0.3/gems/cld3-3.4.3/ext/cld3
make DESTDIR\=
compiling base.cc
compiling embedding_feature_extractor.cc
In file included from embedding_feature_extractor.cc:16:
In file included from ./embedding_feature_extractor.h:23:
In file included from ./feature_extractor.h:45:
./cld_3/protos/feature_extractor.pb.h:17:2: error: This file was generated by an older version of protoc which is
#error This file was generated by an older version of protoc which is
 ^
./cld_3/protos/feature_extractor.pb.h:18:2: error: incompatible with your Protocol Buffer headers. Please
#error incompatible with your Protocol Buffer headers. Please
 ^
./cld_3/protos/feature_extractor.pb.h:19:2: error: regenerate this file with a newer version of protoc.
#error regenerate this file with a newer version of protoc.
 ^
./cld_3/protos/feature_extractor.pb.h:140:33: error: non-virtual member function marked 'final' hides virtual member function
  inline Parameter* New() const final {
                                ^
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/message_lite.h:188:24: note: hidden overloaded virtual
function 'google::protobuf::MessageLite::New' declared here: different number of parameters (1 vs 0)
  virtual MessageLite* New(Arena* arena) const = 0;
                       ^
In file included from embedding_feature_extractor.cc:16:
In file included from ./embedding_feature_extractor.h:23:
In file included from ./feature_extractor.h:45:
./cld_3/protos/feature_extractor.pb.h:197:3: error: unknown type name 'PROTOBUF_MUST_USE_RESULT'
  PROTOBUF_MUST_USE_RESULT std::string* release_name();
  ^
./cld_3/protos/feature_extractor.pb.h:197:33: error: non-friend class member 'string' cannot have a qualified name
  PROTOBUF_MUST_USE_RESULT std::string* release_name();
                           ~~~~~^
./cld_3/protos/feature_extractor.pb.h:197:39: error: expected ';' at end of declaration list
  PROTOBUF_MUST_USE_RESULT std::string* release_name();
                                      ^
                                      ;
./cld_3/protos/feature_extractor.pb.h:215:3: error: unknown type name 'PROTOBUF_MUST_USE_RESULT'
  PROTOBUF_MUST_USE_RESULT std::string* release_value();
  ^
./cld_3/protos/feature_extractor.pb.h:215:33: error: non-friend class member 'string' cannot have a qualified name
  PROTOBUF_MUST_USE_RESULT std::string* release_value();
                           ~~~~~^
./cld_3/protos/feature_extractor.pb.h:215:33: error: duplicate member 'string'
./cld_3/protos/feature_extractor.pb.h:197:33: note: previous declaration is here
  PROTOBUF_MUST_USE_RESULT std::string* release_name();
                                ^
./cld_3/protos/feature_extractor.pb.h:215:39: error: expected ';' at end of declaration list
  PROTOBUF_MUST_USE_RESULT std::string* release_value();
                                      ^
                                      ;
./cld_3/protos/feature_extractor.pb.h:92:9: error: cannot initialize object parameter of type 'const
google::protobuf::MessageLite' with an expression of type 'chrome_lang_id::Parameter'
    if (GetOwningArena() == from.GetOwningArena()
        ^~~~~~~~~~~~~~
./cld_3/protos/feature_extractor.pb.h:92:29: error: cannot initialize object parameter of type 'const
google::protobuf::MessageLite' with an expression of type 'chrome_lang_id::Parameter'
    if (GetOwningArena() == from.GetOwningArena()
                            ^~~~
./cld_3/protos/feature_extractor.pb.h:126:9: error: cannot initialize object parameter of type 'const
google::protobuf::MessageLite' with an expression of type 'chrome_lang_id::Parameter'
    if (GetOwningArena() == other->GetOwningArena()) {
        ^~~~~~~~~~~~~~
./cld_3/protos/feature_extractor.pb.h:126:29: error: cannot initialize object parameter of type 'const
google::protobuf::MessageLite' with an expression of type 'chrome_lang_id::Parameter'
    if (GetOwningArena() == other->GetOwningArena()) {
                            ^~~~~
./cld_3/protos/feature_extractor.pb.h:129:7: error: no matching function for call to 'GenericSwap'
      ::PROTOBUF_NAMESPACE_ID::internal::GenericSwap(this, other);
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/generated_message_util.h:163:22: note: candidate function
not viable: no known conversion from 'chrome_lang_id::Parameter *' to 'google::protobuf::MessageLite *' for 1st argument
PROTOBUF_EXPORT void GenericSwap(MessageLite* m1, MessageLite* m2);
                     ^
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/generated_message_util.h:165:22: note: candidate function
not viable: no known conversion from 'chrome_lang_id::Parameter *' to 'google::protobuf::Message *' for 1st argument
PROTOBUF_EXPORT void GenericSwap(Message* m1, Message* m2);
                     ^
In file included from embedding_feature_extractor.cc:16:
In file included from ./embedding_feature_extractor.h:23:
In file included from ./feature_extractor.h:45:
./cld_3/protos/feature_extractor.pb.h:134:19: error: cannot initialize object parameter of type 'const
google::protobuf::MessageLite' with an expression of type 'chrome_lang_id::Parameter'
    GOOGLE_DCHECK(GetOwningArena() == other->GetOwningArena());
                  ^~~~~~~~~~~~~~
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/stubs/logging.h:192:26: note: expanded from macro
'GOOGLE_DCHECK'
#define GOOGLE_DCHECK    GOOGLE_CHECK
                         ^
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/stubs/logging.h:152:26: note: expanded from macro
'GOOGLE_CHECK'
  GOOGLE_LOG_IF(FATAL, !(EXPRESSION)) << "CHECK failed: " #EXPRESSION ": "
                         ^~~~~~~~~~
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/stubs/logging.h:149:5: note: expanded from macro
'GOOGLE_LOG_IF'
  !(CONDITION) ? (void)0 : GOOGLE_LOG(LEVEL)
    ^~~~~~~~~
In file included from embedding_feature_extractor.cc:16:
In file included from ./embedding_feature_extractor.h:23:
In file included from ./feature_extractor.h:45:
./cld_3/protos/feature_extractor.pb.h:134:39: error: cannot initialize object parameter of type 'const
google::protobuf::MessageLite' with an expression of type 'chrome_lang_id::Parameter'
    GOOGLE_DCHECK(GetOwningArena() == other->GetOwningArena());
                                      ^~~~~
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/stubs/logging.h:192:26: note: expanded from macro
'GOOGLE_DCHECK'
#define GOOGLE_DCHECK    GOOGLE_CHECK
                         ^
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/stubs/logging.h:152:26: note: expanded from macro
'GOOGLE_CHECK'
  GOOGLE_LOG_IF(FATAL, !(EXPRESSION)) << "CHECK failed: " #EXPRESSION ": "
                         ^~~~~~~~~~
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/stubs/logging.h:149:5: note: expanded from macro
'GOOGLE_LOG_IF'
  !(CONDITION) ? (void)0 : GOOGLE_LOG(LEVEL)
    ^~~~~~~~~
In file included from embedding_feature_extractor.cc:16:
In file included from ./embedding_feature_extractor.h:23:
In file included from ./feature_extractor.h:45:
./cld_3/protos/feature_extractor.pb.h:305:49: error: non-virtual member function marked 'final' hides virtual member function
  inline FeatureFunctionDescriptor* New() const final {
                                                ^
/opt/homebrew/Cellar/protobuf/HEAD-b360b9e_1/include/google/protobuf/message_lite.h:188:24: note: hidden overloaded virtual
function 'google::protobuf::MessageLite::New' declared here: different number of parameters (1 vs 0)
  virtual MessageLite* New(Arena* arena) const = 0;
                       ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [embedding_feature_extractor.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/lerk/.rvm/gems/ruby-3.0.3/gems/cld3-3.4.3 for inspection.
Results logged to /Users/lerk/.rvm/gems/ruby-3.0.3/extensions/arm64-darwin-21/3.0.0/cld3-3.4.3/gem_make.out

  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:93:in `run'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:44:in `block in make'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:36:in `each'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:36:in `make'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:63:in `block in build'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/tempfile.rb:317:in `open'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:26:in `build'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:159:in `build_extension'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:193:in `block in build_extensions'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in `each'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in `build_extensions'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/rubygems/installer.rb:845:in `build_extensions'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/rubygems_gem_installer.rb:71:in `build_extensions'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/rubygems_gem_installer.rb:28:in `install'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/source/rubygems.rb:200:in `install'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/installer/gem_installer.rb:54:in `install'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/installer/parallel_installer.rb:186:in `do_install'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/worker.rb:62:in `apply_func'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/worker.rb:57:in `block in process_queue'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/worker.rb:54:in `loop'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/worker.rb:54:in `process_queue'
  /Users/lerk/.rvm/rubies/ruby-3.0.3/lib/ruby/3.0.0/bundler/worker.rb:91:in `block (2 levels) in create_threads'

Failed to locate protobuf

Hi there,

I just saw a similar issue on bundler#5655 and I'm guessing this might be an issue with cld3 itself, could you help me on this?

This is what happens when I try to install the gem:

current directory: ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cld3-3.1.2/ext/cld3
~/.rbenv/versions/2.4.1/bin/ruby -r ./siteconf20170626-4596-1na0dfr.rb extconf.rb
Failed to locate protobuf
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=~/.rbenv/versions/2.4.1/bin/$(RUBY_BASE_NAME)
	--with-protobuf-config
	--without-protobuf-config
	--with-pkg-config
	--without-pkg-config

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/extensions/x86_64-darwin-16/2.4.0-static/cld3-3.1.2/mkmf.log

extconf failed, exit code 1

Thanks in advance

Warning in ruby 3.2

Getting this warning while running cld3 in ruby 3.2 in Linux in docker

/usr/local/bundle/gems/cld3-3.5.1/lib/cld3.rb:53: warning: undefining the allocator of T_DATA class CLD3::NNetLanguageIdentifier::Unstable

Different results returned based on `MAX_NUM_BYTES_TO_CONSIDER`

I was investigating a chunk of text being wrongly labeled as uk (Ukranian), when it was el (Greek). It turned out that the misclassification just happens with certain values of MAX_NUM_BYTES_TO_CONSIDER and when using find_language.

# Kafka text as an example + the word Velcro
text = "Πολυαγαπημένε πατέρα πρόσφατα Velcro με ρώτησες κάποια φορά γιατί ισχυρίζομαι πως σε φοβάμαι. Εγώ δεν ήξερα, ως συνήθως, τι να σου απαντήσω, εν μέρει ακριβώς λόγω του φόβου που νιώθω για σένα, εν μέρει επειδή στην αιτιολόγηση του φόβου αυτού συγκαταλέγονται πάρα πολλές λεπτομέρειες, που εν τη ρύμη του λόγου εγώ ούτε κατά το ήμισυ δεν θα μπορούσα να τις συγκρατήσω. Κι αν εδώ προσπαθώ να σου απαντήσω γραπτώς, μόνο ανολοκλήρωτο κατά πολύ θα αποβεί και τούτο, επειδή και κατά τη γραφή ο φόβος και οι συνέπειές του με κωλύουν έναντί σου κι επειδή το μέγεθος του υλικού εν γένει υπερβαίνει κατά πολύ τη μνήμη μου και το λογικό μου. Για σένα το ζήτημα αποδεικνυόταν πάντοτε πολύ απλό, τουλάχιστον στον βαθμό που μιλούσες εσύ γι’ αυτό ενώπιόν μου και, αδιακρίτως, ενώπιον πολλών άλλων. Εσένα σου φαινόταν να είναι κάπως έτσι: Εσύ εργαζόσουν σκληρά σ’ όλη σου τη ζωή, τα πάντα για τα παιδιά σου, προ πάντων για εμένα τα θυσίαζες, εγώ έκαμνα συνεπώς «ζωή χαρισάμενη», είχα πλήρη ελευθερία να μάθω ό,τι ήθελα, κανέναν λόγο δεν είχα να έχω έγνοιες για την καθημερινή διατροφή, να έχω έγνοιες συνεπώς εν γένει• εσύ αντ’ αυτών καμμίαν ευγνωμοσύνη δεν αξίωνες, γνωρίζεις «την ευγνωμοσύνη των παιδιών, αλλά εν τούτοις τουλάχιστον μια "

200.times { |i|
  max_bytes = 500 + i * 10
  cld3 = CLD3::NNetLanguageIdentifier.new(180, max_bytes)

  lang = cld3.find_language(text)
  lang2 = cld3.find_top_n_most_freq_langs(text, 1)

  puts "When max_bytes is #{max_bytes} probability is less than 0.999: #{lang.probability}" if lang.probability < 0.999

  if lang.language != :el
    puts "When max_bytes is #{max_bytes} then cld3::find_language returns #{lang.language},
      find_top_n_most_freq_langs returns #{lang2.first.language}"
    #pp lang
    #pp lang2
  end
}

puts "Size: #{text.length} - Bytesize: #{text.encode(Encoding::UTF_8).bytesize}"

When I run this, I get a low probability detection for max_bytes == 1660 to 1690.

If I vary the MIN_BYTES_TO_CONSIDER then ruby interpreter crashes with illegal instruction / access violations.

Bad identification for short input

Same as google/cld3#31, I have a few examples where this gem (and/or cld3) performs poorly with small texts:

Another example from v3.6.0:

CLD3::NNetLanguageIdentifier.new(0, 1000).find_language("User ID FA1324102A6E2C72 How to add my name on leader board?")
=> #<struct CLD3::NNetLanguageIdentifier::Result language=:ja, probability=0.7837570905685425, reliable?=true, proportion=1.0, byte_ranges=[]>

CLD3::NNetLanguageIdentifier.new(0, 1000).find_language("AAAA AA A0000000A0A0AAA AAA AA AAA AA AAAA AA AAAAAA")
=> #<struct CLD3::NNetLanguageIdentifier::Result language=:ja, probability=0.7837570905685425, reliable?=true, proportion=1.0, byte_ranges=[]>

How in the world that text is detected as JA 🤯

Failed to locate Protobuf on Windows.

Hi, I'm having this issue, while trying to install cld3. I've read all the issues before. But none of them had the fix I need.

mkmf.log only says: ""pkg-config --exists protobuf"
package configuration for protobuf is not found
"
I tried setting "c:\protobuf\protoc.exe" in the PATH at environment variables.
The cmd was able to get the protoc --version:

and the gcc --version:

Can you help me solve it?

Support Ruby 2.7

cld3-3.2.4 requires ruby version >= 2.3.0, < 2.7.0, which is incompatible with
the current version, ruby 2.7.0p0

Building gem fails on mac os

I forked this gem to have a compatible branch for ruby 3.2 but building from source and installing it fails on Mac OS (Intel)

❯ gem install ./cld3-3.5.0.gem
Building native extensions. This could take a while...
ERROR:  Error installing ./cld3-3.5.0.gem:
	ERROR: Failed to build gem native extension.

    current directory: /Users/manu/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/cld3-3.5.0/ext/cld3
/Users/manu/.asdf/installs/ruby/3.2.1/bin/ruby -I /Users/manu/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0 extconf.rb
creating Makefile

current directory: /Users/manu/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/cld3-3.5.0/ext/cld3
make DESTDIR\= sitearchdir\=./.gem.20230220-22477-6v5sf9 sitelibdir\=./.gem.20230220-22477-6v5sf9 clean

current directory: /Users/manu/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/cld3-3.5.0/ext/cld3
make DESTDIR\= sitearchdir\=./.gem.20230220-22477-6v5sf9 sitelibdir\=./.gem.20230220-22477-6v5sf9
compiling nnet_language_identifier_c.cc
nnet_language_identifier_c.cc:21:10: fatal error: 'nnet_language_identifier.h' file not found
#include "nnet_language_identifier.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [nnet_language_identifier_c.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/manu/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/cld3-3.5.0 for inspection.
Results logged to /Users/manu/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/extensions/x86_64-darwin-21/3.2.0/cld3-3.5.0/gem_make.out

cld3 load error with garbage as error message

Hello.

I got an error while updating mastodon, and I don't know ruby well enough to debut it, but it seems it's cld3-related :

mastodon:~/live$ RAILS_ENV=production bundle exec rails assets:precompile --trace
rails aborted!
Bundler::GemRequireError: There was an error while trying to load the gem 'cld3'.
Gem Load Error is: �����t
Backtrace for gem load error is:
/usr/lib/ruby/gems/2.4.0/gems/ffi-1.9.18/lib/ffi/library.rb:277:in `attach'
/usr/lib/ruby/gems/2.4.0/gems/ffi-1.9.18/lib/ffi/library.rb:277:in `attach_function'
/usr/lib/ruby/gems/2.4.0/gems/cld3-3.1.3/lib/cld3.rb:93:in `<module:Unstable>'
/usr/lib/ruby/gems/2.4.0/gems/cld3-3.1.3/lib/cld3.rb:84:in `<module:CLD3>'
/usr/lib/ruby/gems/2.4.0/gems/cld3-3.1.3/lib/cld3.rb:24:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:82:in `block (2 levels) in require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:77:in `each'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:77:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:66:in `each'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:66:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler.rb:108:in `require'
/home/mastodon/live/config/application.rb:7:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `require'
/home/mastodon/live/Rakefile:4:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `load'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `load'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:286:in `block in load'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:286:in `load'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/command.rb:46:in `invoke'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands.rb:16:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `require'
bin/rails:4:in `<main>'
Bundler Error Backtrace:
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:85:in `rescue in block (2 levels) in require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:77:in `each'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:77:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:66:in `each'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:66:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler.rb:108:in `require'
/home/mastodon/live/config/application.rb:7:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `require'
/home/mastodon/live/Rakefile:4:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `load'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `load'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:286:in `block in load'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:286:in `load'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/command.rb:46:in `invoke'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands.rb:16:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `require'
bin/rails:4:in `<main>'
�����t
/usr/lib/ruby/gems/2.4.0/gems/ffi-1.9.18/lib/ffi/library.rb:277:in `attach'
/usr/lib/ruby/gems/2.4.0/gems/ffi-1.9.18/lib/ffi/library.rb:277:in `attach_function'
/usr/lib/ruby/gems/2.4.0/gems/cld3-3.1.3/lib/cld3.rb:93:in `<module:Unstable>'
/usr/lib/ruby/gems/2.4.0/gems/cld3-3.1.3/lib/cld3.rb:84:in `<module:CLD3>'
/usr/lib/ruby/gems/2.4.0/gems/cld3-3.1.3/lib/cld3.rb:24:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:82:in `block (2 levels) in require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:77:in `each'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:77:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:66:in `each'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler/runtime.rb:66:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.15.1/lib/bundler.rb:108:in `require'
/home/mastodon/live/config/application.rb:7:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `require'
/home/mastodon/live/Rakefile:4:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `load'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `load'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:286:in `block in load'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:286:in `load'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'
/usr/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/command.rb:46:in `invoke'
/usr/lib/ruby/gems/2.4.0/gems/railties-5.1.1/lib/rails/commands.rb:16:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/bootsnap-1.0.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `block in require'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:258:in `load_dependency'
/usr/lib/ruby/gems/2.4.0/gems/activesupport-5.1.1/lib/active_support/dependencies.rb:292:in `require'
bin/rails:4:in `<main>'

library seems to be compiled without error :

mastodon:~/live$ ldd vendor/bundle/ruby/2.4.0/extensions/x86_64-linux/2.4.0/cld3-3.1.2/libcld3.so 
	ldd (0x73d9397f9000)
	libprotobuf.so.11 => /usr/lib/libprotobuf.so.11 (0x73d93913f000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x73d938ded000)
	libc.musl-x86_64.so.1 => ldd (0x73d9397f9000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x73d938bdb000)
	libz.so.1 => /lib/libz.so.1 (0x73d9389c4000)
mastodon:~/live$ ldd vendor/bundle/ruby/2.4.0/extensions/x86_64-linux/2.4.0/ffi-1.9.18/ffi_c.so 
	ldd (0x651465055000)
	libruby.so.2.4 => /usr/lib/libruby.so.2.4 (0x651464a0c000)
	libffi.so.6 => /usr/lib/libffi.so.6 (0x651464804000)
	libc.musl-x86_64.so.1 => ldd (0x651465055000)
	libgmp.so.10 => /usr/lib/libgmp.so.10 (0x6514645a0000)

Can you help me debugging that ?

Detecting protobuf

I'm working on updating the rubygem-cld3 package on FreeBSD. After version 3.1.1, the installation fails because of issues detecting protobuf. mkmf.log says "package configuration for protobuf is not found" and gem_make.out says "Failed to locate protobuf". Protobuf is install under /usr/local (details https://svnweb.freebsd.org/ports/head/devel/protobuf/pkg-plist?revision=439753&view=markup).

Do you have suggestions for fixing the detection of protobuf?

Got error "Too much level of symbolic links"

Hello !

I'm trying to run the bundle install for mastodon which requires cld3, however, I run in to this issue :

feature_extractor.h:45:47: fatal error: cld_3/protos/feature_extractor.pb.h: Trop de niveaux de liens symboliques
 #include "cld_3/protos/feature_extractor.pb.h"

I'm on debian 9 amd64 and run ruby 2.4.1.

Thanks !

gem install cld3 -v '3.2.2' --source 'https://rubygems.org/' fails on MacOS

@mdf092 just referenced this issue on #10 about 5 hours ago. Opening a new issue now that I am also experiencing it.

Behavior:
Running gem install cld3 -v '3.2.2' --source 'https://rubygems.org/' fails with the following output:

Building native extensions. This could take a while...
ERROR:  Error installing cld3:
	ERROR: Failed to build gem native extension.

    current directory: /Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib/ruby/gems/2.5.0/gems/cld3-3.2.2/ext/cld3
/Users/jacobherrington/.asdf/installs/ruby/2.5.1/bin/ruby -r ./siteconf20180920-96617-1fpfe1p.rb extconf.rb
Failed to locate protobuf

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib/ruby/gems/2.5.0/extensions/x86_64-darwin-17/2.5.0-static/cld3-3.2.2/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib/ruby/gems/2.5.0/gems/cld3-3.2.2 for inspection.
Results logged to /Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib/ruby/gems/2.5.0/extensions/x86_64-darwin-17/2.5.0-static/cld3-3.2.2/gem_make.out

cat /Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib/ruby/gems/2.5.0/extensions/x86_64-darwin-17/2.5.0-static/cld3-3.2.2/mkmf.log outputs:

"pkg-config --exists protobuf"
| pkg-config --libs protobuf
=> "-L/usr/local/Cellar/protobuf/3.6.1/lib -lprotobuf -pthread\n"
"clang -o conftest -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include/ruby-2.5.0/x86_64-darwin17 -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include/ruby-2.5.0/ruby/backward -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include/ruby-2.5.0 -I. -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe conftest.c  -L. -L/Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib -L. -L/Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib  -fstack-protector -L/usr/local/lib     -lruby.2.5.1-static -framework Foundation  -lpthread -ldl -lobjc "
ld: warning: text-based stub file /System/Library/Frameworks//Foundation.framework/Foundation.tbd and library file /System/Library/Frameworks//Foundation.framework/Foundation are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation are out of sync. Falling back to library file for linking.
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

"clang -o conftest -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include/ruby-2.5.0/x86_64-darwin17 -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include/ruby-2.5.0/ruby/backward -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include/ruby-2.5.0 -I. -I/Users/jacobherrington/.asdf/installs/ruby/2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe conftest.c  -L. -L/Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib -L. -L/Users/jacobherrington/.asdf/installs/ruby/2.5.1/lib  -fstack-protector -L/usr/local/lib     -lruby.2.5.1-static -framework Foundation -L/usr/local/Cellar/protobuf/3.6.1/lib -lprotobuf -pthread -lpthread -ldl -lobjc "
ld: warning: text-based stub file /System/Library/Frameworks//Foundation.framework/Foundation.tbd and library file /System/Library/Frameworks//Foundation.framework/Foundation are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation are out of sync. Falling back to library file for linking.
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

package configuration for protobuf is not found

For the sake of sanity, protoc --version outputs: libprotoc 3.6.1.

Environment:
MacOS High Sierra 10.13.6, fish shell, Ruby 2.5.1, using asdf.

cat /usr/local/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0/cld3-3.2.5/gem_make.out

current directory: /usr/local/lib/ruby/gems/2.6.0/gems/cld3-3.2.5/ext/cld3
/usr/local/opt/ruby/bin/ruby -I /usr/local/lib/ruby/site_ruby/2.6.0 -r ./siteconf20200103-51270-1o99whl.rb extconf.rb
Could not make proto path relative: feature_extractor.proto: No such file or directory
Could not make proto path relative: sentence.proto: No such file or directory
Could not make proto path relative: task_spec.proto: No such file or directory
creating Makefile

current directory: /usr/local/lib/ruby/gems/2.6.0/gems/cld3-3.2.5/ext/cld3
make "DESTDIR=" clean

current directory: /usr/local/lib/ruby/gems/2.6.0/gems/cld3-3.2.5/ext/cld3
make "DESTDIR="
compiling nnet_language_identifier_c.cc
nnet_language_identifier_c.cc:21:10: fatal error: 'nnet_language_identifier.h' file not found
#include "nnet_language_identifier.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [nnet_language_identifier_c.o] Error 1

make failed, exit code 2