Comments (5)
Working as intended.
The inner call of CLD2 returns the total number of bytes of text found, a list
of three languages, their three percentages of the total text bytes, and a
reliability Boolean.
There are several degenerate cases possible
- UNKNOWN_LANGUAGE is a valid language and may show up in the list (some
webcruft strings such as "http" or "jpg" may deliberately match
UNKNOWN_LANGUAGE to prevent them from falsely indicating Somali or somesuch).
- The three percentages in general will total less than 100%, implying that the
remainder of the text is UNKNOWN_LANGUAGE.
- The percentage of the top language might be small but non-zero, meaning that
any other detected languages are a smaller percentage and the rest is unknown.
- The percentage of the top language might be 0%, meaning 100% unknown.
- Several languages are detected but they differ so slightly or they score much
too low or much too high compared to real text in each language, so the
reliability Boolean is set to false.
In your particular example, only four letter groups score:
fogr fik_ _über_ spie
Other letter groups such as
_inte tive_ _info _vide deos
occur in so many different languages that they are ignored. The letter sequence
"_über_" is strongly German, but not much else is, so the German language
score is too low for a normal 49 bytes of German. (And 49 bytes is too low for
CLD2 to do well -- two sentences is a more reasonable amount of input; CLD2's
design center is real text from web pages, not 1-4 word fragments from searches
or Twitter or suchlike.)
The letter sequence "spie" occurs about equally in German and Latvian, so the
overall score separation between those two ends up too low. In the end, both
languages are dropped entirely with too few useful table hits, leaving 100%
"other". The reliability bit is essentially over the null set of returned
languages in this case.
I haven't looked carefully at the Python wrapper, but Mike may want to expose
the percentages or set the reliability bool to false in more of the degenerate
cases above. /dick
Original comment by [email protected]
on 9 Aug 2013 at 6:17
- Added labels: ****
- Removed labels: ****
from cld2.
Original comment by [email protected]
on 9 Aug 2013 at 6:39
- Changed state: WontFix
- Added labels: ****
- Removed labels: ****
from cld2.
OK I fixed the Python bindings to always return 3 languages even when some of
them are UNKNOWN (previously I would skip UNKNOWN), and added a test case.
Original comment by [email protected]
on 9 Aug 2013 at 8:09
- Added labels: ****
- Removed labels: ****
from cld2.
I'm off on vacation in upstate Wisconsin for a week, back on the 20th. At
that time, I plan to tweak CLD2 to return unreliable if the top language is
less than 2% of the total text -- this will also cover the all-unknown case.
On Friday, August 9, 2013, wrote:
Original comment by [email protected]
on 10 Aug 2013 at 8:04
- Added labels: ****
- Removed labels: ****
from cld2.
Updated to return is_reliable=false if top language is UNKNOWN_LANGUAGE.
Original comment by [email protected]
on 20 Aug 2013 at 9:22
- Changed state: Fixed
- Added labels: ****
- Removed labels: ****
from cld2.
Related Issues (20)
- SIGBUS on ARM32 in utf8statetable.cc:517 HOT 13
- compact_lang_det.h: loadDataFromRawAddress should use types from stdint.h instead of "int" HOT 3
- Can't link "dynamic" and "full" HOT 3
- CLD2DynamicDataLoader calls delete instead of delete[] on array types HOT 4
- The ISO 639 code for Hebrew is outdated HOT 1
- Consider declaring dynamic data methods unconditionally HOT 3
- CLD2 result chunk vector omits portions of input file HOT 6
- Dynamic data loading should not use iostream HOT 5
- Windows build fails: undeclared identifier 'close' HOT 6
- Support mmap-ing dynamic data on win32 HOT 5
- Build warning on Windows with clang HOT 2
- Eliminate redundancy and/or simplify default case for compiling unittest_data.h HOT 4
- Missing include in cld2_dynamic_data_loader.cc HOT 1
- cld2_dynamic_data.cc and cld2_dynamic_data_loader.cc problems on Win32 HOT 10
- Enable dynamic data for 20141015 release HOT 1
- New GCC 5.0 hits problem with narrowing in list-initializers
- CLD should check result of "new" in all use cases
- please use CFLAGS CXXFLAGS CPPFLAGS and LDFLAGS HOT 3
- please provide a SONAME HOT 9
- cld2 testsuite failures HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cld2.