cliqz-oss / keyvi Goto Github PK

see #144

(got somehow closed automatically after git index rewrites, needs to be rebased)

Note: Not sure if the madvise change itself is useful, but some changes in match.h are.

If you add a .kv file with no entries in it and try to merge it, keyvi will fail with sigsev. Reproduce with the attached script.
mergeSigsegv.txt

This is probably the biggest dataset we have tried yet. The error happens at around 88% (Progress: 223600000/253842680 according to Jenkins). The backtrace is (excluding the Python fluff):

#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:167
#1  0x00007efd82de9ceb in memcpy (__len=4, __src=0x7fff198ea2d0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:51
#2  WriteRawValue (length=4, buffer=0x7fff198ea2d0, offset=<optimized out>, this=0x2dce340) at /raid/keyvi/keyvi/src/cpp/dictionary/fsa/internal/sparse_array_persistence.h:160
#3  keyvi::dictionary::fsa::internal::SparseArrayBuilder<keyvi::dictionary::fsa::internal::SparseArrayPersistence<unsigned short> >::WriteTransition (this=0x37f59a0, offset=0, 
    transitionId=<optimized out>, transitionPointer=<optimized out>) at /raid/keyvi/keyvi/src/cpp/dictionary/fsa/internal/sparse_array_builder.h:408
#4  0x00007efd82e01849 in WriteState (unpacked_state=..., offset=4294967180, this=0x37f59a0) at /raid/keyvi/keyvi/src/cpp/dictionary/fsa/internal/sparse_array_builder.h:246
#5  keyvi::dictionary::fsa::internal::SparseArrayBuilder<keyvi::dictionary::fsa::internal::SparseArrayPersistence<unsigned short> >::PersistState (this=<optimized out>, unpacked_state=...)
    at /raid/keyvi/keyvi/src/cpp/dictionary/fsa/internal/sparse_array_builder.h:86
#6  0x00007efd82e2517e in ConsumeStack (end=<optimized out>, this=<optimized out>) at /raid/keyvi/keyvi/src/cpp/dictionary/fsa/generator.h:371
#7  Add<std::basic_string<char> > (value=..., input_key=..., this=0x3372fc8) at /raid/keyvi/keyvi/src/cpp/dictionary/fsa/generator.h:218
#8  keyvi::dictionary::DictionaryCompiler<keyvi::dictionary::fsa::internal::SparseArrayPersistence<unsigned short>, keyvi::dictionary::fsa::internal::JsonValueStore>::Compile(std::function<void (unsigned long, unsigned long, void*)>, void*) (this=0x3372e50, progress_callback=..., user_data=user_data@entry=0x7efd704a6b50)
    at /raid/keyvi/keyvi/src/cpp/dictionary/dictionary_compiler.h:138
#9  0x00007efd82ddbda4 in __pyx_pf_7pykeyvi_22JsonDictionaryCompiler_18Compile (__pyx_v_args=0x7efd7044e050, __pyx_v_self=0x7efd70869d90) at src/pykeyvi.cpp:3946
#10 __pyx_pw_7pykeyvi_22JsonDictionaryCompiler_19Compile (__pyx_v_self=0x7efd70869d90, __pyx_args=0x7efd7044e050, __pyx_kwds=<optimized out>) at src/pykeyvi.cpp:3835

# dpkg -l | grep keyvi
ii  keyvi              0.1.3         amd64        key value index
ii  python-keyvi       0.1.4         amd64        python bindings for keyvi

Could this be the problem?

Is it possible to implement something like regex search using keyvi?

Thanks!

For a new feature I like to discuss modularization. I like to put the new feature into a subpackage ('pykeyvi.index'). Playing a bit with cython it unfortunately turns out, cython does not support it easily:

https://github.com/cython/cython/wiki/FAQ#how-to-compile-cython-with-subpackages

There is a recommendation on the wiki:

https://github.com/cython/cython/wiki/PackageHierarchy

which would build a 'so' for every subpackage, which would mean a lot of wasted space and higher memory usage.

RFC: Move the cython extension into a hidden namespace ('pykeyvi._core') keeping the flat structure, create a python module structure and import accordingly, this would keep a single so file but still provide subpackages:

 - pykeyvi
  - __init__.py
  - index
   - __init__.py

pykeyvi/init.py:

from pykeyvi._core import JsonDictionaryCompiler, ...

pykeyvi/index/init.py:

from pykeyvi._core import IndexReader, IndexWriter

In the longer run this also solves 2 problems:

• cleanup existing structure
• create python code extensions (like the keyvicli library)

Right now it does not work on 32bit.

The -s mode runs OK, but -o results in a segmentation fault.

@hendrik-cliqz I have uploaded the file to s3.

from #213

keyvi cli does not support parameters for compiling.

should speedup completion lookups

Apart from the installations steps mentioned,
one must need to install few things as well, otherwise one get msgrpc package not found and other similar exceptions.
easy_install msgpack-python
apt-get install python-tk python-setuptools
sudo python -m easy_install -U snappy`
these are the requirements to download this on c3.2xlarge machine on AWS.

It would be nice for the FuzzyCompletions api to support an additional parameter to limit number of results. This could speed up fuzzy completions on shorter queries.

The new compressions classes should have versions of the methods that read from / write to streams.

PrefixCompletion does not accept a parameter right now to limit the number of completions. This needs to be added, similar to the implementation in MultiWordCompletion.

https://github.com/cliqz-oss/keyvi/blob/master/pykeyvi/src/pykeyvi.pyx#L511

• Integer and Completion dictionary type mess
• move memory parameter into parameter list

see #125

(got somehow closed automatically after git index rewrites, needs to be rebased)

sample code

compiler = pykeyvi.StringDictionaryCompiler()
compiler.Add('a', 'b')
compiler.WriteToFile('dump1.kv')

Stacktrace

keyvi::dictionary::DictionaryCompiler<keyvi::dictionary::fsa::internal::SparseArrayPersistence<unsigned short>, keyvi::dictionary::fsa::internal::StringValueStore, keyvi::dictionary::sort::TpieSorter<keyvi::dictionary::sort::key_value_pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, keyvi::dictionary::fsa::ValueHandle> > >::WriteToFile<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 243
1   pykeyvi.so                    	0x000000010d0aef1b __pyx_pw_7pykeyvi_24StringDictionaryCompiler_15WriteToFile(_object*, _object*) + 411

OSX, Python 2.7/3.6

keyvicli.py should have some basic unit tests, to prevent e.g. py3 problems, see #229

Currently keyvi only supports zlib, which compresses well, but is very slow and does not do a good job for short strings. Add a framework that

1. allows the user to select the compression method
2. allows the user to select the compression threshold
3. includes also short string compression methods.

it used to work, but I haven't checked for some time.

• to get this done use https://pypi.python.org/pypi/delocate/

Currently, we use keyvi compiler option of "floating_point_precision" for word embeddings in sharding/compiling step. It would be nice to pass this option to command line / python api of keyvi for any keyvi file where the values will be vectors.

Ex: keyvi_compiler_options = {"minimization": "off", "floating_point_precision": "single"}

This will be helpful in reducing the size of massive keyvi files composed of vector values. (Ex. Document Vectors- ~2.1B Vectors ~ 300 Dimensions). I haven't been able to figure out how one can use this feature. A standalone example with documentation will be useful.
@hendrikmuhs

... and maybe the Python package could be renamed to keyvi too. Or is it too late (https://testpypi.python.org/pypi/pykeyvi)?