asmuth / recommendify Goto Github PK

Right now if you're trying to work from a Redis database other than 0, you must use the Ruby similarity implementation, because the database info does not get passed to the C extension.

I don't understand how cosine works. It looks like it does not implement the correct functions.

  input_matrix :thing_other,
#    :native => true,
    :similarity_func => :cosine

recommender.thing_other.process!
Traceback (most recent call last):
        1: from (irb):13
RuntimeError (implemented in subclass)

Looking at the code, it looks like there are unimplemented methods on the class. Is this expected?

Is it right to recommend only p(roduct)4, p5 for "for('p3')" when I input add_set("u1", ["p1","p2","p3"]); add_set("u2", ["p3", "p4", "p5"])?
I think it is some kind of bug in algorithm, because I think it is equivalent p1, p2, p4, and p5.

Am I wrong?

Creating an input matrix (jaccard) with a redis db other than 0 results in the correct items and ccmatrix being created as data is added, but empty similarities values when process! is called. This appears to be because the db is not taken into account by the native recommendify command. AFAICT it never gets passed to it.

I am trying to install and its impossible to me. I have installed hiredis and it is in my directory: /usr/local/include

how is the correct way? Here is the context of what i have made: http://stackoverflow.com/questions/20594338/unable-to-install-recommendify-gem-hiredis-h-not-found

Thanks in advance

Using the native worker, I get the error

RuntimeError: error: dirty exit (pid 2771 SIGSEGV (signal 11))

It happens every time I run recommendify when the item_id has been added in a set. Otherwise, it exits correctly with "item count is zero or one"

Any ideas what could cause this? What information do you need from me?

Turned native: true, and Recommendify would fail silently w/ Jaccard similarity instead of throwing some error when native extensions are failing. (if not built, I do get a proper error message: recommendify_native not found - you need to run rake build_native first)

Note that matrices are built but the final similarities hash in Redis is empty:

redis 127.0.0.1:6379[5]> hgetall recommendify:users:ccmatrix

1. "989:992"
2. "1"
3. "989:994"
4. "1"
5. "992:994"
6. "2"
7. "990:992"
8. "1"
9. "990:994"
10. "1"
11. "991:993"
12. "1"
13. "991:994"
14. "1"
15. "993:994"
16. "1"
    redis 127.0.0.1:6379[5]> hgetall recommendify:similarities
17. "989"
18. ""
19. "992"
20. ""
21. "994"
22. ""
23. "990"
24. ""
25. "991"
26. ""
27. "993"
28. ""

Some debug output added shows exit output as: item count is zero or one <-- this error should be thrown

Further debugging shows that unless you use DB 0 in Redis from Ruby, the native code will fail since it uses the default DB. Will open a new issue or fork for this.

root@bt:~# msfupdate
[]
[] Attempting to update the Metasploit Framework...
[*]

fatal: 'upstream' does not appear to be a git repository
fatal: The remote end hung up unexpectedly
[] Attempting to add remote 'upstream' to your local git repository.
[] Added remote 'upstream' to your local git repository.
HEAD is now at 99b4620 Do final cleanup for sap_configservlet_exec_noauth
Already on 'master'
remote: Counting objects: 197, done.
remote: Compressing objects: 100% (88/88), done.
remote: Total 161 (delta 124), reused 108 (delta 71)
Receiving objects: 100% (161/161), 24.55 KiB, done.
Resolving deltas: 100% (124/124), completed with 19 local objects.
From git://github.com/rapid7/metasploit-framework

• [new branch] Datacut-master -> upstream/Datacut-master

• [new branch] creds-counter-cache -> upstream/creds-counter-cache

• [new branch] feature/railgun/error_msg -> upstream/feature/railgun/error_msg

• [new branch] master -> upstream/master

• [new branch] release -> upstream/release

• [new branch] unstable -> upstream/unstable
  Updating 99b4620..6821c36
  Fast-forward
  Gemfile | 2 +-
  Gemfile.lock | 4 +-
  .../gather/apple_safari_webarchive_uxss.rb | 100 +++++---
  .../unix/webapp/php_wordpress_total_cache.rb | 249 ++++++++++++++++++++
  spec/lib/rex/post/meterpreter/packet_spec.rb | 246 ++++++++++----------
  spec/lib/rex/proto/http/client_request_spec.rb | 8 +-
  6 files changed, 442 insertions(+), 167 deletions(-)
  create mode 100644 modules/exploits/unix/webapp/php_wordpress_total_cache.rb
  [*] Updating gems...
  Fetching gem metadata from http://rubygems.org/.........
  Fetching gem metadata from http://rubygems.org/..
  Using rake (10.0.4)
  Using i18n (0.6.1)
  Using multi_json (1.0.4)
  Using activesupport (3.2.13)
  Using builder (3.0.4)
  Using activemodel (3.2.13)
  Using arel (3.0.2)
  Using tzinfo (0.3.37)
  Using activerecord (3.2.13)
  Using database_cleaner (0.9.1)
  Using diff-lcs (1.2.2)
  Using factory_girl (4.2.0)
  Using json (1.7.7)
  Installing pg (0.15.0) with native extensions
  Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

  /opt/metasploit/ruby/bin/ruby extconf.rb 
  

  checking for pg_config... yes
  Using config values from /opt/metasploit/postgresql/bin/pg_config
  checking for libpq-fe.h... *** extconf.rb failed ***
  Could not create Makefile due to some reason, probably lack of
  necessary libraries and/or headers. Check the mkmf.log file for more
  details. You may need configuration options.
  Provided configuration options:
  --with-opt-dir
  --without-opt-dir
  --with-opt-include
  --without-opt-include=${opt-dir}/include
  --with-opt-lib
  --without-opt-lib=${opt-dir}/lib
  --with-make-prog
  --without-make-prog
  --srcdir=.
  --curdir
  --ruby=/opt/metasploit/ruby/bin/ruby
  --with-pg
  --without-pg
  --with-pg-dir
  --without-pg-dir
  --with-pg-include
  --without-pg-include=${pg-dir}/include
  --with-pg-lib
  --without-pg-lib=${pg-dir}/lib
  --with-pg-config
  --without-pg-config
  --with-pg_config
  --without-pg_config
  /opt/metasploit/ruby/lib/1.9.1/mkmf.rb:381:in try_do': The compiler failed to generate an executable file. (RuntimeError) You have to install development tools first. from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:506:intry_cpp'
  from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:970:in block in find_header' from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:790:inblock in checking_for'
  from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:284:in block (2 levels) in postpone' from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:254:inopen'
  from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:284:in block in postpone' from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:254:inopen'
  from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:280:in postpone' from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:789:inchecking_for'
  from /opt/metasploit/ruby/lib/ruby/1.9.1/mkmf.rb:969:in find_header' from extconf.rb:43:in

  '

Gem files will remain installed in /opt/metasploit/ruby/lib/ruby/gems/1.9.1/gems/pg-0.15.0 for inspection.
Results logged to /opt/metasploit/ruby/lib/ruby/gems/1.9.1/gems/pg-0.15.0/ext/gem_make.out
An error occured while installing pg (0.15.0), and Bundler cannot continue.
Make sure that gem install pg -v '0.15.0' succeeds before bundling.

am using windows xp as host machine and black track 5 r3 as guest

Hi,
Thanks for your effort
I am getting error message when I running the below commands in system, Please throw me light on the same.

vijayram@ubuntu:~/projects/recommendify/recommendify/doc$ ruby example.rb
example.rb:3:in `require': no such file to load -- redis (LoadError)
from example.rb:3

Thanks and Regards
Vijay

DELETED

The native processor has the redis connection information is hardcoded to 127.0.0.2:6379. This should be configurable from ruby. See recommendify.c line 59.

https://github.com/paulasmuth/recommendify/blob/master/src/recommendify.c#L59

Just wondering if there could be any more instructions on how to get started? Would be very helpful for ruby newbies like me.

Once you build the Recommendify gem, where do you go from there? Do you have to write a new model, like in the example?

Trying to copy that code into a model file gives the error "uninitialized constant Recommendify" - does not recognize it as a gem/module. Tips? Thanks! (p.s. running recommendify 0.2.3 on windows)

I put gem 'recommendify' in my Gemfile and ran bundle and got the following:

Installing recommendify (0.3.6) with native extensions 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

        /Users/rahil/.rvm/rubies/ruby-1.9.2-p0/bin/ruby extconf.rb 

make
mkdir -p ../bin
gcc -Wall recommendify.c -lhiredis -o ../bin/recommendify
recommendify.c:4:29: error: hiredis/hiredis.h: No such file or directory
recommendify.c: In function ‘main’:
recommendify.c:20: error: ‘redisContext’ undeclared (first use in this function)
recommendify.c:20: error: (Each undeclared identifier is reported only once
recommendify.c:20: error: for each function it appears in.)
recommendify.c:20: error: ‘c’ undeclared (first use in this function)
recommendify.c:21: error: ‘redisReply’ undeclared (first use in this function)
recommendify.c:21: error: ‘all_items’ undeclared (first use in this function)
recommendify.c:22: error: ‘reply’ undeclared (first use in this function)
recommendify.c:83: warning: implicit declaration of function ‘redisConnectWithTimeout’
recommendify.c:92: warning: implicit declaration of function ‘redisCommand’
recommendify.c:100: warning: implicit declaration of function ‘freeReplyObject’
recommendify.c:111: error: ‘REDIS_REPLY_ARRAY’ undeclared (first use in this function)
recommendify.c:161: warning: implicit declaration of function ‘redisAppendCommand’
recommendify.c:162: warning: implicit declaration of function ‘redisGetReply’
make: *** [build] Error 1


Gem files will remain installed in /Users/rahil/.rvm/gems/ruby-1.9.2-p0@myproject/gems/recommendify-0.3.6 for inspection.
Results logged to /Users/rahil/.rvm/gems/ruby-1.9.2-p0@myproject/gems/recommendify-0.3.6/ext/gem_make.out
An error occured while installing recommendify (0.3.6), and Bundler cannot continue.
Make sure that `gem install recommendify -v '0.3.6'` succeeds before bundling.

Then when I ran gem install recommendify -v '0.3.6' I got the following:

Building native extensions.  This could take a while...
ERROR:  Error installing recommendify:
        ERROR: Failed to build gem native extension.

        /Users/rahil/.rvm/rubies/ruby-1.9.2-p0/bin/ruby extconf.rb

make
mkdir -p ../bin
gcc -Wall recommendify.c -lhiredis -o ../bin/recommendify
recommendify.c:4:29: error: hiredis/hiredis.h: No such file or directory
recommendify.c: In function ‘main’:
recommendify.c:20: error: ‘redisContext’ undeclared (first use in this function)
recommendify.c:20: error: (Each undeclared identifier is reported only once
recommendify.c:20: error: for each function it appears in.)
recommendify.c:20: error: ‘c’ undeclared (first use in this function)
recommendify.c:21: error: ‘redisReply’ undeclared (first use in this function)
recommendify.c:21: error: ‘all_items’ undeclared (first use in this function)
recommendify.c:22: error: ‘reply’ undeclared (first use in this function)
recommendify.c:83: warning: implicit declaration of function ‘redisConnectWithTimeout’
recommendify.c:92: warning: implicit declaration of function ‘redisCommand’
recommendify.c:100: warning: implicit declaration of function ‘freeReplyObject’
recommendify.c:111: error: ‘REDIS_REPLY_ARRAY’ undeclared (first use in this function)
recommendify.c:161: warning: implicit declaration of function ‘redisAppendCommand’
recommendify.c:162: warning: implicit declaration of function ‘redisGetReply’
make: *** [build] Error 1


Gem files will remain installed in /Users/rahil/.rvm/gems/ruby-1.9.2-p0@myproject/gems/recommendify-0.3.6 for inspection.
Results logged to /Users/rahil/.rvm/gems/ruby-1.9.2-p0@myproject/gems/recommendify-0.3.6/ext/gem_make.out

http://stillmaintained.com/

There's too many people running their own forks

In the method #serialize_item in similarity_matrix.rb, I found there could be a bug when converting similarity metrics into a string. the original code is like this

def serialize_item(item_id, max_precision=5)
items = @write_queue[item_id].to_a
items.sort!{ |a,b| b[1] <=> a[1] }
items = items[0..max_neighbors-1]
items = items.map{ |i,s| s>0 ? "#{i}:#{s.to_s[0..max_precision]}" : nil }
items.compact * "|"
end

the issue is at statement "s.to_s[0..max_precision]". For some cases, the value of s could be a fairly small floating point number. when converting it to string, Ruby will use scientific notation as the result of s.to_s. For example,

"8.442380751371887e-05"

After executing "s.to_s[0..max_precision]", given max_precision=5, the return string will become "8.4423", which causes the similarity result being enlarged incorrectly.

I would suggest to change "s.to_s[0..max_precision]" to

"sprint('%.#{max_precision}f", s)"

with sprintf, the above example will output "0.00008".

it is an edge case, there could be other better solution to it. but just want to share my findings on this issue.

Thank you very much for the great gem!

MetaSync

Hi,
I just tried to install recommendify on my machine but gem install always fails with the errors below during the build of the native extension. I tried to install on Mac OS 10.7, Archlinux and Debian and had no luck. So it seems to be an error on the end of the gem. ;-)

mkdir -p ../bin
gcc -Wall recommendify.c -lhiredis -o ../bin/recommendify
recommendify.c:4:29: error: hiredis/hiredis.h: No such file or directory
recommendify.c: In function ‘main’:
recommendify.c:20: error: ‘redisContext’ undeclared (first use in this function)
recommendify.c:20: error: (Each undeclared identifier is reported only once
recommendify.c:20: error: for each function it appears in.)
recommendify.c:20: error: ‘c’ undeclared (first use in this function)
recommendify.c:21: error: ‘redisReply’ undeclared (first use in this function)
recommendify.c:21: error: ‘all_items’ undeclared (first use in this function)
recommendify.c:22: error: ‘reply’ undeclared (first use in this function)
recommendify.c:83: warning: implicit declaration of function ‘redisConnectWithTimeout’
recommendify.c:92: warning: implicit declaration of function ‘redisCommand’
recommendify.c:100: warning: implicit declaration of function ‘freeReplyObject’
recommendify.c:111: error: ‘REDIS_REPLY_ARRAY’ undeclared (first use in this function)
recommendify.c:161: warning: implicit declaration of function ‘redisAppendCommand’
recommendify.c:162: warning: implicit declaration of function ‘redisGetReply’
make: *** [build] Error 1``` 

I'm getting this error. Can you help?

Thanks

Installing recommendify (0.3.6) with native extensions 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

        /Users/xxx/.rvm/rubies/ruby-1.9.3-p125/bin/ruby extconf.rb 

make
mkdir -p ../bin
gcc -Wall recommendify.c -lhiredis -o ../bin/recommendify
recommendify.c:4:29: error: hiredis/hiredis.h: No such file or directory
recommendify.c: In function ‘main’:
recommendify.c:20: error: ‘redisContext’ undeclared (first use in this function)
recommendify.c:20: error: (Each undeclared identifier is reported only once
recommendify.c:20: error: for each function it appears in.)
recommendify.c:20: error: ‘c’ undeclared (first use in this function)
recommendify.c:21: error: ‘redisReply’ undeclared (first use in this function)
recommendify.c:21: error: ‘all_items’ undeclared (first use in this function)
recommendify.c:22: error: ‘reply’ undeclared (first use in this function)
recommendify.c:83: warning: implicit declaration of function ‘redisConnectWithTimeout’
recommendify.c:92: warning: implicit declaration of function ‘redisCommand’
recommendify.c:100: warning: implicit declaration of function ‘freeReplyObject’
recommendify.c:111: error: ‘REDIS_REPLY_ARRAY’ undeclared (first use in this function)
recommendify.c:161: warning: implicit declaration of function ‘redisAppendCommand’
recommendify.c:162: warning: implicit declaration of function ‘redisGetReply’
make: *** [build] Error 1