kidpollo / tanker Goto Github PK

The search_tank(query, options) method alters the options hash. Might be good for it to dup them first (in search_results) so that the user can repeat the search without recreating the options (or having to dup them itself).

Hi,
I'd like to know how to install and how to use tanker on rails 2.3.8

Tks
Pedro

Is there a way to sort results of search? I use indextank for e-commerce app for facet search & catalog list. I need to have the ability to sort by name in ASC & DESC orders, and sort by (price * product.currency.index). As results returned as paginated results I'm not able to sort them as array bevcause each page begins from the start. Paginate => false didn't work right now. Any ideas hot to accomplish this?

When I try the following for search,

List.published.search_tank ("android", :page => 1, :per_page => 10)

I get 4 records back, but if I do:

List.published.search_tank ("android", :page => 2, :per_page => 10)

I get another four records. I would have expected all 8 to be returned on page 1.

List.published.search_tank ("android", :page => 1, :per_page => 20) returns all the 8 records.

Is there a way to increase the accepted document size?

Can someone explain me how to work with facets?
Thanks

Hello guys,
i don't know if i make something wrong. here is my problem:

Tour.all.count

89
t = Tour.search_tank "deutsch*"
t.count
8
t.total_entries
87

The 8 is the real (correct) resultset (all tours with "deutsch*" in the content). But why is total_entries 87?
As far as i understand this, will_paginate uses the total_entries to calculate the paginate links ( < 1 2 3 > ).
So i get 9 paginate links for only 8 results (with 10 per page). I don't know, maybe it's a paginate bug, but i use will_paginate at other places without tanker without problems. Any help is appreciated, thanks!

I'm trying to search multiple models with conditions like so:

@results = Tanker.search([Topic, Post], 'blah', :conditions => "topics.public is true")

The results I got didn't filter the results based on conditions (i.e. the results still searched for all topics instead of just public topics). I also tried using named scopes too:

@results = Tanker.search([Topic.public, Post], 'blah')

...but got an undefined method "per_page" error. How would I search multiple models with conditions correctly (if at all possible)?

Hi there, I just started using Indextank on Heroku recently, and google points me to use this gem rather than thinkingtank (the one suggested by indextank).

Anyway, I added tanker into my Gemfile, ran bundle install, tried to follow through the example, and try to run the rake task.
That is where I ran into the rake task error.

I'm using rake gem version 0.8.7 and it gave me this error when trying to run 'rake' command:
Rake tasks fetching failed with (APP_PATH)
Tanker-1.1.1/lib/Tanker.rb:289:syntax error, unexpected ')', expecting '='

So, I took a look at the library and there seems to be a missing ampersand on the block for ModelConfig's initialize method, tried to fix that (rake runs now) and ran into ArgumentError (3 for 2) when trying to run the rake tanker:reindex task. I'm pretty new to Ruby so I'm not sure how to proceed.
Tried running version 1.0.9, no luck. Reverted to 1.0.0 and it ran fine.
Have I been doing something very wrong or is the gem broken?

I get a deprecation warning when using Tanker 1.1.6 with Rails 3.0.7, ruby 1.9.2-p290.

DEPRECATION WARNING: Yajl's JSON gem compatibility API is going to be removed in 2.0

I didn't find in documentation if this gem supports search through all models. Does this gem supports such feature? and how can I implement this?

Shouldn't reindex clear the index?

From my console:

Profile.search_tank("Kris")

Returns:

+-----+------------+-----------+-------------------------+------------+
| id  | first_name | last_name | bio                     | department |
+-----+------------+-----------+-------------------------+------------+
| 240 | Miss       | Kris      | Atque consequuntur l... | Dolor      |
+-----+------------+-----------+-------------------------+------------+

So I'm expecting this:

Profile.search_tank("Kris", :conditions => {'-id' => [204]})

To return nothing, but instead I get:

+-----+------------+-----------+-------------------------+------------+
| id  | first_name | last_name | bio                     | department |
+-----+------------+-----------+-------------------------+------------+
| 240 | Miss       | Kris      | Atque consequuntur l... | Dolor      |
+-----+------------+-----------+-------------------------+------------+

Two thoughts:

1. Am I missing something simple?
2. In this case, does id refer to the index tank document id rather than the id of the resource I'm searching on (i.e. the profile's id)?

If tanker is filtering on the index_tank doc id, I suspect that's not the behavior that most people would expect. Ideally, we shouldn't need to concern ourselves with index_tank references.

Please let me know if I'm missing something easy. Otherwise, let me know if you'd like me to start working on a fork.

Thanks, Cory.

I'd like to be able to only show results with a relevance above a certain threshhold.

Since we're doing some query expansion, the results set contains a lot of fuzz - documents which are only vaguely relevant. From reading the API docs it seems that the HTTP response body does include the relevance score of each result, which sounds perfect for this situation. However, as far as I can tell from the documentation Tanker doesn't support it.

Example

results = Article.search_tank('keyword')
top_results = results.select { |a| a.relevance >= 2 }

Relevant API Section

The response body will contain a JSON map with these fields:
    "matches": the total number of matches for the query
    "facets": a map from category name to a values count map
    "results": a list of objects with the "docid" field
        query_relevance_score: query specific document relevance score
        variable_<N>: variable value, from 0 to N
        category_<NAME>: category value for the NAME document category / facet
    "search_time": the time it took to search in seconds

Possible solutions

• Instead of returning an array of whichever model, return an array of Result which extends the model and adds a new field called relevance, which is fetched from the IndexTank response.
• Just expose the raw JSON response when you call results.index_tank_json, as a fallback for situations where Tanker doesn't cover all the API's functionality.

Let me know your thoughts, when I have some time I'd love to help work on this if you think it sounds sensible.

  def indexable?
    false
  end

Still get all models indexed!

If index tank can not be reached (or do not have the index supplied or give any other error) a search will throw an uncaught exception.

The problem is that tanker always expects the response hash to contain 'results'.

    results = index_result['results']
    return [] if results.empty?

The solution is of course simple

    results = index_result['results']
    return [] if results.nil? || results.empty?

In production you do not want your app to get an error when your search is down but in development it would be nice to see the error. Me and @arvida was discussing that maybe we should have some sort of general handling for communication exceptions? (log, send with hoptoad if available or so)

Mainly want to take it up for discussion, I can send a pull request for just fixing so it fails silently if you think that is enough for now.

I have defined category like this:

    category :type do
      tags.select{|tag| tag.tag_category_id==4 }.map(&:name)
    end

I'm returning array of tags from particular category as "type" category.
But how do i search by tags?

I've noticed things like *-. are special characters, should they be filtered out by tanker automatically?

I'm having trouble getting Indextank's AutoComplete feature to work when using Tanker. When I run it I get no results back regardless of the query. I know the same issue was reported here - #25 - but i'm confused with the resolution that was given. I'm not sure what is meant by :indexes => :text in this instance. Can anyone shed a bit more light on what I need to do?

Given the same search term:

When I use :paginate => false, I get 10 results

when I put the normal pagination options back in (:page => params[:page], :per_page => x) I get 145 results

How do i get categories and count of results in it?

Is there a way to disable the update/delete indexes in test environment?
In my case I have many models with:
after_save :update_tank_indexes
after_destroy :delete_tank_indexes
Then each time a test object created, the update_tank_indexes will be called. This make the tests run very slow, and also break the test when I don't have Internet connection. It will be nice to be able to disable tanker's indexes update, and be able to enable it on per test case basis.

As described in the documentation, I define

MY_APPLICATION::Application.config.tanker_pagination_backend = :kaminari

in config/initializers/tanker.rb. Calling Tanker.configuration in the console results in

{:url=>"http://:[email protected]"}

As far as I can see, the tanker_pagination_backend wasn't applied. If I now try to use pagination in a view with the help of

<%= paginate @products %>

I get:

undefined method `num_pages' for #WillPaginate::Collection:0x000001042ac538

As far as I can interpret this, will_paginate is used instead of kaminari. What am I doing wrong?

$ rake tanker:functions
reindexing all IndexTank functions
No IndexTank functions defined.
Define your server-side functions inside your model's tankit block like so:

  tankit 'myindex' do
    functions do
      {
        1 => "-age",
        2 => "relevance / miles(d[0], d[1], q[0], q[1])"
      }
    end
  end

I have the following defined in my model:

tankit do
    functions do
      {
          0 => "-age",
          1 => "relevance / miles(q[0], q[1], d[0], d[1])",
          2 => "miles(q[0], q[1], d[0], d[1])"
      }
    end
end

Indexing doesn't work with Mongoid as on ORM because Mongoid stores IDs as a BSON::ObjectId, not as a String or a Number (which yields the error "ArgumentError: "Value for field __id is not a String nor a number""). Any chance of adding a fix for this? I think I'll fork and give it a whirl.

Basically I would like to do queries without pagination turned on, so I just get an array of all the documents from #search. I also think new users don't expect the results to be paginated by default?

I am about to go a head and implement this, just want to check with you guys first if you have any thoughts on how it should work and if its a feature for the master branch? I few things I was thinking about:

• Should pagination be enabled/disabled by default?
• Should it be a setting on Tanker.configuration or an option per #search call?
• Maybe you should be able to select pagination lib if enabled? Like having a setting that says
  Tanker.configuration.pagination_backend = :will_paginate or Tanker.configuration.pagination_backend = :kaminari (haven't checked much on the kaminari source yet, so might not be possible to use that lib..)

So, any thoughts on this? =)

The current Geospatial example code does not produced the desired result. Should the defined index functions be using d[0] q[0] notation or should it be doc.var[0] query.var[0]? And should the Tanker.search function be accepting :var0 :var1 parameters or should it be acception :variables => {0 => ..., 1 => ...}?

Hi, I'm a beginner at Rails and I was wondering how errors are handled in Tanker? For example, what happens when the IndexTank API is unreachable? Will my model fail to save? Ideally, my model data would be saved regardless and any failed attempts at indexing would be placed in a queue for retrying later.

Thanks!

I encountered another difficulty while using the gem. I'm using Rails 3 and still using 1.8.7 Ruby (some gems are not compatible with the newer version)

Anyway, I have this code in my view

<%= form_tag search_path, :id => 'search_bar', :method => 'get' do %>
<%= text_field_tag :search, params[:search], :id =>"header-search-textbox", :placeholder => 'Search your moments here' %>
<%= submit_tag "Search", :id => 'header-search-btn', :name => nil %>

and my controller:

@moments = Moment.search_tank(params[:search], :variables => {0 => @group.id})
render 'some_page'


The problem is that my search function is broken for some keywords, for example: 'facebook', 'photo', or 'I'. Other random strings e.g. 'qwerty' or 'asdfg' actually ran and returned nil result and some other keywords return the expected results. However, for the few keywords mentioned above, instead of returning me nil result for my moments instance variable, it stops there and the page redirects to a 'page not found' route. It didn't run my render code.

Once again, I am really not sure how those few keywords break the search function. Some help here, anyone?

For now, I am using Indextank's client to get the query and there's no problem, just filing this report to check whether there's something that I'm missing...

hey guys,

i ran into a problem when i tried to search words with special characters like german umlauts (Ä, Ü and so on). I can't imagine that it is an intextank issue or is it? I hope you can help! thanks

I'd like to fetch results which include both a reference to my model instance as well as a snippet showing where the search term appeared in the indexed doc.

Looking thru the index tank docs, there's built in support for snippets functionality (also saw some reference to it in the tanker code). After poking around, however, I can't seem to solve a few problems:

1. Tanker returns only model instances. That's good – and half of what I want. I would also like it to return the snippets (preferably as a virtual attribute so I can call @post.snippets along with the other attributes already being returned thru the model instance)

2. Even when sprinkling trace statements throughout the Tanker code, I can't seem to figure out where/when/why my searches fail to fetch snippets. The snippets functionality works fine when strictly following the basic ruby tutorial in the index tank docs. Something in the Tanker code seems to be swallowing up the snippets (or preventing them from being fetched in the first place?).

If I'm missing something basic, let me know. Otherwise, I think this would be a feature worth adding. I'd be willing to help out. I just need a little direction as I'm not sure at what point the snippets are getting lost.

Thank for the work on this gem. It beats the pants off thinking_tank.

Cory.

From time to time, things are getting into my index that do not exist in my database. When this happens, I get an ActiveRecord::RecordNotFound exception (I'm using rails). I have fixed this locally by monkey patching the instantiate_results_from_db method. I know I could use rescue to handle that exception, but it seems like a shame when tanker could return 1,000 id's and one of them might not be in the database anymore. Here is how I fixed it:

#tanker.rb - instantiate_results_from_db
...
id_map.each do |klass, ids|
  # narrow ids down to records that exist 
  ids = ids.find_all {|x| constantize(klass).exists? x}
  ...
end

I hear this is what Sunspot does. What do you think? I suppose it could be configurable.

I am using indextank-1.1.6 and facing above exception

following is my model definition

include Tanker
tankit "regions" do
indexes :name
indexes :full_name
indexes :slug
end

trace is like below

IndexTank::HttpCodeException: Incorrect api call
/home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/indextank_client.rb:64:in execute' /home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/indextank_client.rb:22:inPUT'
/home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/indextank_client.rb:143:in add_documents' /home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker.rb:61:inbatch_update'
/home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker.rb:302:in block in tanker_reindex' /home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker.rb:301:ineach'
/home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker.rb:301:in each_with_index' /home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker.rb:301:intanker_reindex'
/home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker/utilities.rb:39:in block in reindex_all_models' /home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker/utilities.rb:38:ineach'
/home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker/utilities.rb:38:in reindex_all_models' /home/loken/.rvm/gems/ruby-2.1.0@corental/gems/tanker-1.1.6/lib/tanker/tasks/tanker.rake:7:inblock (2 levels) in <top (required)>'

Any tanker forks that allow for creating a custom docid?

Let's say I have a user:

+------------+-----------+------------+-----------+
| first_name | last_name | department | position  |
+------------+-----------+------------+-----------+
| Joe        | Smith     | History    | Professor |
+------------+-----------+------------+-----------+

When running version 1.0.0...

User.search_tank("Joe")
User.search_tank("Smith")
User.search_tank("Joe Smith")

...would all return the model instance for Joe Smith. After upgrading to version 1.1.2, I can still find Joe with...

User.search_tank("Joe")
User.search_tank("Smith")

but...

User.search_tank("Joe Smith")

...returns an empty array. Any idea what might be going on?

PS: New snippets are working great for me. Thanks a bunch! You should add a donate button to the repo.

With tanker, is there a way to make it so that a model is only indexed based on certain conditions? In my case, I only want to index if the model joins to another model. I've tried some different approaches, but so far nothing has been compatible with rake tanker:reindex.

def execute(req)
res = Net::HTTP.new(@uri.host).start { |http| http.request(req) }
if res.is_a? Net::HTTPSuccess
if res.body.nil? or res.body.empty?
return res.code, nil
else
begin
return res.code, JSON.parse(res.body)
rescue
raise "Invalid JSON response: #{res.body}"
end
end
elsif res.is_a? Net::HTTPUnauthorized
raise SecurityError, "Authorization required"
elsif res.is_a? Net::HTTPBadRequest
raise ArgumentError, res.body
else
raise HttpCodeException.new(res.code, res.body)
end
end

it seems like the code only forward the host part of the uri, and not the port.

I'm using HoundSleuth indexer. And it looks like it returns as an answer plain text instead of json:

Error Message: There was an error clearing or creating the charities index: Invalid JSON response: "Index deleted."

I get a lot of "getaddrinfo: Name or service not known" from my app, with the following log :

/usr/ruby1.9.2/lib/ruby/1.9.1/net/http.rb:644:in `initialize'
/usr/ruby1.9.2/lib/ruby/1.9.1/net/http.rb:644:in `open'
/usr/ruby1.9.2/lib/ruby/1.9.1/net/http.rb:644:in `block in connect'
/usr/ruby1.9.2/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
/usr/ruby1.9.2/lib/ruby/1.9.1/timeout.rb:87:in `timeout'
/usr/ruby1.9.2/lib/ruby/1.9.1/net/http.rb:644:in `connect'
/usr/ruby1.9.2/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
/usr/ruby1.9.2/lib/ruby/1.9.1/net/http.rb:626:in `start'
/app/.bundle/gems/ruby/1.9.1/gems/tanker-1.1.4/lib/indextank_client.rb:48:in `execute'
/app/.bundle/gems/ruby/1.9.1/gems/tanker-1.1.4/lib/indextank_client.rb:22:in `PUT'
/app/.bundle/gems/ruby/1.9.1/gems/tanker-1.1.4/lib/indextank_client.rb:138:in `add_document'
/app/.bundle/gems/ruby/1.9.1/gems/tanker-1.1.4/lib/tanker.rb:348:in `update_tank_indexes'

Is there a way to fix this ?

Hi all,

My results are limited to 10! I can't find for the life of me where to change it. Appreciate its simple, any help would be appreciated :)

Geoff

Indextank's autocomplete doesn't work properly when using Tanker. The problem seems to be due to Tanker putting multiple models into one index. Have you run into this, and if so, do you have a solution?

Hi guys,

Thank you very much for this work. I have a question: IndexTank isn't signing up new users. How do I get an api url? Is there a proper way to get around this? I am deploying my app on Heroku and I keep getting this message when I try to activate indexTank " That add-on is only available to selected users". Is this related to IndexTank's acquisition? Any idea how will the situation be?

Thank you very much!

I just installed the latest tanker gem and began following the instructions in the readme, but the conditions block doesn't seem to be recognized:

undefined method `conditions' for #<Tanker::ModelConfig:0x007fdd84551270>

My model looks like this:

class Region < ActiveRecord::Base

  include Tanker

  ....

  tankit 'my_index' do
    conditions do
      indexable?
    end
    indexes :name
  end

  def indexable?
    self.name?
  end

end