SearchQuery doesn't seem to handle multiple exclude attributes correctly about xapian-haystack HOT 22 CLOSED

Just as a sanity check, could you try the same filter/exclude without the date_lte?

from xapian-haystack.

Thanks for the quick response! You bet. It works correctly without the date__lte:

ipdb> sqs
Out[0]: [<SearchResult: restaurants.restaurant (pk=621)>, <SearchResult: places.place (pk=5159)>, <SearchResult: events.event (pk=509)>, <SearchResult: restaurants.restaurant (pk=487)>, <SearchResult: restaurants.restaurant (pk=518)>, <SearchResult: restaurants.restaurant (pk=589)>, <SearchResult: restaurants.restaurant (pk=604)>, <SearchResult: restaurants.restaurant (pk=649)>, <SearchResult: restaurants.restaurant (pk=673)>, <SearchResult: restaurants.restaurant (pk=548)>, <SearchResult: restaurants.restaurant (pk=557)>, <SearchResult: restaurants.restaurant (pk=608)>, <SearchResult: restaurants.restaurant (pk=626)>, <SearchResult: restaurants.restaurant (pk=704)>, <SearchResult: restaurants.restaurant (pk=508)>, <SearchResult: restaurants.restaurant (pk=592)>, <SearchResult: restaurants.restaurant (pk=593)>, <SearchResult: restaurants.restaurant (pk=347)>, <SearchResult: restaurants.restaurant (pk=989)>, '...(remaining elements truncated)...']
ipdb> sqs.filter(contenttype='events.event')
Out[0]: [<SearchResult: events.event (pk=509)>]
ipdb> sqs.exclude(contenttype='events.event')
Out[0]: [<SearchResult: restaurants.restaurant (pk=621)>, <SearchResult: places.place (pk=5159)>, <SearchResult: restaurants.restaurant (pk=487)>, <SearchResult: restaurants.restaurant (pk=518)>, <SearchResult: restaurants.restaurant (pk=589)>, <SearchResult: restaurants.restaurant (pk=604)>, <SearchResult: restaurants.restaurant (pk=649)>, <SearchResult: restaurants.restaurant (pk=673)>, <SearchResult: restaurants.restaurant (pk=548)>, <SearchResult: restaurants.restaurant (pk=557)>, <SearchResult: restaurants.restaurant (pk=608)>, <SearchResult: restaurants.restaurant (pk=626)>, <SearchResult: restaurants.restaurant (pk=704)>, <SearchResult: restaurants.restaurant (pk=508)>, <SearchResult: restaurants.restaurant (pk=592)>, <SearchResult: restaurants.restaurant (pk=593)>, <SearchResult: restaurants.restaurant (pk=347)>, <SearchResult: restaurants.restaurant (pk=989)>, <SearchResult: restaurants.restaurant (pk=357)>, '...(remaining elements truncated)...']

from xapian-haystack.

Great. In that case, what exactly is today? Is it a string or a DateTime?

from xapian-haystack.

It's a datetime.date.today() object.

from xapian-haystack.

One final question before I try to write a test-case and patch; what is the date field of Event 509 set to?

from xapian-haystack.

It's set to datetime.datetime(2010, 1, 21, 0, 0).

Here's some more ipdb output verifying that:
ipdb> sqs
Out[0]: [<SearchResult: restaurants.restaurant (pk=621)>, <SearchResult: places.place (pk=5159)>, <SearchResult: events.event (pk=509)>, <SearchResult: restaurants.restaurant (pk=487)>, <SearchResult: restaurants.restaurant (pk=518)>, <SearchResult: restaurants.restaurant (pk=589)>, <SearchResult: restaurants.restaurant (pk=604)>, <SearchResult: restaurants.restaurant (pk=649)>, <SearchResult: restaurants.restaurant (pk=673)>, <SearchResult: restaurants.restaurant (pk=548)>, <SearchResult: restaurants.restaurant (pk=557)>, <SearchResult: restaurants.restaurant (pk=608)>, <SearchResult: restaurants.restaurant (pk=626)>, <SearchResult: restaurants.restaurant (pk=704)>, <SearchResult: restaurants.restaurant (pk=508)>, <SearchResult: restaurants.restaurant (pk=592)>, <SearchResult: restaurants.restaurant (pk=593)>, <SearchResult: restaurants.restaurant (pk=347)>, <SearchResult: restaurants.restaurant (pk=989)>, '...(remaining elements truncated)...']
ipdb> sqs.filter(contenttype='events.event')
Out[0]: [<SearchResult: events.event (pk=509)>]
ipdb> sqs.filter(contenttype='events.event')[0].date
Out[0]: datetime.datetime(2010, 1, 21, 0, 0)
ipdb> today
Out[0]: datetime.date(2010, 1, 21)

And here's some ipdb output verifying that excluding with just the date works, as well. It seems to only be a problem with multiple exclude attributes.

ipdb> sqs.filter(date__lt=today)
Out[0]: [<SearchResult: restaurants.restaurant (pk=621)>, <SearchResult: places.place (pk=5159)>, <SearchResult: restaurants.restaurant (pk=487)>, <SearchResult: restaurants.restaurant (pk=518)>, <SearchResult: restaurants.restaurant (pk=589)>, <SearchResult: restaurants.restaurant (pk=604)>, <SearchResult: restaurants.restaurant (pk=649)>, <SearchResult: restaurants.restaurant (pk=673)>, <SearchResult: restaurants.restaurant (pk=548)>, <SearchResult: restaurants.restaurant (pk=557)>, <SearchResult: restaurants.restaurant (pk=608)>, <SearchResult: restaurants.restaurant (pk=626)>, <SearchResult: restaurants.restaurant (pk=704)>, <SearchResult: restaurants.restaurant (pk=508)>, <SearchResult: restaurants.restaurant (pk=592)>, <SearchResult: restaurants.restaurant (pk=593)>, <SearchResult: restaurants.restaurant (pk=347)>, <SearchResult: restaurants.restaurant (pk=989)>, <SearchResult: restaurants.restaurant (pk=357)>, '...(remaining elements truncated)...']
ipdb> sqs.exclude(date__lt=today)
Out[0]: [<SearchResult: events.event (pk=509)>]

Thanks again for your help! It's much appreciated. I'm on a lovely accelerated timeline, and we're trying to bring this new search to production this weekend. ;-)

from xapian-haystack.

Here's the query description that this results in:

Xapian::Query(((Zcafe OR cafe) AND ( AND_NOT (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event)) AND VALUE_RANGE 15 20100221000000 99990101000000))

I'm thinking that should actually be:

Xapian::Query(((Zcafe OR cafe) AND ( AND_NOT ((ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event**)** AND VALUE_RANGE 15 20100221000000 99990101000000)))

Am I anywhere close? :-)

from xapian-haystack.

Maybe:

Xapian::Query(((Zcafe OR cafe) AND ( AND_NOT (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event) AND_NOT VALUE_RANGE 15 20100221000000 99990101000000)))

from xapian-haystack.

I think that either Xapian::Query(((Zcafe OR cafe) AND (<alldocuments> AND_NOT ((ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event) AND VALUE_RANGE 15 20100221000000 99990101000000))) or Xapian::Query(((Zcafe OR cafe) AND_NOT (<alldocuments> AND (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event)) AND_NOT VALUE_RANGE 15 20100221000000 99990101000000)) is correct.

Writing up some test cases..

from xapian-haystack.

Can you try this as a (temporary) workaround:
sqs.exclude(contenttype='events.event').exclude(date__lte=today)
and tell me if that works?

from xapian-haystack.

Unfortunately, no dice.

sqs.exclude(contenttype='events.event').exclude(date__lte=today) gives me:

Xapian::Query(((Zcafe OR cafe) AND (<alldocuments> AND_NOT (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event)) AND (<alldocuments> AND_NOT VALUE_RANGE 14 00010101000000 20100121000000)))

And I'm not sure what I did differently, but the original query - sqs.exclude(contenttype='events.event', date__lte=today) - now gives me:

Xapian::Query(((Zcafe OR cafe) AND (<alldocuments> AND_NOT (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event)) AND (<alldocuments> AND_NOT VALUE_RANGE 14 00010101000000 20100121000000)))

...which I now notice are the same thing.

from xapian-haystack.

Wait, this looks like what I'd expect.

If I'm reading that query correctly, it'd return anything that has the word "cafe" in it, but not an Event and not in the date range 1/1/0001 to today.

I think I'm confusing myself. What is the behaviour that you're expecting?

from xapian-haystack.

I want it to exclude results that are events which have a date less than or equal to today. Event 509 has today's date, so it should be excluded from the original result set, leaving just the restaurants and places. In production, the query will be a little more complex, but I've boiled it down to this in order to isolate the issue.

Here's the search_node description that should be converted to a Xapian query:

<SQ: AND (content__exact=cafe AND NOT (contenttype__exact=events.event AND date__lte=2010-01-21))>

from xapian-haystack.

Oh, I see. You want to combine the two parameters. The issue here is that the backend sees them as two independent filters. Essentially, the query is like this:

cafe AND_NOT Event AND_NOT some_date

I think you want this:

cafe AND_NOT (Event AND some_date)

Can you try this:

sqs.exclude(SQ(contenttype='events') | SQ(date_lte=date.today()))

from xapian-haystack.

I changed that to sqs.exclude(SQ(contenttype='events.event') | SQ(date__lte=datetime.date.today())). Instead of excluding it, that selected it.

Xapian::Query(((Zcafe OR cafe) AND (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event OR VALUE_RANGE 15 00010101000000 20100121000000)))

I tried changing that to this: sqs.filter(~SQ(contenttype='events.event') & ~SQ(date__lte=today)) and it seems to work:

Xapian::Query(((Zcafe OR cafe) AND (<alldocuments> AND_NOT (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event)) AND (<alldocuments> AND_NOT VALUE_RANGE 15 00010101000000 20100121000000)))

I think that achieves what I was looking for, right? sqs.filter() and sqs.exclude() return the exact same results here - probably because django-haystack is converting that to query.add_filter(SQ(~SQ(contenttype='events.event') & ~SQ(date__lte=today))) and query.add_filter(~SQ(~SQ(contenttype='events.event') & ~SQ(date__lte=today))), respectively.

from xapian-haystack.

Hmm - now that I'm getting down into the more complex query, I'm having trouble again. Here is the search_node description:

<SQ: AND (content__exact=cafe AND NOT ((contenttype__exact=events.event AND (date__lt=2010-01-14 OR date__gt=2010-04-21))))>

And no mater what I do I can't get the results to come out correctly. I'm continuing to debug the _query_from_search_node method, which I suspect is the culprit. :-)

from xapian-haystack.

Okay, I came up with a minor change to xapian_backend.py that resolves the issue for me. It doesn't appear that it was handling negation correctly. at the end of the _query_from_search_node method. I'm going to go ahead and create a fork because I need to check out the changes to our testing server. Let me know what you think of the change.

Thanks!

from xapian-haystack.

I was wrong, the change I made only fixed one test case, the rest were still broken. I reverted it back. Here's the problem I'm dealing with now. I'm trying to exclude old or far-future events from our search results:

The search_node I'm using evaluates to:

<SQ: AND (content__exact=old AND content__exact=97 AND NOT ((contenttype__exact=events.event AND (date__lt=2010-01-14 OR date__gt=2010-04-21))))>

It gets converted to:

Xapian::Query(((Zold OR old) AND (Z97 OR 97) AND (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event) AND ((<alldocuments> AND_NOT VALUE_RANGE 15 20100114000000 99990101000000) OR (<alldocuments> AND_NOT VALUE_RANGE 15 00010101000000 20100421000000))))

To be the most efficient, I believe it should be:

Xapian::Query(((Zold OR old) AND (Z97 OR 97) AND_NOT ((ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event) AND VALUE_RANGE 15 20100114000000 99990101000000 OR VALUE_RANGE 15 00010101000000 20100421000000)))

Or at the very least, it should come out as:

Xapian::Query(((Zold OR old) AND (Z97 OR 97) AND_NOT ((ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event) AND ((<alldocuments> AND VALUE_RANGE 15 20100114000000 99990101000000) OR (<alldocuments> AND VALUE_RANGE 15 00010101000000 20100421000000)))))

Does that look right?

from xapian-haystack.

I've been working for this all day now, and I'm just going to have to give up for now. Here's the SearchQuerySet line that gives the search_node I listed above:

sqs = sqs.exclude(
    SQ(contenttype='events.event') &
    (
        SQ(date__lt=(today - SITE_SEARCH_EVENTS_FROM)) |
        SQ(date__gt=(today + SITE_SEARCH_EVENTS_TO))
    )
)

It constructs a good search_node, but the xapian.Query it's building just doesn't work. Once again, I greatly appreciated your help, and I hope you have a good day despite this craziness! :-)

from xapian-haystack.

Alright, I'm giving up on this. I've deleted my fork, because it doesn't work. :-) I'm going to open a new issue for the real problem, and lay this issue to rest.

from xapian-haystack.

Confirmed with Daniel that Haystack should behave as follows: ".exclude() should use an AND between all kwargs provided and then NOT the whole thing."

Will write up some tests to confirm whether or not this is what's happening now and fix (or not) as required.

from xapian-haystack.

Closing this issue as I think that Issue #45 better describes it going forward.

from xapian-haystack.