Data refreshment at heroku about opentaba-server HOT 5 CLOSED

can you expand on this? what differences do you see?

from opentaba-server.

Bumped into this while building the tests. look into 30035/plans from the current Heroku app

and try on your computer to rescrape from mmi. the results are different. even the len of the returned json.
maybe I'm doing something wrong.

from opentaba-server.

I think I understand what you're referring to. The way MMI works, they put new plans as belonging to all blocks - ie instead of the plan appearing on the one (sometimes few) gushim it's relevant for, it appears on all >400 blocks in JLM.

Our solution for this is to run clean_db.py, that moves plans which appear in >n blocks into a blacklist, and then filter out results based on that blacklist.

Perhaps a cleaner solution would be to start scraping the plan address, and cross check that with the address->block API (in the longer run maybe drop blocks altogether and just show plans based on street address).

from opentaba-server.

I guess this is the explanation to the problem (I can't recreate the problem on my computer, since I didn't figure out how to run rq on windows - but from I see in the test logs - where the results array for this gush is much longer then the one in the api)
But the tests run clean_db. So I guess that means that clean_db doesn't work right :(
about the changing of the api that would mean moving the address>block locator to server side, I guess that makes more sense any way doesn't it?

from opentaba-server.

AHA I think I figured our why clean_db doesn't work in the functional tests!
Since clean_db searches for a plans that appear over a certain threshold and since The tests only scrape one gush. clean_db Doesn't do anything..

from opentaba-server.