currently we scrape all the plan info from the gush results list page. if we start scr

related to <a class="issue-link js-issue-link" data-error-text="Failed to load title"

done in <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https:

scrape plan address about opentaba-server HOT 3 CLOSED

niryariv commented on September 16, 2024

scrape plan address

from opentaba-server.

niryariv commented on September 16, 2024

related to #58

from opentaba-server.

niryariv commented on September 16, 2024

need 3 things:

add scraping location to the scraper (ie go into the plan URL, read data next to מקום התוכנית)
have some way to know if the scraper already looked up an address and didn't find it (so it wont do so again)
for plans already in the DB, write code that'll scrape 50 plans addresses each night, starting from the most recent non-scraped one

from opentaba-server.

florpor commented on September 16, 2024

done in e04d691
no need to actually follow plan links or limit ourselves, the field can be scraped by our scraper, just need to ask for it

from opentaba-server.