rafikahmed / web-scraping-course Goto Github PK

Thanks a lot Rafik
Warm Regards,
Hiran

Hi Ahmed,
Thanks for the prompt reply for a previous query that I posted in your GitHub account.
Right now, I am facing another issue with dynamic pagination. I was trying to replicate the Airbnb.com example and wanted to do similar activity in holidayiq.com.
My objective is to scrape the review comments for one location in Holidayiq.com.

The url is : https://www.holidayiq.com/page/Rohtang-Pass-Manali-Sightseeing-451-8189-p2.html

When I try to open the fetch type in XHR to a new tab to get the json file it throws an error. I am unable to proceed. Could you please help me resolve this issue.

Thanks in advance,
Hiran

Hi Ahmed,

I was trying to scrape a website to extract data as part of learning . Th problem is that I am unable to extract the url's correctly.

I am pasting the code for your kind perusal.
=====> code begins
import scrapy

class GoodReadsSpider(scrapy.Spider):
#identity
name = "holidayiqmp"

#Request
start_urls= [
    'https://www.holidayiq.com/Manali-Sightseeing-451.html'
]

#Response
def parse(self, response):
    for holiday in response.selector.xpath("//div[@class='media-body']"):
                        
        yield {
            
            'Name': holiday.xpath(".//h2[@class='media-heading']/text()").extract_first(),
            'Type': holiday.xpath(".//h2[@class='media-heading']//span/text()").extract_first(),
            'url' : holiday.xpath("//div[@class='infinite-scroll-example__scrollable-parent wpscroll']/div/a/@href").extract(),
            
        }

<=======Code ends.

I an able to get Name and Type as expected. I am having problems with URL's.
Please help me resolve this.
Thanks,
Hiran