The first three records returned from our above code. We use the replace method to get rid of it and replace it with empty space. segment (Each site is different after all). The formatting on the returned URLs is rather weird, as it is preceded by a. The next line adds the base url into the returned URL to complete it. Basically this XPath expression will only locate URLs within headings of size h3. Hence we created an XPath expression, '//h3/a' to avoid any non-books URLs. Url = self.base_url + url.replace('./.', '')Īll the URLs of the books were located within heading tags. The init method of LxmlLinkExtractor takes settings that determine which links may be extracted. You have learnt that you need to get all the elements on the first page, scrap them individually, and how to go to the next page to repeat this process.Rules = [Rule( LinkExtractor(allow = 'books_1/'), A link extractor is an object that extracts links from responses. Now you are able to extract every single element from a website. Next_page_partial_url = next_page_partial_url: Yield scrapy.Request(book_url, callback=self.parse_book) This is the final code: # -*- coding: utf-8 -*. Run the spider again: scrapy crawl spider -o next_page.json. Then, we add the base_url and we have our absolute URL. You can see the pattern: We get the partial URL, we check if /catalogue is missing and if it does, we add it. Next_page_url = self.base_url + next_page_partial_url Next_page_partial_url = "catalogue/" + next_page_partial_url If 'catalogue/' not in next_page_partial_url: If you couldn’t solve it, this is my solution: next_page_partial_url = next_page_partial_url: Why don’t you try? Again, you just need to check the link and prefix /catalogue in case that sub-string isn’t there. We have the same problem we had with the books: Some links have /catalogue, some others don’t.Īs we have the same problem, we have the same solution. Let’s go to the second page and see what’s going on with the next button and compare it with the first one (and its link to the second one) We didn’t get the third page from the second one. We managed to get the first 20, then the next 20. Let’s run the code again! It should work, right? scrapy crawl spider -o next_page.json You can check my code here: for book in all_books:īook_url = 'catalogue/' not in book_url: They didn’t add it to make you fail.Īs /catalogue is missing from some URLs, let’s have a check: If the routing doesn’t have it, let’s prefix it to the partial URL. There is a /catalogue missing on each routing. Compare the successful URLs (blue underline) with the failed ones (red underline). The is a website made by Scraping Hub to train people on web scraping, and they have little traps you need to notice. We managed to get the first 20 books, but then, suddenly, we can’t get more books… What’s going on? There is only 20 elements in the file! Let’s check the logging to see what’s going on. Run the code with scrapy crawl spider -o next_page.json and check the result. Yield scrapy.Request(next_page_url, callback=self.parse) Next_page_partial_url = self.base_url + next_page_partial_url This is how I did it: for book in all_books: As we did it before, you can do it yourself. Beware, it is a partial URL, so you need to add the base URL. You know how to extract it, so create a next_page_url we can navigate to. The next page URL is inside an a tag, within a li tag. Since this is currently working, we just need to check if there is a ‘Next’ button after the for loop is finished. Tax = table-striped"]/tr/td/text()').extract_first() Price_inc_tax = table-striped"]/tr/td/text()').extract_first() Price_excl_tax = table-striped"]/tr/td/text()').extract_first() Price = "instock")]/text()').extract().strip()ĭescription = table-striped"]/tr/td/text()').extract_first() Relative_image = self.base_url + relative_image.replace('./.', '') Title = response.xpath('//div/h1/text()').extract_first() Let’s start from the code we used in our second lesson, extract all the data: # -*- coding: utf-8 -*-īook_url = self.start_urls + scrapy.Request(book_url, callback=self.parse_book) And that’s what we are going to start using right now.Ĭhecking if there is a ‘next page’ available
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |