16 Scrapy爬取二级目录

import scrapy
from ..items import SecondaryZufangItem

class ZufangSpider(scrapy.Spider):
    name = 'zufang'
    start_urls = ['http://bj.ganji.com/wblist/haidian/zufang/']
    #host_name = 'http://bj.ganji.com{}'

    def parse(self, response):
        zf = SecondaryZufangItem()
        next_link = response.xpath('//div[@class="pageBox"]/a[@class="next"]/@href').extract()[0]
        print(next_link,'*****'*20)
        yield scrapy.Request(next_link, callback = self.parse)

返回结果

SobersMacBookPro2016:secondary_zufang miraco$ scrapy crawl zufang
2018-12-20 14:58:24 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: secondary_zufang)
2018-12-20 14:58:24 [scrapy.utils.log] INFO: Versions: lxml 4.2.4.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i  14 Aug 2018), cryptography 2.3.1, Platform Darwin-17.7.0-x86_64-i386-64bit
2018-12-20 14:58:24 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'secondary_zufang', 'NEWSPIDER_MODULE': 'secondary_zufang.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['secondary_zufang.spiders']}
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled item pipelines:
['secondary_zufang.pipelines.SecondaryZufangPipeline']
2018-12-20 14:58:24 [scrapy.core.engine] INFO: Spider opened
2018-12-20 14:58:24 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-20 14:58:24 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-12-20 14:58:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/robots.txt> (referer: None)
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/wblist/haidian/zufang/> (referer: None)
http://bj.ganji.com/haidian/zufang/pn2/ ****************************************************************************************************
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn2/> (referer: http://bj.ganji.com/wblist/haidian/zufang/)
http://bj.ganji.com/haidian/zufang/pn3/ ****************************************************************************************************
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn3/> (referer: http://bj.ganji.com/haidian/zufang/pn2/)
http://bj.ganji.com/haidian/zufang/pn4/ ****************************************************************************************************
2018-12-20 14:58:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn4/> (referer: http://bj.ganji.com/haidian/zufang/pn3/)
http://bj.ganji.com/haidian/zufang/pn5/ ****************************************************************************************************
2018-12-20 14:58:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn5/> (referer: http://bj.ganji.com/haidian/zufang/pn4/)
http://bj.ganji.com/haidian/zufang/pn6/ ****************************************************************************************************
2018-12-20 14:58:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn6/> (referer: http://bj.ganji.com/haidian/zufang/pn5/)
http://bj.ganji.com/haidian/zufang/pn7/ ****************************************************************************************************
2018-12-20 14:58:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn7/> (referer: http://bj.ganji.com/haidian/zufang/pn6/)
http://bj.ganji.com/haidian/zufang/pn8/ ****************************************************************************************************
2018-12-20 14:58:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn8/> (referer: http://bj.ganji.com/haidian/zufang/pn7/)
http://bj.ganji.com/haidian/zufang/pn9/ ****************************************************************************************************
2018-12-20 14:58:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn9/> (referer: http://bj.ganji.com/haidian/zufang/pn8/)
http://bj.ganji.com/haidian/zufang/pn10/ ****************************************************************************************************
2018-12-20 14:58:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn10/> (referer: http://bj.ganji.com/haidian/zufang/pn9/)
http://bj.ganji.com/haidian/zufang/pn11/ ****************************************************************************************************
2018-12-20 14:58:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn11/> (referer: http://bj.ganji.com/haidian/zufang/pn10/)
http://bj.ganji.com/haidian/zufang/pn12/ ****************************************************************************************************
2018-12-20 14:58:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn12/> (referer: http://bj.ganji.com/haidian/zufang/pn11/)
http://bj.ganji.com/haidian/zufang/pn13/ ****************************************************************************************************
2018-12-20 14:58:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn13/> (referer: http://bj.ganji.com/haidian/zufang/pn12/)
http://bj.ganji.com/haidian/zufang/pn14/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn14/> (referer: http://bj.ganji.com/haidian/zufang/pn13/)
http://bj.ganji.com/haidian/zufang/pn15/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn15/> (referer: http://bj.ganji.com/haidian/zufang/pn14/)
http://bj.ganji.com/haidian/zufang/pn16/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn16/> (referer: http://bj.ganji.com/haidian/zufang/pn15/)
http://bj.ganji.com/haidian/zufang/pn17/ ****************************************************************************************************
2018-12-20 14:58:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn17/> (referer: http://bj.ganji.com/haidian/zufang/pn16/)
http://bj.ganji.com/haidian/zufang/pn18/ ****************************************************************************************************
2018-12-20 14:58:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn18/> (referer: http://bj.ganji.com/haidian/zufang/pn17/)
http://bj.ganji.com/haidian/zufang/pn19/ ****************************************************************************************************
2018-12-20 14:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn19/> (referer: http://bj.ganji.com/haidian/zufang/pn18/)
http://bj.ganji.com/haidian/zufang/pn20/ ****************************************************************************************************
2018-12-20 14:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn20/> (referer: http://bj.ganji.com/haidian/zufang/pn19/)
http://bj.ganji.com/haidian/zufang/pn21/ ****************************************************************************************************
2018-12-20 14:58:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn21/> (referer: http://bj.ganji.com/haidian/zufang/pn20/)
http://bj.ganji.com/haidian/zufang/pn22/ ****************************************************************************************************
2018-12-20 14:58:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn22/> (referer: http://bj.ganji.com/haidian/zufang/pn21/)
http://bj.ganji.com/haidian/zufang/pn23/ ****************************************************************************************************
2018-12-20 14:58:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn23/> (referer: http://bj.ganji.com/haidian/zufang/pn22/)
http://bj.ganji.com/haidian/zufang/pn24/ ****************************************************************************************************
2018-12-20 14:58:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn24/> (referer: http://bj.ganji.com/haidian/zufang/pn23/)
http://bj.ganji.com/haidian/zufang/pn25/ ****************************************************************************************************
2018-12-20 14:58:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn25/> (referer: http://bj.ganji.com/haidian/zufang/pn24/)
http://bj.ganji.com/haidian/zufang/pn26/ ****************************************************************************************************
2018-12-20 14:58:36 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://bj.ganji.com/haidian/zufang/pn26/> (referer: http://bj.ganji.com/haidian/zufang/pn25/)
2018-12-20 14:58:36 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://bj.ganji.com/haidian/zufang/pn26/>: HTTP status code is not handled or not allowed
2018-12-20 14:58:36 [scrapy.core.engine] INFO: Closing spider (finished)

是吧,要将获取的url地址丢回调度器:

yield scrapy.Request(next_link[0], callback = self.parse)

这里不能写return,因为还要接着爬呢。

import scrapy
from ..items import SecondaryZufangItem

class ZufangSpider(scrapy.Spider):
    name = 'zufang'
    start_urls = ['http://bj.ganji.com/wblist/haidian/zufang/']
    #host_name = 'http://bj.ganji.com{}'

    def parse(self, response):
        zf = SecondaryZufangItem()
        title_list = response.xpath('.//div[@class="f-list-item ershoufang-list"]//dd[@class="dd-item title"]/a/text()').extract()
        money_list = response.xpath('.//span[@class="num"]/text()').extract()

        for i,j in zip(title_list,money_list):
            zf['title']=i
            zf['money']=j
            yield zf

        next_link = response.xpath('//div[@class="pageBox"]/a[@class="next"]/@href').extract()


        if len(next_link) > 0:
            print(next_link,'*****'*20)
            yield scrapy.Request(next_link[0], callback = self.parse)
        else:
            pass #自行添加,一般是处理验证页面啥的

返回的挺多呢

《16 Scrapy爬取二级目录》 image.png

《16 Scrapy爬取二级目录》

    原文作者:夏威夷的芒果
    原文地址: https://www.jianshu.com/p/114170e65bdb
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系博主进行删除。
点赞