一、原生
1、模块
from scrapy.dupefilters import RFPDupeFilter
2、RFPDupeFilter方法
a、request_seen
核心:爬虫每执行一次yield Request对象,则执行一次request_seen方法
作用:用来去重,相同的url只能访问一次
实现:将url值变成定长、唯一的值,如果这个url对象存在,则返回True表名已经访问过,若url不存在则添加该url到集合
1)、request_fingerprint
作用:对request(url)变成定长唯一的值,如果使用md5的话,下面的两个url值不一样
注意:request_fingerprint() 只接收request对象
from scrapy.utils.request import request_fingerprint from scrapy.http import Request # url1 = 'https://test.com/?a=1&b=2' url2 = 'https://test.com/?b=2&a=1' request1 = Request(url=url1) request2 = Request(url=url2) # 只接收request对象 rfp1 = request_fingerprint(request=request1) rfp2 = request_fingerprint(request=request2) print(rfp1) print(rfp2) if rfp1 == rfp2: print('url相同') else: print('url不同')
2)、request_seen
def request_seen(self, request): # request_fingerprint 将request(url) -> 唯一、定长 fp = self.request_fingerprint(request) if fp in self.fingerprints: return True # 返回True,表明已经执行过一次 self.fingerprints.add(fp)
b、open
父类BaseDupeFilter中的方法,爬虫开始时,执行
def open(self): # 爬虫开始 pass
c、close
爬虫结束时执行
def close(self, reason): # 关闭爬虫时执行 pass
d、log
记录日志
def log(self, request, spider): # 记录日志 pass
e、from_settings
原理及作用:和pipelines中的from_crawler一致
@classmethod def from_settings(cls, settings): return cls()
二、自定义
待续
1、配置文件(settings.py)
# 原生 # DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter' DUPEFILTER_CLASS = 'toscrapy.dupefilters.MyDupeFilter'
2、自定义去重类(继承BaseDupeFilter)
from scrapy.dupefilters import BaseDupeFilter from scrapy.utils.request import request_fingerprint # class MyDupeFilter(BaseDupeFilter): def __init__(self): self.visited_fp = set() @classmethod def from_settings(cls, settings): return cls() def request_seen(self, request): # 判断当前的request对象是否,在集合中,若在则放回True,表明已经访问,否则,访问该request的url并将该url添加到集合中 if request_fingerprint(request) in self.visited_fp: return True self.visited_fp.add(request_fingerprint(request)) def open(self): # can return deferred print('开启爬虫') def close(self, reason): # can return a deferred print('结束爬虫') def log(self, request, spider): # log that a request has been filtered pass
3、前提条件
yield request的对象
yield scrapy.Request(url=_next, callback=self.parse, dont_filter=True)
dont_filter不能为True,这个值默认为False