[scrapy] 2024-04-18 圈点122
摘要:scrapy监听各事件的勾子设置方法的示例:示例为监听当scrapy停止后将执行spider_closed函数。
scrapy监听各事件的勾子设置方法的示例:
示例为监听当scrapy停止后将执行spider_closed函数。
from scrapy import signals from scrapy import Spider class DmozSpider(Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = ['http://www.xoxxoo.com',] @classmethod def from_crawler(cls, crawler, *args, **kwargs): spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs) crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed) return spider def spider_closed(self, spider): spider.logger.info('Spider closed: %s', spider.name) def parse(self, response): pass
备注:
在from_crawler中必须指明“类名”,监控的事件,比如示例为:spider_closed