快捷导航

Scrapy:Python3下的第一次运行测试

2016-9-27 17:33| 发布者: xandy| 查看: 2299| 评论: 2|原作者: 集搜客网络爬虫|来自: 集搜客社区

摘要: 《Scrapy的架构初探》一文讲解了Scrapy的架构,本文就实际来安装运行一下Scrapy爬虫。本文以官网的tutorial作为例子,完整的代码可以在github上下载。

1,引言

Scrapy的架构初探》一文讲解了Scrapy的架构,本文就实际来安装运行一下Scrapy爬虫。本文以官网的tutorial作为例子,完整的代码可以在github上下载。

2,运行环境配置

  • 本次测试的环境是:Windows10, Python3.4.3 32bit
  • 安装Scrapy : $ pip install Scrapy #实际安装时,由于服务器状态的不稳定,出现好几次中途退出的情况

3,编写运行第一个Scrapy爬虫

3.1. 生成一个新项目:tutorial

$ scrapy startproject tutorial

项目目录结构如下:

3.2. 定义要抓取的item

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy

class DmozItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()

3.3. 定义Spider

import scrapy
from tutorial.items import DmozItem

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            item = DmozItem()
            item['title'] = sel.xpath('a/text()').extract()
            item['link'] = sel.xpath('a/@href').extract()
            item['desc'] = sel.xpath('text()').extract()
            yield item

3.4. 运行

$ scrapy crawl dmoz -o item.json

1) 结果报错:

  • A) ImportError: cannot import name '_win32stdio'
  • B) ImportError: No module named 'win32api'

2) 查错过程:

查看官方的FAQstackoverflow上的信息,原来是scrapy在python3上测试还不充分,还有小问题

3) 解决过程:

  • A) 需要手工去下载twisted/internet下的 _win32stdio 和 _pollingfile,存放到python目录的lib\sitepackages\twisted\internet下
  • B) 下载并安装pywin32

再次运行,成功!在控制台上可以看到scrapy的输出信息,待运行完成退出后,到项目目录打开结果文件items.json, 可以看到里面以json格式存储的爬取结果。

[
{"title": ["        About       "], "desc": [" ", " "], "link": ["/docs/en/about.html"]},
{"title": ["   Become an Editor "], "desc": [" ", " "], "link": ["/docs/en/help/become.html"]},
{"title": ["            Suggest a Site          "], "desc": [" ", " "], "link": ["/docs/en/add.html"]},
{"title": [" Help             "], "desc": [" ", " "], "link": ["/docs/en/help/helpmain.html"]},
{"title": [" Login                       "], "desc": [" ", " "], "link": ["/editors/"]},
{"title": [], "desc": [" ", " Share via Facebook "], "link": []},
{"title": [], "desc": [" ", "  Share via Twitter  "], "link": []},
{"title": [], "desc": [" ", " Share via LinkedIn "], "link": []},
{"title": [], "desc": [" ", " Share via e-Mail   "], "link": []},
{"title": [], "desc": [" ", " "], "link": []},
{"title": [], "desc": [" ", "  "], "link": []},
{"title": ["        About       "], "desc": [" ", " "], "link": ["/docs/en/about.html"]},
{"title": ["   Become an Editor "], "desc": [" ", " "], "link": ["/docs/en/help/become.html"]},
{"title": ["            Suggest a Site          "], "desc": [" ", " "], "link": ["/docs/en/add.html"]},
{"title": [" Help             "], "desc": [" ", " "], "link": ["/docs/en/help/helpmain.html"]},
{"title": [" Login                       "], "desc": [" ", " "], "link": ["/editors/"]},
{"title": [], "desc": [" ", " Share via Facebook "], "link": []},
{"title": [], "desc": [" ", "  Share via Twitter  "], "link": []},
{"title": [], "desc": [" ", " Share via LinkedIn "], "link": []},
{"title": [], "desc": [" ", " Share via e-Mail   "], "link": []},
{"title": [], "desc": [" ", " "], "link": []},
{"title": [], "desc": [" ", "  "], "link": []}
]

第一次运行scrapy的测试成功。

4,接下来的工作

接下来,我将使用GooSeeker API来实现网络爬虫,省掉对每个item人工去生成和测试xpath的工作量。目前有2个计划:

  1. 在gsExtractor中封装一个方法:从xslt内容中自动提取每个item的xpath。
  2. 从gsExtractor的提取结果中自动提取每个item的结果。

具体选择哪个方案,将在接下来的实验中确定,并发布到gsExtractor新版本中。

5,文档修改历史

2016-06-11:V1.0,首次发布

若有疑问可以集搜客网络爬虫

鲜花

握手

雷人

路过

鸡蛋

相关阅读

发表评论

最新评论

评论 chenxj888 2018-5-8 18:23
Williamslife: 大家能不能不要互相抄,<a href="http://www.gooseeker.com/doc/thread-1901-1-1.html" target="_blank">http://www.gooseeker.com/doc/thread-1901-1-1.html</a>,我一条一条搜,打开看半天,内容都一样,截图都一样,意义是什么呢? ...
这篇文章发表的时候,在python3下安装scrapy还很不成熟,会碰到很多坑。这篇文章就是记录当时的过程
评论 Williamslife 2018-5-8 17:57
大家能不能不要互相抄,http://www.gooseeker.com/doc/thread-1901-1-1.html,我一条一条搜,打开看半天,内容都一样,截图都一样,意义是什么呢?

查看全部评论(2)

GMT+8, 2019-10-15 18:49