如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?

<tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405598726″ data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image> <tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405606323″ data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image>

前沿:

加群:683380553 获取如何爬取日本老师的源码!!

这是一个命令行python程序,用于搜索Google Images上的关键字/关键短语,并可选择将图像下载到您的计算机。还可以从另一个python文件调用此脚本。

如果只想为每个关键字下载最多100个图像,则无需安装依赖项。如果您想要每个关键字超过100个图像,需要同时安装Selenium库chromedriver。故障排除部分中的详细说明。

Github地址https://github.com/hardikvasa/google-images-download

<tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405625197″ data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image>

当然,这么热门的项目,也不是一个人短时间完成的。Github上显示,此项目的贡献者就高达26人,代码总共提交了133次。此项目能收到这么多star,还是在于更多的贡献者和后期的不断维护升级。

这个项目在github也做了具体的使用说明和介绍,大家可以根据自身的项目情况进行配置即可(为了方便阅读,通过google翻译为中文)。

<tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405632698 ql-align-center” data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image>

关于此项目使用这里多逼逼几点:

  • 版本要求

  • 该项目作者GitHub上说python2x与Python3x都可以,推荐Python3。

  • 项目使用:有2种使用方式

    方式一:使用pip安装报的方式(推荐)

<pre spellcheck=”false” style=”box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>pip install google_images_download
</pre>

    方式二:手动下载

<pre spellcheck=”false” style=”box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>git clone https://github.com/hardikvasa/google-images-download.git
cd google-images-download && sudo python setup.py install
</pre>

  • 图片数量:
  • google搜索关键字首页默认100张图片,要抓取更多的图片,就得增加翻页功能。此项目已经实现了翻页功能,只需要使用者同时安装Selenium库chromedriver,代码里面配置即可。
  • 参数方式:
  • 原作者对参数的传递形式和每一个参数都做了详细的说明,大家可以github详细了解。

使用案例

这里给大家分享一下如何将此项目用来爬取去自己需要的图片。原作者介绍了几种关键字输入的方式。这里以文件读取的形式将关键字传入。这里爬取图片的数量选为500张,需要下载、安装Selenium库chromedriver。

1、搭建爬虫环境

<pre spellcheck=”false” style=”box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>pip install selenuium
pip install requests
pip install google_images_download
</pre>

2、下载chromedriver驱动

因为下载的图片数量大于100,所以还需要在安装chromedriver,在代码配置chromedriver位置即可。

首先要查看自己电脑上安装的google浏览器版本号

<tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405703015 ql-align-center” data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image>

▲谷歌浏览器版本号▲

github给的下载路径国内是无法下载的,不过小编已经将大多数驱动下载下来,需要的可以后台私信获取。当然也可以通过别的渠道获取。

<tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405703018 ql-align-center” data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image>

▲github给的下载地址▲

在chrome官网找到与本机的google浏览器版本号一致的chromedriver

<tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405703019 ql-align-center” data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image>

▲chrome官网对应的驱动版本号▲

下载完后,安装的路径根据操作系统自己指定。我使用的时win系统,安装在了D盘,具体路径如下:

<pre spellcheck=”false” style=”box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>”D:downloadchromedriver.exe”
</pre>

3、编辑爬取的关键字文件:

如图所示,只需要在keywords.csv文件里面按图所示填写对应老师的名称即可。

<tt-image data-tteditor-tag=”tteditorTag” contenteditable=”false” class=”syl1553405703022 ql-align-center” data-render-status=”finished” data-syl-blot=”image” style=”box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: “PingFang SC”, “Hiragino Sans GB”, “Microsoft YaHei”, “WenQuanYi Micro Hei”, “Helvetica Neue”, Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;”>
《如何利用Python爬取当年在硬盘存过的日本老师!你认识哪些呢?》 image

<input class=”pgc-img-caption-ipt” placeholder=”图片描述(最多50字)” value=”” style=”box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1);”></tt-image>

4、代码展示:

<pre spellcheck=”false” style=”box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>import csv
import os
import sys
from google_images_download import google_images_download

实例化一个下载器

downloader = google_images_download.googleimagesdownload()
BASE_DIR = os.path.dirname(os.path.abspath(file))
sys.path.insert(0, BASE_DIR)

读取关键字文件

csv_file = csv.reader(open(BASE_DIR + “google_imageskeywords.csv”, “r”))
def download_images(csv_file):
“””
传入关键字等参数,下载对应的图片文件
files: 读取的关键字文件
limit: 爬取的图片数量
print_urls: 是否显示爬取的图片url
chromedriver: chromedriver安装的路径。不填此参数,默认爬取前100张图片
output_directory:自定义保存图片的位置
“””
for key_word in csv_file:
arguments = {“keywords”: key_word, “limit”: 500, “print_urls”: True,
“chromedriver”: “D:downloadchromedriver.exe”,
“output_directory”: BASE_DIR + “\files”}
downloader.download(arguments)
if name == main :
download_images(csv_file)
</pre>

运行:python google_download.py,你就会发现有源源不断的老师图片进入你的硬盘,接下来就是坐等爬完所有你要的图片了。

在实际中,我们可以利用这个python脚本去爬取基于google的任何图片了。

    原文作者:不谈风月_0eb8
    原文地址: https://www.jianshu.com/p/325d2c706370
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系博主进行删除。
点赞