python采集500彩票网，获取所有双色球的开奖结果，写入csv文件！

1、分析网站：（加小编Python学习群：813542856即可自动获取大量python视频教程以及各类PDF！）

<pre class=”ql-align-justify” style=”-webkit-tap-highlight-color: transparent; box-sizing: border-box; font-family: Consolas, Menlo, Courier, monospace; font-size: 16px; white-space: pre-wrap; position: relative; line-height: 1.5; color: rgb(153, 153, 153); margin: 1em 0px; padding: 12px 10px; background: rgb(244, 245, 246); border: 1px solid rgb(232, 232, 232); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>1.1 目标网站：http://kaijiang.500.com/shtml/ssq/18131.shtml
</pre>

1.2谷歌浏览器打开该网站，右击检查查看分析网站。

我们需要爬取的数据信息，如下图

python采集500彩票网，获取所有双色球的开奖结果，写入csv文件！

(1) 分析标题

通过分析，发现标题信息在如下地方显示：

python采集500彩票网，获取所有双色球的开奖结果，写入csv文件！

(2) 分析开奖号码

通过分析，发现开奖号码在如下地方显示：

python采集500彩票网，获取所有双色球的开奖结果，写入csv文件！

(3) 分析往期开奖结果url

通过分析，发现网页是通过一个下拉菜单，选择相应的期号，会自动跳转对应的网站。

<pre class=”ql-align-justify” style=”-webkit-tap-highlight-color: transparent; box-sizing: border-box; font-family: Consolas, Menlo, Courier, monospace; font-size: 16px; white-space: pre-wrap; position: relative; line-height: 1.5; color: rgb(153, 153, 153); margin: 1em 0px; padding: 12px 10px; background: rgb(244, 245, 246); border: 1px solid rgb(232, 232, 232); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>网站规律就是http://kaijiang.500.com/shtml/ssq/+“期号”+.shtml
</pre>

python采集500彩票网，获取所有双色球的开奖结果，写入csv文件！

2、思路分析

先提取网页数据，用Beautiful Soup从网页抓取数据，然后通过正则表达式，提取想要的数据。

3、代码：

<pre style=”-webkit-tap-highlight-color: transparent; box-sizing: border-box; font-family: Consolas, Menlo, Courier, monospace; font-size: 16px; white-space: pre-wrap; position: relative; line-height: 1.5; color: rgb(153, 153, 153); margin: 1em 0px; padding: 12px 10px; background: rgb(244, 245, 246); border: 1px solid rgb(232, 232, 232); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;”>import requests
from bs4 import BeautifulSoup
import re

获取网页数据，伪装成浏览器

def gethtml(url):
headers = {
“Use-Agent”: “Mozilla/5.0 (Windows NT 6.1; Win64; x64) Apple
WebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36″
}
req = requests.get(url,headers = headers)
req.encoding = “GBK”
html = req.text
bf = BeautifulSoup(html, “html.parser”)
return bf

爬取标题

def gettitle(html):
titlehtml = html.find_all(“td”, class_=”td_title01″)
titletxt = str(titlehtml).strip()
p1 = r’shtml”>(.?)<f.?<strong>(.?)</strong>.?</font>(.?)</a>
.?right”>(.*?)</span>’
titles = re.compile(p1, re.S).findall(titletxt)
qi = list(titles[0])
qi[2] = (‘期’)
return ”.join(qi)

爬取红色球

def getred(html):
redhtml = html.find_all(“li”, class_=”ball_red”)
redtxt = str(redhtml).strip()
p1 = r’red”>(.*?)</li>’
reds = re.compile(p1, re.S).findall(redtxt)
return ‘红球:’ + ‘ ‘.join(reds)

爬取蓝色球

def getbule(html):
bulehtml = html.find_all(“li”, class_=”ball_blue”)
buletxt = str(bulehtml).strip()
p1 = r’blue”>(.*?)</li>’
bules = re.compile(p1, re.S).findall(buletxt)
return ‘蓝球:’ + ‘ ‘.join(bules)

获取所有url

def getlistnum(html):
listnumhtml = html.find_all(“span”, class_=”iSelectBox”)
p1 = r’href=”(.*?)”>’
listnums = re.compile(p1, re.S).findall(str(listnumhtml))
return listnums[1:]
url = ‘http://kaijiang.500.com/shtml/ssq/18131.shtml’
def main():
html = gethtml(url)
htmlurls = getlistnum(html)
for htmlurl in htmlurls:
ssqhtml = gethtml(htmlurl)
a = gettitle(ssqhtml)
b = getred(ssqhtml)
c = getbule(ssqhtml)

写入txt文件

with open(r’D:�.txt’,’a’) as f:
print(htmlurl)
f.write(a + ‘
‘+ b + ‘
‘+ c + ‘
‘)
f.close()
if name == “main“:
main()

</pre>

输出文件：

python采集500彩票网，获取所有双色球的开奖结果，写入csv文件！