[No.003-0]爬虫网易赔率数据并导入到mysql数据库

    首先,我拥有这个导入赔率的需求,并且,我需要的是所有的数据赔率,我需要把某些脏赔率(极有可能不会出现的赔率,误导彩迷的赔率)删除,并且我需要知道这些赔率的比分,删除这些赔率,可以逆推正确比分以及赔率的区间。
    所以我不得不做的就是把每天的赔率数据导入到我自己的数据库,根据一定的运算法则,将可能性筛选出来,然后我再做进一步的判断。
#encoding:utf-8
import urllib2
from bs4 import BeautifulSoup

website = "http://caipiao.163.com/order/jczq-hunhe/#from=leftnav"
page = urllib2.urlopen(website)
soup = BeautifulSoup(page)
for incident in soup('td'):
    print incident
得到类似于以下的结果集:
<td gametype="bf" index="30" sp="120.00">负其他

<div>120.00</div>

</td>
<td gametype="zjq" index="0" sp="14.00">0

<div>14.00</div>

</td>
<td gametype="zjq" index="1" sp="5.20">1

<div>5.20</div>

</td>
<td gametype="zjq" index="2" sp="3.55">2

<div>3.55</div>

</td>
<td gametype="zjq" index="3" sp="3.50">3

<div>3.50</div>

</td>
<td gametype="zjq" index="4" sp="4.70">4

<div>4.70</div>

</td>
<td gametype="zjq" index="5" sp="7.50">5

<div>7.50</div>

</td>
<td gametype="zjq" index="6" sp="13.00">6

<div>13.00</div>

</td>
<td gametype="zjq" index="7" sp="18.00">7+

<div>18.00</div>

</td>
……

这里得到的结果,仅仅是赔率的结果,而且需要吧gametype的内容筛选出来,得到紧缺的,总进球 7球 18赔率的结果;

接下来

提取td中的内容,使用re正则来提取;
直接在for循环中使用re,避免使用文件作为缓存

#查询半全场的赔率
for item in soup.findAll("td",{"gametype":"bqc"}):
    print item.find("div").string
#查询表中的标签,并将内容筛选出来
#半全场赔率,依次为"胜胜","胜平","胜负","平胜","平平","平负","负胜","负平","负负"
for item in soup.findAll("td",{"gametype":"bqc"}):
     print item.find("div").string

#再查询比分赔率
#先是胜赔,1:0~胜其他,之后是平赔,0:0~3:3,平其他,并将其导入比分赔率bfpl
temp = ["1:0","2:0","2:1","3:0","3:1","3:2","4:0","4:1","4:2","5:0","5:1","5:2","胜其他","0:0","1:1","2:2","3:3","平其他","0:1","0:2","1:2","0:3","1:3","2:3","0:4","1:4","2:4","0:5","1:5","2:5","负其他"]
i = 1
bfpl = []
for item in soup.findAll("td",{"gametype":"bf"}):
    bfpl.append(item.find("div").string)

#---------------------
#构建比分赔率字典
i = 1
temp = ["1:0","2:0","2:1","3:0","3:1","3:2","4:0","4:1","4:2","5:0","5:1","5:2","胜其他","0:0","1:1","2:2","3:3","平其他","0:1","0:2","1:2","0:3","1:3","2:3","0:4","1:4","2:4","0:5","1:5","2:5","负其他"]
len



#再查询总进球赔率
for item in soup.findAll("td",{"gametype":"zjq"}):
     print item.find("div").string

#----------------------------------------
#查询所有的主队、客队名字数据以及场次数据
#主队hostTeam
i = 1
hostTeam = []
for item in soup.findAll("em",{"class":"hostTeam"}):
    hostTeam.append(item.b.string)
    i+=1

for item in hostTeam:
    print hostTeam[item]

#客队guestTeam
i = 1
guestTeam = []
for item in soup.findAll("em",{"class":"guestTeam"}):
    guestTeam.append(item.b.string)
    i+=1

for item in guestTeam:
    print guestTeam[item]

#------------------
#场次以及主队客队数据
#------------------
i = 1
for item in hostTeam:
    print '---------'
    print screening[i],hostTeam[i],guestTeam[i]
    i+=1

#-----------------------
#场次信息 jtip
i = 1
screening = []
for item in soup.findAll("span",{"class":"co1"}):
    screening.append(item.i.string)
    i+=1

#遍历场次数据
i=1
for item in screening:
    print screening[i]
    i+=1

#------------------
#做出场次+比分的list-->scbf[]
for item in screening:
    i=0
    while i<len(temp):
        scbf[i]=screening[i]+temp[i]
        i+=1


#=====================

    原文作者:Profeel
    原文地址: https://segmentfault.com/a/1190000000648573
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系博主进行删除。
点赞