pyhton案例： 爬网页显示乱码

2024年1月27日 203次阅读 来源: 随风化作雨

最近想写一个爬天气的小爬虫，百度了一个天气API （json）的网址：
http://wthrcdn.etouch.cn/weather_mini?city=%E5%8C%97%E4%BA%AC

用火狐浏览器打开后显示乱码：

{"data":{"yesterday":{"date":"1鏃ユ槦鏈熷洓","high":"楂樻俯 31鈩�","fx":"鍗楅","low":"浣庢俯 17鈩�","fl":"3-4绾�","type":"鏅�"},"city":"鍖椾含","aqi":"60","forecast":[{"date":"2鏃ユ槦鏈熶簲","high":"楂樻俯 24鈩�","fengli":"3-4绾�","low":"浣庢俯 13鈩�","fengxiang":"涓滈","type":"闃甸洦"},{"date":"3鏃ユ槦鏈熷叚","high":"楂樻俯 28鈩�","fengli":"寰绾�","low":"浣庢俯 16鈩�","fengxiang":"鍗楅","type":"鏅�"},{"date":"4鏃ユ槦鏈熷ぉ","high":"楂樻俯 31鈩�","fengli":"寰绾�","low":"浣庢俯 19鈩�","fengxiang":"鍗楅","type":"澶氫簯"},{"date":"5鏃ユ槦鏈熶竴","high":"楂樻俯 27鈩�","fengli":"寰绾�","low":"浣庢俯 17鈩�","fengxiang":"鍗楅","type":"闃�"},{"date":"6鏃ユ槦鏈熶簩","high":"楂樻俯 25鈩�","fengli":"寰绾�","low":"浣庢俯 16鈩�","fengxiang":"鍗楅","type":"闃�"}],"ganmao":"鍚勯」姘旇薄鏉′欢閫傚疁锛屽彂鐢熸劅鍐掓満鐜囪緝浣庛€備絾璇烽伩鍏嶉暱鏈熷浜庣┖璋冩埧闂翠腑锛屼互闃叉劅鍐掋€�","wendu":"20"},"status":1000,"desc":"OK"}

调试改变网页编解码格式为：unicode，显示正常：

{"data":{"yesterday":{"date":"1日星期四","high":"高温 31℃","fx":"南风","low":"低温 17℃","fl":"3-4级","type":"晴"},"city":"北京","aqi":"60","forecast":[{"date":"2日星期五","high":"高温 24℃","fengli":"3-4级","low":"低温 13℃","fengxiang":"东风","type":"阵雨"},{"date":"3日星期六","high":"高温 28℃","fengli":"微风级","low":"低温 16℃","fengxiang":"南风","type":"晴"},{"date":"4日星期天","high":"高温 31℃","fengli":"微风级","low":"低温 19℃","fengxiang":"南风","type":"多云"},{"date":"5日星期一","high":"高温 27℃","fengli":"微风级","low":"低温 17℃","fengxiang":"南风","type":"阴"},{"date":"6日星期二","high":"高温 25℃","fengli":"微风级","low":"低温 16℃","fengxiang":"南风","type":"阴"}],"ganmao":"各项气象条件适宜，发生感冒机率较低。但请避免长期处于空调房间中，以防感冒。","wendu":"20"},"status":1000,"desc":"OK"}

原以为是编解码不对的问题，后调试了各种编解码，无济于事，后发现是网页压缩的问题。

#!/usr/bin/env python
# -*- coding: utf-8 -*

import re
import urllib2
import json
import sys
import zlib
from bs4 import BeautifulSoup

#encodings.aliases.aliases['ascii'] = 'windows-1252'
#reload(sys)
#sys.setdefaultencoding( "utf-8" )



def getHtml(url):
    user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
    headers = {'User-Agent':user_agent}
    req = urllib2.Request(url,headers=headers)
    print req.info()
    resp = urllib2.urlopen(req)
    html = resp.read()
    return html
html = getHtml('http://wthrcdn.etouch.cn/weather_mini?city=北京')

#html = html.encode('unicode')
html = zlib.decompress(html ,16+zlib.MAX_WBITS)
print html

只要引用zlib库，添加html = zlib.decompress(html ,16+zlib.MAX_WBITS)这行就可以了。

    原文作者：随风化作雨
    原文地址: https://www.jianshu.com/p/bf5306c0ab23
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞