python爬虫系列(2.2-requests库的高级使用)

一、设置代理ip

1、直接在请求的时候加上proxies就可以,注意我们一般会写上http和https的,这样当遇到http请求就会走http字典对应的代理

2、具体代码

import requests

if __name__ == “__main__”:

# 定义一个请求头(模拟浏览器)

headers = {

‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36’}

# 定义代理的字典(可以去http://www.xicidaili.com查找最新的代理)

proxies = {

‘http’: ‘http://115.223.223.29:9000‘,

‘https’: ‘https://197.232.21.141:59075

}

response = requests.get(‘http://httpbin.org/get‘, headers=headers, proxies=proxies)

response.encoding = ‘utf-8’

print(response.text)

二、关于requests库操作cookie

1、从网站上获取cookie

import requests

if __name__ == “__main__”:

# 定义一个请求头(模拟浏览器)

headers = {

‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36’}

response = requests.get(‘https://www.baidu.com‘, headers=headers)

# 迭代出全部的

for k, v in response.cookies.items():

print(k, ‘===’, v)

print(response.cookies)

print(response.cookies.get_dict())

2、使用session会话存储当前网上的cookies

import requests

if __name__ == “__main__”:

url = ‘https://www.lagou.com/jobs/positionAjax.json?needAddtionalResult=false

headers = {

‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36’,

‘Referer’: ‘https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=

}

data = {

‘first’: ‘true’,

‘pn’: ‘1’,

‘kd’: ‘python’,

}

# 使用session包装下

session = requests.session()

response = session.post(url=url, headers=headers, data=data)

print(response.json())

三、不安全证书网站的请求(12306是最典型的)

1、只需要在请求的时候加上verify=False就可以

2、具体代码

import requests

if __name__ == “__main__”:

# 定义一个请求头(模拟浏览器)

headers = {

‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36’}

response = requests.get(‘http://www.12306.cn/mormhweb/‘, headers=headers, verify=False)

response.encoding = ‘utf-8’

print(response.text)

    原文作者:七月
    原文地址: https://zhuanlan.zhihu.com/p/48955023
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系博主进行删除。
点赞