如何Python下载大文件？

2023年11月23日 266次阅读

我想用python脚本下载很多文件，但是经常就有那么几个出错，写了个error handling，跳了过去，但是把出错的链接保存了一下。

转过天来，研究了一下出的什么错。

一个报错如下:

PS C:\temp> python .\DownloadFromList.py
Downloading https://github.com/Unity-Technologies/ScriptableRenderPipeline/archive/master.zip
Traceback (most recent call last):
   File “.\DownloadFromList.py”, line 20, in <module>
     r = requests.get(url)
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py”, line 72, in get
     return request(‘get’, url, params=params, **kwargs)
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py”, line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py”, line 512, in request
     resp = self.send(prep, **send_kwargs)
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py”, line 644, in send
     history = [resp for resp in gen] if allow_redirects else []
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py”, line 644, in <listcomp>
     history = [resp for resp in gen] if allow_redirects else []
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py”, line 222, in resolve_redirects
     **adapter_kwargs
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py”, line 662, in send
     r.content
   File “C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\models.py”, line 827, in content
     self._content = b”.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b”
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File “.\DownloadFromList.py”, line 28, in <module>
     print(“Error happened:”, e.message)
AttributeError: ‘MemoryError’ object has no attribute ‘message’
PS C:\temp>

上网搜索了一下, 找到了解决方案.

为了防止这个参考资料的网页消失(以前经常发生的), 所以我就直接把代码抄过来放在这里, 备用（抄袭，嗯，注明了出处就可以光明正大的抄袭）.

使用request

def download_file(url):
local_filename = url.split('/')[-1]
# NOTE the stream=True parameter
r = requests.get(url, stream=True)
with open(local_filename, 'wb') as f:
    for chunk in r.iter_content(chunk_size=1024): 
        if chunk: # filter out keep-alive new chunks
            f.write(chunk)
            f.flush()
return local_filename

使用urllib2

file = urllib2.urlopen('url')
with open('filename','w') as f:
    while True:
        tmp = file.read(1024)
        if not tmp:
            break 
        f.write(tmp)

参考资料

==================

https://ox0spy.github.io/post/python/python-download-large-file-without-out-of-memory/

参考资料所援引的代码来自下面的两个链接。

http://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py

http://stackoverflow.com/questions/27053028/how-to-download-large-file-without-memoryerror-in-python

点赞