1. 把内存变成硬盘,把需要读的数据塞到里面去,加快了io。
Optimizing PyTorch training code
如何给你PyTorch里的Dataloader打鸡血
轻轻松松为你的Linux系统创建RAM Disk
Linux创建使用内存硬盘(RAM DISK)
A library containing both highly optimized building blocks and an execution engine for data pre-processing in deep learning applications
博客: Introducing GPU Instances: Using Deep Learning to Obtain Frontal Rendering of Facial Images
DALI 文档 : dali-pytorch
4. 将原始图像保存为pt或hdf5文件
hdf5: Saving and loading a large number of images (data) into a single HDF5 file (图片转换成HDF5文件(加载,保存))
pt : https://discuss.pytorch.org/t/how-to-speed-up-the-data-loader/13740/3
5. 将原始数据保存为lmdb格式
博客:Efficiently processing large image datasets in Python
6. Python简易实现并行操作
一行 Python 代码实现并行