我在包含我的
Python(2.7)类的目录中的VM上.我试图将我的类的实例pickle到我的HDFS中的目录.
我正在努力做一些事情:
import pickle
my_obj = MyClass() # the class instance that I want to pickle
with open('hdfs://domain.example.com/path/to/directory/') as hdfs_loc:
pickle.dump(my_obj, hdfs_loc)
从我做过的研究中,我认为像snakebite这样的东西可能会有所帮助……但是有没有人有更具体的建议呢?
最佳答案 如果您在具有足够权限的Jupyter笔记本中运行,这是一个解决方法:
import pickle
my_obj = MyClass() # the class instance that I want to pickle
local_filename = "pickle.p"
hdfs_loc = "//domain.example.com/path/to/directory/"
with open(local_filename, 'wb') as f:
pickle.dump(my_obj, f)
!!hdfs dfs -copyFromLocal $local_filename $hdfs_loc