我在
this dataset of Jeopardy questions上运行频谱聚类,这是我面临的数据令人沮丧的问题.请注意,我只是在“问题”列中聚集所有值.
当我在数据集上运行双聚类时,显然存在“除以零”的ValueError.
/usr/local/lib/python3.6/dist-packages/sklearn/cluster/bicluster.py:38: RuntimeWarning: divide by zero encountered in true_divide
row_diag = np.asarray(1.0 / np.sqrt(X.sum(axis=1))).squeeze()
/usr/local/lib/python3.6/dist-packages/sklearn/cluster/bicluster.py:286: RuntimeWarning: invalid value encountered in multiply
z = np.vstack((row_diag[:, np.newaxis] * u,
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
...
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
该错误显然表明我的数据中潜伏着NaN或无限值(这只是问题的单一列).这是我正在处理的完全文本数据,我已经尝试过大多数NumPy和Pandas函数来过滤NaN和inf,以及Stack Overflow上的许多解决方案.我找不到任何东西.
为了确保我的代码没有错误,同样的事情完全适用于二十个新闻组数据集.
Here’s the code on Kaggle if you want to run it and see for yourself.但是,如果SO的政策禁止这样做,这里的代码简而言之:
dat = pd.DataFrame(pd.read_csv('../input/jarchive_cleaned.csv'))
qlist = []
def cleanhtml(raw_html):
cleanr = re.compile('<.*?>')
cleantext = re.sub(cleanr, '', raw_html)
return cleantext
for row in dat.iterrows():
txt = row[1]['text'].lower()
txt = cleanhtml(txt)
txt = re.sub(r'[^a-z ]',"",txt)
txt = re.sub(r' ',' ',txt)
# txt = ' '.join([stem(w) for w in txt.split(" ")])
qlist.append([txt,row[1]['answer'],row[1]['category']])
print(qlist[:10])
swords = set(stopwords.words('english'))
tv = TfidfVectorizer(stop_words = swords , strip_accents='ascii')
queslst = [q for (q,a,c) in qlist]
qlen = len(set([c for (q,a,c) in qlist]))
mtx = tv.fit_transform(queslst)
cocluster = SpectralCoclustering(n_clusters=qlen, svd_method='arpack', random_state=0) #
t = time()
cocluster.fit(mtx)
最佳答案 一些字符串序列如’down out’导致TfidfVectorizer()返回零值.这会导致错误从零除错误开始,这会导致mtx稀疏矩阵中的inf值,这会导致第二个错误.
作为解决此问题的一个解决方法是在由TfidfVectorizer.fit_transform()创建后删除此序列或从mtx矩阵中删除零矩阵元素,由于稀疏矩阵运算,这有点棘手.
我做了第二个解决方案,因为我没有潜入原始任务,如下:
swords = set(stopwords.words('english'))
tv = TfidfVectorizer(stop_words = swords , strip_accents='ascii')
queslst = [q for (q,a,c) in qlist]
qlen = len(set([c for (q,a,c) in qlist]))
mtx = tv.fit_transform(queslst)
indices = []
for i,mx in enumerate(mtx):
if np.sum(mx, axis=1) == 0:
indices.append(i)
mask = np.ones(mtx.shape[0], dtype=bool)
mask[indices] = False
mtx = mtx[mask]
cocluster = SpectralCoclustering(n_clusters=qlen, svd_method='arpack', random_state=0) #
t = time()
cocluster.fit(mtx)
最后它有效.我希望,它有所帮助,祝你好运!