我有一个用例,我想重复快速过滤数据表.我有一种感觉,如果我使用像df [val> 0,]这样的简单过滤器,扫描是O(N),但如果我只是将val字段设置为键,那么我应该能够更快地过滤,但不是确定如何.作为一个例子,我做了以下几点
排序的data.table
df_sort = data.table(id = seq(1, 100000), val = runif(100000, 0, 1))
setkeyv(df_sort, c('val')
一个常规的data.table
df = data.table(id = seq(1, 100000), val = runif(100000, 0, 1))
我想要做的是找到val>的唯一id.托尔.使用rbenchmark我做了以下
benchmark('raw' = { l = length(unique(df$id[df$val >=0.1]))}, 'sort' = {l = length(unique(df_sort$id[df_sort$val >=0.1]))}, replications=20)
我得到的结果显示两者没有区别.
test replications elapsed relative user.self sys.self user.child sys.child
1 raw 20 0.440 1.005 0.494 0 0 0
2 sort 20 0.438 1.000 0.540 0 0 0
有人可以就如何做到这一点给出一些指示吗?它甚至可能吗?
编辑
在评论中尝试了一个建议,回来更糟.不知道为什么.
benchmark('raw' = { l = df[val >=0.1, uniqueN(id)]}, 'sort' = {l = df_sort[val >=0.1, uniqueN(id)]}, replications=200)
test replications elapsed relative user.self sys.self user.child sys.child
1 raw 200 0.770 1.000 0.928 0 0 0
2 sort 200 1.361 1.768 1.691 0 0 0
最佳答案 好的,这是一些适当的基准.两个基准测试中的时间单位可能不一样.
library(rbenchmark)
library(data.table)
set.seed(42)
#using sample to rule out ALTREP influence
df <- data.table(id = sample(1e7, replace = TRUE), val = runif(1e7, 0, 1))
df_sort <- copy(df)
setkey(df_sort, val)
benchmark('raw' = { l = length(unique(df$id[df$val >=0.1]))},
'sort' = {l = length(unique(df_sort$id[df_sort$val >=0.1]))},
'raw_DT' = {df[val >=0.1, uniqueN(id)]},
'sort_DT' = {df_sort[val >=0.1, uniqueN(id)]},
'raw_DT_length' = {df[val >=0.1, length(unique(id))]},
replications=20, order = "relative")
# test replications elapsed relative user.self sys.self user.child sys.child
#3 raw_DT 20 9.422 1.000 8.742 0.680 0 0
#4 sort_DT 20 9.617 1.021 8.901 0.697 0 0
#2 sort 20 15.237 1.617 14.016 1.222 0 0
#1 raw 20 15.313 1.625 14.128 1.190 0 0
#5 raw_DT_length 20 16.719 1.774 15.473 1.233 0 0
benchmark(unique_length = length(unique(df$id)),
uniqueN = uniqueN(df$id),
replications=20, order = "relative")
# test replications elapsed relative user.self sys.self user.child sys.child
#2 uniqueN 20 7.708 1.000 7.340 0.368 0 0
#1 unique_length 20 15.683 2.035 14.765 0.920 0 0
benchmark(subset_base = df$id[df$val >=0.1],
subset_data.table = df[val >=0.1, id],
subset_data.table_keyed = df_sort[val >=0.1, id],
replications=20, order = "relative")
# test replications elapsed relative user.self sys.self user.child sys.child
#1 subset_base 20 1.546 1.000 1.206 0.340 0 0
#3 subset_data.table_keyed 20 2.360 1.527 2.046 0.314 0 0
#2 subset_data.table 20 2.493 1.613 2.184 0.309 0 0
一些评论:您的基本R方法根本不使用data.table功能. Base R逻辑运算符不使用排序;它们是一个简单的矢量扫描. uniqueN击败长度(unique())应该如此. data.table子集未针对> =(尚未)进行优化.如果你真的想改进时间,你应该用Rcpp编写一个唯一的NSubset函数,它使用被排序的数据(我也会对ID进行排序).
> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.4
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
locale:
[1] de_DE.UTF-8/de_DE.UTF-8/de_DE.UTF-8/C/de_DE.UTF-8/de_DE.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.11.2 rbenchmark_1.0.0
loaded via a namespace (and not attached):
[1] compiler_3.5.0 tools_3.5.0