快速data.table过滤

2024年1月15日 201次阅读

我有一个用例,我想重复快速过滤数据表.我有一种感觉,如果我使用像df [val> 0,]这样的简单过滤器,扫描是O(N),但如果我只是将val字段设置为键,那么我应该能够更快地过滤,但不是确定如何.作为一个例子,我做了以下几点

排序的data.table

df_sort = data.table(id = seq(1, 100000), val = runif(100000, 0, 1))
setkeyv(df_sort, c('val')

一个常规的data.table

df = data.table(id = seq(1, 100000), val = runif(100000, 0, 1))

我想要做的是找到val>的唯一id.托尔.使用rbenchmark我做了以下

benchmark('raw' = { l = length(unique(df$id[df$val >=0.1]))}, 'sort' = {l = length(unique(df_sort$id[df_sort$val >=0.1]))}, replications=20)

我得到的结果显示两者没有区别.

  test replications elapsed relative user.self sys.self user.child sys.child
1  raw           20   0.440    1.005     0.494        0          0         0
2 sort           20   0.438    1.000     0.540        0          0         0

有人可以就如何做到这一点给出一些指示吗？它甚至可能吗？

编辑

在评论中尝试了一个建议,回来更糟.不知道为什么.

benchmark('raw' = { l = df[val >=0.1, uniqueN(id)]}, 'sort' = {l = df_sort[val >=0.1, uniqueN(id)]}, replications=200)

test replications elapsed relative user.self sys.self user.child sys.child
1  raw          200   0.770    1.000     0.928        0          0         0
2 sort          200   1.361    1.768     1.691        0          0         0

最佳答案 好的,这是一些适当的基准.两个基准测试中的时间单位可能不一样.

library(rbenchmark)
library(data.table)

set.seed(42)
#using sample to rule out ALTREP influence
df <- data.table(id = sample(1e7, replace = TRUE), val = runif(1e7, 0, 1))
df_sort <- copy(df)
setkey(df_sort, val)

benchmark('raw' = { l = length(unique(df$id[df$val >=0.1]))}, 
          'sort' = {l = length(unique(df_sort$id[df_sort$val >=0.1]))},
          'raw_DT' = {df[val >=0.1, uniqueN(id)]},
          'sort_DT' = {df_sort[val >=0.1, uniqueN(id)]},
          'raw_DT_length' = {df[val >=0.1, length(unique(id))]},
          replications=20, order = "relative")
#           test replications elapsed relative user.self sys.self user.child sys.child
#3        raw_DT           20   9.422    1.000     8.742    0.680          0         0
#4       sort_DT           20   9.617    1.021     8.901    0.697          0         0
#2          sort           20  15.237    1.617    14.016    1.222          0         0
#1           raw           20  15.313    1.625    14.128    1.190          0         0
#5 raw_DT_length           20  16.719    1.774    15.473    1.233          0         0


benchmark(unique_length = length(unique(df$id)), 
          uniqueN = uniqueN(df$id),
          replications=20, order = "relative")
#           test replications elapsed relative user.self sys.self user.child sys.child
#2       uniqueN           20   7.708    1.000     7.340    0.368          0         0
#1 unique_length           20  15.683    2.035    14.765    0.920          0         0

benchmark(subset_base = df$id[df$val >=0.1], 
          subset_data.table = df[val >=0.1, id],
          subset_data.table_keyed = df_sort[val >=0.1, id],
          replications=20, order = "relative")
#                     test replications elapsed relative user.self sys.self user.child sys.child
#1             subset_base           20   1.546    1.000     1.206    0.340          0         0
#3 subset_data.table_keyed           20   2.360    1.527     2.046    0.314          0         0
#2       subset_data.table           20   2.493    1.613     2.184    0.309          0         0

一些评论：您的基本R方法根本不使用data.table功能. Base R逻辑运算符不使用排序;它们是一个简单的矢量扫描. uniqueN击败长度(unique())应该如此. data.table子集未针对> =(尚未)进行优化.如果你真的想改进时间,你应该用Rcpp编写一个唯一的NSubset函数,它使用被排序的数据(我也会对ID进行排序).

> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.4

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] de_DE.UTF-8/de_DE.UTF-8/de_DE.UTF-8/C/de_DE.UTF-8/de_DE.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.11.2 rbenchmark_1.0.0 

loaded via a namespace (and not attached):
[1] compiler_3.5.0 tools_3.5.0
点赞