使用杂乱的字符串和多个字符串指示符R读取大数据

2023年2月5日 193次阅读

我有一个大的(8GB)csv文件(逗号分隔),我想读入R.该文件包含三列

>日期#in 2017-12-27格式
> text #a string
>每个字符串键入#a标签(NA,typeA或typeB)

我遇到的问题是text列包含各种字符串指示符：'(单引号.),“(双引号.标记),无”.标记,以及多个分隔的字符串.

例如.

date        text                        type
2016-01-01  great job!                  NA
2016-01-02  please, type "submit"       typeA
2016-01-02  "can't see the "error" now" typeA
2016-01-03  "add \\"/filename.txt\\""   NA

为了阅读这些大数据,我试过：

>基本read.csv和readr的read_csv函数：对一部分工作正常但失败(可能是由于内存)或需要很长时间才能读取
>通过Mac终端将数据分块成1m行的批次：失败,因为线条似乎任意破坏
>使用fread(首选,因为我希望这将解决另外两个问题)：失败,错误：期望3个cols,但是行1103在处理完所有cols后包含文本.

我的想法是通过使用我所知道的数据的细节来解决这些问题,即每行以日期开头并以NA,typeA或typeB结束.

我怎么能实现这个(使用纯readLines或fread)？

编辑：
使用Mac TextWrangler打开的示例数据(匿名)：

"date","text","type"
"2016-03-30","Maybe use `tapply` from `base`, and check how that works.",NA
"2016-04-01","Fiex this now. Please check.","typeA"
"2016-04-01","Does it work? Maybe try the other approach.","typeB"
"2016-04-01","This won't work. You should remove ABC ... each line starts with a date and ends with ... and this line is veeeeeeeeeeeeeeeeeery long.",NA
"2014-05-02","Tried to remove ""../"" but no success @myid",typeA

样本数据2：

"date","text","type"
"2018-05-02","i try this, but it doesnt work",NA
"2018-05-02","Thank you very much. Cheers !!",NA
"2018-05-02","@myid. I'll change this.",NA

可重复的fread错误的示例数据“Expecting 3 cols,但第3行包含处理所有cols后的文本.”：

"date","text","type"
"2015-03-02","Some text, some text, some question? Please, some question?",NA
"2015-03-02","Here you have the error ""Can’t access {file \""Macintosh HD:abc:def:filename\"", \""/abc.txt\""} from directory."" something -1100 from {file ""Macintosh HD:abc:def:filename"", ""/abc.txt""} to file",NA
"2015-03-02","good idea",NA
"2015-03-02","Worked perfectly :)",NA

SessionInfo：

R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.5

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.10.4-3 readr_1.1.1        

loaded via a namespace (and not attached):
[1] compiler_3.5.0   assertthat_0.2.0 R6_2.2.2         cli_1.0.0       
[5] hms_0.4.2        tools_3.5.0      pillar_1.2.2     rstudioapi_0.7  
[9] tibble_1.4.2     yaml_2.1.19      crayon_1.3.4     Rcpp_0.12.16    
[13] utf8_1.1.3       pkgconfig_2.0.1  rlang_0.2.0

最佳答案 readLines方法可以

infile <- file("test.txt", "r")
txt <- readLines(infile, n = 1)
df <- NULL

#change this value as per your requirement
chunksize <- 1

while(length(txt)){
  txt <- readLines(infile, warn=F, n = chunksize)
  df  <- rbind(df, data.frame(date = gsub("\\s.*", "", txt),
                              text = trimws(gsub("\\S+(.*)\\s+\\S+$", "\\1", txt)),
                              type = gsub(".*\\s", "", txt),
                              stringsAsFactors = F))
  }

这使

> df
        date                          text  type
1 2016-01-01                    great job!    NA
2 2016-01-02         please, type "submit" typeA
3 2016-01-02   "can't see the "error" now" typeA
4 2016-01-03 "add \\\\"/filename.txt\\\\""    NA

示例数据：test.txt包含

date        text                        type
2016-01-01  great job!                  NA
2016-01-02  please, type "submit"       typeA
2016-01-02  "can't see the "error" now" typeA
2016-01-03  "add \\"/filename.txt\\""   NA

更新：
您可以使用以下正则表达式解析器修改上面的代码,以解析另一组示例数据

df  <- rbind(df, data.frame(date = gsub("\"(\\S{10}).*", "\\1", txt),
                            text = gsub(".*\"\\,\"(.*)\"\\,(\"|NA).*", "\\1", txt),
                            type = gsub(".*\\,|\"", "", txt),
                            stringsAsFactors = F))

另一组样本数据：

"date","text","type"
"2016-03-30","Maybe use `tapply` from `base`, and check how that works.",NA
"2016-04-01","Fiex this now. Please check.","typeA"
"2016-04-01","Does it work? Maybe try the other approach.","typeB"
"2016-04-01","This won't work. You should remove ABC ... each line starts with a date and ends with ... and this line is veeeeeeeeeeeeeeeeeery long.",NA
"2014-05-02","Tried to remove ""../"" but no success @myid","typeA"
点赞