R：巨大(> 20GB)文件的xmlEventParse期间的内存管理

2023年11月2日 232次阅读

基于前一个问题(
see here),我试图通过xmlEventParse读取许多大型xml文件,同时保存节点变化数据.使用此示例xml：
https://www.nlm.nih.gov/databases/dtd/medsamp2015.xml.

下面的代码使用xpathSapply来提取必要的值和一系列if语句,以便将值与唯一值(PMID)匹配到记录中的每个非唯一值(LastName)的方式组合 – 对于这些值不是LastNames.目标是沿途编写一系列小csv(此处,每1000个LastNames之后)以最小化使用的内存量.

当在完整大小的数据集上运行时,代码成功地批量输出文件,但是仍然存储在内存中,一旦使用所有RAM,最终会导致系统错误.我在代码运行时看过任务管理器,可以看到R的内存随着程序的进展而增长.如果我在中途停止程序然后清除R工作区,包括隐藏的项目,则内存似乎仍被R使用.直到我关闭R才重新释放内存.

自己运行几次,即使清除了工作区,你也会看到R的内存使用量增长.

请帮忙！对于以这种方式读取大型XML文件的其他人来说,这个问题似乎很常见(参见例如注释in this question).

我的代码如下：

library(XML)

filename <- "~/Desktop/medsamp2015.xml"

tempdat <- data.frame(pmid=as.numeric(),
                      lname=character(), 
                      stringsAsFactors=FALSE) 
cnt <- 1
branchFunction <- function() {
  func <- function(x, ...) {
    v1 <- xpathSApply(x, path = "//PMID", xmlValue)
    v2 <- xpathSApply(x, path = "//Author/LastName", xmlValue)
    print(cbind(c(rep(v1,length(v2))), v2))

    #below is where I store/write the temp data along the way
    #but even without doing this, memory is used (even after clearing)

    tempdat <<- rbind(tempdat,cbind(c(rep(v1,length(v2))), v2))
    if (nrow(tempdat) > 1000){
      outname <- paste0("~/Desktop/outfiles",cnt,".csv")
      write.csv(tempdat, outname , row.names = F)
      tempdat <<- data.frame(pmid=as.numeric(),
                            lname=character(), 
                            stringsAsFactors=FALSE)
      cnt <<- cnt+1
    }
  }
  list(MedlineCitation = func)
}

myfunctions <- branchFunction()

#RUN
xmlEventParse(
  file = filename, 
  handlers = NULL, 
  branches = myfunctions
)

最佳答案 这是一个例子,我们有一个启动脚本invoke.sh,它调用一个R脚本并将url和filename作为参数传递…在这种情况下,我之前已经下载了测试文件
medsamp2015.xml并放入./data目录.

>我的意思是在invoke.sh脚本中创建一个循环并遍历目标文件名列表.对于每个文件,您调用R实例,下载它,处理文件并继续下一个.

警告：我没有检查或更改您的功能与任何其他下载文件和格式.我将通过删除第62行上的print()包装来关闭输出的打印.

print( cbind(c(rep(v1, length(v2))), v2))

>见：runtime.txt打印出来.
>输出.csv文件放在./data目录中.

注意：这是我在此主题上提供的先前答案的衍生物：
R memory not released in Windows.我希望它有助于举例.

启动脚本

  1 #!/usr/local/bin/bash -x
  2
  3 R --no-save -q --slave < ./47162861.R --args "https://www.nlm.nih.gov/databases/dtd" "medsamp2015.xml"

R文件 – 47162861.R

# Set working directory

projectDir <- "~/dev/stackoverflow/47162861"
setwd(projectDir)

# -----------------------------------------------------------------------------
# Load required Packages...
requiredPackages <- c("XML")

ipak <- function(pkg) {
  new.pkg <- pkg[!(pkg %in% installed.packages()[, "Package"])]
  if (length(new.pkg))
    install.packages(new.pkg, dependencies = TRUE)
  sapply(pkg, require, character.only = TRUE)
}

ipak(requiredPackages)

# -----------------------------------------------------------------------------
# Load required Files
# trailingOnly=TRUE means that only your arguments are returned
args <- commandArgs(trailingOnly = TRUE)

if ( length(args) != 0 ) {
  dataDir <- file.path(projectDir,"data")
  fileUrl = args[1]
  fileName = args[2]
} else {
  dataDir <- file.path(projectDir,"data")
  fileUrl <- "https://www.nlm.nih.gov/databases/dtd"
  fileName <- "medsamp2015.xml"
}

# -----------------------------------------------------------------------------
# Download file

# Does the directory Exist? If it does'nt create it
if (!file.exists(dataDir)) {
  dir.create(dataDir)
}

# Now we check if we have downloaded the data already if not we download it

if (!file.exists(file.path(dataDir, fileName))) {
  download.file(fileUrl, file.path(dataDir, fileName), method = "wget")
}

# -----------------------------------------------------------------------------
# Now we extrat the data

tempdat <- data.frame(pmid = as.numeric(), lname = character(),
  stringsAsFactors = FALSE)
cnt <- 1

branchFunction <- function() {
  func <- function(x, ...) {
    v1 <- xpathSApply(x, path = "//PMID", xmlValue)
    v2 <- xpathSApply(x, path = "//Author/LastName", xmlValue)
    print(cbind(c(rep(v1, length(v2))), v2))

    # below is where I store/write the temp data along the way
    # but even without doing this, memory is used (even after
    # clearing)

    tempdat <<- rbind(tempdat, cbind(c(rep(v1, length(v2))),
      v2))
    if (nrow(tempdat) > 1000) {
      outname <- file.path(dataDir, paste0(cnt, ".csv")) # Create FileName
      write.csv(tempdat, outname, row.names = F) # Write File to created directory
      tempdat <<- data.frame(pmid = as.numeric(), lname = character(),
        stringsAsFactors = FALSE)
      cnt <<- cnt + 1
    }
  }
  list(MedlineCitation = func)
}

myfunctions <- branchFunction()

# -----------------------------------------------------------------------------
# RUN
xmlEventParse(file = file.path(dataDir, fileName),
              handlers = NULL,
              branches = myfunctions)

测试文件和输出

~/dev/stackoverflow/47162861/data/medsamp2015.xml

$ll                                                            
total 2128
drwxr-xr-x@ 7 hidden  staff   238B Nov 10 11:05 .
drwxr-xr-x@ 9 hidden  staff   306B Nov 10 11:11 ..
-rw-r--r--@ 1 hidden  staff    32K Nov 10 11:12 1.csv
-rw-r--r--@ 1 hidden  staff    20K Nov 10 11:12 2.csv
-rw-r--r--@ 1 hidden  staff    23K Nov 10 11:12 3.csv
-rw-r--r--@ 1 hidden  staff    37K Nov 10 11:12 4.csv
-rw-r--r--@ 1 hidden  staff   942K Nov 10 11:05 medsamp2015.xml

运行时输出

> ./invoke.sh > runtime.txt
+ R --no-save -q --slave --args https://www.nlm.nih.gov/databases/dtd medsamp2015.xml
Loading required package: XML

档案：runtime.txt

点赞