我在我的c#项目中使用CSVHelper并将一个大的csv数据文件(大约2000条记录)读入内存.
https://github.com/JoshClose/CsvHelper
如果记录小于500,它可以正常工作,它总是在不同阶段抛出IOException取决于网络或数字是否上升.我目前部署到Azure云平台,所以从blob存储到服务器的读取,应该不是任何网络问题.
CsvHelper.CsvParserException: A parsing error occurred.
Row: '995' (1 based)
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
at CsvHelper.CsvParser.GetChar(Int32& fieldStartPosition, Int32& rawFieldStartPosition, String& field, Boolean prevCharWasDelimiter, Int32& recordPosition, Int32& fieldLength, Boolean isPeek) in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 445
at CsvHelper.CsvParser.ReadLine() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 247
at CsvHelper.CsvParser.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 108
--- End of inner exception stack trace ---
at CsvHelper.CsvParser.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 136
at CsvHelper.CsvReader.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvReader.cs:line 173
它会抛出(csv.read())
var wc = new WebClient();
using (var sourceStream = wc.OpenRead(fileUrl))
{
using (var csv = new CsvReader(new StreamReader(sourceStream)))
{
while (csv.Read())
{
try
{
//some reading operation
}
catch (Exception ex)
{
_logger.Error(ex);
}
}
_logger.InfoFormat("Finished {0} reading data #{1}");
}
}
在哪里设置streamreader超时值?
最佳答案 使用云资源(无论是Azure还是任何其他云资源)时,您不应直接读取文件.最好你应该实现重试逻辑,以确保绕过任何瞬态错误(阅读有关瞬态错误
here,
here和
there或只是在互联网上搜索术语“瞬态错误”).
在你的情况下,我建议你通过CloudBlockBlob.DownloadToStream方法包裹你的电话.因此,您仍然可以使用Stream来解析文件,但您也将使用.NET Azure Blob API库的安全性,该库可以代表您处理所有瞬态错误.
您的代码看起来像这样:
// get the CloudblockBlob object
using(MemoryStream blobStream = new MemoryStream())
{
blobObject.DownloadToStream(blobStream);
using (var csv = new CsvReader(new StreamReader(blobStream)))
{
while (csv.Read())
{
try
{
//some reading operation
}
catch (Exception ex)
{
_logger.Error(ex);
}
}
_logger.InfoFormat("Finished {0} reading data #{1}");
}
}