c# – 多次调用GetStringAsync的更有效方法？

2023年11月14日 340次阅读

我有(我的网址列表大约是1000个网址),我想知道是否有更有效的来自同一网站的多个网址(已经更改了ServicePointManager.DefaultConnectionLimit).

同样最好重复使用相同的HttpClient或在每次调用时创建一个新的HttpClient,下面只使用一个而不是多个.

using (var client = new HttpClient { Timeout = new TimeSpan(0, 5, 0) })
{
    var tasks = urls.Select(async url =>
    {
        await client.GetStringAsync(url).ContinueWith(response =>
        {
           var resultHtml = response.Result;
           //process the html

        });
    }).ToList();

    Task.WaitAll(tasks.ToArray());
}

正如@cory所建议的那样
这里是使用TPL的修改代码,但是我必须设置MaxDegreeOfParallelism = 100以达到与基于任务的速度大致相同的速度,下面的代码可以改进吗？

var downloader = new ActionBlock<string>(async url =>
{
    var client = new WebClient();
    var resultHtml = await client.DownloadStringTaskAsync(new Uri(url));


}, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 100 });

foreach(var url in urls)
{
    downloader.Post(url);
}
downloader.Complete();
downloader.Completion.Wait();

最后

public void DownloadUrlContents(List<string> urls)
{
    var watch = Stopwatch.StartNew();

    var httpClient = new HttpClient();
    var downloader = new ActionBlock<string>(async url =>
    {
        var data = await httpClient.GetStringAsync(url);
    }, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 100 });

    Parallel.ForEach(urls, (url) =>
    {
        downloader.SendAsync(url);
    });
    downloader.Complete();
    downloader.Completion.Wait();

    Console.WriteLine($"{MethodBase.GetCurrentMethod().Name} {watch.Elapsed}");    
}

最佳答案 虽然您的代码可以正常工作,但为ActionBlock引入缓冲区块是一种常见做法.为什么要这样做？第一个原因是任务队列大小,您可以轻松地平衡队列中的消息数.第二个原因是将消息添加到缓冲区几乎是即时的,之后TPL Dataflow负责处理您的所有项目：

// async method here
public async Task DownloadUrlContents(List<string> urls)
{
    var watch = Stopwatch.StartNew();

    var httpClient = new HttpClient();

    // you may limit the buffer size here
    var buffer = new BufferBlock<string>();
    var downloader = new ActionBlock<string>(async url =>
        {
            var data = await httpClient.GetStringAsync(url);
            // handle data here
        },
        // note processot count usage here
        new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = Environment.ProcessorCount });
    // notify TPL Dataflow to send messages from buffer to loader
    buffer.LinkTo(downloader, new DataflowLinkOptions {PropagateCompletion = true});

    foreach (var url in urls)
    {
        // do await here
        await buffer.SendAsync(url);
    }
    // queue is done
    buffer.Complete();

    // now it's safe to wait for completion of the downloader
    await downloader.Completion;

    Console.WriteLine($"{MethodBase.GetCurrentMethod().Name} {watch.Elapsed}");
}
点赞