tpl-dataflow – Tarik Billa

Duplicate exceptions with BroadcastBlock in TPL Dataflow

December 29, 2023 by Tarik

I’ve written a TPL DataFlow example (https://github.com/squideyes/PodFetch) that takes a slightly different approach to completion and error handling. Here’s the relevant code from Line’s 171 to 201 of Program.cs: scraper.LinkTo(fetcher, link => link != null); scraper.LinkTo(DataflowBlock.NullTarget<Link>()); scraper.HandleCompletion(fetcher); Status.Info.Log(“Fetching APOD’s archive list”); links.ForEach(link => scraper.Post(link)); scraper.Complete(); try { await fetcher.Completion; Status.Finished.Log(“Fetched: {0:N0}, Skipped: {1:N0}, Errors: {2:N0}, … Read more

TPL Dataflow, guarantee completion only when ALL source data blocks completed

September 27, 2023 by Tarik

The issue is exactly what casperOne said in his answer. Once the first transform block completes, the processor block goes into “finishing mode”: it will process remaining items in its input queue, but it won’t accept any new items. There is a simpler fix than splitting your processor block in two though: don’t set PropagateCompletion, … Read more

Implementing correct completion of a retryable block

April 21, 2023 by Tarik

Maybe a ManualResetEvent can do the trick for you. Add a public property to TransformManyBlock private ManualResetEvent _signal = new ManualResetEvent(false); public ManualResetEvent Signal { get { return _signal; } } And here you go: var retries = new HashSet<RetryingMessage<TInput>>(); TransformManyBlock<RetryableMessage<TInput>, TOutput> target = null; target = new TransformManyBlock<RetryableMessage<TInput>, TOutput>( async message => { try … Read more

TPL Dataflow, whats the functional difference between Post() and SendAsync()?

March 24, 2023 by Tarik

To see the difference, you need a situation where blocks will postpone their messages. In this case, Post will return false immediately, whereas SendAsync will return a Task that will be completed when the block decides what to do with the message. The Task will have a true result if the message is accepted, and … Read more

Throttling asynchronous tasks

March 5, 2023 by Tarik

As suggested, use TPL Dataflow. A TransformBlock<TInput, TOutput> may be what you’re looking for. You define a MaxDegreeOfParallelism to limit how many strings can be transformed (i.e., how many urls can be downloaded) in parallel. You then post urls to the block, and when you’re done you tell the block you’re done adding items and … Read more