Link to home
Start Free TrialLog in
Avatar of Mr Knackered
Mr KnackeredFlag for United Kingdom of Great Britain and Northern Ireland

asked on

C# Background Workers

I am using background workers in c# and there seems to be a bottle neck somewhere. I have a process that runs for customer records and I have split this up and am running blocks on different background workers. I am using SQLBulkcopy to write data to SQL when a cache reaches a certain number of records and when running on 2 Threads the process takes about 10 mins and consumes about 20% cpu for the app and SQL is using about 10% cpu.

When running the process on 6 threads (6 background workers) the process takes about 7 mins to complete and the app is using about 35% cpu and SQL bobs along at about 10%. I have pushed it up to 12 Threads (12 background workers) and it takes about 7 mins to complete and the app is using the same CPU as 6 Threads.

I have an 8 core processor with 16 threads and it just wont exceed 45% cpu usage.

Is there a limit to the number of worker threads the thread pool will deal with? Its not a problem with network traffic as its running on the same box as SQL and its running in Shared Memory. Data is being cached into a memory data table and when it reaches 20,000 records in each thread the data is bulk copied to SQL.

It feels like there is a limit of 4 or 5 worker threads.
Avatar of AndyAinscow
AndyAinscow
Flag of Switzerland image

As far as I know there isn't a limit to the number of background workers.  (That doesn't mean more is better - that is rather different.)
Each thread will get a timeslice to perform work in then the next thread is given time and so on.
One thing not to forget is that your app is not the only thing running, even if it is the only app you open.  There will be others such as an anti virus (I hope!) running.  Various system services...  All of these require processor time.
Another thing is that any disc access (unless it is an SSD) is massively slower than anything in memory.

These sorts of problems are typically difficult to pin down to any particular thing and often turn out not to be anything you can do about it.
ASKER CERTIFIED SOLUTION
Avatar of Miguel Oz
Miguel Oz
Flag of Australia image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial