C# Background Workers

I am using background workers in c# and there seems to be a bottle neck somewhere. I have a process that runs for customer records and I have split this up and am running blocks on different background workers. I am using SQLBulkcopy to write data to SQL when a cache reaches a certain number of records and when running on 2 Threads the process takes about 10 mins and consumes about 20% cpu for the app and SQL is using about 10% cpu.

When running the process on 6 threads (6 background workers) the process takes about 7 mins to complete and the app is using about 35% cpu and SQL bobs along at about 10%. I have pushed it up to 12 Threads (12 background workers) and it takes about 7 mins to complete and the app is using the same CPU as 6 Threads.

I have an 8 core processor with 16 threads and it just wont exceed 45% cpu usage.

Is there a limit to the number of worker threads the thread pool will deal with? Its not a problem with network traffic as its running on the same box as SQL and its running in Shared Memory. Data is being cached into a memory data table and when it reaches 20,000 records in each thread the data is bulk copied to SQL.

It feels like there is a limit of 4 or 5 worker threads.
Mr KnackeredAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AndyAinscowFreelance programmer / ConsultantCommented:
As far as I know there isn't a limit to the number of background workers.  (That doesn't mean more is better - that is rather different.)
Each thread will get a timeslice to perform work in then the next thread is given time and so on.
One thing not to forget is that your app is not the only thing running, even if it is the only app you open.  There will be others such as an anti virus (I hope!) running.  Various system services...  All of these require processor time.
Another thing is that any disc access (unless it is an SSD) is massively slower than anything in memory.

These sorts of problems are typically difficult to pin down to any particular thing and often turn out not to be anything you can do about it.
Miguel OzSoftware EngineerCommented:
What version of .NET framework are you running?
This value depends on version: In 2.0 it was 50 times the number of cores and  4.0 this value depends on OS resources.
Quoting MSDN:
There is one thread pool per process. Beginning with the .NET Framework version 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the GetMaxThreads method to determine the number of threads. The number of threads in the thread pool can be changed by using the SetMaxThreads method.
As previous post explained the issues could vary from HD to slow SQL (e.g. your affected tables may have triggers  or far too many indexes to update when inserting data)
My suggestion will be:
1) Use a profiler tool (VS2013 Ultimate has one in the product) and find out what it makes the code slow.
2) Use a tool like Process Explorer to find out how many threads/CPU,  Physical Memory + Pagefile info, etc.
3) Enable SQL profiler in yourDB server to find out if you issues there.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Server Hardware

From novice to tech pro — start learning today.