I am using background workers in c# and there seems to be a bottle neck somewhere. I have a process that runs for customer records and I have split this up and am running blocks on different background workers. I am using SQLBulkcopy to write data to SQL when a cache reaches a certain number of records and when running on 2 Threads the process takes about 10 mins and consumes about 20% cpu for the app and SQL is using about 10% cpu.
When running the process on 6 threads (6 background workers) the process takes about 7 mins to complete and the app is using about 35% cpu and SQL bobs along at about 10%. I have pushed it up to 12 Threads (12 background workers) and it takes about 7 mins to complete and the app is using the same CPU as 6 Threads.
I have an 8 core processor with 16 threads and it just wont exceed 45% cpu usage.
Is there a limit to the number of worker threads the thread pool will deal with? Its not a problem with network traffic as its running on the same box as SQL and its running in Shared Memory. Data is being cached into a memory data table and when it reaches 20,000 records in each thread the data is bulk copied to SQL.
It feels like there is a limit of 4 or 5 worker threads.