HttpWebRequest maximum throughput

Using HttpWebRequest, I appear to be hitting a ceiling (regarding connections) with a spider I'm working on when going beyond a certain number of threads. The machine is very beefy, quad core, 4gb. It just seems that connections fail when I try to utilize 100+ requests.

It appears as if it is a PER application restriction, because I can load up another app and do the same with (what looks like no problems). I'm not 100% sure though because the more connections I introduce, the more errors I'm noticing.

I have a 2 part question:

1. Is there any way to raise this limit in the application?

2. Are there OS limitations (Windows Server 2003) to how many socket connections / httpwebrequests can be made? If so, how can I tweak the system to allow more?
mcaincAsked:
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

x
 
w00teConnect With a Mentor Commented:
Well here's your first answer:

"According to the Microsoft documentation., "The thread pool is created the first time you create an instance of the ThreadPool class. The thread pool has a default limit of 25 threads per available processor, which could be changed using CorSetMaxThreads as defined in the mscoree.h file."

25 * 4 cores = 100, the number you're stuck at.

From: http://www.csharpfriends.com/Articles/getArticle.aspx?articleID=201

Second question:

There is a parameter that limits the maximum number of connections that TCP may have open simultaneously.
[HKEY_LOCAL_MACHINE \System \CurrentControlSet \Services \Tcpip \Parameters]
TcpNumConnections = 0x00fffffe (Default = 16,777,214)

Note a 16 Million connection limit sounds very promising, but there are other parameters (See below), which keeps us from ever reaching this limit.

http://smallvoid.com/article/winnt-tcpip-max-limit.html

Have a good night man,

-w00te
0
All Courses

From novice to tech pro — start learning today.