Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 446
  • Last Modified:

HttpWebRequest maximum throughput

Using HttpWebRequest, I appear to be hitting a ceiling (regarding connections) with a spider I'm working on when going beyond a certain number of threads. The machine is very beefy, quad core, 4gb. It just seems that connections fail when I try to utilize 100+ requests.

It appears as if it is a PER application restriction, because I can load up another app and do the same with (what looks like no problems). I'm not 100% sure though because the more connections I introduce, the more errors I'm noticing.

I have a 2 part question:

1. Is there any way to raise this limit in the application?

2. Are there OS limitations (Windows Server 2003) to how many socket connections / httpwebrequests can be made? If so, how can I tweak the system to allow more?
0
mcainc
Asked:
mcainc
1 Solution
 
w00teCommented:
Well here's your first answer:

"According to the Microsoft documentation., "The thread pool is created the first time you create an instance of the ThreadPool class. The thread pool has a default limit of 25 threads per available processor, which could be changed using CorSetMaxThreads as defined in the mscoree.h file."

25 * 4 cores = 100, the number you're stuck at.

From: http://www.csharpfriends.com/Articles/getArticle.aspx?articleID=201

Second question:

There is a parameter that limits the maximum number of connections that TCP may have open simultaneously.
[HKEY_LOCAL_MACHINE \System \CurrentControlSet \Services \Tcpip \Parameters]
TcpNumConnections = 0x00fffffe (Default = 16,777,214)

Note a 16 Million connection limit sounds very promising, but there are other parameters (See below), which keeps us from ever reaching this limit.

http://smallvoid.com/article/winnt-tcpip-max-limit.html

Have a good night man,

-w00te
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now