?
Solved

HttpWebRequest maximum throughput

Posted on 2010-09-16
1
Medium Priority
?
437 Views
Last Modified: 2012-06-27
Using HttpWebRequest, I appear to be hitting a ceiling (regarding connections) with a spider I'm working on when going beyond a certain number of threads. The machine is very beefy, quad core, 4gb. It just seems that connections fail when I try to utilize 100+ requests.

It appears as if it is a PER application restriction, because I can load up another app and do the same with (what looks like no problems). I'm not 100% sure though because the more connections I introduce, the more errors I'm noticing.

I have a 2 part question:

1. Is there any way to raise this limit in the application?

2. Are there OS limitations (Windows Server 2003) to how many socket connections / httpwebrequests can be made? If so, how can I tweak the system to allow more?
0
Comment
Question by:mcainc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 
LVL 12

Accepted Solution

by:
w00te earned 2000 total points
ID: 33698234
Well here's your first answer:

"According to the Microsoft documentation., "The thread pool is created the first time you create an instance of the ThreadPool class. The thread pool has a default limit of 25 threads per available processor, which could be changed using CorSetMaxThreads as defined in the mscoree.h file."

25 * 4 cores = 100, the number you're stuck at.

From: http://www.csharpfriends.com/Articles/getArticle.aspx?articleID=201

Second question:

There is a parameter that limits the maximum number of connections that TCP may have open simultaneously.
[HKEY_LOCAL_MACHINE \System \CurrentControlSet \Services \Tcpip \Parameters]
TcpNumConnections = 0x00fffffe (Default = 16,777,214)

Note a 16 Million connection limit sounds very promising, but there are other parameters (See below), which keeps us from ever reaching this limit.

http://smallvoid.com/article/winnt-tcpip-max-limit.html

Have a good night man,

-w00te
0

Featured Post

Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Wouldn’t it be nice if you could test whether an element is contained in an array by using a Contains method just like the one available on List objects? Wouldn’t it be good if you could write code like this? (CODE) In .NET 3.5, this is possible…
Today I had a very interesting conundrum that had to get solved quickly. Needless to say, it wasn't resolved quickly because when we needed it we were very rushed, but as soon as the conference call was over and I took a step back I saw the correct …
NetCrunch network monitor is a highly extensive platform for network monitoring and alert generation. In this video you'll see a live demo of NetCrunch with most notable features explained in a walk-through manner. You'll also get to know the philos…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question