Solved

HttpWebRequest maximum throughput

Posted on 2010-09-16
1
436 Views
Last Modified: 2012-06-27
Using HttpWebRequest, I appear to be hitting a ceiling (regarding connections) with a spider I'm working on when going beyond a certain number of threads. The machine is very beefy, quad core, 4gb. It just seems that connections fail when I try to utilize 100+ requests.

It appears as if it is a PER application restriction, because I can load up another app and do the same with (what looks like no problems). I'm not 100% sure though because the more connections I introduce, the more errors I'm noticing.

I have a 2 part question:

1. Is there any way to raise this limit in the application?

2. Are there OS limitations (Windows Server 2003) to how many socket connections / httpwebrequests can be made? If so, how can I tweak the system to allow more?
0
Comment
Question by:mcainc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 
LVL 12

Accepted Solution

by:
w00te earned 500 total points
ID: 33698234
Well here's your first answer:

"According to the Microsoft documentation., "The thread pool is created the first time you create an instance of the ThreadPool class. The thread pool has a default limit of 25 threads per available processor, which could be changed using CorSetMaxThreads as defined in the mscoree.h file."

25 * 4 cores = 100, the number you're stuck at.

From: http://www.csharpfriends.com/Articles/getArticle.aspx?articleID=201

Second question:

There is a parameter that limits the maximum number of connections that TCP may have open simultaneously.
[HKEY_LOCAL_MACHINE \System \CurrentControlSet \Services \Tcpip \Parameters]
TcpNumConnections = 0x00fffffe (Default = 16,777,214)

Note a 16 Million connection limit sounds very promising, but there are other parameters (See below), which keeps us from ever reaching this limit.

http://smallvoid.com/article/winnt-tcpip-max-limit.html

Have a good night man,

-w00te
0

Featured Post

Is Your Team Achieving Their Full Potential?

74% of employees feel they are not achieving their full potential. With Linux Academy, not only will you strengthen your team's core competencies but also their knowledge of of the newest IT topics.

With new material every week, we'll make sure that you stay ahead of the game.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Welcome my friends to the second instalment and follow-up to our Minify and Concatenate Your Scripts and Stylesheets (http://www.experts-exchange.com/Programming/Languages/.NET/ASP.NET/A_4334-Minify-and-Concatenate-Your-Scripts-and-Stylesheets.html)…
In my previous article (http://www.experts-exchange.com/Programming/Languages/.NET/.NET_Framework_3.x/A_4362-Serialization-in-NET-1.html) we saw the basics of serialization and how types/objects can be serialized to Binary format. In this blog we wi…
There are cases when e.g. an IT administrator wants to have full access and view into selected mailboxes on Exchange server, directly from his own email account in Outlook or Outlook Web Access. This proves useful when for example administrator want…
In this video, viewers are given an introduction to using the Windows 10 Snipping Tool, how to quickly locate it when it's needed and also how make it always available with a single click of a mouse button, by pinning it to the Desktop Task Bar. Int…

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question