Solved

HttpWebRequest maximum throughput

Posted on 2010-09-16
1
428 Views
Last Modified: 2012-06-27
Using HttpWebRequest, I appear to be hitting a ceiling (regarding connections) with a spider I'm working on when going beyond a certain number of threads. The machine is very beefy, quad core, 4gb. It just seems that connections fail when I try to utilize 100+ requests.

It appears as if it is a PER application restriction, because I can load up another app and do the same with (what looks like no problems). I'm not 100% sure though because the more connections I introduce, the more errors I'm noticing.

I have a 2 part question:

1. Is there any way to raise this limit in the application?

2. Are there OS limitations (Windows Server 2003) to how many socket connections / httpwebrequests can be made? If so, how can I tweak the system to allow more?
0
Comment
Question by:mcainc
1 Comment
 
LVL 12

Accepted Solution

by:
w00te earned 500 total points
Comment Utility
Well here's your first answer:

"According to the Microsoft documentation., "The thread pool is created the first time you create an instance of the ThreadPool class. The thread pool has a default limit of 25 threads per available processor, which could be changed using CorSetMaxThreads as defined in the mscoree.h file."

25 * 4 cores = 100, the number you're stuck at.

From: http://www.csharpfriends.com/Articles/getArticle.aspx?articleID=201

Second question:

There is a parameter that limits the maximum number of connections that TCP may have open simultaneously.
[HKEY_LOCAL_MACHINE \System \CurrentControlSet \Services \Tcpip \Parameters]
TcpNumConnections = 0x00fffffe (Default = 16,777,214)

Note a 16 Million connection limit sounds very promising, but there are other parameters (See below), which keeps us from ever reaching this limit.

http://smallvoid.com/article/winnt-tcpip-max-limit.html

Have a good night man,

-w00te
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Normally the drop down box control found in the .Net framework tools is able to select just one data and value at a time, which is displayed on the text area.   But what if you want to have multiple values to be selected in the drop down box? As …
A basic question.. “What is the Garbage Collector?” The usual answer given back: “Garbage collector is a background thread run by the CLR for freeing up the memory space used by the objects which are no longer used by the program.” I wondered …
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now