.NET HTTPWebRequests

I've been dealing with this problem for a while. I've been working on creating a spider to connect to various web addresses and collect some data.

The problem is, after a certain number of threads (30-40) in 1 application (exe) requests start to fail. In the meantime I just limited the application to 25 threads.

However, if I start another instance of the exe, I can run another 25 threads with no problems. I've ran up to 6 instances of the same application with no apparent issues.

My question is, how can I achieve this with a single application instance? Do I need to use some kind of 3rd party tool (like curl) or some crude sockets method?
Who is Participating?
kris_perConnect With a Mentor Commented:

This link mentions that maximum no. of connections can be configured using ServicePointManager.DefaultConnectionLimit  ==> http://stackoverflow.com/questions/3500563/c-http-request-max-connections

This link mentiond about changing the maximum no. of connections by changing the value in registry => http://bytes.com/topic/c-sharp/answers/563646-httpwebrequest-maximum-http-connection

See if this would help you.
is it possible its maxing out the number of simultaneous connections to a single server, and that the reason multiple apps work is because they follow different paths and don't have the same number of connections per server?

Many http servers have a limit of 3 - 5 max connections from a single ip address to prevent dos attacks.
If this is your issue, then simply add a counter while any thread is on a particular ip address (resolved domain name, some domains have the same ip) and if it exceeds 5 then that thread sleeps untill the number drops, and spawns a new thread with a new psuedo random starting point.
mcaincAuthor Commented:
they're all hitting the same site on the same machine (same IP)

it's like there is a limit to how many requests you can make per application instance

if i run 5-6 of the same exe they all work just fine
mcaincAuthor Commented:
this looks like it was it!
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.