asked on

.NET HTTPWebRequests

I've been dealing with this problem for a while. I've been working on creating a spider to connect to various web addresses and collect some data.

The problem is, after a certain number of threads (30-40) in 1 application (exe) requests start to fail. In the meantime I just limited the application to 25 threads.

However, if I start another instance of the exe, I can run another 25 threads with no problems. I've ran up to 6 instances of the same application with no apparent issues.

My question is, how can I achieve this with a single application instance? Do I need to use some kind of 3rd party tool (like curl) or some crude sockets method?

themrrobert

is it possible its maxing out the number of simultaneous connections to a single server, and that the reason multiple apps work is because they follow different paths and don't have the same number of connections per server?

Many http servers have a limit of 3 - 5 max connections from a single ip address to prevent dos attacks.
If this is your issue, then simply add a counter while any thread is on a particular ip address (resolved domain name, some domains have the same ip) and if it exceeds 5 then that thread sleeps untill the number drops, and spawns a new thread with a new psuedo random starting point.

mcainc

ASKER

they're all hitting the same site on the same machine (same IP)

it's like there is a limit to how many requests you can make per application instance

if i run 5-6 of the same exe they all work just fine

ASKER CERTIFIED SOLUTION

kris_per

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

mcainc

ASKER

this looks like it was it!