Posted on 2010-11-29
I've been dealing with this problem for a while. I've been working on creating a spider to connect to various web addresses and collect some data.
The problem is, after a certain number of threads (30-40) in 1 application (exe) requests start to fail. In the meantime I just limited the application to 25 threads.
However, if I start another instance of the exe, I can run another 25 threads with no problems. I've ran up to 6 instances of the same application with no apparent issues.
My question is, how can I achieve this with a single application instance? Do I need to use some kind of 3rd party tool (like curl) or some crude sockets method?