[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

.NET HTTPWebRequests

Posted on 2010-11-29
4
Medium Priority
?
377 Views
Last Modified: 2012-05-10
I've been dealing with this problem for a while. I've been working on creating a spider to connect to various web addresses and collect some data.

The problem is, after a certain number of threads (30-40) in 1 application (exe) requests start to fail. In the meantime I just limited the application to 25 threads.

However, if I start another instance of the exe, I can run another 25 threads with no problems. I've ran up to 6 instances of the same application with no apparent issues.


My question is, how can I achieve this with a single application instance? Do I need to use some kind of 3rd party tool (like curl) or some crude sockets method?
0
Comment
Question by:mcainc
  • 2
4 Comments
 
LVL 13

Expert Comment

by:themrrobert
ID: 34235918
is it possible its maxing out the number of simultaneous connections to a single server, and that the reason multiple apps work is because they follow different paths and don't have the same number of connections per server?

Many http servers have a limit of 3 - 5 max connections from a single ip address to prevent dos attacks.
If this is your issue, then simply add a counter while any thread is on a particular ip address (resolved domain name, some domains have the same ip) and if it exceeds 5 then that thread sleeps untill the number drops, and spawns a new thread with a new psuedo random starting point.
0
 

Author Comment

by:mcainc
ID: 34235960
they're all hitting the same site on the same machine (same IP)

it's like there is a limit to how many requests you can make per application instance

if i run 5-6 of the same exe they all work just fine
0
 
LVL 16

Accepted Solution

by:
kris_per earned 2000 total points
ID: 34237746

This link mentions that maximum no. of connections can be configured using ServicePointManager.DefaultConnectionLimit  ==> http://stackoverflow.com/questions/3500563/c-http-request-max-connections

This link mentiond about changing the maximum no. of connections by changing the value in registry => http://bytes.com/topic/c-sharp/answers/563646-httpwebrequest-maximum-http-connection

See if this would help you.
0
 

Author Closing Comment

by:mcainc
ID: 34288498
this looks like it was it!
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Parsing a CSV file is a task that we are confronted with regularly, and although there are a vast number of means to do this, as a newbie, the field can be confusing and the tools can seem complex. A simple solution to parsing a customized CSV fi…
The ECB site provides FX rates for major currencies since its inception in 1999 in the form of an XML feed. The files have the following format (reducted for brevity) (CODE) There are three files available HERE (http://www.ecb.europa.eu/stats/exch…
Screencast - Getting to Know the Pipeline
Is your OST file inaccessible, Need to transfer OST file from one computer to another? Want to convert OST file to PST? If the answer to any of the above question is yes, then look no further. With the help of Stellar OST to PST Converter, you can e…
Suggested Courses
Course of the Month17 days, 14 hours left to enroll

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question