?
Solved

.NET HTTPWebRequests

Posted on 2010-11-29
4
Medium Priority
?
375 Views
Last Modified: 2012-05-10
I've been dealing with this problem for a while. I've been working on creating a spider to connect to various web addresses and collect some data.

The problem is, after a certain number of threads (30-40) in 1 application (exe) requests start to fail. In the meantime I just limited the application to 25 threads.

However, if I start another instance of the exe, I can run another 25 threads with no problems. I've ran up to 6 instances of the same application with no apparent issues.


My question is, how can I achieve this with a single application instance? Do I need to use some kind of 3rd party tool (like curl) or some crude sockets method?
0
Comment
Question by:mcainc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 13

Expert Comment

by:themrrobert
ID: 34235918
is it possible its maxing out the number of simultaneous connections to a single server, and that the reason multiple apps work is because they follow different paths and don't have the same number of connections per server?

Many http servers have a limit of 3 - 5 max connections from a single ip address to prevent dos attacks.
If this is your issue, then simply add a counter while any thread is on a particular ip address (resolved domain name, some domains have the same ip) and if it exceeds 5 then that thread sleeps untill the number drops, and spawns a new thread with a new psuedo random starting point.
0
 

Author Comment

by:mcainc
ID: 34235960
they're all hitting the same site on the same machine (same IP)

it's like there is a limit to how many requests you can make per application instance

if i run 5-6 of the same exe they all work just fine
0
 
LVL 16

Accepted Solution

by:
kris_per earned 2000 total points
ID: 34237746

This link mentions that maximum no. of connections can be configured using ServicePointManager.DefaultConnectionLimit  ==> http://stackoverflow.com/questions/3500563/c-http-request-max-connections

This link mentiond about changing the maximum no. of connections by changing the value in registry => http://bytes.com/topic/c-sharp/answers/563646-httpwebrequest-maximum-http-connection

See if this would help you.
0
 

Author Closing Comment

by:mcainc
ID: 34288498
this looks like it was it!
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Creating an analog clock UserControl seems fairly straight forward.  It is, after all, essentially just a circle with several lines in it!  Two common approaches for rendering an analog clock typically involve either manually calculating points with…
Entity Framework is a powerful tool to help you interact with the DataBase but still doesn't help much when we have a Stored Procedure that returns more than one resultset. The solution takes some of out-of-the-box thinking; read on!
Add bar graphs to Access queries using Unicode block characters. Graphs appear on every record in the color you want. Give life to numbers. Hopes this gives you ideas on visualizing your data in new ways ~ Create a calculated field in a query: …
In this video, Percona Solution Engineer Dimitri Vanoverbeke discusses why you want to use at least three nodes in a database cluster. To discuss how Percona Consulting can help with your design and architecture needs for your database and infras…
Suggested Courses

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question