[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

.NET HTTPWebRequests

Posted on 2010-11-29
4
Medium Priority
?
376 Views
Last Modified: 2012-05-10
I've been dealing with this problem for a while. I've been working on creating a spider to connect to various web addresses and collect some data.

The problem is, after a certain number of threads (30-40) in 1 application (exe) requests start to fail. In the meantime I just limited the application to 25 threads.

However, if I start another instance of the exe, I can run another 25 threads with no problems. I've ran up to 6 instances of the same application with no apparent issues.


My question is, how can I achieve this with a single application instance? Do I need to use some kind of 3rd party tool (like curl) or some crude sockets method?
0
Comment
Question by:mcainc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 13

Expert Comment

by:themrrobert
ID: 34235918
is it possible its maxing out the number of simultaneous connections to a single server, and that the reason multiple apps work is because they follow different paths and don't have the same number of connections per server?

Many http servers have a limit of 3 - 5 max connections from a single ip address to prevent dos attacks.
If this is your issue, then simply add a counter while any thread is on a particular ip address (resolved domain name, some domains have the same ip) and if it exceeds 5 then that thread sleeps untill the number drops, and spawns a new thread with a new psuedo random starting point.
0
 

Author Comment

by:mcainc
ID: 34235960
they're all hitting the same site on the same machine (same IP)

it's like there is a limit to how many requests you can make per application instance

if i run 5-6 of the same exe they all work just fine
0
 
LVL 16

Accepted Solution

by:
kris_per earned 2000 total points
ID: 34237746

This link mentions that maximum no. of connections can be configured using ServicePointManager.DefaultConnectionLimit  ==> http://stackoverflow.com/questions/3500563/c-http-request-max-connections

This link mentiond about changing the maximum no. of connections by changing the value in registry => http://bytes.com/topic/c-sharp/answers/563646-httpwebrequest-maximum-http-connection

See if this would help you.
0
 

Author Closing Comment

by:mcainc
ID: 34288498
this looks like it was it!
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

It’s quite interesting for me as I worked with Excel using vb.net for some time. Here are some topics which I know want to share with others whom this might help. First of all if you are working with Excel then you need to Download the Following …
Parsing a CSV file is a task that we are confronted with regularly, and although there are a vast number of means to do this, as a newbie, the field can be confusing and the tools can seem complex. A simple solution to parsing a customized CSV fi…
How to fix incompatible JVM issue while installing Eclipse While installing Eclipse in windows, got one error like above and unable to proceed with the installation. This video describes how to successfully install Eclipse. How to solve incompa…
This lesson discusses how to use a Mainform + Subforms in Microsoft Access to find and enter data for payments on orders. The sample data comes from a custom shop that builds and sells movable storage structures that are delivered to your property. …

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question