Solved

.NET HTTPWebRequests

Posted on 2010-11-29
4
372 Views
Last Modified: 2012-05-10
I've been dealing with this problem for a while. I've been working on creating a spider to connect to various web addresses and collect some data.

The problem is, after a certain number of threads (30-40) in 1 application (exe) requests start to fail. In the meantime I just limited the application to 25 threads.

However, if I start another instance of the exe, I can run another 25 threads with no problems. I've ran up to 6 instances of the same application with no apparent issues.


My question is, how can I achieve this with a single application instance? Do I need to use some kind of 3rd party tool (like curl) or some crude sockets method?
0
Comment
Question by:mcainc
  • 2
4 Comments
 
LVL 13

Expert Comment

by:themrrobert
ID: 34235918
is it possible its maxing out the number of simultaneous connections to a single server, and that the reason multiple apps work is because they follow different paths and don't have the same number of connections per server?

Many http servers have a limit of 3 - 5 max connections from a single ip address to prevent dos attacks.
If this is your issue, then simply add a counter while any thread is on a particular ip address (resolved domain name, some domains have the same ip) and if it exceeds 5 then that thread sleeps untill the number drops, and spawns a new thread with a new psuedo random starting point.
0
 

Author Comment

by:mcainc
ID: 34235960
they're all hitting the same site on the same machine (same IP)

it's like there is a limit to how many requests you can make per application instance

if i run 5-6 of the same exe they all work just fine
0
 
LVL 16

Accepted Solution

by:
kris_per earned 500 total points
ID: 34237746

This link mentions that maximum no. of connections can be configured using ServicePointManager.DefaultConnectionLimit  ==> http://stackoverflow.com/questions/3500563/c-http-request-max-connections

This link mentiond about changing the maximum no. of connections by changing the value in registry => http://bytes.com/topic/c-sharp/answers/563646-httpwebrequest-maximum-http-connection

See if this would help you.
0
 

Author Closing Comment

by:mcainc
ID: 34288498
this looks like it was it!
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction Although it is an old technology, serial ports are still being used by many hardware manufacturers. If you develop applications in C#, Microsoft .NET framework has SerialPort class to communicate with the serial ports.  I needed to…
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …

820 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question