Using HttpWebRequest, I appear to be hitting a ceiling (regarding connections) with a spider I'm working on when going beyond a certain number of threads. The machine is very beefy, quad core, 4gb. It just seems that connections fail when I try to utilize 100+ requests.
It appears as if it is a PER application restriction, because I can load up another app and do the same with (what looks like no problems). I'm not 100% sure though because the more connections I introduce, the more errors I'm noticing.
I have a 2 part question:
1. Is there any way to raise this limit in the application?
2. Are there OS limitations (Windows Server 2003) to how many socket connections / httpwebrequests can be made? If so, how can I tweak the system to allow more?