My current place of employ has recently moved development to the .NET development platform (for better or for worse). Much of the old software is written in perl. While I see no trouble migrating much of it into .NET languages, I fear what I will miss the most is the ease of using perl for web spidering/scraping applications. I have used extensively the WWW::Mechinize suite of modules, and (maybe through ignorance of my own) see no corollary within the .Net languages.
What I am looking for in response to my question is the communities input (2 cents) into how to approach spidering/scraping activities within the .NET languages. While perl is my background, I am not necessarily looking for Mechanize.NET, just the best way to go about automated web navigation within .NET.
100 points will go to the first five persons with a thoughtful response.
Thanks a lot; I really value real world opinions over the boasts of printed documentation!