MSDN library address: regular express

Hello, I want to look up and print out all MSDN library web pages. The sample web page likes
http://msdn.microsoft.com/en-us/library/e7sf90t3.aspx

Open in new window

Can you please look at my code?
Thanks.
string pattern = @"http://msdn.microsoft.com/en-us/library/[a-zA-Z0-9].+.aspx";
Regex r = new Regex(pattern);
foreach (Match match in r.Matches) //not sure how
 {
         Console.WriteLine("Address: {0}", match);
  }

Open in new window

zhshqzycAsked:
Who is Participating?
 
käµfm³d 👽Commented:
Yes, actually I want to download all of the MSDN pages in the MSDN library.

Yikes! I hope your hard drive is big!

In any event, there is a WebClient class in the System.Net namespace that can be used to download web pages. You could scrape the returned page for links using the code above. Here is a quick (and crude) example of how to do so. I can't guarantee you won't run out of memory, though, trying to use the following. I have no clue, ultimately, how big the MSDN library is.

I included a capture group which will grab the name of the file for writing out the file.
using System;
using System.IO;
using System.Net;
using System.Collections.Generic;
using System.Text.RegularExpressions;

namespace ConsoleApplication10
{
    class Program
    {
        static void Main(string[] args)
        {
            WebClient client = new WebClient();     // Used to download pages
            Stack<string> urls = new Stack<string>(new string[] { "http://msdn.microsoft.com/en-us/library/ms123401.aspx" });   // Used to remember where we are in the site tree
            List<string> downloaded = new List<string>();   // Pages link to each other, so we only want to download what hasn't already been downloaded

            while (urls.Count > 0)
            {
                string toDownload = urls.Pop();     // Grab what's on top of the stack

                if (!downloaded.Contains(toDownload))   // Check that we haven't downloaded already
                {
                    string pageData = client.DownloadString(toDownload);    // Download
                    string pattern = @"http://msdn\.microsoft\.com/en-us/library/([a-zA-Z0-9]+\.aspx)";

                    foreach (Match match in System.Text.RegularExpressions.Regex.Matches(pageData, pattern))    // Find all links
                    {
                        Console.WriteLine("Address: {0}", match.Value);
                        urls.Push(match.Value);

                        File.WriteAllText("myLocalFolder\\" + match.Groups[1].Value, pageData); // Groups[1] corresponds to the stuff inside the parentheses in the pattern
                    }
                }
            }
        }
    }
}

Open in new window

0
 
käµfm³d 👽Commented:
Corrected:
string pattern = @"http://msdn\.microsoft\.com/en-us/library/[a-zA-Z0-9]+\.aspx";

Open in new window

0
 
käµfm³d 👽Commented:
Also:
foreach (Match match in r.Matches) //not sure how
{
    Console.WriteLine("Address: {0}", match.Value);
}

Open in new window

0
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

 
zhshqzycAuthor Commented:
I got an error
Foreach cannot operate on a 'method group'. 
Did you intend to invoke the 'method group'?

Open in new window

0
 
zhshqzycAuthor Commented:
I want to grab all web pages.
0
 
käµfm³d 👽Commented:
Ooops! Silly oversight on my part. You need parentheses on the call to Matches(). You also need to pass the string that holds the data:
foreach (Match match in r.Matches(source_data_string))
{
    Console.WriteLine("Address: {0}", match.Value);
}

Open in new window

0
 
käµfm³d 👽Commented:
I want to grab all web pages.
I'm not quite sure what you mean by this. Are you saying you want to extract all the MSDN URLs from a string of data, or are you saying you want to download all of the MSDN pages in the MSDN library? You cannot do the latter with regex alone.
0
 
zhshqzycAuthor Commented:
Yes, actually I want to download all of the MSDN pages in the MSDN library. I know regex is not enough. But I think that I should get all web pages address first. I saw that there is a similar link DownloadString
So in your code
foreach (Match match in r.Matches(source_data_string))

Open in new window

How can I get
source_data_string

Open in new window

0
 
zhshqzycAuthor Commented:
I have another web
bash command
I used the pattern
string pattern = @"([a-zA-Z0-9-]+\.html)";

Open in new window

Is it right?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.