scraping a web page using System.Net.HttpWebReques causes 403 error

Posted on 2012-09-01
Last Modified: 2012-09-18
hi i am trying to scrape a web page but when i try to run it it falls over at
 objResponse = objRequest.GetResponse();
The remote server returned an error: (403) Forbidden.
any help have been reading up but not sure what to do here maybe a user agent but not sure

private void button2_Click(object sender, EventArgs e)
            string url = "";
            string strResult = "";

     //       WebRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20060728 Firefox/1.5";
            WebResponse objResponse;
            WebRequest objRequest = System.Net.HttpWebRequest.Create(url);

            objResponse = objRequest.GetResponse();

            using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
                strResult = sr.ReadToEnd();
                // Close and clean up the StreamReader

            // Display results to a webpage
     //       Response.Write(strResult);
Question by:sydneyguy
    LVL 35

    Assisted Solution

    by:Miguel Oz

    Author Comment

    its not the problem with pharsing the html it the fact the the web site data cannot be seen and throws up a 403,
    it works for google but not clusty is this the same problem or are we looking at two different prob here
    LVL 17

    Assisted Solution

    Maybe I missed it?

    What is your Referrer?

    Try is as

    Open in new window

    LVL 74

    Assisted Solution

    by:käµfm³d 👽
    It's entirely possible that the host uses cookies or Javascript to ensure that bots (automated  programs) don't access the pages. I would first ensure that you are not violating the site's terms of service.  Then I would suggest using a tool like Fiddler to examine the requests your browser sends to see if such behavior is occurring.
    LVL 17

    Expert Comment

    kaufmed. stated use Fiddler.  I'd listen to him for sure.

    i had actually ran a fiddle earlier for a brief run on
    I did not look deep enough to conclude anything.

    Again  kaufmed stated cookies.

    I had noticed you script did not contain and Cookie jar, referrer or Follow instructions for you bot.

    I do quite a bit a scraping. I'll admit not in  C#.
    But the process is the same.

    You will find some site Require a Referrer from that site.
    Some a cookie.
    Some a Redirect.
    Some 3 redirects 2 cookie and a referrer.
    Some none.

    LVL 10

    Accepted Solution

    Are you behind a proxy/firewall? That error (403) its that you are required to send an authorization header that you did not send, could be basic authentication or not, either way, what you could do to check it, is, logoff / logon at the machine, start Fiddler and navigate the website, if you are behind a http proxy, you should see not only the request to the site, but also a post to the proxy,

    Author Closing Comment

    thanks for all your help

    Featured Post

    What Is Threat Intelligence?

    Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

    Join & Write a Comment

    Does the idea of dealing with bits scare or confuse you? Does it seem like a waste of time in an age where we all have terabytes of storage? If so, you're missing out on one of the core tools in every professional programmer's toolbox. Learn how to …
    Whether you've completed a degree in computer sciences or you're a self-taught programmer, writing your first lines of code in the real world is always a challenge. Here are some of the most common pitfalls for new programmers.
    An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
    Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

    754 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    25 Experts available now in Live!

    Get 1:1 Help Now