How do I make a 404 page crawler

Posted on 2008-11-14
Last Modified: 2012-05-05
I need some direction on how to make a 404 page crawler for internal page development.
I should write to a log file or somthing.

Question by:gvilla23
    LVL 23

    Expert Comment

    I don't quite understand your question.

    Are you asking for a tool that crawls your internal website and finds links to missing pages?

    Author Comment

    yes, its only for internal use.
    LVL 23

    Accepted Solution

    Theres a free tool called xenu which did most of what I wanted:

    There are quite a few tools out there. Try the following google searches:

    link checker
    link varifier
    website link verifier
    website link crawler

    I'm also working on one myself. currently in alpha stage...


    Featured Post

    Maximize Your Threat Intelligence Reporting

    Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

    Join & Write a Comment

    New Relic: Our company recently started researching several products to figure out what were the best ways for us to increase our web page speed and to quickly identify performance problems that we may be having. One of the products we evaluated wa…
    Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
    This video teaches users how to migrate an existing Wordpress website to a new domain.
    Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now