Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.
One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.
Solved
wget to find broken links on a website
Posted on 2010-08-25
So, as the title says, I'm trying to get a list of broken links. More importantly, though, I'm looking for what sites they're on.
This is what I tried:
wget --spider -r -o /log.txt will get me the list of broken links.
I then used wget -r to download the entire website.
I then had to use "cat log.txt | SED 's/http:\/\/www.website.com\//\//' > brokenhtml.txt " to strip the "http://www.website.com/" from the broken links, so that it matched the HREFs in the html of the downloaded pages
I then used grep -rlf brokenhtml.txt www.website.
Everything up to the last steps produces the expected results. When I cat the log.txt and randomly paste the URL into a browser window, it correctly gives me a 404. When I do the last step to find the pages that the HREFs are in, I get pages with no broken links on them.
This is what I tried:
wget --spider -r -o /log.txt will get me the list of broken links.
I then used wget -r to download the entire website.
I then had to use "cat log.txt | SED 's/http:\/\/www.website.com\//\//' > brokenhtml.txt " to strip the "http://www.website.com/" from the broken links, so that it matched the HREFs in the html of the downloaded pages
I then used grep -rlf brokenhtml.txt www.website.
Everything up to the last steps produces the expected results. When I cat the log.txt and randomly paste the URL into a browser window, it correctly gives me a 404. When I do the last step to find the pages that the HREFs are in, I get pages with no broken links on them.
2
4 Comments
You grep should probably be an iteration through the contents of brokenhtml.txt.... I.E.
while read line
do
grep -rlf $line www.website.
done <brokenhtml.txt
or...
for f in `cat brokenhtml.txt`
do
grep -rlf $f www.website.
don
or something to that effect.
while read line
do
grep -rlf $line www.website.
done <brokenhtml.txt
or...
for f in `cat brokenhtml.txt`
do
grep -rlf $f www.website.
don
or something to that effect.
apparently grep works better when you separate the parameters with spaces.
Current command that produces the desired results (thus far, it's still running):
grep -r -i -l -f b-links.txt www.website.com
the -f command allows you to use a file for matching patterns, one line per pattern. I did also strip out a leading blank line from the pattern file... perhaps that was also breaking it.
Current command that produces the desired results (thus far, it's still running):
grep -r -i -l -f b-links.txt www.website.com
the -f command allows you to use a file for matching patterns, one line per pattern. I did also strip out a leading blank line from the pattern file... perhaps that was also breaking it.
sed -n '/www.website.com/,$ p' brokenhtml.txt > <out_put_file>
Featured Post
Question has a verified solution.
Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.
Have a better answer? Share it in a comment.
Join & Write a Comment Already a member? Login.
The title says it all. Writing any type of PHP Application or API code that provides high throughput, while under a heavy load, seems to be an arcane art form (Black Magic). This article aims to provide some general guidelines for producing this typ…
Often times it's very very easy to extend a volume on a Linux instance in AWS, but impossible to shrink it. I wanted to contribute to the experts-exchange community a way of providing a procedure that works on an AWS instance. It can also be used on…
Learn how to get help with Linux/Unix bash shell commands.
Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses
Course of the Month6 days, 1 hour left to enroll
568 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.