Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users
Course Of The Month
11 days, 3 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
wget to find broken links on a website
Posted on 2010-08-25
So, as the title says, I'm trying to get a list of broken links. More importantly, though, I'm looking for what sites they're on.
This is what I tried:
wget --spider -r -o /log.txt will get me the list of broken links.
I then used wget -r to download the entire website.
I then had to use "cat log.txt | SED 's/http:\/\/www.website.com\//\//' > brokenhtml.txt " to strip the "http://www.website.com/" from the broken links, so that it matched the HREFs in the html of the downloaded pages
I then used grep -rlf brokenhtml.txt www.website.
Everything up to the last steps produces the expected results. When I cat the log.txt and randomly paste the URL into a browser window, it correctly gives me a 404. When I do the last step to find the pages that the HREFs are in, I get pages with no broken links on them.
This is what I tried:
wget --spider -r -o /log.txt will get me the list of broken links.
I then used wget -r to download the entire website.
I then had to use "cat log.txt | SED 's/http:\/\/www.website.com\//\//' > brokenhtml.txt " to strip the "http://www.website.com/" from the broken links, so that it matched the HREFs in the html of the downloaded pages
I then used grep -rlf brokenhtml.txt www.website.
Everything up to the last steps produces the expected results. When I cat the log.txt and randomly paste the URL into a browser window, it correctly gives me a 404. When I do the last step to find the pages that the HREFs are in, I get pages with no broken links on them.
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
2
4 Comments
You grep should probably be an iteration through the contents of brokenhtml.txt.... I.E.
while read line
do
grep -rlf $line www.website.
done <brokenhtml.txt
or...
for f in `cat brokenhtml.txt`
do
grep -rlf $f www.website.
don
or something to that effect.
while read line
do
grep -rlf $line www.website.
done <brokenhtml.txt
or...
for f in `cat brokenhtml.txt`
do
grep -rlf $f www.website.
don
or something to that effect.
apparently grep works better when you separate the parameters with spaces.
Current command that produces the desired results (thus far, it's still running):
grep -r -i -l -f b-links.txt www.website.com
the -f command allows you to use a file for matching patterns, one line per pattern. I did also strip out a leading blank line from the pattern file... perhaps that was also breaking it.
Current command that produces the desired results (thus far, it's still running):
grep -r -i -l -f b-links.txt www.website.com
the -f command allows you to use a file for matching patterns, one line per pattern. I did also strip out a leading blank line from the pattern file... perhaps that was also breaking it.
sed -n '/www.website.com/,$ p' brokenhtml.txt > <out_put_file>
Featured Post
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
The purpose of this article is to show how we can create Linux Mint virtual machine using Oracle Virtual Box. To install Linux Mint we have to download the ISO file from its website i.e. http://www.linuxmint.com. Once you open the link you will see …
The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
Learn several ways to interact with files and get file information from the bash shell.
ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses
Earn Certification
HTML5 Specialist - Certification
Free withPremium
Course of the Month11 days, 3 hours left to enroll
628 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.