Solved

crawl site to remove http links

Posted on 2014-01-02
3
153 Views
Last Modified: 2014-01-23
Hi,
I am looking for a crawler which will extract all url on a page and place them in a file.  Any recommendations?  Thanks.
0
Comment
Question by:NYGiantsFan
  • 2
3 Comments
 
LVL 52

Assisted Solution

by:Scott Fell, EE MVE
Scott Fell,  EE MVE earned 250 total points
Comment Utility
I used to use this when I had a PC.  http://www.httrack.com/  I don't know something similar for the MAC though.
0
 
LVL 14

Accepted Solution

by:
Giovanni Heward earned 250 total points
Comment Utility
You can download curl and run the following command:

curl http://www.example.com 2>nul|findstr /i "<a"

Open in new window


You can redirect the output to a file like so:

curl http://www.example.com 2>nul|findstr /i "<a" >>links.txt

Open in new window


If you want to clean up that output, try:

for /f tokens^=2^ delims^=^" %l in ('curl http://www.example.com 2^>nul^|findstr /i "<a"') do echo %l >>links.txt

Open in new window

0
 
LVL 14

Expert Comment

by:Giovanni Heward
Comment Utility
If you're looking for a GUI method, try:

http://www.focalmedia.net/urlextract.html
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

A high-level exploration of how our ever-increasing access to information has changed the way we do our jobs.
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…
In this Experts Exchange video Micro Tutorial, I'm going to show how small business owners who use Google Apps can save money by setting up what is called a catch-all email address in their Gmail accounts. By using the catch-all feature, small busin…

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now