crawl site to remove http links

Hi,
I am looking for a crawler which will extract all url on a page and place them in a file.  Any recommendations?  Thanks.
NYGiantsFanAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
Giovanni HewardConnect With a Mentor Commented:
You can download curl and run the following command:

curl http://www.example.com 2>nul|findstr /i "<a"

Open in new window


You can redirect the output to a file like so:

curl http://www.example.com 2>nul|findstr /i "<a" >>links.txt

Open in new window


If you want to clean up that output, try:

for /f tokens^=2^ delims^=^" %l in ('curl http://www.example.com 2^>nul^|findstr /i "<a"') do echo %l >>links.txt

Open in new window

0
 
Scott Fell, EE MVEConnect With a Mentor Developer & EE ModeratorCommented:
I used to use this when I had a PC.  http://www.httrack.com/  I don't know something similar for the MAC though.
0
 
Giovanni HewardCommented:
If you're looking for a GUI method, try:

http://www.focalmedia.net/urlextract.html
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.