Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Spider

Posted on 1998-08-01
11
Medium Priority
?
413 Views
Last Modified: 2013-12-25
Does anyone know where I can get a free script or program
that can list info it finds for a given URL or address?

Eg.
prompt> spider http://where.com/files

and have it index and record what it finds at that address and whatever is underneath it.

It should be for Unix or Linux.

Thanks.
0
Comment
Question by:scottg
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
11 Comments
 
LVL 10

Expert Comment

by:MasseyM
ID: 1858794
Search Download.com Winfiles.com or Shareware.com
0
 

Author Comment

by:scottg
ID: 1858795
All those sites are for garbage windows software.  I need a script or program for a real OS.  Sorry.
0
More Than Just A Video Library

Train for your certification. Learn the latest DevOps tools. Grow your skillset to do better work.

At Linux Academy, we release new training modules every week so you'll always be up to date on the latest tech.

 
LVL 2

Expert Comment

by:ndnet
ID: 1858796
0
 

Author Comment

by:scottg
ID: 1858797
Thanks, but they're primarily for index local directories, not crawling web sites.

0
 
LVL 1

Expert Comment

by:evilgreg
ID: 1858798
Is this something similar to the "web-crawling" question in the CGI topic area? If so, it should not be too hard to make a perl script to do what you are asking. You'll also need to provide more information (see my questions under that topic).
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858799
See: http://WWW.InContext.COM
for the InContext "Spider" software.
Not quite $0.00, but under $99.
0
 

Author Comment

by:scottg
ID: 1858800
Thats web authoring software, not a crawler.

Thanks anyway.
0
 
LVL 12

Accepted Solution

by:
Otta earned 100 total points
ID: 1858801
InContext also markets a "crawler".
0
 

Author Comment

by:scottg
ID: 1858802
Thanks,but I can't even find it on the site.   It's probably too much cash anyway.
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858803
The InContext software is named "FlashSite".
That URL contains the text:

| Speed up your Web browsing by using FlashSite,
| winner of the Editor's Choice award from PC Magazine,
| to download all or any part of a Web site
| to your hard drive for customized power-browsing experience.


0

Featured Post

CHALLENGE LAB: Troubleshooting Connectivity Issues

Goal: Fix the connectivity issue in the lab's AWS environment so that you can SSH into the provided EC2 instance.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When the s#!t hits the fan, you don’t have time to look up who’s on call, draft emails, call collaborators, or send text messages. An instant chat window is definitely the way to go, especially one like HipChat. HipChat is a true business app. An…
Without even knowing it, most of us are using web applications on a daily basis.  In fact, Gmail and Yahoo email, Twitter, Facebook, and eBay are used by most of us daily—and they are web applications. We generally confuse these web applications to…
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
The viewer will learn how to count occurrences of each item in an array.
Suggested Courses

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question