Solved

Spider

Posted on 1998-08-01
11
355 Views
Last Modified: 2013-12-25
Does anyone know where I can get a free script or program
that can list info it finds for a given URL or address?

Eg.
prompt> spider http://where.com/files

and have it index and record what it finds at that address and whatever is underneath it.

It should be for Unix or Linux.

Thanks.
0
Comment
Question by:scottg
11 Comments
 
LVL 1

Expert Comment

by:dagon
ID: 1858793
0
 
LVL 10

Expert Comment

by:MasseyM
ID: 1858794
Search Download.com Winfiles.com or Shareware.com
0
 

Author Comment

by:scottg
ID: 1858795
All those sites are for garbage windows software.  I need a script or program for a real OS.  Sorry.
0
Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

 
LVL 2

Expert Comment

by:ndnet
ID: 1858796
0
 

Author Comment

by:scottg
ID: 1858797
Thanks, but they're primarily for index local directories, not crawling web sites.

0
 
LVL 1

Expert Comment

by:evilgreg
ID: 1858798
Is this something similar to the "web-crawling" question in the CGI topic area? If so, it should not be too hard to make a perl script to do what you are asking. You'll also need to provide more information (see my questions under that topic).
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858799
See: http://WWW.InContext.COM
for the InContext "Spider" software.
Not quite $0.00, but under $99.
0
 

Author Comment

by:scottg
ID: 1858800
Thats web authoring software, not a crawler.

Thanks anyway.
0
 
LVL 12

Accepted Solution

by:
Otta earned 50 total points
ID: 1858801
InContext also markets a "crawler".
0
 

Author Comment

by:scottg
ID: 1858802
Thanks,but I can't even find it on the site.   It's probably too much cash anyway.
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858803
The InContext software is named "FlashSite".
That URL contains the text:

| Speed up your Web browsing by using FlashSite,
| winner of the Editor's Choice award from PC Magazine,
| to download all or any part of a Web site
| to your hard drive for customized power-browsing experience.


0

Featured Post

The New “Normal” in Modern Enterprise Operations

DevOps for the modern enterprise offers many benefits — increased agility, productivity, and more, but digital transformation isn’t easy, especially if you’re not addressing the right issues. Register for the webinar to dive into the “new normal” for enterprise modern ops.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
Get to know the ins and outs of building a web-based ERP system for your enterprise. Development timeline, technology, and costs outlined.
This video teaches users how to migrate an existing Wordpress website to a new domain.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question