Solved

Spider

Posted on 1998-08-01
11
346 Views
Last Modified: 2013-12-25
Does anyone know where I can get a free script or program
that can list info it finds for a given URL or address?

Eg.
prompt> spider http://where.com/files

and have it index and record what it finds at that address and whatever is underneath it.

It should be for Unix or Linux.

Thanks.
0
Comment
Question by:scottg
11 Comments
 
LVL 1

Expert Comment

by:dagon
ID: 1858793
0
 
LVL 10

Expert Comment

by:MasseyM
ID: 1858794
Search Download.com Winfiles.com or Shareware.com
0
 

Author Comment

by:scottg
ID: 1858795
All those sites are for garbage windows software.  I need a script or program for a real OS.  Sorry.
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 
LVL 2

Expert Comment

by:ndnet
ID: 1858796
0
 

Author Comment

by:scottg
ID: 1858797
Thanks, but they're primarily for index local directories, not crawling web sites.

0
 
LVL 1

Expert Comment

by:evilgreg
ID: 1858798
Is this something similar to the "web-crawling" question in the CGI topic area? If so, it should not be too hard to make a perl script to do what you are asking. You'll also need to provide more information (see my questions under that topic).
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858799
See: http://WWW.InContext.COM
for the InContext "Spider" software.
Not quite $0.00, but under $99.
0
 

Author Comment

by:scottg
ID: 1858800
Thats web authoring software, not a crawler.

Thanks anyway.
0
 
LVL 12

Accepted Solution

by:
Otta earned 50 total points
ID: 1858801
InContext also markets a "crawler".
0
 

Author Comment

by:scottg
ID: 1858802
Thanks,but I can't even find it on the site.   It's probably too much cash anyway.
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858803
The InContext software is named "FlashSite".
That URL contains the text:

| Speed up your Web browsing by using FlashSite,
| winner of the Editor's Choice award from PC Magazine,
| to download all or any part of a Web site
| to your hard drive for customized power-browsing experience.


0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I've been asked to discuss some of the UX activities that I'm using with my team. Here I will share some details about how we approach UX projects.
An enjoyable and seamless user experience can go a long way on an eCommerce site. While a cohesive layout and engaging copy play roles in creating a positive user experience, some sites neglect aspects that seem marginal but in actuality prove very …
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
The viewer will get a basic understanding of what section 508 compliance can entail, learn about skip navigation links, alt text, transcripts, and font size controls.

774 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question