Solved

Spider

Posted on 1998-08-01
11
393 Views
Last Modified: 2013-12-25
Does anyone know where I can get a free script or program
that can list info it finds for a given URL or address?

Eg.
prompt> spider http://where.com/files

and have it index and record what it finds at that address and whatever is underneath it.

It should be for Unix or Linux.

Thanks.
0
Comment
Question by:scottg
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
11 Comments
 
LVL 10

Expert Comment

by:MasseyM
ID: 1858794
Search Download.com Winfiles.com or Shareware.com
0
 

Author Comment

by:scottg
ID: 1858795
All those sites are for garbage windows software.  I need a script or program for a real OS.  Sorry.
0
Automating Terraform w Jenkins & AWS CodeCommit

How to configure Jenkins and CodeCommit to allow users to easily create and destroy infrastructure using Terraform code.

 
LVL 2

Expert Comment

by:ndnet
ID: 1858796
0
 

Author Comment

by:scottg
ID: 1858797
Thanks, but they're primarily for index local directories, not crawling web sites.

0
 
LVL 1

Expert Comment

by:evilgreg
ID: 1858798
Is this something similar to the "web-crawling" question in the CGI topic area? If so, it should not be too hard to make a perl script to do what you are asking. You'll also need to provide more information (see my questions under that topic).
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858799
See: http://WWW.InContext.COM
for the InContext "Spider" software.
Not quite $0.00, but under $99.
0
 

Author Comment

by:scottg
ID: 1858800
Thats web authoring software, not a crawler.

Thanks anyway.
0
 
LVL 12

Accepted Solution

by:
Otta earned 50 total points
ID: 1858801
InContext also markets a "crawler".
0
 

Author Comment

by:scottg
ID: 1858802
Thanks,but I can't even find it on the site.   It's probably too much cash anyway.
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858803
The InContext software is named "FlashSite".
That URL contains the text:

| Speed up your Web browsing by using FlashSite,
| winner of the Editor's Choice award from PC Magazine,
| to download all or any part of a Web site
| to your hard drive for customized power-browsing experience.


0

Featured Post

[Live Webinar] The Cloud Skills Gap

As Cloud technologies come of age, business leaders grapple with the impact it has on their team's skills and the gap associated with the use of a cloud platform.

Join experts from 451 Research and Concerto Cloud Services on July 27th where we will examine fact and fiction.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Because your company can’t afford for you to make SEO mistakes, you’ll want to ensure you’re taking the right steps each and every time you post a new piece of content. This list of optimization do’s and don’ts can help you become an SEO wizard.
FAQ pages provide a simple way for you to supply and for customers to find answers to the most common questions about your company. Here are six reasons why your company website should have a FAQ page
The viewer will learn how to count occurrences of each item in an array.
The viewer will get a basic understanding of what section 508 compliance can entail, learn about skip navigation links, alt text, transcripts, and font size controls.

632 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question