Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Spider

Posted on 1998-08-01
11
Medium Priority
?
431 Views
Last Modified: 2013-12-25
Does anyone know where I can get a free script or program
that can list info it finds for a given URL or address?

Eg.
prompt> spider http://where.com/files

and have it index and record what it finds at that address and whatever is underneath it.

It should be for Unix or Linux.

Thanks.
0
Comment
Question by:scottg
11 Comments
 
LVL 10

Expert Comment

by:MasseyM
ID: 1858794
Search Download.com Winfiles.com or Shareware.com
0
 

Author Comment

by:scottg
ID: 1858795
All those sites are for garbage windows software.  I need a script or program for a real OS.  Sorry.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 2

Expert Comment

by:ndnet
ID: 1858796
0
 

Author Comment

by:scottg
ID: 1858797
Thanks, but they're primarily for index local directories, not crawling web sites.

0
 
LVL 1

Expert Comment

by:evilgreg
ID: 1858798
Is this something similar to the "web-crawling" question in the CGI topic area? If so, it should not be too hard to make a perl script to do what you are asking. You'll also need to provide more information (see my questions under that topic).
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858799
See: http://WWW.InContext.COM
for the InContext "Spider" software.
Not quite $0.00, but under $99.
0
 

Author Comment

by:scottg
ID: 1858800
Thats web authoring software, not a crawler.

Thanks anyway.
0
 
LVL 12

Accepted Solution

by:
Otta earned 100 total points
ID: 1858801
InContext also markets a "crawler".
0
 

Author Comment

by:scottg
ID: 1858802
Thanks,but I can't even find it on the site.   It's probably too much cash anyway.
0
 
LVL 12

Expert Comment

by:Otta
ID: 1858803
The InContext software is named "FlashSite".
That URL contains the text:

| Speed up your Web browsing by using FlashSite,
| winner of the Editor's Choice award from PC Magazine,
| to download all or any part of a Web site
| to your hard drive for customized power-browsing experience.


0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When crafting your “Why Us” page, there are a plethora of pitfalls to avoid. Follow these five tips, and you’ll be well on your way to creating an effective page.
Australian government abolished Visa 457 earlier this April and this article describes how this decision might affect Australian IT scene and IT experts.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to dynamically set the form action using jQuery.
Suggested Courses

916 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question