We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you two Citrix podcasts. Learn about 2020 trends and get answers to your biggest Citrix questions!Listen Now


Site Search Engine

tlajoie asked
Medium Priority
Last Modified: 2013-12-25
I'm looking for a WebSite that provides a Search Engine for Web Sites.  The website should be able to crawl our site, hold the database for our search engine and then we would add the appropriate code to our site for the search engine, which when accessed, would log into the site that has the search engine and the user would get the search results.
Watch Question

Check out Alta-Vista at http://www.altavista.com .  I think they may have what you are looking for.

I may be able to help you with that.  I have a site with my own cgi-bin and have a working search script.  I am not entirely certain if I can get it to search an outside site, but I can certainly try.  Please email all the info about your site to seanrobertson@hotmail.com


Thanks Sean, we can give it a try.  I'll drop you an email.
A good introductory Linux/Unix search engine for you to try would be SWISH.  You can find info and the download at http://www.eit.com/software/swish/swish.html

It takes about 30 minutes to setup and install SWISH.  Then you will need to load one of the suggested CGI gateways or add your own.  This assumes you can install and run software on your site.

NT's IIS of course comes with a search engine.


I'm not looking to run the search engine on our site.  I'm looking for a company that has a search engine on their site that we could code into our site, and their search engine would handle the crawling, indexing and database for our site.  I know I seen a company offering this service last year but I can't seem to track them down now.

Thanks for the help anyway.
The URL http://www.altavista.digital.com/cgi-bin/query?pg=aq&what=web&kl=XX&q=host%3Amysite.com+AND+text%3Afrogs&r=&d0=21%2FMar%2F86&d1=&search.x=37&search.y=1
Will give you all web pages at mysite.com that contain the word "frogs".

Try the advanced help section of http://www.altavista.digital.com/

It should be easy for you to HTML a form to do the front end for this type of query.  You can also have a CGI script do a lynx -dump on the generated URL, saving to a file so that you can use the search results yourself.


How will it work if AltaVista dosen't have our entire site indexed?
Getting AltaVista to index your site is the easy part.  Submit your site at the following URL


Then you can use the process I mentioed above to use the AltaVista engine to do your searches.


Our site has over 40,000 web pages.  Are you saying that by submitting the one URL it will index ALL pages?  
You asked to have a way for a search engine to crawl your site.  That is done by submitting one URL.  If your 40,000 web pages are structured such that they are not connected, in a web, then crawling your site will not be an effective way to index them.

With 40,000 web pages under your control you obviously have the use of, or control of significant server resourcces.  You should consider hosting your own search engine as it takes little extra CPU time and an insignificant cost in hard disk space.


Well, I know I have seen Companies that provide this service and that is what I am trying to find.
My Company provides such a service for free, but it's restricted to Italian sites.

Is this your case?

1. Submit your "home-page" to the AltaVista site's scanner.  It will follow
all the "links", and will thus fetch every page which is accessible.
Use a tool like the "InContext Spider",
at http://WWW.InContext.CA/ to ensure
that all your pages are "linked".

2. To search, submit the search to the AltaVista site, adding the string:


to the user-specified search-string.

Thus, all the "hits" will be restricted
to your WWW-server which also contain
the user-specified string(s).


Thanks Otta.  We have been using the Link Promotion Spider for over a year.  What we are looking for is a service that would allow us to place a search engine on OUR site, so users could search just our site.

Thanks for the reply,
Have a Great New Year

See the URL:
for a comparison among the search-engines
offered by AltaVista, Excite, HotBot, InfoSeek, and Lycos.


Thanks Otta.  I'll take a look at what they have to say.
Basically, these pages list the various companies
which have the technology you want, and, of course,
list how you can license their technology,
and run their software, either on "your" or "their"
platform, depending on whether you want to purchase
the platform, i.e., one "fat" PC, or you want to
use their platform.

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts


Ok, I haven't checked it out yet, but it sure sounds like thats what I'm looking for.  Yet get the points :)

PS.  We will need the engine to be run on a third party site as we have our site hosted and its just not a service they offer at present.

Thanks for the help Otta.


Thanks again Otta.   Currently I'm waiting for AltaVista to reply back to a message I sent them about their VAL program.
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.