I'm looking for a search engine that we can use to index content and links on our portal. Specifically, I'm looking for something that will spider the links on our portal to a certain depth, like, say, 2 layers, and then store that somewhere where we can search it as part of our portal search engine.
Anyone know a tool that can do that? Or a vendor product that's relatively inexpensive (between hundreds of dollars and thousands of dollars)? Is it possible to run google desktop for this?