webcs
asked on
Telling a spider to crawl a page not on my index page
I would like to tell the search engine spiders to crawl a page that is not indexed in my html paged not linked to directly anywhere in my site.
I have a search engine and the results are all html files however I can't think of a way to let the spider know where they are so it can index them.
Also do spiders index file directory lists. IE if it comes to a directory and there is just a list of files on the server, will it crawl those files?
thanks, Will
I have a search engine and the results are all html files however I can't think of a way to let the spider know where they are so it can index them.
Also do spiders index file directory lists. IE if it comes to a directory and there is just a list of files on the server, will it crawl those files?
thanks, Will
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
sorry there is no way to do that unless there is a inbond link to that file
You could consider placing a spacer.gif with the link there - that will at least get the hyperlink on the page
-Corey
-Corey
Corey -
You're not suggesting a hidden link are you....?
Number one on the list of Google's "Quality Guidelines - Specific recommendations:" is "Avoid hidden text or hidden links".
- duz
You're not suggesting a hidden link are you....?
Number one on the list of Google's "Quality Guidelines - Specific recommendations:" is "Avoid hidden text or hidden links".
- duz
Well worse comes to worse you know. :) The other option woud possibly be to create a sitemap as well.
-Corey
-Corey
webcs,
If your web server allows directory browsing, and you submitted the directory url rather than or in addition to the index.html, then the spider will index the directory.
In such a setup, the web server in effect creates an HTML page that lists the directory contents.
ShalomC
If your web server allows directory browsing, and you submitted the directory url rather than or in addition to the index.html, then the spider will index the directory.
In such a setup, the web server in effect creates an HTML page that lists the directory contents.
ShalomC
You could also do a 'paid submit' of the direct url - it will get independantly listed quicker than doing a 'free' submit.
You could, actually, you SHOULD get a domain name for the page if it is important enough that you want it found! Then optimize the heck out of it for the specific information.
You could, actually, you SHOULD get a domain name for the page if it is important enough that you want it found! Then optimize the heck out of it for the specific information.
ASKER
I did think of that...but will a spider actually index every file in a list like that or just ignore them. Also thought of putting a chatacter like an asterist on the bottom of the page and linking in that way. That essentially would not be hidden text correct?
webcs,
"If your web server allows directory browsing, and you submitted the directory url rather than or in addition to the index.html, then the spider will index the directory.
In such a setup, the web server in effect creates an HTML page that lists the directory contents."
ShalomC
webcs,
"If your web server allows directory browsing, and you submitted the directory url rather than or in addition to the index.html, then the spider will index the directory.
In such a setup, the web server in effect creates an HTML page that lists the directory contents."
ShalomC
Hey,
A spider doesn't care whether the HTML page was created manually or by the web server. If the browser can display it - a spider can read it and will index it.
ShalomC
A spider doesn't care whether the HTML page was created manually or by the web server. If the browser can display it - a spider can read it and will index it.
ShalomC
ASKER
I suppose it wouldnt help the rank or weuight to be found that way though,
The rank is calculated by popularity, or incoming links.
If you have Apache, you can add static headers and footers to the automatic directory index, and include there any descriptive text that will help the search engine calculate your site's relevance.
ShalomC
If you have Apache, you can add static headers and footers to the automatic directory index, and include there any descriptive text that will help the search engine calculate your site's relevance.
ShalomC