avarma
asked on
URL Submission to multiple search engines
Is anyone familiar with those sites on the web that charge you X amount of dollars for submitting your URL to a 100 (or whatever)
search-engines? Example - worldsubmit.com
My question is: Do you know how they do this ? I am assuming they use a CGI interface for each search-engine - and have their own script that simply executes the CGI for each of the 100 search engines.
I am really looking for a script that will let me submit a URL
to multiple search engines. I have heard of some off-the-shelf
software that lets you do this - but am really more interested
in something that I can interface to my web-application - like
a CGI script.
Thanks in advance
Anuj Varma (avarma@adversite.com)
search-engines? Example - worldsubmit.com
My question is: Do you know how they do this ? I am assuming they use a CGI interface for each search-engine - and have their own script that simply executes the CGI for each of the 100 search engines.
I am really looking for a script that will let me submit a URL
to multiple search engines. I have heard of some off-the-shelf
software that lets you do this - but am really more interested
in something that I can interface to my web-application - like
a CGI script.
Thanks in advance
Anuj Varma (avarma@adversite.com)
to be fair - i dont think it's really worth registering to 200 search engines - but 20-30 yes..
...Register-It! will do 10-15 or so of the most popular ones in one gor for free... ...i tihnk they're at www.register-it.com
there are a coupla other free ones out there but i dunno where
i think they just basically fake a response from the form on each search engine's site - the "Add My site" form, but with a different one for each engine (cos they all ask for slightly different stuff)
...Register-It! will do 10-15 or so of the most popular ones in one gor for free... ...i tihnk they're at www.register-it.com
there are a coupla other free ones out there but i dunno where
i think they just basically fake a response from the form on each search engine's site - the "Add My site" form, but with a different one for each engine (cos they all ask for slightly different stuff)
How THEY do it I'm not sure. How I'd do it is relatively simple. You start a cgi program which starts several threads. Each thread connects to a search engine by posting the query. (You'd need to know the exact format of the query for each server). The thread reads the response and, in simple cases, writes the data it got to the cgi output (ie: std_out). The thread then proceeds with the next engine until no engines are left. This executes the query in parallel for a fast response (the response is the slowest of the engines, assuming one thread per engine) (More that 10 threads is not really on).
A more sophisticated approach is to parse the html reply from the engines separating out the answers from the advertising and other junk and merging the results into one result. This is made easier if you can agree on a search interface with the provider of the search engine.
A more sophisticated approach is to parse the html reply from the engines separating out the answers from the advertising and other junk and merging the results into one result. This is made easier if you can agree on a search interface with the provider of the search engine.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER