Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


how to create a meta search engine

Posted on 2014-03-24
Medium Priority
Last Modified: 2014-11-12
How do I create a meta search engine getting results from (1)google custom search (2) bing api via azure datamarket and (3) yahoo boss using php and curl and json format for results? I already have the api keys from all three search engines. I want to have the results cached for a time period but I don't know a lot about php programming but I can follow directions. I seen a code on this site about it but not sure how it pertains to my project. I hope I can be helped threw this project.
Question by:Domcast8
LVL 54

Accepted Solution

Scott Fell,  EE MVE earned 2000 total points
ID: 39952244
I think it will help to break this request out in small parts.  First, by creating a meta search engine, are you trying to simultaneously take a user search term and look up both Google and Bing?  

If so, you only need Google and only one of yahoo or bing since bing powers yahoo search.

Since each has their own api, it might be a good idea to first make a custom search for each of the search api's you have.  For instance, google has a php library for this.  https://developers.google.com/custom-search/json-api/v1/libraries   Use the library just for google and figure out how it works and the different options.

Bing also has a php sample ready for you.  I don't guarantee this link will last forever.  And I found a php library for boss https://code.google.com/p/yahoo-boss-php/wiki/HowToUse

This is not a small project and you should first be able to create 3 separate (or just 2) php pages for each of the different search api's.  This ensures you understand the concepts and can move on to the next step which will be combining the search's to one view.

Lastly, you want to create a cache somehow.  Do you mean store search's on your server?

I would start by getting your google search working.  Use the php library, if you have troubles with that, post a new question just on using google search api and the php library.  Make sure you give it a try first before asking for help.  This is not a beginners request and will take some time on your part.  

Once you have google going, repeat for yahoo and bing.  At that point, if you are unsure how to merge all the results together, you can start another question on that topic.

The hard part of all of this will be mashing the results together and that will take a bit of computer science.  If you are not sure about creating the php libraries for the individual api's, mashing this together may be a bit of a stretch.

Dogpile is already doing this http://www.dogpile.com/.  I don't think they get much play.  One of the  newest search engines is duckduckgo https://duck.co/duckduckhack/ddh-intro

Personally, I think you should stick with one search engine.  Dogpile has been doing this for a long time and I don't find much use for them.  Google and Bing really have this covered.  I know from my own websites, of all search generated traffic, Google represents 70% to 80% and Bing/Yahoo represent the difference where Bing brings in a little more than yahoo.
LVL 111

Expert Comment

by:Ray Paseur
ID: 39953036
I especially agree about the Dogpile comment.  This could be a fun academic project, but it is a really useless activity in terms of achieving business success.  For an object lesson look how much (or how little) money DogPile is worth today after years of doing exactly what you've described.  And where are Alta Vista, Lycos, etc.?  They all got subsumed by Google.  Bing would not exist at all unless Microsoft was supporting it; it would fail if it had to live or die on its own P&L statement.

If you're new to PHP and want some good learning resources to jump-start your work, this article will help.  I agree with Scott, that you need to break the project down into smaller bites that can be solved individually.  A good guide for asking questions is the SSCCE.

Author Closing Comment

ID: 39953337
Thank you for you response. This does help me one step at a time. This is a long project but I believe I can do it with the help of experts exchange. I will first try to set up the search engines separately and then go from there.
I've seen other meta search engine scripts like inoutscripts that do cache results for a time period on a server. They say its to become more independent in the future. I would like to do it for that reason and more.
Anyway, thanks for your quick reply helping me with me project.
I will start on what you said and I will get back with you with any other questions that may come up.
Thanks Experts-exchange.com. Day 1 and your already helping me get somewhere.

Featured Post

NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

As managed cloud service providers, we often get asked to intervene when cloud deployments go awry. Attracted by apparent ease-of-use, flexibility and low computing costs, companies quickly adopt leading public cloud platforms such as Amazon Web Ser…
Herein one will find an aggregate of some of my experience building and deploying virtualization stacks both in standalone, clustered Hyper-V, clustered Hyper-V with a Scale-Out File Server (SOFS) backend, and Storage Spaces Direct (S2D).
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The viewer will learn the basics of jQuery including how to code hide show and toggles. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery…
Suggested Courses

580 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question