Bing vs Google vs Yahoo: the Blind search engine test

Blind search engine test

A couple of months ago, Microsoft conducted a research study where they found search engine branding had an impact on the perceived quality of results. Now, months later, Bing is launched and people are once again ranking the quality of search results. At first, there were direct side-by-side comparisons, but of course there’s no better challenge than a double-blind experiment.

Michael Kordahi, which I feel compelled to disclose is a Microsoft employee, has set up such an experiment, dubbed “BlindSearch” which you can try here. On the site, you enter a single search query which results three sets of unbranded results. After viewing the results, you are given an option to vote on one set of results, which then reveals the provider and adds it to the tally.

At the time of writing, Google is in the lead with the tally standing at “Google: 40%, Bing: 31%, Yahoo: 29%” out of 355 votes.

Update: Approximately half an hour after posting, Bing takes a slight lead over Google. (Google: 36%, Bing: 38%, Yahoo: 26% | 641 votes)

Update 2: A thousand votes later, Google and Bing are tied. (Google: 39%, Bing: 39%, Yahoo: 22% | 1757 votes)

Update 3: Three’s the magic number, for Google. (Google: 40%, Bing: 37%, Yahoo: 23% | 2554 votes)

Update 4: Google is taking a stronger lead with another thousand votes. (Google: 43%, Bing: 32%, Yahoo: 25% | 3520 votes)

Update 5: It appears the application has exceed Yahoo API’s daily limit. Michael is aware of the problem and looking for a workaround. Yahoo results has since been fixed.

Update 6: Some people’s attempt at gaming the system has forced the tallies to be now removed.

110 insightful thoughts

  1. That is awesome. Sometimes it’s (well, so far at least) hard to tell who’s who… more than usual.
    I accidently voted for Yahoo. The shame.

    1. Google is good,But 1 Problem,they know everything you type even if you dont click on what you type in the address bar,dont take my word for it,You will be worred,Bing is better more sucure and getting faster every day.Right Now Ping at 5:31 google is Average=24ms thats preaty good.Now lets try bing,
      Average is 36ms at 5:35 am,google is faster at this time but not everytime,Bing wont stop till they are in the lead on everything.

    2. well, i think google should win bing just takes you to the site you want but google can take it to any site you want and it can search things for you so I disagree with whoever says bing is better.

  2. Bing is closing in on Google now: 37 vs 35%. I’ve personally always found Live Search to be better in most situations.

  3. @hello I haven’t as much as those around me would like to call me a loyal MSfanboy >.<

    The statistics don’t really mean much yet… Wait until it gets to like… 10,000.

    1. I don’t really consider this to be accurate at all. For my first search, Google returned results in a different language, which never happens to me normally. Yahoo’s search wasn’t even available, and Bing’s was the only one that worked normally. As far as we know, Yahoo’s search could be best.

    2. i like Google and Bing which one should i pick. because bing is in the lead and ggole i have been using sice i was little

  4. Thats really quite interesting… I just picked some keywords and run the test five times and it appears I picked Bing 3 out of 5 times!

    When I was just there, there had been 715 votes with Bing on 38% and Google on 35%.

  5. Very interesting test. The truth of the matter is, without the branding and specific engine interfaces, I really couldn’t tell much difference between them.

    Every search I performed revealed roughly the same 10 results just in different order of ranking. I usually voted for the one whose ranking order was most diverse (so that duplicate sites weren’t together). My results were always for Bing or Google with that approach, and with about the same number of votes each.

    I tested using simple one-word searches, specific fully worded questions, and with complex strings like:

    “Active Directory” “PowerShell” Create User Account

    I’m very impressed by the test in the sense that branding definitely has an impact, but so does the usability of the site, and quite frankly, Bing’s interface is fantastic. Now that I’ve seen the results are exceptionally comparable to Google’s, I feel very comfortable setting Bing as my default search engines… err, decision maker! 😉

  6. I tried the same keyword search on Blind Search and on Google Search. The result from Google as shown on Blind Search page is different from the result on Google Search. I’m doing the search on Google.com (not the local version). This is kind of interesting!

  7. Bing is wining 41 to 36 to 23 right now. I’ve discovered that its pretty easy to find Bing, Wikipedia is generally 2nd or 3rd

  8. @Justin and Tommo,

    I just did the exact same type of test search and couldn’t duplicate Justin’s results.

  9. @Justin: That is typical of Google. Due to their infrastructure, results vary with region and which datacenter served your query. The Google API, which this uses, would explain the difference between search results if you searched vanilla Google.

  10. Hmm, I notice if you can cheat if you look at the page source AFTER searching… the result images are embedded into the code and hidden by CSS styles. Unfortunately, the images are named with the provider names instead of randomized strings.

  11. LOL, no no… I didn’t for any of the searches I did. I just took a peak at the code to see if it was possible once I saw Christo’s comment about easy to identify Bing.

  12. The more I think about it… there should be multiple, randomized filenames for each engine, and the results page should randomize which graphics are used.

  13. I found Bing better for results concerning people (it links to their blogs, profiles, and stuff better) — but when searching for computer terms, Google comes out better.

    Interesting experiment :)

  14. @GoodThings2Life sure it’s possible to cheat. actually rather than view source, you could open another browser window and do a parallel search (most of the time)

    I reckon the cheaters are very much fewer in number than legit searches so they kinda become irrelevant.

    Moral of the story, no one likes a cheat :-)

    Thanks all for your positive comments around this little experiment.

  15. @Michael,

    Very true, although my thought was more along the lines of the ones realizing they can purposely skew your results.

    I love the experiment though, and it is definitely revealing! :)

  16. The only problem with this is that some of the best parts of bing search (like instant results/sidebar results) don’t appear.

  17. This is an awesome idea…and I applaud the guy for making this…and my results are tied about 50/50 for Google/Bing…

    But I’ll tell you why I won’t switch to Bing:

    Terrible name, not as in-depth “beyond first page” results and likely terrible support in the future (how long before MS ditches this brand?)

    I loved Live Search when it first came out…….but MS showed no initiative (at least not up to Google levels) when it came to analytics, custom search site integration, advertising, etc. the way Google does. Google is a one-stop shop with a cohesive brand that screams “quality”….Bing, on the other hand? It says nothing.

    Like I said too — the results BEYOND the first page are what truly matters. Sure, top 10 links are great and all…but for people searching for REAL information, we nee dto know that there’s a ton of relevant links in the top 50 or 100 links. That’s where Google (and even Yahoo) wins. Plus, Google has been indexing sites for ages, which means I can easily find cached versions of old, old pages (that maybe Internet Archive doesn’t even have) with relative ease.

    Live Search/Bing? Not so much. Info from the past couple of years…sure. But say, before 2004/2005? Good luck finding lots of links for that content.

    I’m a huuuuuuuuge MS fanboy, but I have no desire whatsoever to switch back from Google. They should have just stuck with Live Search and upgraded that. :/

  18. Interesting, but isn’t Bing’s big selling point it’s filtering UI?
    At the moment, that’s the reason why I use it over Google. Actual hits don’t mean that much – sometimes Google is better sometimes it isn’t. Filtering saves much more time – especially with words with more than one meaning.

  19. This would be more interesting if the author removed the “score” at the top of the page. It’s just going to be a stupid magnet for gaming rather than an opportunity for people to “let the scales fall from their eyes”.

    Apart from that (and the technical tweaks suggested earlier) it’s a great idea.

  20. Like Sam & Joe mention, this test misses off the main point of Bing, namely the ability to filter and refine the results. The ‘raw’ results are still important but the MUCH better UI on Bing makes all the difference. Otherwise its just a google clone.

  21. I would actually like *this* to be my default search site. Basically a site that will display results from the 3 major search engines.

    I hope someone can improve on this idea and add in additional features, especially a way to see more results.

  22. Hi Long,

    Interesting post. It’s true that our preconceptions alter our ability to assess accurately.

    As any good cook will confirm, “The first bite is with the eye.” If we think that food will taste great, we will enjoy it more than if it looks less appealing but is equally delicious.

  23. @Imran How do you know those figures wouldn’t have been the same in a comparison between Google and Live Search?

    Percentages are great and all, but don’t tell the entire story. I do feel that Bing really taps into an entirely separate segment of the market, almost drawing the lines in the existing search market and picking out the group of individuals who want simple answers like what item to buy or when to buy a plane ticket. It then becomes obvious that another segment of the search market, you have those who query just for results. Whenever I use google, I wholeheartedly welcome not just relevant search results but also the junk results. I find that having both allows me a greater array of perspectives on the topic I’m searching.

    With the existence of multiple search behaviors, it seems weird to me to just lump everyone into the same category and ask them to vote. It might be more statistically significant to map the votes to the specific queries so you can really see which engine excels in each particular search type.

  24. Yahoo gained 11 percent in one hour against bing and 15 percent against google in one hour.

    Blows away both Google and Bing within one hour after being bef?

    Something funny going on. I think some sort of cheating is going. To much movement in such a short space

  25. What a brilliant effort. I wish if this goes down any time, some one else (MS themselves?) will resurrect this forever.

  26. I’ve been doing most of my searches using this for the past couple days. In my own usage, I’m getting (about) the following result:

    Google – 55%
    Bing – 40%
    Yahoo – 5%

    For general purpose stuff, Bing and Google were very comparable my choice to pick one over the other in this blind test was essentially a “coin-flip” For technical searches, especially engineering research, Google’s results were usually the best. In every case however — Yahoo stood out clearly as the worst of the three…

    One thing I noticed is that Google is that it will often ask you “did you mean to search for —-” if it thinks I made a typo. Yahoo and Bing on the other hand simply ASSUME I made a typo, and I have to explicitly click a link to see the results I actually asked for. For example a few queries of mine containing the word “silicon” were replaced with the word “silicone” automatically…

    If I actually make a typo, I don’t mind clicking an extra link. But (more often) when I’m searching for an obscure topic, I’d rather not be fowarded to one that has nothing to do with my query just because changing one character results in more results…

  27. Simple reason Bing will never be used as much as Google – with Google I go to the upper right hand side of my Firefox browser and type in the search. With Bing, I would have to go to the site and type in my search… Google wins on ease of use hands down.

  28. The douche is probably using the thumbnail image in the source to determine which column is which engine before voting. It’s pretty clear in the source code that YHO.png is Yahoo, MSF.png is Bing, and GOO.png is Google. You might want to figure out a way to obscure those thumbnail images, or throw them out altogether.

  29. Blindsearch got some friends and I talking.

    Said resident genius friend, Abram: “I dunno about you guys, but I’m just going to reset my default search provider to be “blindsearch.fejus.com”. It gives me the top search results from all three major search providers in a simple, spam-free, ad-free format. Woo hoo!”

    I responded: “It would be cool if there was a Firefox add-on that lets you search blindsearch.fejus.com from the Firefox search bar…”

    After a few minutes of searching and noodling I was able to whip one up. Here it is:
    http://drinkspiller.cannonballinteractive.com/temp/add_blindsearch.html

  30. @Timothy: Your comment on using search from Firefox: Google paid a big fee to Mozilla for the rights to have Google as the default Mozilla search for most versions of Mozilla (except for some Asian versions that use Yahoo because Yahoo paid a fee for that exclusivity). Few people realize that this arrangement of defaulting to Google search in Firefox, coupled with Firefoxes increasing popularity, is one of the driving factors behind Google’s growth and dominance.

    Bottom line: people don’t just use Google because it gives good results, they use it because of a multitude of factors, including prominent placement in Firefox and because it is a habit.

    I tried the Blindsearch 20 times, and found that I chose Bing 9 of those times, followed by Yahoo at 7,and Google at 4 times.

    I’ll try it again to see if I get different results.

  31. @no punfs: Interesting note about Mozilla and Google. I did not know this.

    One thing I can say is that Microsoft is doing what it knows best: throwing money at problems. Bing is not the “next generation” of search, it is simply a re-branded Live. I doubt that Microsofts mammoth marketing campaign will pay-off in the long term.

  32. I have to say, I am surprised, very surprised. In my quick test, I searched for so called “backups”. Bing nailed the two sites I ended up using after 30 minute google searching, and both were as it’s first and second link, in the order that I preferred (the best one – first, the second best – second). I wish I knew this yesterday, it would have saved me a lot of searching when I was trying to find “backups” of my favourite Russian musician. Way to go, Microsoft. Now, I just have to read your EULA.

  33. Lol, this is hilarious! I tried to find the EULA of bing, by searching google, of course. This is the exact query: http://www.google.ca/search?hl=en&q=bing+EULA&btnG=Google+Search&meta=&aq=f&oq= .

    I searched for two words: bing EULA . I found it! err… her!

    http://cid-dd066d21917bfdcc.profile.live.com/

    There is, indeed, a person named Eula Bing. I though this is just a really cool coincidence. Seeing as how she is on Microsoft Live, I wonder what Microsoft will, or *can* do.

    And no, I did not find bing’s EULA yet.

  34. If you are trying to find something and don’t get the right results in Google then try Bing coz it does return the results it does return the results your are looking for and it also works the other way around. But if you are searching for Videos then the best thing to use will be Bing as it has always returned better results than Google which returns results mostly from its very own Youtube and also you get to play the videos without having to click on them to know their contents. I have got so used to typing Google to search that I have to consciously make an effort to type Bing. So way to go…….

  35. I find it interesting, that of all the search engines, yahoo is the only search engine that provides it’s own url in the top three results when completing a search on search engines. Google (Dogpile, AltaVista, Bing), Cuil (Search engine history, web search engine, regular columns: search engines), Bing (images of search engines, dogpile, ask.com), AltaVista (search engine, monster crawler, people search). One would think, they would use this opportunity to reinforce their own url.

  36. Interesting site. My work involves lots of search and I always google. I am not able to take the test honestly since in most cases the results give sufficient clue regarding which one google, few of them are

    1. Few of the results start with date
    2. Wikipedia is given highest ranking
    3. for ‘How can I” kind of queries ‘Yahoo answers’ have highest ranking (yeah, yahoo search have lower ranking for ‘yahoo answers’ than google search)
    4. The famous websites / companies first result of google has different structure (it lists important links within the website)

    Anyways even if Bing results are a tad better, I am too much habituated to google to switch :)

  37. typing “google.cmo” on the address bar of internet explorer and bing comes out but the results does not show any sign of google…first preview….this bing is baaaaaad.

    i’m stiking with google for now and backing up with yahoo/altavista.

  38. Even if Bing was the next best thing since sliced bread (in terms of search engine technology), people are so loyal to the Google brand, that it will be very unlikely that Bing will make any kind of significant dent.

  39. Um….. i just discovered something. Do you see those ads on the side?!

    Err…. it says “Ads by google”—What does this mean?—-IT MEANS THIS PAGE IS OWNED BY FUCKING GOOGLE!

  40. GOOGLE KEEPS CENCORING THE TRUTH! I TRY TO ADD A COMMENT ABOUT THE TRUTH AND IT DOSN’T GET ON!

    THIS PAGE HAS GOOGLE ADS WHICH MEANS IT’S OWNED BY GOOGLE PEOPLE!

  41. @wtf?!: Ever heard of Google Adsense? Do you know a heck of a thing about computers? This is not a Google owned site. Or so they say. *rolls eyes*.

  42. Wow, you all have been duped. This test says you get results from all three searches but you really don’t. Type in “How big is the sun” in the test. Hold the results. Then type the same thing in a real Google page, and compare. You will see that none of the results from the first test matches up at all to the real Google page. You all been dubed by a old trick.

  43. Blindsearch favors Bing. For example, I wrote ‘salmon’ and clicked on the column that seemed more informative and Bing it was! But the column that corresponds to yahoo didn’t have anything but Google had! So I checked Yahoo website and wrote ‘salmon’ and there were some pages with information on salmon. So, could there be an agenda here to entertain some people who are not thinking or what?

  44. BROS WTF THIS TEST IS RIGGED AND BIASED U ALL R DUMB PEOPLE WE ALL KNOW GOOGLE IS BETTER!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  45. I feel Google has the best search engine. They are so clever, they know everything. It is really amazing. Their results are very accurate, clear and easy to read. Their searches are awesome. Nobody can ever beat Google.

    1. Yes, I am right the Google search is the greatest. Yahoo is second. I have not tried Bing. I spend a lot of of time with Google search, they are super. Kudos.

Leave a Reply