Last year Google made more than 520 changes to its search algorithm software and conducted about 58,000 experiments, 7000 of those on live users.
With 200 users acting as Google's guinea pigs at any one time, that means, if you're a Google user, you've probably been involved in the company's quest for the ideal algorithm.
Scott Huffman is Google's head of experimentation, leading a team of engineers who gauge how good the company's search results are.
"We measure the effects of changes and measure if they're really positive or not. Every day we're changing something."
The team constantly monitors huge sets of random searches and uses paid, trained human evaluators in countries around the world who choose the most relevant algorithms among options devised by Google's engineers.
Unlike other products that come to the market with a lag between versions, such as spreadsheet software, Google "can't be turned off for a week" so changes must be made on the fly.
Mr Huffman likes the analogy of a jumbo jet engine being tinkered with at 30,000 feet – continuous upgrades to the search machine while it is bombarded by billions of queries requires a delicate touch, and tweaks must be made without any "collateral damage" to Google's meta-motor.
"Almost any change improves and hurts some searches, but overall, all changes improve searches."
Users don't realise how different search results depend on time and location – for example, a search on "rugby" from New Zealand will give "radically" different results from the same search in the United States.
From Google's Californian headquarters, Mr Huffman types in the keyword "rugby" and his search returns a top result for Ralph Lauren's ultra-preppy Rugby leisurewear brand.
In New Zealand, the top results are Rugby World Cup and Wikipedia's entry on rugby union.
"In the US you don't really get the All Blacks or the World Cup because in the US the sport of rugby is not as popular as the sport of buying things," Mr Huffman says.
When asked why a New Zealand search for a generic English language service such as a phone directory or government department returns Australia's Yellow Pages or Britain's Transport Department, Mr Huffman concedes Google could be doing better.
"That means the algorithms are not quite balanced right."
However, he stresses the response to this sort of bug must be automated because there is "no little room" at Google HQ where he can go to fix the problem physically. "Everything we do has to be completely automated and algorithmic."
Another challenge is dealing with the 16 per cent of all queries that have never been seen before. "We're not making lists by hand of these results, everything we do has to have a general approach that can apply to any query so that we can handle that 16 per cent without looking stupid."
Experiments are also done in live web traffic where change is "unleashed" on real users with software that "flips a coin" deciding if a user will get the new result or the old result in any given search. Google then compares what the users did with the results and moves from there.
Mr Huffman is at pains to point out that Google's search engine is far less static than people think, with answers varying in tandem with the ever-morphing algorithms.
There's also a constant battle to keep "spammy" and low-quality sites out of search results and Google is always trying to make searches faster and more relevant.
The experimentation is ceaseless as mobile phones and tablets take a bigger slice of the giant question mark hanging over the web, and Google is responding to the new search patterns that emerge with advancing gadgetry.
"From year to year, Google's almost a completely different search engine," Mr Huffman says.
A few years ago Google tested this theory, taking 1000 random queries on January 1 and saving the results. They ran the same 1000 questions on December 31 and polled their staff on what percentage of results they thought would come back the same.
Apart from Facebook and a handful of other search results remaining static, Mr Huffman says the outcome was amazing.
"It turns out just about every result was different."
Google Searches Top 5 fastest rising searches in Wellington in the past 7 days: 1. The GC 2. Earthquake 3. Mother's Day 4. TV3 On Demand 5. Yahoo Mail Top 10 searches in Wellington in the past 7 days: 1. Wellington 2. Facebook 3. YouTube 4. Google 5. Trade Me 6. Games 7. Hotmail 8. Stuff.co.nz 9. On Demand 10. Weather Top 5 fastest rising searches in New Zealand in the past 7 days: 1. The GC 2. Cool maths games 3. Geonet 4. Mother's Day 5. TV3 On Demand Top 10 searches in New Zealand in the past 7 days: 1. NZ 2. Facebook 3. YouTube 4. Google 5. Games 6. Trade Me 7. Hotmail 8. Yahoo 9. Westpac 10. ASB --------------------
- © Fairfax NZ News
Do you check your smartphone in bed?Related story: Smartphone etiquette - should you take it to bed?