Big Data is watching you
Hal the computer in 2001: A Space Odyssey is ranked as one of the great villains of the screen - but his computer cousins could now be helping decide if you get a discount in a shop or get out of prison.
Some are worried that like HAL, there could be a dangerous streak in the new world of "Big Data"- and it could be costing you money, the chance of a job or even finger you as a terrorist threat.
Cheap computer processors and sophisticated mathematical software mean more and more decisions that were once left to human judgment are now computer-assisted.
In New Zealand, "business analytics" software helps decide whether prisoners are paroled, whether financial transactions are flagged as fraudulent and whether businesses offer wavering customers "special offers" to secure their loyalty.
United States analyst and blogger Alistair Croll raised eyebrows this month when he said "Big Data" - an industry buzzword that describes the use of such software to glean insights from vast amounts of data - was "our generation's biggest civil rights issue and we don't know it".
The changing economics of information technology mean we can now collect information long before we decide what it's for, and that is a dangerous thing, Croll argued.
A frequently cited example of Big Data gone bad was when American Express cut the credit limit of Atlanta public relations executive Kevin Johnson by two-thirds, explaining that other customers who shopped where he shopped had proved to have a poor record of making repayments.
If nothing else, the greater use of business analytics software spells bad news for time-poor consumers who may be reluctant to change their telco, bank or electricity company and who instead rely on the movement of the herd to ensure they benefit from the forces of competition.
The new trend towards "one-on-one marketing", whereby companies analyse Big Data so they can identify and pick off individuals who are likely to "churn" by giving them special offers, means the sheep among us are more likely to be left in a field of less-green grass with no special deals coming our way.
That worries Telecommunications Users Association chief executive Paul Brislen, who has said such tactics can make pro-competition tools, such as websites that allow people to compare prices from utilities, trickier to operate and less valuable.
In another development that tilts the playing field between consumers and businesses, supermarkets are analysing the Big Data captured at their checkouts to help work out when and by how much to discount their products to maximise profits.
There is nothing new about big corporations and government agencies harnessing computer technology to analyse and predict people's behaviour and segment customers.
But what is rapidly changing is the scale and efficiency with which that is happening, and the surprising range of circumstances in which business analytics software is helping make what were previously purely judgment calls.
The Corrections Department first developed a computer system, RocRol, in 1995 that calculates the chances of prisoners being reconvicted within five years of their release. It works by analysing 14 variables, including the offender's age at first imprisonment, their number of convictions in the previous five years, their association with gangs, and drug use.
Corrections analyst Arul Nadesu told a conference at Te Papa in February that new software developed by business analytics firm SAS that incorporates neural networking technology - a technique for processing data that mimics the way signals are passed between neurons in the brain - could reduce the risk of RocRol "misclassifying" an offender to just one in seven.
RocRol is used to create pre-sentencing reports and determine the security level of prisons that offenders are sent to and whether they get access to rehabilitation programmes.
Corrections' manager of strategic analysis and research, Peter Johnston, says RocRol's scores are also one factor, among many, that parole boards can consider when deciding whether to free offenders from jail early.
OTHER organisations such as ACC and Child, Youth and Family could also reap benefits if they could draw up predictive models that let them forecast with a high degree of accuracy who would return to their care, Nadesu said.
What's driving the increased use of business analytics and Big Data is not just smarter software and the continued commoditisation of powerful computer hardware needed to process it, but also the emergence of a new breed of professional experienced in putting the two together.
Asian Cloud Computing Association chairman Par Botes, a vice president of US computer storage giant EMC, says the tighter regulations governing automated trading on financial markets that were imposed following the global economic crisis have given the sector a big boost.
They meant many bright mathematicians employed on Wall Street to predict stock movements had to look for alternative employment, fuelling the Big Data boom. "Those people haven't disappeared. They are applying their skills to new problems," he says.
On a visit to Wellington, Jim Davis, the vice president and chief marketing officer of SAS - the world's largest privately-owned software firm - acknowledged analytics could be a double-edged sword.
It had helped unravel the human genome and could help work out what medicines to give individuals, but the same information, if used inappropriately, could create extreme discrimination, he said. "If I am an employer and I see someone has a propensity for serious illnesses I may not want to employ them.
"The question is ‘Do the benefits outweigh the risks?' There is some social responsibility there not to step over the line and use data and analytics for the wrong reason and as we get into the area of Big Data and technologies that allow us to process more data, the risk is there."
Davis says "99 per cent of the time" he has been comfortable with the way SAS tools have been used. Ethics are very high in governments' minds, he says. "It is our job to work with organisations and customers to help them understand [how] analytics can be used to customise the user experience so people are served better, so their junk mail is reduced and the offers presented to them are relevant based on their lifestyle.
"This is the balance between the good things with analytics and making sure people don't abuse it. People should be called out on the abuse."
But that is only possible if people know what is being done. Davis says SAS software is being used in ways "it can't talk about" in government "fusion centres" that bring together data from multiple law enforcement agencies.
"A lot of governments are sharing that information in a central location so they can get more of a 360-degree view of the citizen to determine what the ‘total threat' is around each individual," he says.
The Social Development Ministry is looking at applying SAS' software in its programme of social welfare reform, but it declines to say how, or even to outline what problems it is seeking to solve.
EMC's Botes says one example of where Big Data has been used to positive effect is in the health sector. One US health provider has managed to accurately predict how many hospital beds it needs three months ahead by analysing data on admissions and variables such as weather forecasts, he says.
New Zealand has its own software exporter in that field, Christchurch company Emendo, which has sold its demand-forecasting software in multimillion-dollar deals to hospitals in Britain, Canada, Australia and New Zealand.
Even here there is arguably a flipside. Accurate forecasting could lead to super-efficient rostering that provides no slack for already hard-working staff. But Emendo co-founder Nick Burns counters that using analytics to "resource to the correct volumes" is safer and less stressful for staff when demand for medical services peaks.
Botes is a self-confessed geek who believes that thanks in part to Big Data, the "internet of things", where machines as well as people inform and shape the lives we lead, is just around the corner.
He lives in Singapore in an apartment equipped with net-connected thermostats and sensors that track his movements from room to room and read weather forecasts to keep the temperature constant. When he travels to and from the apartment it is in a car that monitors his speed and checks traffic reports and social networks to estimate his arrival time.
Looking out from EMC's offices across a sunny Wellington harbour, he beams. The data suggests his world has dawned.
- © Fairfax NZ News