The spy system is in a political uproar, with claims of cronyism and illegal spying, and a damning official report into the GCSB, the largest intelligence agency. At the heart of all the scandals is the key question: who controls the spies? Anthony Hubbard reports.
It was the week when the house of spies caught fire. A lot of myths died in the political smoke.
Successive governments have said that the foreign intelligence service, the GCSB, did not spy on New Zealanders. An official report revealed that was wrong.
In fact, more than 80 Kiwis may have been snooped on over nearly a decade. What's more, the spying might have been illegal.
The report, by Cabinet Secretary Rebecca Kitteridge, reveals that the Government Communications Security Bureau was a mess.
It was bloated with middle managers and afraid to sack time-servers out of fear they would leak secrets. Poor performers were tolerated, and staff stayed too long in their jobs and grew stale.
Ms Kitteridge also painted an unflattering picture of the spy services' official watchdog, the Inspector-General of Intelligence and Security. The watchdog, it seems, was skinny and even a bit slack.
The political uproar was extraordinary.
A former head of the GCSB, Sir Bruce Ferguson, accused Prime Minister John Key of being "on dope".
The Opposition accused Mr Key of either lying, or being asleep at the wheel, while the spies lounged around and possibly broke the law.
Ian Fletcher, the head of the GCSB and the man accused of being the prime minister's crony, emerged blinking in the spotlight and admitted that the scale of change needed at the GCSB was "probably more than I expected".
It was a triumph of British-style euphemism.
Former prime minister Sir Geoffrey Palmer put it more bluntly: "This whole thing has been a disaster for the agencies, a disaster for the Government and a disaster for the country."
By the end of the week, the only question was: How big a change is coming to New Zealand's spy system?
Mr Key, showing signs of irritation as the scandal kept raining on his goodwill parade through China, promised "substantial change" to the way spies were held to account.
Critics, however, say much greater reforms will be needed than those suggested in the Kitteridge report.
Now is the time, they say, to build a new house altogether.
A WEAK AND GUMMY WATCHDOG
At the heart of the democratic control and oversight of the spy system is Inspector-General Paul Neazor, an unflappable and courteous 79-year-old former judge of the High Court. He has a staff of one (a part-time secretary).
Mr Neazor's counterpart in Australia, as Ms Kitteridge notes in her report, has a staff of 12.
The inspector-general has often been accused of being toothless. The report describes the gumminess in detail.
"The overwhelming impression one gets" about the Australian inspector's office, Ms Kitteridge says, "is that it is very muscular. All parties to whom I spoke described it as robust and assertive."
Its staff include former spies and review officers and visits are paid to the Australian spies every week or two.
The New Zealand inspectors-general, on the other hand, have "operated on their own rather than managing an office of staff". They have "visited the bureau's premises approximately quarterly to review documentation".
The former inspector-general, Laurie Greig, tended to be "reactive", and "did not tend to initiate [his] own inquiries or audits", Ms Kitteridge notes.
However, since 2008, Mr Neazor had carried out a "proactive review programme".
He received "some" operational briefings, reviewed the bureau's internal guidance and recommended some improvements.
He had asked to have his own website but this had not happened.
"These initiatives and ideas were all constructive," Ms Kitteridge wrote.
"In my view, though, the current New Zealand oversight model could be strengthened further to ensure a really robust level of external oversight."
Ms Kitteridge recommends a beefed- up office, more along Australian lines, and the Government will clearly make the changes.
But will they be enough?
The critics say no.
BREEDING A REAL BULLDOG
The inspector-general is appointed by the prime minister. The prime minister also appoints the head of the GCSB.
So it's the PM who appoints both the spymaster and the watchdog.
Critics say the watchdog should be appointed instead by Parliament. That would underline the independence of their role, and make it clear that the inspector was not just a creature of the government.
It would also remove any whiff of cronyism, such as the accusation that Mr Key shoulder-tapped Mr Fletcher because he was a family friend.
Auckland University's associate professor of law, Bill Hodge, says the inspector would then be like other officers of parliament such as the auditor-general, the parliamentary commissioner for the environment and the ombudsman. "The ombudsman reports to Parliament and that's what makes him special, as opposed to being inside the executive wing," he says.
As Parliament's man, or woman, the inspector "would be a salutary control on an executive mindset," says Dr Hodge.
Sir Geoffrey agrees: "There needs to be some separation between him [the inspector] and the agency he oversees."
The inspector-general's office had never been properly resourced.
"We're a very small nation," Sir Geoffrey says. " We have only limited resources.
"We try to do everything that larger nations that are better resourced do, and we can't do it properly because we won't pay for it.
"And that is characteristic of public administration in New Zealand. You find it everywhere."
Greens co-leader Russel Norman says the inspector's office also needs a culture change.
At present, he says, the inspector is chosen to be "one of us", a "reliable chap" who won't rock the boat unnecessarily.
"He goes in there [the GCSB] and has a cup of tea and says, 'Are you doing anything you shouldn't be doing?'
"And they say, 'No, we're not.' And he says, 'Oh well then, can you show me a few things?'
"And they show him a few things and then he goes off."
The inspector-general, Dr Norman says, has apparently failed for many years to spot the possible illegality of the GCSB's spying on New Zealanders.
The Greens co-leader points out that the former inspector-general, former High Court judge Laurie Greig, had to resign when the High Court found he had shown "apparent bias".
In an interview about the controversy over Algerian immigrant Ahmed Zaoui, suspected by the Security Intelligence Service of being a security risk, Mr Greig seemed to suggest that, if it was up to him, "it would be 'outski' [for Zaoui] on the next plane".
He had also said: "We don't want lots of people coming in on false passports that they've thrown down the loo on the plane and saying, 'I'm a refugee, keep me here'. And perhaps having some association elsewhere."
Ms Kitteridge says in her report that Mr Greig had equated a lack of complaints about the GCSB with compliance with the law.
"The fact that there are very few complaints and little need for any inquiry into the activities of the . . . GCSB," he had said in his 1999 annual report, "indicates, I believe, that the performance of their activities does not impinge adversely on New Zealand citizens."
There was an obvious fallacy here. People can only complain about things they know are happening, as Mr Greig's successor, Mr Neazor, points out.
But that's the thing about the spy agencies.
Most of the time you don't know what they're doing.
THE SECRET COMMITTEE
The other main check on the spies is a committee of MPs chaired by the prime minister. Here again the critics say this is a largely toothless tiger - and there is so far no sign that Mr Key wants to change that.
The critics want to give the tiger some teeth.
Parliament's intelligence and security committee meets in secret and its members are not allowed to say anything about what happens behind closed doors. Only the prime minister can convene the committee, and he has draconian powers over what it can discuss.
Opposition leader David Shearer, one of five members of the committee, says: "I don't think it serves as a check on our security agencies as the public would like it to."
The committee meets in private and "it doesn't necessarily bring up very sensitive types of issues," Mr Shearer says.
Dr Norman says the committee usually meets just twice a year - and the prime minister can rule out discussion on any matter he deems operationally sensitive.
"It's of very limited use," Dr Norman says.
The brevity of the committee's meetings became something of a joke around Parliament - its notoriously brief annual reports would say that the committee "met for 45 minutes" - but Dr Norman says that has changed since he joined.
"The meetings have become rather elongated," he says, but rarely enlightening.
The other two members of the committee, UnitedFuture leader Peter Dunne and ACT leader John Banks, did not respond to The Dominion Post's invitation to discuss it.
Dr Norman and others say the committee should be turned into a normal select committee of Parliament, but with some special rules to protect security secrets.
"If you're not told what's going on," he says, "it's impossible to keep the spies accountable."
The committee would then be a parliamentary forum free of the control of the prime minister, and able to delve deep into the secret kingdom.
Dr Norman points out that American spies are accountable to the intelligence committees of the House and the Senate. These committees are routinely described as powerful, determined, and with a large number of expert staffers, who have a brief to investigate all aspects of spy activity, including operational matters.
The committees "get a lot of information about what is going on, but they are all sworn to secrecy about what they get," Dr Norman says.
If the United States can do it, he says, why not New Zealand?
American-born Dr Hodge agrees. The idea "makes absolute sense" even though governments in the British tradition have been reluctant to take such a step.
New Zealand followed that tradition, which places less emphasis than the US does on the doctrine of a separation of powers between the government or executive and the legislature.
But having an intelligence select committee along American lines would provide "salutary checks and balances, which is the thing you want."
THE AMERICAN BEAST
Some, however, say the American system is not as effective a watchdog as is often claimed. A 2011 book by Stanford University academic Amy Zegart concludes that "Congress is not designed to oversee intelligence agencies well".
Her book Eyes on Spies: Congress and the United States Intelligence Community, points to a wide range of systemic problems.
There is, despite the usual claims, limited expertise among the staff and a lack of interest from the elected representatives. She points to fragmentation, competition and outright turf wars among different committees, and a determined effort by the spies to elude accountability.
"What former Senate Intelligence Committee chairman John D. Rockefeller IV has called the 'long and sordid history of congressional oversight weaknesses' is not widely known," says Ms Zegart.
Defenders of change in New Zealand, however, say the flaws in the American system are not an argument against change in this country.
One of the problems Ms Zegart identifies, for instance, is the lack of incentive for a politician to become an expert in intelligence. Intelligence is difficult and takes a lot of time and trouble to master. There are no powerful lobby groups to reward the politician who wins spoils for them.
There are, in short, very few votes in intelligence issues. Far better to spend your time on the glamour projects with big political pork barrels attached.
Sir Geoffrey says these incentives work in all areas of politics, not just intelligence oversight.
For example, "there are very few incentives on MPs to do their select committee work. They would be far better off to prance and posture in front of television".
Yet the MPs' work in select committees was essential.
In any case, Sir Geoffrey says: "Even if they miss things, the oversight [of the intelligence committees] has a deterrent effect on the agencies and that is a useful check and balance."
Dr Hodge makes a similar point: "It's not perfect. But with these types of agencies you simply do your best."
The courts can sometimes act as a check on the spies. It was Kim Dotcom's extradition case, after all, that led to the revelation that the GCSB was involved in his business. This led to the revelation that it has spied on him illegally.
"But for the Dotcom case and the decisions on it, this entire matter would not have come to public view," says Sir Geoffrey.
"And it just shows you the importance of having an independent judiciary who can adjudicate the law without fear or favour.
"They are part of the government system, but they are a separate part and they are the least dangerous branch."
In the Dotcom case, the courts have even granted Dotcom the right to sue the GCSB.
Courts are certainly less deferential to governments over security issues than they used to be. However, the record is patchy.
The Aziz Choudry affair in the late 1990s is sometimes recalled as a victory by the courts over the spies. The truth is more complex.
The SIS was caught breaking into the Christchurch house of Mr Choudry, an anti-globalisation campaigner, during a demonstration against an Apec meeting.
The then inspector-general, Mr Greig, ruled that the break-in was not illegal. Mr Choudry sued, and the then prime minister, Jenny Shipley, refused to give him access to official documents bearing on the case.
She told the court that it was a matter of national security.
It was during a Court of Appeal hearing over this that Justice Ted Thomas made his celebrated remark: "The courts today are not prepared to be awe-struck by the 'mantra' of national security."
However, a majority of the bench agreed with Ms Shipley - and Mr Choudry was denied the documents. The Crown did then apologise and paid him an undisclosed sum.
But clearly the courts will not always side with the complainants against the spies.
THE BIGGER PROBLEM
Some of the biggest spy scandals have come to light not through official watchdogs or the work of clever lawyers in court cases, but through the media.
The big changes in the American intelligence system came after a long-running series of scandals and leaks, including the rolling disaster of Watergate.
The obvious issue here is that media coverage of spy matters is patchy. The journalists don't know what they don't know, just like intelligence inspectors and law-makers. And Sir Geoffrey says the media in New Zealand are "so supine and badly resourced and ineffective" that they are not a very good watchdog.
One of the bigger problems is that governments tend to move slowly on questions of espionage. And better oversight rules often go hand in hand with increased powers for the spies.
In 1996, for instance, the government brought in "a new era" for intelligence oversight, setting up the inspector-general's office and the Intelligence and Security Committee of MPs. At the same time, it also expanded the definition of national security to include New Zealand's economic wellbeing.
Critics charged that this would allow spying on critics of the prevailing free-market and free- trade political consensus.
Supporters of Aziz Choudry said it showed that these fears had indeed been justified.
They also noted that the "reforms" of 1996 had been drafted by the then director of the SIS, brigadier Lin Smith, and Sir Thaddeus McCarthy, who held the watchdog post under the previous arrangement.
The system, the critics said, was cloning itself but leaving most of the power with the spies themselves.
Now critics say the same thing is happening.
The Kitteridge blueprint will increase oversight in some ways, says Dr Norman, but won't go nearly far enough.
Reform, say the critics, always lags behind the problem. A scandal erupts and there are changes to make spies more accountable. Later, they will be shown as inadequate when another scandal erupts.
Sir Geoffrey, for instance, praised the 1996 reforms at the time, saying they got the checks and balances "about right".
Today, he says the reforms were done on the cheap and don't go nearly far enough.
Sir Geoffrey notes the change in attitudes himself.
"I remember when [Australian Cabinet Minister] Kim Beazley came to a Cabinet meeting when I was acting prime minister and I'd just made a decision to build the Waihopai extension.
"And he said to me as I was taking him to our Cabinet meeting, as you often do take senior Australian ministers when they come over, 'I'll thank them for Waihopai'.
"I said, 'Don't do that, Kim, they don't know anything about it.'
"There were only three ministers who knew - the prime minister, me and the minister of defence - and it [the Waihopai construction] was hidden in the Ministry of Defence's estimates.
"There was far less transparency in those days than there is now.
"We are going on a journey towards transparency. It's a difficult journey and it's got a few hiccups on the way.
"But you do have to go there."
- © Fairfax NZ News
Do you think Wellington deserves its spot in CNN's list of top coffee cities?Related story: CNN rates Wellington coffee among best