Crowdsourcing MP expenses
BY REUBEN SCHWARZStuff launched its first crowdsourcing project on Monday. We expected it to take weeks to check the more than 16,000 documents relating to MPs' credit card expenses that we uploaded. Thanks to you, it should only take days.
Crowdsourcing - basically putting things online and letting the public help to sort and analyse it - is something we've always been interested in. It's a great way to quickly sift through a lot of information and get you - the reader - involved in creating the news.
When we heard we had eight boxes of receipts and other documents coming from an Official Information Act request, we took inspiration from Britain's Guardian and decided to let our readers pick which receipts they thought were worth investigating.
Professional reporters quickly noticed some outrageous claims, which led to major political shakeups. We didn't really know what the public would think of our crowdsourcing idea.
The response was a gush of interest. So far more than 15,000 people have looked through the documents, with more than 4000 people submitting reviews. We've had more than 6000 documents marked by you for us to look into, and about 7000 notes on documents.
Now we're going to go through what you've told us and see if there's anything we can turn into a story.
Each receipt had to be reviewed by three people before we considered it checked. We used a threshold of three to stop mistakes and anyone who tried to abuse the system (for example, by targeting the opposing party).
More than half the receipts had been checked by three people within 24 hours. That translates into more than 50,000 views of documents, which is incredible. We had about 80 percent checked within 48 hours.
Here are some of the highlights:
* "$1412 on a meal. That's more than some of us make in a month!" - on receipt from staff of Winston Peters, 2008
* "Paying for nurofen, sleeping pills and insect repellent?" - on receipt from staff of Phil Goff, 2008
* "Minibar charges, $12 on cashews!!!" - on receipt from staff of Pete Hodgson, 2005
* "Every time this man eats it is over a hundred dollars" - on receipt from Parekura Horomia, 2004
* "$500 a night at the Ritz-Carlton. Living it up on our dollar. Also, $233.10 in 'Miscellaneous Charges' " - on receipt from staff of Michael Cullen, 2006
* "Should staff not supply their own umbrellas. What next...we buy their underwear?" - on receipt from staff of John Key, 2009
* "Cocktails? 623USD??" - on receipt from staff of Jim Sutton, 2003
* "Raiding the minibar!" - on receipt from Harry Duynhoven, 2005
* "A very expensive dinner for four people" - on receipt from Chris Carter, 2006
* "Digital camera $1400 very expensive! And lost receipt." - on receipt from Annette King, 2003
How we did it
Eight boxes of documents arrived at Fairfax's (owner of Stuff, the Dominion Post, the Press and more) Parliamentary bureau on Thursday last week. We started scanning right away and kept going for five days straight, hiring extra staff to scan over the weekend.
On Thursday morning we also met with our developers at Catalyst and sussed out the details of what we wanted to build. It took us three working days to get the first version up and running. After a few hours of testing and bug fixing we were ready to launch it. We added improvements (like the MP search and the zoom and rotate buttons) as soon as we could get them ready.
The documents were scanned and saved as pdfs. Each page of these pdfs was converted to an image, which was embedded inside a Stuff page.
We used a clever randomiser that pushed people toward pages that were close to being checked three times, so we'd get a steady stream of data. Once something was checked by three people it was removed from the randomiser.
When you reviewed a receipt (as "don't know", "investigate this" or "not interesting"), you sent a message to our database telling it what you chose, and any notes you put in. We then exported that data in a spreadsheet and filtered it for our reporters.
* So what do you think about this crowdsourcing project? Are there others you'd like us to tackle? Let us know in the comments.
» If you like this blog, vote for it in the NetGuide People's Choice Web Awards! Go here to cast your vote for it as Best Blog. Voting closes on July 30.
Sponsored links
Newest First
Oldest First
This is a good local win for crowd sourcing, transparency and open data. Well done to Stuff, Catalyst and the citizens who participated. I hope Stuff continues to promote similar efforts: what about regional and local government expenses, government procurement and contracts, environmental data, lobbying and more?
It would be a fantastic service to continue doing this over time.
It would also be very telling to graph expenses by MP, and seperately by Role, over time. Peaks (and troughs, errr, valleys) would then be visible -- I agree standard spending by Trade & Industry is going to be higher than a backbencher, but it should be fairly constant year to year.
Well done on this initiative.
Lazy? Did you even read the post?
You think turning around development work in 5 days, scanning 16 000 documents, loading and displaying them... and then managing all that live data coming in over the next few days... is lazy?
If anything, it sounds like more work. They even hired extra staff to do it!
You shouldn't be looking for things to "turn into a story"; you should be looking for things that are actually worthy of being publicly discussed. I suspect many are not.
The item on the MPs expenses is old hat now, it was journalists who put in the Official Information request and cost the taxpayer 50k in getting all the documentation together and photocopied etc so the journalists should be going through the paperwork, not the taxpayer/ordinary citizen!!! Another example of milking a newsworthy item as there is seemingly nothing better to report on.
Agreed Anita! I think all this hoopla over expenses in such a waste of time. Politician are traveling all the time, away from the family, the least we can do is shout him a movie =)
Brilliant. Well done, Stuff. This is what the govt should have done instead of churning out all that paper - another attempt to obfuscate perhaps. I would love to see the expenses of porky old Jonathan Hunt and the Governor General (who most of us wouldn't recognise if we fell over him). Can he still import all those Mercedes tax free?
Well done on bringing all this out for public scrutiny. Appears that many have well overstepped the mark. My gut feeling is that as taxpayers are the ones being fleeced it is taxpayers (voters) who should deal out at least part of the punishment by sacking offenders from office at the next election. I suspect this is the only punishment most politicians are capable of understanding and the present batch of offenders are no doubt hoping that it will all be forgotten by November next year. Applying this sanction may mean that one has to vote for a party one might not particularly favor. However, I would prefer to see honest politicians with whom I did not always agree in parliament than dishonest ones of a party I would normally have voted for. Also I hope we will in the near future take a look at local body politicians and some of the means they use to spend ratepayers money on themselves. Overseas trips to attend conferences and foster "sister city relationships" seem to have become the latest lark.
This whole exercise seems to be a smokescreen to divert attention from what really is happening behind closed doors.How about some investigation into the pollution of our waterways by massive ammounts of effluent from dairy cows and why nothing appears to be done about it. And what about the millions of taxpayer dollars being diverted to minority ethnic groups to buy their support for the present govt.
Houston died in bathtub - coroner
Christchurch cricket bat murder admitted
Woman crushed, friend watched 'helplessly'
Daily trivia quiz: February 13
Hundreds of unfit teachers in class
Superbike champion dies after race crash
Tainui leader ousted from board
Bus survivor praises her heroic rescuers
Hundreds of unfit teachers in class
Houston died in bathtub - coroner
Your top 10 cheesy pickup lines
Kiwi women obsessed with weight
Ethnic rights advice stuns communities
NZ, mate, you might have a drinking problem
Paul Henry's disjointed return to TV
Warning hearing has power to kill Transmission Gully
Innovative... Or lazy?