Power surge fallout sinks health systems

A power surge caused a major failure of clinical computer systems at Auckland Hospital, a report has found.

The four-hour system failure in April led to "significant clinical risk", according to the Health Ministry's third report of serious and sentinel events in New Zealand hospitals.

Colin McArthur, medical adviser for quality and safety at the Auckland District Health Board, says a power surge at the board's Grafton data centre damaged the main circuit board in one of the datacentre's two uninterruptible power supplies beyond repair.

The circuit board was replaced the following day, he says. "While some core systems were identified as requiring a shutdown while the circuit board was replaced, a number of other systems lost power when the repair started because it was not documented that they were on the circuit board being repaired."

Several clinical and non-clinical systems were affected by the power surge but the areas "moved to business continuity plan process and procedures".

The ministry report said key servers supporting the clinical systems had single power supplies, and there was "inadequate knowledge of power supply requirements".

Dr McArthur says the board has since revised its documentation and sign-off procedures for maintenance operations. The servers were running off mains power and the board is "upgrading all servers to fully separated dual power supply systems".

The board has also spent $1 million on software that tracks requests for follow-ups on X-rays after a chest X-ray report identifying "incidental lung cancer" was not reviewed by clinicians treating the patient.

The ministry report said the hospital's manual system was unable to ensure that appropriate clinical signoff occurred.

Dr McArthur says the new software can escalate requests if there is any delay in meeting them.

"It is also integrated with the current X-ray dictation and reporting system to ensure that all reports are electronically viewed, accepted and signed off by an appropriate clinician."

Air New Zealand's computer systems were crippled in October, delaying dozens of flights, after a power generator at IBM's Newton data centre failed.

It is understood IBM was relying on a single generator to power the airline's mainframe during scheduled maintenance on an uninterruptible power supply attached to the data centre's main power supply.

A UPS is designed to smooth out brown-outs and spikes in the electricity supply and provide battery back-up in case of a power cut.

Tech Troubles

The Health Ministry report found IT issues were implicated in several serious events:

* After 10 medication overdoses involving Counties Manukau patients, a review found an automated medicines dispensing cabinet did not have safety software to alert staff to overdoses.

* Patient management systems at a hospital in Northland failed after water leaked on to a computer switchboard in extreme weather.

* Incompatible systems meant a report for an invasive X-ray was entered against the wrong South Canterbury patient, and a colonoscopy was performed based on the inaccurate information.

- © Fairfax NZ News