Covid-19: How raw vaccine and case data can be (very) misleading

Director-General of Health Dr Ashley Bloomfield details modelling based on current cases in the outbreak.

At Monday’s 1pm press conference, Director-General of Health Ashley Bloomfield presented a slide that showed the vaccination status of Covid-19 cases.

This particular slide showed that 10 per cent of the Covid cases between October 23 and October 28 were fully vaccinated.

Between September 11 and September 24 only 2.87 per cent of cases were fully vaccinated.

At first glance, this seems very odd. If the vaccines protect people from Covid-19, why is the proportion of fully vaccinated cases increasing?

* Covid-19 NZ: How reopening at 90 per cent vaccinated stacks up globally
* Covid-19: What we've learned about the virus over 18 months in the wilderness
* Ireland and Covid-19: More than 1000 cases every day but normality looms

Bloomfield commented: “But just to point out here: as you see in every country, and as is happening here, as you get high vaccination rates, an increasing proportion of cases will be people who are fully vaccinated.”

In early September, about 30 per cent of the entire New Zealand population was fully vaccinated. By late October, that number had doubled.

The fact that those numbers seem unusual is because of a statistical trait called Simpson’s Paradox. This particular quirk has been weaponised by some to question the effectiveness of the vaccines – so much so that I was somewhat reluctant to write about it, not wanting to give oxygen to a potentially dangerous misunderstanding of data that can give rise to confusion about just how effective the vaccines are.

But it is true that in recent weeks, for example, more vaccinated people have died in the UK, than unvaccinated people. This was (and is) very much expected because the vast majority of the population, particularly those much more likely to die, are fully vaccinated. And the vaccines, although incredibly effective, are not perfect.

My colleague Charlie Mitchell recently offered this excellent equivalent. In New Zealand, around 20 or 30 per cent of road fatalities are among people not wearing a seatbelt. That means most people killed on the road are wearing their seatbelts. Does that mean seatbelts don’t work? Of course it doesn’t.

Covid-19 modeller Michael Plank told me he uses the following analogy to explain this phenomenon: the majority of crimes are committed by right-handed people. But that doesn’t mean right-handed people are more likely to be criminals.

(“The analogy only takes you so far – because right-handed people aren’t less likely to be criminals either,” he says, “but vaccinated people are less likely to die”).

These are just two examples of how data can play tricks on us. They’re reminders that you should not just believe raw numbers you read online and take them at face value with zero context.

Let’s look at the UK in more detail.

A dig into the raw data in the UK

The UK government puts out weekly reports on Covid cases. In this story we’ll look at a surveillance report for October 28.

The report includes a range of data including, for example, the vaccination coverage across England and the vaccination status of Covid-19 cases, people hospitalised and those who died over the four weeks to October 24.

You can see the cases in the table below. At first glance, this data is counterintuitive. Why, for example, are there only 434 Covid cases in those over 80 who are unvaccinated and more than 11,000 in the fully vaccinated?

And let’s check out the deaths. Again, why did 1209 fully vaccinated people aged 80+ die and only 143 who were not vaccinated?

What’s going on then? Well, it’s a numbers game. Some 95 per cent of people over 80 in England have had two vaccine doses. (In some parts of the country the rates are even higher.)

That’s not everyone but it’s nearly everyone. So stick with me for a moment and imagine every single person aged 80 or older was fully vaccinated. Then every single death would be in the vaccinated age group.

As the report explains it was expected that “a large proportion” of Covid-19 cases, hospitalisations and deaths would occur in vaccinated people after almost everyone was vaccinated.

This is because the vaccines are not perfect. UK health officials believe that the Pfizer vaccine is effective at preventing:

  • 75-85 per cent of infections.
  • 80-90 per cent of symptomatic disease.
  • 95-99 per cent of hospitalisations.
  • 90-99 per cent of mortalities.

That’s for the first 3-4 months after vaccination, by the way, and they note the issue of waning immunity, which is primarily to do with protection against infection. It appears the the vaccine protects the vast majority of people from serious illness for much longer.

So, think of it this way:

  • There are 1000 people on an island.
  • 90 per cent of those are vaccinated (900 people) and 10 per cent are not (100).
  • The ‘y’ virus arrives at this island and every single person is exposed. All 100 unvaccinated people catch this virus.
  • This particular vaccine is 75 per cent effective at stopping spread, which means 225 fully vaccinated people (25 per cent of 900) catch the virus.

Does that mean the vaccine doesn’t work? Of course not. Without it, all 1000 people would have got sick.

And now let’s talk about deaths

According to The Financial Times, which analysed global data, a vaccinated 80-year-old is about at the same risk of dying as an unvaccinated 50-year old.

(Age defines, more than anything else, the risk of an individual dying from the virus.)

And according to this US analysis from The Economist, the estimated risk of death in an unvaccinated 50-year-old man (with no underlying conditions) is 0.4 per cent. Sure, that may not seem very high, but if 100,000 unvaccinated 50-year-olds – or fully-vaccinated 80 year-olds – catch Covid-19, 400 would die.

The point I’m trying to make here is that even fully vaccinated elderly people are at risk from Covid-19. The risk has been massively reduced by the vaccines – an unvaccinated 80-year-old man, according to The Economist’s analysis, faces a 12.2 per cent chance of death – but there’s still a risk.

Let’s run some numbers again.

  • There’s 10,000 80-year-old men living on an island. 98 per cent (9800) of them are vaccinated against virus ‘x’.
  • Virus ‘x’ arrives and, again, let’s assume everyone is infected.
  • The 9800 vaccinated people face a 0.4 per cent chance of dying – 39 die.
  • The remaining 200 unvaccinated face a 12.2 per cent chance of dying. This means 24 die.

Again, does this mean this particular vaccine doesn’t work? Not at all.

This is a numbers game. A small percentage of a big number can still be much larger than a big percentage of a small number, as Stuff reporter Nikki Macdonald has explained.

What’s more, and this goes well beyond the simple illustrative examples above, it’s important to keep in mind the people prioritised for vaccination in the UK were those most at risk from Covid-19.

What this means is that the vaccinated people are the groups who, without vaccination, are most at risk of dying from Covid-19 but also, as the report explains: “more at risk of hospitalisation or death due to non-Covid-19 causes, and thus may be hospitalised or die with Covid-19 rather than because of Covid-19”.

Dr Ashley Bloomfield speaks about case numbers on Monday.
Dr Ashley Bloomfield speaks about case numbers on Monday.

This is important. The UK counts Covid deaths as those who tested positive for Covid-19 and either died within a certain number days since their first positive test or had the virus mentioned on their death certificate.

So this may well mean some of those 1492 people I mentioned above may not necessarily have died of Covid-19.

And finally in the UK, there’s the issue of waning immunity. While the vaccines seem to protect the vast majority of people from serious illness for a long time, there have been suggestions the vaccine effectiveness at keeping the elderly (over 75) out of hospital has decreased over time. Hence, why the UK is rolling out a booster programme. But the science on this is not settled.

Going back to the case numbers for a moment

There’s another thing to keep in mind with all this data. You can’t rely on this particular set of numbers to tell you how good the vaccines are. The UK report acknowledges this.

Firstly, the data only covers a small period.

Secondly, fully vaccinated people are more likely to socialise and therefore be exposed to the virus, and fully vaccinated people are more likely to get tested (because they care for their health).

This creates all sorts of statistical anomalies which mean it’s very hard to compare like for like over that particular period. The UK surveillance report still includes a table that shows ‘rates’, but cautions against using them as the be all and end all. You can see them below, by the way.

As the report outlines: “The rate of death within 28 days or within 60 days of a positive Covid-19 test increases with age, and again is substantially greater in unvaccinated individuals compared to fully vaccinated individuals.”

And here's another representation of the rates from the previous week’s surveillance report.

Is there something more solid?

Yes. There is. The UK’s Office for National Statistics has recently released a report comparing deaths in vaccinated and unvaccinated people between January 2 and September 24 this year.

This particular work looks to smooth out all the discrepancies.

Its findings are quite striking. The age-adjusted risk of deaths involving Covid-19 was 32 times greater in unvaccinated people than in fully vaccinated individuals in a particular time period.

Clarification: The ONS published a blog subsequently explaining the limitations of this data. We have updated this story to acknowledge that. You can read it here.