Estimating underreporting in adverse events using ambulance call records

Comparing the South Australian Ambulance Service calls with the Database of Adverse Event Notifications

As discussed in more detail in my previous article, South Australia (SA) has provided an important study group with the new COVID-19 vaccines. It contains a population where only about 0.05% of the population contracted the virus by the end of October 2021. Other locations where a significant proportion of the population were exposed to SARS-CoV-2 mean that it would be difficult to determine if the deterioration in the health of the population was due to long-term effects of COVID-19 or the vaccine.

It might be possible to estimate the rate of underreporting in adverse events if there was a data set which recorded the true number of a specific adverse event from the vaccine. An increase in emergency calls to the South Australian Ambulance Service (SAAS) might be useful for this. Naturally it is not possible to know the cause from this limited information. An increase in calls could be due to adverse events from the vaccine, and it also could be from something unrelated, so care needs to be taken when drawing conclusions about analysis of this data set. I suspect that one of the calls that the SAAS receives in high proportion relative to people making their own way to the hospital or GP is a call for 'Convulsions / Fitting'. I think it is likely that most people would immediately ring 000 (emergency services) if they saw that event occurring. Other medical events like a chest pain could result in an ambulance call, a visit to the hospital, or a visit to the GP. Ambulance calls for 'Convulsions / Fitting' probably represents a large portion of the events in people that have not had a history of epilepsy.

The data sets considered here are South Australian Ambulance Service calls for October 2021 (treatment month) and October 2019 (control month), the Database of Adverse Event Notifications, and population data from the Australian Bureau of Statistics (ABS). Table 1 shows the calculation for 12 to 15 year old South Australians. Please see my previous article for how the number of calls was interpreted. There were 19 calls in October 2019 for 'Convulsions / Fitting' in 12 to 15 year old children and 42 in October 2021. The 2019 figure was adjusted for population changes to give 19.8 calls (this was only used to calculate the percentage increase, not for Fisher's exact test). The p-value was 0.0067, indicating a significant increase in October 2021 over October 2019. The total increase in calls over the expected amount (assuming that October 2019 was a 'normal' year) was 22.2 calls.

Looking at the Database of Adverse Event Notifications (DAEN) reports that contain either the word 'seizure' or 'convulsion' for the age group of 12 to 15 years in October 2021 results in 12 reports. There may be a delay between the symptom and the report being added to DAEN so the number of reports from a two and four week delay was also determined. This gives an underreporting factor of 28 (DAEN only records about 3.6% of these adverse events). If all of the assumptions above were correct, then the numbers in DAEN need to be multiplied by 28 to be accurate for seizures or convulsions. This multiplication factor will not be directly transferrable to other adverse events; it is likely that less severe events will have a greater level of underreporting and vice-versa. The underreporting is probably also a function of the time between the injection and the onset of the symptom. It is also likely there is link between underreporting and the age of the patient.

Table 1. Calculation to estimate the level of underreporting in DAEN using ambulance calls for 'Convulsions / Fitting' in 12 to 15 year old South Australians.

Links ABS1 ABS2 DAEN SAAS FOI

Table 2 performs the same calculation as discussed above, but uses 12 to 19 year old South Australians. The result of that calculation is an underreporting factor of 38.6 (DAEN only records about 2.6% of these adverse events).

Table 2. Calculation to estimate the level of underreporting in DAEN using ambulance calls for 'Convulsions / Fitting' in 12 to 19 year old South Australians.

To check that the increase in calls for 'Convulsions / Fitting' is not related to some other cause, the age group of 0 to 11 years old is used. That age group had not received COVID-19 vaccines in 2021. As seen in Table 3, an increase in calls is observed, however it is not statistically significant with a p-value of 0.14. If we did apportion 28% of the increase in calls to causes other than the vaccine, then the underreporting factor from 12 to 15 year old children would be 20.9 times and from 12 to 19 year old children would be 27.4 times.

Table 3. Comparison of ambulance calls for 'Convulsions / Fitting' in 0 to 11 year old South Australians.

The terms 'Seizure' and 'Convulsion / Fitting' are not directly interchangeable and so there may be some differences in ambulance calls for 'Convulsions / Fitting' and the MedDRA reaction terms in DAEN containing 'seizure' or 'convulsion'. If South Australia had an unusual uptake rate of vaccines in the 12 to 19 year old population compared with the rest of Australia then this could introduce an error in the calculations above.

Conclusion

There appears to be a huge level of underreporting in DAEN. This issue is not new and has been documented before. This article attempts to discover the level of underreporting for a serious issue (seizures/convulsions) and finds a possible underreporting factor ranging from 21 to 39 for children/teenagers. An alternative way of looking at this that DAEN only records around 2.6% to 4.8% of seizure/convulsion adverse events in children/teenagers. The Therapeutics Goods Administration should be giving serious consideration to the level of underreporting for other issues such as Myocarditis.

Coming soon…

It looks like there are differing levels of underreporting across vaccine batches recorded on the Theraputics Goods Administration’s Adverse Event Management System. I will be discussing this in a future article.