Oxford Covid-19’s response database does not meet its intended purpose
Over the last two years, the volume of research on public health issues has likely surpassed all previous records. Not just epidemiologists and public health experts, but also a slew of physical scientists, psychologists, sociologists, economists, and just about everyone else has weighed in.
Many of these researchers rely on OxCGRT (the Oxford database), a large-scale database created by the Blavatnik School of Government at the University of Oxford, which tracks different 23 policy responses (such as school closures, travel restrictions, and vaccination policy) in more than 180 countries since January 1, 2020. The Stringency Index, which aims to inform scholars about how harsh certain rules were, is one of the most prominent indexes built from this collection.
Firstly, I want to applaud the University on this effort, which does have some usefulness. Furthermore, having any data is generally preferable to having none; thus we are fortunate to have this dataset. But, after thoroughly examining the database in recent days, I’ve formed serious reservations about its architecture. Through this post, I’d like to draw attention to what I believe are its fatal flaws.
Researchers are typically aware that all indices have significant flaws, yet most indices may be used with caution. The Stringency Index, on the other hand, is absolutely useless in my opinion. Some aspects of the database, on the other hand, might be utilized with caution. If researchers do not grasp this, they will inevitably make policy recommendations that are harmful.
I’m not talking about code discrepancies here. Because the coding is done by a tiny army of volunteers all around the world, there are still significant irregularities despite rigorous training. Any discrepancies can be corrected. The database’s design is at the heart of the problem.
1. Sweden scores lower on the stringency index than the United States
Lockdowns have been the world’s greatest public health experiment in history, with Anders Tegnell giving the world a lesson in public health by explaining his actions publicly at each stage. “It was as if the world had gone insane, and everything we had discussed had been forgotten,” he said on June 24, 2020.
Except for Sweden, all countries have accepted Chinese lockdowns, which are strictly prohibited by Western laws and pandemic preparations. Lockdowns that be used indiscriminately violate international human rights commitments and are much outside the scope of public health regulations.
Over the last two years, a heated discussion about Sweden and the rest of the globe has raged. Sweden has been vilified by the media, politicians, and public health authorities throughout the world for:
(a) purportedly taking a “soft” attitude to the epidemic in comparison to other countries
(b) having more deaths than its neighbours, Norway and Finland. Of course, the evidence shows that Sweden’s strategy has resulted in one of Europe’s lowest excess fatalities. Its neighbours had policies that were similarly “loose.”
The Stringency Index, on the other hand, says nothing about this huge controversy. While Sweden has consistently ranked low on the Index over the last two years, it has been categorized as being more rigorous than the United States at times. That is an “out-and-out” absurdity, indicating that the database is severely flawed.
2. No mental model
The Oxford database’s authors appear to have had no idea what they were aiming to gauge.
Our econometric and statistical job will wind up being a fishing expedition if we don’t have a clear mental picture of what we’re attempting to achieve, according to econometrics 101. The nature and design of the measuring instrument are perhaps more significant than the results: with a lousy measuring tool, we’ll end up with garbage-in, garbage-out.
3. Zero-COVID vs. mitigation
The standard science of pandemic management is mitigation, which ensures that the demands on the health system are kept to a minimum. Except for Sweden and maybe a handful of other countries, the majority of countries proclaimed a zero-COVID strategy, i.e., eradication of the virus within their borders.
As a result, policies were enacted that were not only unusual but also prohibited by public health literature. As a result, the initial inquiry should have been: How can we differentiate zero-COVID countries from others?
Coercive measures were generally used by zero-COVID countries, whereas voluntary measures were mostly used by mitigation countries. The first port of call should have been this clear differentiation.
The police use their sheer power to enforce a prescribed regulation. It is a state-sanctioned act of violence against the community for the community’s general good. But, just as we can’t assault someone unless it’s in self-defense, the government must adhere to a far higher (human rights) standard when using force.
Yes, the Oxford database tries to account for this distinction. Consider the difference between a government (such as Sweden) that encourages individuals to work from home whenever feasible and another (such as Australia or India) that relies on the police to guarantee that no one is on the streets.
The Oxford database assigns a 1 to the first and a 2 or 3 to the second. However, this is missing the point. These two measurements have a dramatically different influence on society, by an order of magnitude. It’s the difference between telling someone what to do and physically attacking them if they don’t. The Universal Declaration of Human Rights states unequivocally that freedom of movement is unaffected unless extraordinary circumstances exist, which was never the case during the epidemic. Instead, it was generally understood that lockdowns have a significant negative impact on society.
People can go on with their lives while taking the recommended safeguards. They are not required to provide any evidence to justify their presence on the streets.
For moving about, they are not punished, assaulted, or arrested. Similarly, mask advice is a gentle reminder of our common sense (I’m not disputing the research, which indicates that there is no method to prevent virus propagation in aerosols except with N95 masks). People are beaten up by the police as a result of the mask mandate, which has a significant negative impact on community well-being. Unnecessary requirements are a violation of our humanity.
4. Proportionate vs indiscriminate
There are occasions when coercive measures are necessary to comply with public health legislation and pandemic planning. Coercion, on the other hand, is subject to a slew of human rights legislation and ethical obligations. As a result, the principles of proportionality and ethics, both of which need risk-based methods, are profoundly established in the field of public health. Any forceful intervention must be tailored to the specific danger.
If the science of virology, immunology, epidemiology, and public health was a blank slate before the COVID epidemic, and if OxCGRT was the first attempt to find what works, the Oxford database’s naïve approach makes sense. However, there was already a large and thorough body of knowledge regarding the dangers of lockdowns and opposed border closures (e.g. Donald Henderson) as well as masks worn by the general public. Instead, as Sunetra Gupta noted, keeping borders open was necessary to maintain a population’s immunity.
The Oxford designers evidently have no understanding of public health or the World Health Organization’s October 2019 guidelines. On the OxCGRT website, I found a working paper that listed no public health professional or epidemiologist among the team members. Prof. Sunetra Gupta, the world’s most skilled theoretical epidemiologist, works at their institution, so they had easy access to her.
Everyone who mattered understood exactly what to do in the event of various pandemics.
Formal recommendations were incorporated into national and state pandemic planning. Depending on the virus’s lethality and risk distribution, several mandates/restrictions were allowed. Public health studies found that targeted limitations to avoid high levels of harm (such as for the elderly in aged care institutions) were required, with just guidelines elsewhere.
On March 10, 2020, Victoria’s pandemic plan said, “COVID-19 is regarded as being of intermediate clinical severity.” It adopted a risk-based approach and “focused on safeguarding vulnerable Victorians.” The paper concluded, “Older Victorians and those with chronic illnesses are recognized to be at increased risk of COVID-19 infection.” It will also “intensify risk-reduction initiatives for at-risk groups,” according to the statement. Lockdowns were not even close to being a part of the plan with such a focused strategy.
The database architecture needs to be based on a risk-based matrix, not a tick-a-box mechanical method that ignored risk, severity, harm, costs, or science. In any sphere of public policy, such a mechanistic database would never pass muster.
I’ve worked in the field of occupational health and safety for a long time (OHS). A mechanistic approach to OHS coding might lead to an index in which equipment testing and tagging (such as the electrical lead on a toaster or a computer) are ranked on par with preventative measures for construction workers falling from heights.
However, in these two circumstances, the likelihood of injury multiplied by the size of harm is completely different, with the monetary magnitude of compensation claims, which includes the standardized cost of death, giving significantly more meaningful insights than any ordinal scale or arithmetic sum. OHS experts would never accept risk classification that is unrelated to the severity of the damage.
The Oxford database design team was able to consult the literature as well as experts: What are the usual, widely recognized public health strategies for COVID, in which the benefits outweigh the risks? A set of proportional measurements like this should have been marked as 0. Restrictions that were less than this should have been coded as -1 or -2, and restrictions that were beyond the standard package should have been coded as positive.
For example, the database now assigns a value of 0 to “no measures,” despite the fact that inactivity may indicate a failure of the public health response and, in many situations, should be assigned a value of -1. The most serious problem is that most proposed policies are now classed as 1. However, a suggestion for a civilized, not totalitarian approach. It must be coded with a 0 value.
5. Dollar value, QALYs or WELLBYs
Instead of using an ordinal scale, pandemic actions should be prioritized using a standardized monetary cost imposed on society, similar to how cost-benefit analysis is done. As a standardization tool, we may utilize QALYs or WELLBYs. However, because this is a difficult task, a simpler option may have been devised.
6. An alternative that is simpler
Recognizing that mandated rules have a disproportionately negative influence on the community, the ordinal scale has to be non-linear at the very least. We should have used at least 3 and 9 instead of 2 and 3 for different types of lockdowns. The 23-hour lockdowns in Melbourne with an 8-hour curfew would have been rated as 9 under such a regime. These had an unintended impact on millions of individuals, causing widespread terror and hysteria in society.
The database fatally conflates the radically varied rules by awarding 1 to suggested stay-at-home orders and 3 harsh lockdowns. Any “findings” based on the Stringency Index, which is completely unsuited for the purpose, should be questioned.
7. Utilizing the database to retrieve some utilities
)
If time allows, re-code the old database to accommodate a more risk-based approach. However, there may be a quicker way to obtain anything valuable from the database. To identify whether variables are valuable, researchers should thoroughly investigate the underlying codes and physically inspect the notes in each code.
I believe the total of C2, C6, C7, and H6 (Workplace closures, internal mobility restrictions, stay-at-home regulations, and facial coverings) might provide a more accurate estimate of the severity of lockdowns than the Stringency Index. Once a shortlist of such important factors has been compiled, cross-national comparisons may be made. More significantly, meaningful analysis such as OLS regressions should take into account the non-linearities that are present in the data.
