By Deborah Borfitz
September 7, 2021 | The proper role of COVID-19 diagnostics, and especially rapid antigen tests, was the subject of a wide-ranging and mathematically oriented keynote presentation by Gerald Kost, M.D., Ph.D., director of the Point-of-Care Testing Center for Teaching and Research at the University of California, Davis, at the recent Next Generation Dx Summit. During his 30-minute talk, he took both the U.S. Food and Drug Administration (FDA) and the Cambodian government to task on their decision-making during the pandemic.
Kost began by walking attendees through the mathematics of COVID-19 diagnostics—specifically, how to use “predictive value geometric mean-squared” (PV GM2) visual logistics to interpret test metrics. The FDA Emergency Use Authorization (EUA) threshold creates excessive false-positives when disease prevalence is low and excessive false-negatives when disease prevalence is high, he says.
This is particularly problematic for antigen tests, which have weakened sensitivity, creating an “undue numbers of false-negatives,” Kost notes.
The FDA EUA documents the analog of sensitivity (90%) as positive percent agreement (PPA) and the analog of specificity (95%) as negative percent agreement (NPA). The agency has created “performance tiers” to forge consistent connections between the tier and prevalence boundary, or the prevalence at which the risk of a false omission exceeds set targets, he explains.
At the 5% target prevalence boundary, for example, acceptance of a rate of false omission for tier 1 is 33% and for tier 2 is 50%, Kost says. At a higher level of risk, and shift toward herd immunity, prevalence boundaries correspondingly rise. “We recommend tier 2 as the minimum threshold for quality of performance for COVID-19 diagnostics,” with a sensitivity of 95% and a specificity of 97.5%.
“We have to consider not just the PPA, NPA, or the sensitivity/specificity, and how a test performs in actual clinical evaluations, but also the degree of risk we’re willing to accept when we use a COVID-19 diagnostic,” says Kost. This is particularly true for rapid antigen tests.
The PV GM2 visual technique involves multiplying PPA by NPA to get a read on performance for the entire range of prevalences for different tests on one graphic, says Kost, and is not intended for point comparisons.
In his cited example, a multiplex test for influenza A and B and SARS-CoV-2 with their associated PPAs (100%, 100%, and 96.4%) and NPAs (99.6%, 99.7%, and 98%) is displayed alongside the FDA EUA thresholds (90% sensitivity and 93% specificity) and performs well across all ranges of prevalence. Another multiplex test, which also includes respiratory syncytial virus, also shows satisfactory performance using the PV GM2 technique.
The “satisfying part,” he says, is that most such tests tend to perform well when prevalences range between 10% and 80%. “But on the low end and the high end they do not.”
Using the same technique in a performance analysis on the first portable PCR test kit and the first rapid COVID antibody test for the point of care, Kost introduces the additional concept of uncertainty. The sample size per EUA is 60 and 110, respectively, which is “inadequate,” he says, noting several hundred samples from a diverse population in a multi-center study is required to get “a good general idea of performance.”
Despite all the uncertainty on the low and high ends, the PCR test claims “perfect metrics” for PPA and NPA, Kost says. “Perfection tends to fall off when we go to clinical evaluations.”
A similar pattern is seen with the magnitude of the confidence interval, says Kost. “Toward the low and high prevalances, uncertainty is great.”
Tier-Based Performance
Rapid antigen tests are “fast, inexpensive, readily accessible and, now… ubiquitous worldwide” for avoiding and managing risk, says Kost.
The performance patterns for the first fully “at home” antigen diagnostic test for COVID-19 look dramatically different for symptomatic and asymptomatic subjects, as reflected in their PV GM2 curves. The test performs more poorly among asymptomatic individuals and “the risk tends to be more broadband,” Kost says.
In each case, maximum performance looked best at prevalences ranging from 20% to 80%—conveniently, “pretty much what prevalence is these days,” he adds. Again, “at the extremes things tend to fall off.”
By continuously mapping false omission rates for all FDA EUA tests through May 2021, Kost calculates that the prevalence boundaries for five of them are “sub-tier” because the PPA is below 90%. Given prevalences around the world, that would be “quite risky” in terms of false-negatives and people unknowingly spreading the disease to others—most especially with the Delta variant. On the other hand, the prevalence boundary on the six, tier-2 tests inches out to between 52.3% and 69.6%.
The analysis suggests both the sub-tier and tier-1 tests should be “improved or retired from EUA status,” he says.
Manufacturer claims in EAU documents for the rapid antigen tests also don’t match up with clinical reality, Kost continues. For tests used in a pediatric setting, PV GM2 curves fall off and the false omission rates rise, both “rather strikingly.” Together with lack of sensitivity in actual clinical evaluations, prevalence boundaries fall to an “almost unusable level… [with] considerable risk for false-negative results.”
Guidelines For Use
Among the standards of care for COVID-19 rapid antigen testing at home is to use the kit five to seven days after symptoms arise and repeat the test after 24 to 36 hours or follow the protocol specified by the manufacturer. Using iterative calculations, Kost shows the logic of recursive prevalence—that is, testing twice improves performance. The positive predictive value maps through to a new prevalence, but the formula for finding COVID among all negatives is “one minus the negative predictive value.”
In terms of managing risk, he continues, the prevalence boundary is where risk exceeds the false omission rate. Prevalence boundaries for rapid antigen tests vary from 30% to 70% “and we really need to push that up.” It’s important to determine if a specific test is compatible with the prevailing community prevalence, he adds. “Testing twice extends the prevalence boundary, which should not be crossed.”
In a screenshot of his continuous mapping of negative predictive value and rate of false omissions, and the impact of testing twice, Kost demonstrates a jump in prevalence boundaries “nearly to the extent of herd immunity.” In other words, “the expected performance of tests can be improved by testing twice,” pushing up their negative predictive value and pushing down their rate of false omissions, he says.
The need for easy access and early detection is “fundamentally different than what is happening in many parts of the world,” he continues. “We want to maintain access to widespread, free community testing in order to facilitate early detection of breakthrough and variant infections through surges.”
Delta viral loads can be “up to 1,000 times higher,’ Kost says, “which increases transmissibility up to 10-fold but also improves the diagnostic effectiveness of rapid antigen tests for this variant.”
New Normal
In a mobile setting with one of his former students, Kost says, COVID-19 testing was provided inside a van in Vacaville, California—the site of the first community transmission of COVID-19 in early 2020—a few miles from the UC Davis campus. It is a “clever, needed approach” where individuals can choose from a menu of testing options and is supported by Wi-Fi internet communication. “If you want a travel certificate with a negative test, you’re provided a follow-up email according to the schedule of your travel.”
Mobile testing in the van has recently “skyrocketed to over 200 per day,” Kost reports. The rapid antigen test has proven to be particularly popular, followed by the rapid RT-PCR test, which represents the future of diagnostic testing. Kost recently recommended that Cambodia adopt a mobile testing format of this type so it can “move where the outbreaks are.”
It is important to avoid exposure and vulnerability. “When rapid antigen tests are negative in the presence of COVID-19 symptoms and signs or deterioration of clinical condition,” he says, “rapid response multiplex molecular diagnostics will help clarify the differential diagnosis, whether quarantine or isolation is needed, and the treatment plan. This is our failsafe pathway when we consider the clinical context of the person who is testing.”
The “new normal” in America is self- and home testing, says Kost, and such tests are readily available online and in neighborhood pharmacies. The antigen tests are “reasonably priced for America” (starting at $23.99) but far too expensive for resource-limited countries where affordability is in the $3 to $5 range.
In unique settings—e.g., “musicians performing, teachers teaching, or worshipers praying—Kost recommends that people screen temperatures and then deploy rapid antigen tests before engaging in the activity. As an example, a musician in Davis might obtain a test at a local pharmacy in minutes via a bicycle ride, complete the test at home, and have results in 15 minutes. The overall process would take less than 30 minutes and would ideally be repeated 24 to 36 hours later.
Manufacturers generally have a disclaimer for the second test protocol, Kost says, which is why more large, multicenter studies in diverse populations are needed to prove whether double testing follows the “theoretical prediction” of improved performance.
Having recently spent several months in Cambodia, Kost says, he knows international airline travel right now is rough, requiring a negative rapid antigen test or PCR molecular assay results within 72 hours of departure as well as prior to returning to the U.S. “This can be extremely problematic if you go abroad and end up with a positive COVID test, throwing yourself into what can be incredibly long periods and ordeals of quarantine.”
On the other hand, individuals might carry the home kits with them and test negative along the way, he adds. Some airlines are accepting self-testing results with online documentation of the results.
Point-of-Care Culture
Everyone in society needs to be empowered with point-of-care strategies to keep them safe from what is now, thanks to the Delta variant, an endemic disease, says Kost. “Delta is spreading asynchronously in Africa, America, the Indonesian Archipelago, and other regions and thereby establishing itself in communities.
“Troublingly,” he continues, “low vaccination rates are permitting more dangerous variants to arise. New novel variants are rendering, potentially, vaccines partially ineffective. Vaccinated people are becoming infected now and they spread variants rapidly.”
All of this is characteristic of an endemic disease, Kost says. “The logical conclusion is that COVID is here to stay. That means a lot of testing has to be done in the future, new techniques, cheaper techniques, more access, and more speed.”
An “interesting new question” that has recently been posed by the print media in the U.S. is whether governments are prolonging the pandemic, Kost says, and one could say yes in several respects. In Cambodia, for example, people who test positive or are suspected of Delta infection are being held in designated treatment centers for at least three weeks and release depends on two negative tests 72 hours apart on days 16 and 19 or repeated indefinitely until negative, all followed by two more weeks of quarantine at home.
That means five weeks, minimum, of involuntary detention, says Kost, who disagrees with the approach. “I think Cambodia is trying to control Delta contagion with testing after the fact rather than being anticipatory” and discovering where it is and managing the risk.
Private industry could perhaps “save the day,” he says. United Airlines, for example, is encouraging people to get test 72 hours before international departure and to carry kits for web-guided self-testing before returning to the U.S.
“I like the freedom choice,” Kost says. “I want people to have access to rapid response testing in homes, communities [with, for example, mobile stations], workplaces, and abroad.” He advocates “anytime, anywhere” testing with a wide variety of free diagnostics available. This, he says, would empower risk avoidance and management by individuals, families, and society.
He refers to this as the “point-of-care culture, [the] vision for the future coming alive now,” which could be implemented in Cambodia by ramping up testing along its borders with Thailand and the Mekong region. In one of the provinces in West Cambodia, he has identified the optimal geospatial sites for COVID-19 rapid antigen testing at borders and in emergency rooms.
In addition to anywhere, anytime testing access, Kost concludes, people need an FDA that promotes higher-quality rapid antigen tests and he believes “the practical need” will ensure that happens.
Test results need to be used to avoid and manage risk rather than punish individuals, he says. “People living in a resource-limited environment can’t possibly tolerate being out of work for five weeks. Their families will starve.”
If people are not vaccinated, Kost advises testing them weekly and improving the performance of the assays “so this routine will be highly effective.” California recently became the first state in the nation to require teachers to be vaccinated or be tested weekly, he notes.
Furthermore, Kost says, “don’t stigmatize positive test results by requiring unnecessary prolonged quarantine or shameful detention… police do not belong in the mix.”
Instead, he says, conserve resource and end quarantine when evidence-based test results turn negative. Once an infected individual tests negative, they can get back to work within seven to 10 days “with almost no risk to others.”