Communicating the Imperfect Diagnostic Accuracy of COVID-19 Rapid Antigen Self-Tests: An Online Randomized Experiment

Objective To investigate the potential impacts of optimizing coronavirus disease 2019 (COVID-19) rapid antigen test (RAT) self-testing diagnostic accuracy information. Design Online randomized experiment using hypothetical scenarios: in scenarios 1 to 3 (RAT result positive), the posttest probability was considered to be very high (likely true positives), and in scenarios 4 and 5 (RAT result negative), the posttest probability was considered to be moderately high (likely false negatives). Setting December 12 to 22, 2022, during the mixed-variant Omicron wave in Australia. Participants Australian adults. Intervention: diagnostic accuracy of a COVID-19 self-RAT presented in a health literacy-sensitive way; usual care: diagnostic accuracy information provided by the manufacturer; control: no diagnostic accuracy information. Main Outcome Measure Intention to self-isolate. Results A total of 226 participants were randomized (control n = 75, usual care n = 76, intervention n = 75). More participants in the intervention group correctly interpreted the meaning of the diagnostic accuracy information (P = 0.08 for understanding sensitivity, P < 0.001 for understanding specificity). The proportion who would self-isolate was similar across scenarios 1 to 3 (likely true positives). The proportion was higher in the intervention group than in the control for scenarios 4 and 5 (likely false negatives). These differences were not statistically significant. The largest potential effect was seen in scenario 5 (dinner party with confirmed cases, the person has symptoms, negative self-RAT result), with 63% of the intervention group and 49% of the control group indicating they would self-isolate (absolute difference 13.3%, 95% confidence interval: −2% to 30%, P = 0.10). Conclusion Health literacy sensitive formatting supported participant understanding and recall of diagnostic accuracy information. This may increase community intentions to self-isolate when there is a likely false-negative self-RAT result. Trial registration: Australia New Zealand Clinical Trial Registry (ACTRN12622001517763) Highlights Community-based diagnostic accuracy studies of COVID-19 self-RATs indicate substantially lower sensitivity (and higher risk of false-negative results) than the manufacturer-supplied information on most government public Web sites. This online randomized study found that a health literacy–sensitive presentation of the imperfect diagnostic accuracy COVID-19 self-RATs supported participant understanding and recall of diagnostic accuracy information. Health literacy–sensitive presentation may increase community intentions to self-isolate after a negative test result where the posttest probability is still moderately high (i.e., likely false-negative result). To prevent the onward spread of infection, efforts to improve communication about the high risk of false-negative results from COVID-19 self-RATs are urgently needed.

Although the pandemic has been officially declared as over, 1 the SARS-CoV-2 virus continues to spread in communities, causing morbidity and mortality from COVID-19. 2 A key strategy to prevent onward spread is testing to facilitate detection and isolation of SARS-CoV-2-infected people until they are no longer infectious. 3,4The use of reverse-transcription polymerase chain reaction (RT-PCR) tests was previously the mainstay for identifying COVID-19-positive cases.However, access to RT-PCR tests is now more limited and requires a referral from a general practitioner or nurse practitioner.Self-testing using a rapid antigen test (RAT) is now the mainstay method of identifying COVID-19 infection in Australia. 5ATs offer several advantages over RT-PCR tests, including allowing self-testing at home with results available within 15 to 20 min 6 and high specificity for infectiousness. 7The use of RATs for self-testing was first approved by the Australian Therapeutic Goods Administration in November 2021. 8The World Health Organization recommends that RATs have a minimum sensitivity of 80% and a minimum specificity of 97% among symptomatic individuals. 9However, when RATs are used in a real-world setting, their clinical sensitivity often fails to meet this minimum threshold, 6 especially when used as a self-test by consumers. 10Despite findings of research studies of suboptimal sensitivity, the publicly available information on the accuracy of RATs in use in Australia is that provided by test manufacturers. 8This is likely to lead consumers to believe the tests have very high sensitivity and can be relied on for ruling out COVID-19 infection.
Overestimation of the sensitivities of RATs may lead to underappreciation of the chance of false-negative test results, resulting in avoidable onward transmission of SARS-CoV-2 and new cases of COVID-19.The understanding and interpretation of test results may improve if information provided is based on decision science principles, 11 is health literacy sensitive, 12 and includes graphics such as pictograms or photographs. 13e aimed to investigate the potential impacts of providing diagnostic accuracy information that is more applicable to consumers undertaking self-testing 14 and presented in a health literacy-sensitive way, compared with the accuracy information currently provided to consumers.Sydney School of Public Health, Faculty of Medicine and Health, University of Sydney, Sydney, NSW, Australia (HL, MK, LZ, DMA, MT, CB, KJLB).The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Financial support for this study was provided in whole by a grant from the Australian National Health and Medical Research Council (NHMRC) investigator grant No. 1174523 (recipient: Bell).The funding agreement ensured the authors' independence in designing the study, interpreting the data, writing, and publishing the report.*These authors contributed equally to this work and are co-lead authors.

Methods
A summary of the methods is provided below.and a more detailed description is provided in the prespecified study protocol. 15

Setting and COVID-19 Context
The study was conducted from December 12 to 22, 2022, during the Omicron wave in Australia, when there was an average of 15,937 new COVID-19 cases per day nationwide. 16At this time, there was no longer mandatory reporting of positive RATs nor isolation of COVID-19-positive cases in Australia, both requirements having ceased in October 2022.Although health policies continued to encourage these behaviors, the responsibility to do these and other preventative behaviors was left up to the individual.

Study Population
Participants were recruited from the general Australian public through an independent social research company (Dynata), which has a panel of 600,000 participants whose demographic characteristics align closely with those of the national population.Dynata has a points system in which participants receive points after completing surveys.The points can then be used to redeem vouchers, cash, or other rewards.Participants were eligible if they were at least 18 y of age, able to read and understand English, and resided in Australia.Stratified sampling was used, with quotas in place for gender (50% male, 50% female or other), age (50% younger than 40 y, 50% 40 y or older), and education (50% no university degree, 50% university degree).

Design
Participants were randomly assigned to 1 of 3 study arms: intervention arm, diagnostic accuracy information from a community-based study of self-testing provided in a way that was health literacy sensitive; usual care arm, manufacturer supplied diagnostic information provided; or control arm, no diagnostic accuracy information provided (allocation ratio, 1:1:1).

Interventions
Hypothetical scenarios.Participants in each of the 3 randomized groups were asked to read through the materials for the condition they were allocated to and to imagine they were in 5 different hypothetical scenarios that were presented sequentially.In all scenarios, there was a high probability that the individual was infectious with COVID-19, but the RAT results differed across the 5 scenarios.In scenarios 1 to 3, the RAT result was positive, and posttest probability was considered to be very high (likely true positives).In scenarios 4 and 5, the RAT result was negative, but the posttest probability was still considered to be moderately high (likely false negatives; Box 1).Diagnostic accuracy information.We used diagnostic accuracy information for the Flowflex RAT.This test is approved for use in Australia, and diagnostic accuracy information was available from a large independent community-based study of self-testing 14 as well as from the manufacturer. 8,18he intervention arm used clinical sensitivity and specificity from the community-based study of the Flowflex RAT. 14 This study was considered to be at low risk of bias. 10 The diagnostic accuracy information was presented in health literacy-sensitive information using graphical methods (Figure 1).The content and format of the intervention was developed and evaluated using a ''universal precautions'' approach to health literacy.19 This included using the Sydney Health Literacy Lab Health Literacy Editor 20 to simplify the language and reduce the grade reading level of the text, use of white space and formatting to reduce information overload, and inclusion of supporting images including icon arrays, in line with recommended best practice for patient decision aids.21 The usual care arm instructions were the Flowflex RAT instructions and diagnostic accuracy information provided in the Flowflex RAT kit by the manufacturer, downloaded from the Therapeutic Goods Administration Web site at the time of the study 8,18 (Figure 2).The control arm had no information provided on the diagnostic accuracy of the test.

Procedure
The questionnaire and randomization were created using Qualtrics, which uses the Mersenne Twister pseudorandom number generator to randomize participants. 22Participants and researchers were blinded to the allocation sequence until completion of data collection.After eligibility screening and provision of consent, participants were randomized, read the diagnostic accuracy information, and answered questions to see how well they understood the information (if in the intervention or usual care arms).They then read the hypothetical scenarios and completed outcome measures.Finally, participants answered demographic questions and a health literacy We assumed a high pretest probability across all 5 scenarios, consistent with the information provided and the context of high community transmission in the Australian community at the time of the study (December 2022).Using the sensitivity of 52% and specificity of 99% found in the community-based study of the Flowflex RAT (positive likelihood ratio 52, negative likelihood ratio 0.48) and assuming a pretest probability of 50%, we calculated that posttest probabilities were .80% across scenarios 1 to 3 with a positive RAT result, which we considered to be very high.Assuming a pretest probability of 50% in scenarios 4 and 5 with a negative RAT result, we calculated posttest probabilities of 25%, which we considered moderately high (i.e., probability is still well above suggested management decision thresholds to self-isolate). 17question (single-item literacy screener 23 ).The full questionnaire is provided in the Supplement.
Outcome measures.Box 2 outlines the prespecified primary, secondary, and process outcome measures.For scenarios 4 and 5, an additional post hoc secondary outcome (not prespecified) was also analyzed: participants who indicated they would neither stay at home nor avoid people at higher risk.This variable was created from responses to the primary outcome and secondary outcome 5.
Analysis.We summarized participants' characteristics at baseline for the 3 randomized groups using frequency and percentages.We estimated the effects of the intervention and of usual care (relative to control) by comparing outcomes across randomized groups.To address the issue of repeated measurements nested within participants across the 5 scenarios, we used generalized estimated equation statistical models.All statistical analyses were conducted using R. 24 Sample size.A sample size of 219 participants (73 per group) with 1:1:1 allocation to intervention or usual care or control groups was calculated to ensure that we would have 80% power to detect a pairwise difference in the proportion choosing to self-isolate as small as 10%.The assumptions were that 76% would choose to isolate in the control group for each scenario, a 10% dropout rate (individuals who do not complete the survey or who complete it faster than 1 min who are likely to have not read the survey questions), a = 0.05, the normal approximation to the binomial distribution, and the standard formula for comparing proportions in independent equalsized groups.

Ethics/Trial Registration
The Human Research Ethics Committee of the University of Sydney approved the study (2022/419), and it was Graphical presentation of clinical diagnostic accuracy estimates from a population-based study of the Flowflex RAT. 14 registered with Australia New Zealand Clinical Trial Registry (ACTRN12622001517763).This study was piloted with a convenience sample of 25 individuals prior to data collection.This identified problems with the branching logic, typos, wording, formatting, and other minor changes, allowing correction.
The study is reported according to the Consolidated Standards of Reporting Trials (CONSORT) guidelines. 25

Results
The sample comprised 226 individuals randomized to the 3 arms (control n = 75, usual care n = 76, intervention n = 75; see Figure 3 for the flow of participants).There were no dropouts (all participants completed the survey, with the shortest completion time being 2 min).
Characteristics were well-balanced across randomized label conditions, including the quota characteristics of location (state and territories), age, gender, and education (Table 1).Most participants were living with someone, born in Australia, non-Indigenous, spoke English at home, and had high health literacy.Nearly one-quarter (23%) had used a COVID-19 test in the past month, just less than half (42%) had a history of COVID-19 infection, and more than three-quarters (73%) had at least 3 doses of a COVID-19 vaccine.Just more than half (55%) were in full-or part-time work, and just less than half (46%) had an employer who supported them to work from home if sick.The results of separate scenario evaluations are presented in Table 2 and Figure 4, while Figure 5 presents the results when all scenarios are evaluated together within the generalized estimated equation model.

Process Outcomes
The proportion of participants who correctly recalled the test sensitivity provided to them was 72% in the intervention group (n = 54/75) and 80% in the usual care group (n = 61/76) (sensitivity provided was 52% for intervention and 97.1% for usual care groups; no information on test sensitivity was provided for the control group).The proportion of participants who indicated understanding of what this meant in terms of number of infected people who would test positive on the RAT was 64% in the intervention group (n = 48/75) and 50% in the usual care group (n = 38/76).The proportion of participants who correctly recalled the test specificity provided to them was 75% in the intervention (n = 56/75) and 78% in the usual care group (n = 59/76; specificity provided was 99% for intervention and 99.5% for usual care groups).The proportion of participants who indicated understanding of what this meant in terms of number of Box 2 Outcomes 1.Primary outcome: intention to self-isolate (yes/no; this was dichotomized from a 5-point scale: Yes (Stay at home without exception, except for shopping, except for work, except for shopping and work) v.No (Continue to leave the house as normal) Secondary outcomes: 2. Would do a RAT (yes/no) 3. Would isolate from other household members (yes/no/not applicable) 4. Would do further tests (yes/no) 5. Would avoid visiting people at higher risk of developing complications from COVID-19 (such as older people) (yes/no) 6. Would avoid crowds (yes/no) 7. Would keep 1.5 m away from others (yes/no) 8. Would wash hands more often (yes/no) 9. Would wear a mask (yes/no) Process Outcomes 10.Understanding of the test information.This was measured in 3 ways: 1) perceived sensitivity and specificity of the test after reading the test information (0-100 for each), 2) perceived reading ease of the material (5-point item from very difficult to very easy), 3) perception on whether the information provided in the study helped the participant know what to do if the test result is positive/negative (5-point item ranging from strongly disagree to strongly agree).11.Participants' perceived chance of being infected with COVID-19 or still being infectious with COVID-19.This outcome was measured in 2 ways: 1) 5-point response item ranging from very unlikely to very likely and 2) perceived chance between 1 and 100.
RAT, rapid antigen test.noninfected people who would test negative on the RAT was 57% in the intervention group (n = 43/75) and 24% in the usual care group (n = 18/76).The perceived reading ease and usefulness of the provided information was similar for the 2 groups, with most participants indicating that the information was easy/very easy to read and agreeing/strongly agreeing that the information helped them (Supplemental Table S1).The participants' perceived probability of being infectious with COVID-19 was similar across all 3 randomized groups and was higher for scenario 1 than for the other scenarios (Supplemental Table S2).

Primary Outcome
Table 2 and Figure 4 present the findings for the primary and secondary outcomes for the 5 scenarios.The proportions of participants who would self-isolate (primary outcome) were similar across the 3 randomized conditions for scenarios 1 to 3 (likely true positives).The proportions who would self-isolate were higher for those in the intervention condition than the usual care or control conditions for scenarios 4 and 5 (likely false negatives), although these differences were not statistically significant.The largest potential effect of the intervention was seen in scenario 5 (dinner party with confirmed cases, person has symptoms).Here, 62.7% of the intervention group indicated that they would self-isolate, compared with 49.3% of the control group (absolute difference 13.3%, 95% confidence interval [CI]: 22% to 30%, P = 0.10).When participants' responses across scenarios were combined in the generalized estimated equation model, the odds ratio for self-isolation was 1.43 (95% CI: 0.91 to 2.23) for intervention versus control (P = 0.12; Figure 5).

Secondary Outcomes
The proportion of participants who stated they would use a RAT after being presented with the scenario information was consistently high across the 3 randomized groups.The lowest proportions were seen for scenario 3 (day 6 after a positive RAT result, 2 d without symptoms, Figure 2 second row), in which 71% of the 226 participants indicated they would do a test.The proportion who would report the positive RAT result (only asked for scenario 1) was also high and similar across the 3 randomized groups, with 85% of the overall sample stating that they would do this (control group: 80% [95% CI 70% to 88%], usual care: 90% [95% CI 81% to 95%], intervention 87% [95% CI 78% to 93%]).Of those who indicated they would self-isolate, most would also isolate from other household members, and this was again similar across the 3 randomized groups.This was lowest for scenario 3, in which 80% of n = 150 participants who would self-isolate and who did not live alone indicated they would also isolate from others in their house.
Although the ''further preventative actions'' was intended to indicate actions taken after self-isolation, many participants appeared to not interpret it this way.In retrospect the wording for this question was unclear as it did not specify the timing for further actions (''What further actions would you take (select all that apply)?''; see full questionnaire in the Supplement).This resulted in a lower proportion stating they would avoid visiting people at higher risk of developing complications from COVID-19 (such as older people) in the Intervention group compared to the Control and Usual Care groups in all Scenarios (Table 2, Figure 2 row E).The differences were largest in in the negative RAT result Scenarios 4 and 5.The odds ratio for avoiding visiting higher risk people for the 5 scenarios combined in the generalized estimating equation model was 0.63 (0.38 to 1.05; P = 0.07; Figure 5).Post hoc analyses found that these differences were driven by people who said they would not avoid visiting people at higher risk because they would be self-isolating at home (Supplemental Table S3).After viewing the likely false-negative test results in scenario 4 (household contact of confirmed case, with symptoms) and scenario 5 (close contact of known cases at a dinner party, with symptoms), the proportion of each randomized group who said they would neither stay at home nor avoid people at higher risk was the same for both scenarios: 24.0% of the intervention group compared with 20.0% of the control group (absolute difference 4.0%, 95% CI: 210.6% to 18.6%, P = 0.69).In the overall sample, 42 people (18.6%) stated they would take neither preventative action.
The proportions who would get further tests, avoid crowds, keep 1.5 m away from others, wash hands more often, and wear a mask in crowded situations were similar across the 3 randomized groups.The proportion of the overall sample who would take each of these further actions ranged from 51% (wear a mask in crowded situations, scenario 5) to 82% (get further tests, scenario 1; Table 2, Figure 2 rows D-I).

Discussion
We found strong evidence that health literacy-sensitive formatting supported participant understanding and recall of diagnostic accuracy information for a COVID-19 RAT.Most participants correctly recalled the diagnostic test accuracy estimates that were provided to them, and more of the intervention group correctly interpreted the meaning of these (P = 0.08 for understanding sensitivity, P \ 0.001 for understanding specificity).More than double the participants in the intervention group indicated that they understood what the specificity meant relative to those in the usual care group.
Despite participants being more informed in the intervention group, this did not translate into statistically significant increases in intentions to isolate and prevent onward spread.Nevertheless, we found possible beneficial effects on intentions to isolate across all scenarios, strongest in scenario 5 (where the overall sample were least likely to isolate).In this scenario, there was an absolute difference of 13% more who would choose to self-isolate.
An apparent possible harm of the intervention, with a higher proportion of people in the intervention group expressing intent to visit high-risk individuals, appears to be an artifact of the questionnaire design.Further analysis revealed that the proportion who reported they would neither self-isolate nor avoid visiting higher risk people was similar across randomized groups.In the overall sample, most people would do and report the test, but fewer people would do additional actions in the likely false-negative scenarios such as wearing a mask in a crowded place (51% in scenario 5).Of concern, nearly 20% of the overall sample stated that they would neither self-isolate nor avoid visiting higher risk people after receiving a negative result in scenario 5, which we considered to have a moderately high posttest probability of infection.
Many studies have found that the sensitivity of RATs is lower in a real-world setting because of factors relating to the study population and the implementation of the test. 6,10,26,27Patient factors include the viral load for infected patients, whether the individual is asymptomatic, and the severity of the illness if symptomatic. 6mplementation factors include whether the testing is completed within 7 d from symptom onset and whether the sample is self-collected or collected by a health professional. 10,26,27However, knowledge of the poorer diagnostic accuracy of RATs when used by a self-test by consumers in real-world settings appears to low in the general community. 28At the time of our study (and still the case now), published information on the accuracy of RATs on the Australian Government's Therapeutic Goods Association Web site (provided by test manufacturers) indicated substantially higher sensitivity than actual performance when used as intended-as a self-test by community members. 10thers have also undertaken research to address the need for better communication about RATs for selftesting use.A randomized online study of the interpretation of COVID-19 self-test results by 360 adults in the United States recruited in April 2021 found that after receiving likely false-negative results, participants receiving the intervention instructions (based on decision science principles) were more likely to self-isolate than participants receiving the standard authorized instructions or no instructions. 11That study focused on optimizing the instructions to support the person to undertake appropriate further actions, whereas our study focused on optimizing communicating the high risk of a falsenegative result.Both studies found positive effects for the interventions that were similar size (absolute difference of 13% for our study and 14% for the US study).Whereas the US study found statistically significant benefits against the usual care condition but not the control, we found larger benefits against the control than usual care (and in our study, neither comparison was statistically significant).
The strengths of the study include its randomized design to investigate the effects of optimizing communication about COVID-19 RAT diagnostic accuracy on people's intentions to self-isolate and take other preventative actions.This study contributes high-quality evidence to the body of research investigating how to best communicate test results to patients and other community members and to the communication about COVID-19 RATs in particular.This study sample included geographical, gender, and age diversity; however, it was not representative of the overall Australian population on many other characteristics (higher proportion of participants were born in Australia, spoke English at home, had a university diploma or degree, and had higher health literacy than the general population did).
As well as a sample with several indicators of more socioeconomic advantage than the general community, there were several other limitations.Although the sample size was chosen based on reasonable assumptions, it is likely that we did not have sufficient statistical power to detect effects of the intervention that did truly exist.Furthermore, the hypothetical nature of the online experimental design of the study means that our findings may not generalize to what participants would do in real life-although the high proportion who had past COVID-19 infection and the prominence of the pandemic in day-to-day life at the time of the study may mitigate against that.A follow-up study investigating what participants actually did after contracting COVID-19 infection, rather than their stated intentions, would be informative.Further research to explore the relative weighting that people place on symptoms versus test results may also be helpful.The likely misinterpretation of the questions regarding further preventative actions that indicated actions taken instead of self-isolation instead of after self-isolation indicates poor wording of this element of the questionnaire that was not detected on pilot testing.We did not provide information to participants about why the sensitivity of the test was suboptimal, and doing so might have helped them to conceptualize why results may be unreliable.This may have more effectively countered manufacturer information on the Australian Government's Therapeutic Goods Authority Web page and in the test kits themselves that indicated high sensitivity for COVID-19 RATs.The study was conducted in December 2022 when mandatory self-isolation had only recently been uplifted.The proportion of people in the control group indicating they would take preventative behaviors was very high, suggesting likely ceiling effects.It is likely that fewer people in the control group would self-isolate now, given the longer period since the restrictions were lifted and changing societal expectations about preventative behaviors.It is possible that the intervention would have a larger effect now.In addition, some participants may have already been aware of the risk of a false-negative result from information outside of the study, including past experience with RATs; this would have lessened the contrast across randomized groups.Finally, our study design did not allow separate evaluation of the 2 components of the intervention (different diagnostic accuracy information and health literacy sensitive presentation).
Australia now relies on the public to choose to selfisolate and take other precautionary measures for COVID-19 prevention and control, and self-RATs play a pivotal role in this.The evidence base for the diagnostic accuracy of COVID-19 self-RAT when performed in a real-life setting and as intended (i.e., unsupervised consumer collection and interpretation) suggests a high risk of false-negative results. 10Concerted efforts are needed to increase community awareness of the issue and to prevent the false interpretation that 1 negative result can rule out infection when there is suspicion of COVID-19.
Although our intervention increased the understanding of both test sensitivity and specificity, we found a stronger effect for the latter.Future research (for COVID-19, or other conditions) could build on this finding and explore the impacts of providing health literacysensitive diagnostic accuracy information in scenarios where it is likely the individual does not have the target condition (i.e., likely true-negative and likely false-positive results).For COVID-19 RATs, these results would indicate scenarios where self-isolation is unnecessary.

Conclusion
This study shines a light on the need for better communication on the limitations of self-tests.In this way, the public can make better informed decisions about preventative behaviors that limit the onward spread of infection.

Figure 1
Figure 1 Diagnostic accuracy information for intervention.

Figure 2
Figure 2 Diagnostic accuracy information for usual care.

Figure 3
Figure 3 Study design and flow of participants.

Figure 4
Figure 4 Effects of the intervention and usual care compared with control in each of 5 scenarios.(A) Would do a RAT (rapid antigen test), (B) Would stay at home, (C) Would isolate from other household members, (D) Would do further tests, (E) Would avoid visiting people at higher risk, (F) Would avoid crowds, (G) Would keep 1.5m away from others, (H) Would wash hands more often, and (I) Would wear a mask in crowded situations.

Figure 5
Figure 5 Effects of the intervention and usual care compared with control across the 5 scenarios (generalized estimating equation models).
Box 1 The 5 Hypothetical Scenarios in the Survey a 17Posttest probabilities were calculated using the methods outlined in an article presenting Fagan's nomogram for coronavirus disease 19 (COVID-19).17

Table 1
Baseline Characteristics of Randomized Groups

Table 2
Effects of the Intervention and Usual Care Compared with Control in Each of the 5 Scenarios a RAT, rapid antigen test.Primary outcome = would stay at home.