Examining self-described policy-relevant evidence base for policymaking: an evidence map of COVID-19 literature

Study design

We employ a variant of the Evidence Gap Map in this study. Evidence mapping allows the synthesis of information available in an area or subarea of interest.10 This approach provides information that can be used by policymakers, civil society, scientists and other interested stakeholders seeking a clear view of the current state of knowledge in public health policymaking, which is foundational for monitoring, evaluating and criticising public policymaking in a specific area. Our approach intentionally restricts consideration to self-described policy-relevant evidence to reflect how policymakers would search the literature rather than survey the literature as a scientist to identify research gaps. This method was employed to answer the overarching research question of ‘how has evidence relevant to evidence-based COVID-19 pandemic policy been presented in the literature from inception to 2022?’ Note that evidence was not restricted but included any discipline, for example, scientific, social, cultural and economic disciplines. This question was answered on the premises of six secondary research questions as follows:

1. What is the stated purpose of evidence related to COVID-19 pandemic policy?

2. Where in the policy hierarchy and the policy cycle is evidence directed or used?

3. What type of evidence is provided and what methods are used to obtain evidence of relevance to policy?

4. Are there differences between contributions based on developing vs developed countries?

5. Are there differences in evidence availability, for example, open access (OA), which would influence policymaker ability to engage with articles?

6. What are the article authors’ stated ‘challenge(s)’ for perceived (non)effectiveness of the policy?

Search strategy

We systematically reviewed the literature in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) as outlined by Moher et al.15 Figure 1 shows the study selection flowchart following the PRISMA guidelines. We searched the SCOPUS database including PubMed from 2020 to 2022 for the keyword combinations ‘evidence based’, ‘evidence informed’, ‘evidence led’ AND ‘policy’, Science in/and/to policy, Science in/and/to governance AND ‘COVID’ OR ‘SARS’ in the title, abstract or keywords (see online supplemental material for detailed search string). We consulted a reference librarian who confirmed our search string to be appropriate and the number of papers detected to be within the bounds. We also searched LexisNexis using the same search terms but did not detect any additional materials.

Inclusion and exclusion criteria

Article metadata were extracted and exported in Microsoft Excel (Office V.365) for screening and selection. Articles were excluded if duplicated, or not in English, while peer-reviewed contributions identified as ‘article’, ‘book chapter’, ‘review’ were retained. However, whole books, articles with no authorship attribution and/or no abstracts, were filtered out and excluded directly from the search data bases and were not picked up for screening. The study included only peer-reviewed literature covering the period from January 2020 to December 2022. The full search terms used for this study are found in online supplemental material 1.

A secondary search of title/abstract/keywords within the first set of articles was conducted to identify the COVID-19 pandemic-relevant subset using the terms ‘COVID’ or ‘SARS’. At least two authors independently screened the titles, abstracts and keywords of all articles to assess if the focus was relevant to the COVID pandemic. Discrepancies were resolved through discussions between the two authors and if cases were still unresolved, a third author would serve as a tiebreaker. Subsequently, three authors independently examined the abstracts to determine the ‘pandemic response focus’ and these were categorised as—(a) prevention (including animal-vector management), (b) risk or surveillance (risk assessment and infection surveillance including epidemiological models), (c) impact studies, (d) response action (immediate actions aimed at addressing the medical consequences for individuals including pandemic response strategies, risk management and communication), (e) control (relating to control of spread of the disease including declaration of endemic status), (f) recovery (short-term and long-term stabilisation and sustainability plan including rehabilitation) and (g) others.

Data extraction and evidence map analysis

Full texts of the articles categorised as having ‘COVID-19 pandemic policy relevance’ were retrieved for data extraction and analysis. Two hundred and thirteen articles were fully inspected and categorised to generate evidence maps. Bibliometric information and metadata were extracted, including publication date, number of authors, country of first author affiliation (UNCTAD 2022 regional categories).16 We categorised article authorship by sector (ie, academia or research organisation, government at international, national or local levels, Non-Governmental Organisations (NGOs), industry, consultant). Additionally, we identified OA categories (green, gold, bronze and hybrid gold) to highlight ready availability to policy and decision-makers. Bibliometric analyses were performed on extracted data and presented as medians (and IQRs), proportions and descriptive statistics. Binary logistic regression analysis was used to model the odds of authors publishing alone, publishing from the same country against the gross domestic product (GDP) classification (developing and developed countries). We extracted information from full texts to evaluate information relevant to COVID-19 policymaking including policy cycle focus, policy level, evidence type and the authors’ assessment of policy success.

Residual questions’ analysis/search criteria

Each of the 213 articles was searched by at least two authors to identify residual or further pandemic research-related and/or policy-related questions. We read the discussion and conclusion sections of each article and additionally used the following search terms: “question”, “?”, “need” and/or “gap” to help in detection of questions across full articles. Residual research questions, policy needs or gaps and the context such questions/needs/gaps were framed were extracted into MS Excel. The residual questions were further grouped into five categories (clinical; society/public health; economic; policy/governance and technological/technical). Where necessary, questions were assigned to multiple categories.

Purpose of evidence/policy cycle focus

The most common pandemic response focus of articles analysed was response strategy (43.7%), followed by general measures (36.6%) and then with risk/surveillance (26.3%) and control measures (23.5%). The recovery, impact and prevention measures were each less than 3% of articles (figure 2).

The most common policy purpose of articles was the presentation of research findings the article authors considered of relevance (60.1%), followed by work that examined the impact of policy (28.6%) or highlighted or supported a policy need (22.5%).

Policy-level focus

The majority (86.4%) of articles were oriented towards operational policies including management strategies, regulations and action or implementation plans, with 26 focused at government-level policies, 14 supporting legislation and 21 that were unspecified (online supplemental figure 3).

Type of evidence

About 61% of articles presented primary evidence (130 articles), 27.7% presented secondary evidence (59 articles) while 11.3% (24 articles) were opinion papers. The evidence map of research types (figure 3) highlights that two types of primary evidence are dominant, (a₁) experimental research (laboratory, field, computational studies) and (a₂) surveys (questionnaire surveys, interviews, group elicitation), mostly presenting findings that are deemed of relevance to policy. The majority of secondary evidence were reviews providing findings or evaluating policy impact.

Developed and developing economy disparities

Lead author affiliations were from 50 countries with highest outputs from developed countries (149 articles; 70.0%) such as the USA (43 articles; 20.2%), the United Kingdom (39 articles; 18.3%) and Australia (16 articles; 7.5%), whereas lead authors from 29 developing countries produced 64 articles (30.0%), including China (12 articles; 5.6%) and India (9 articles; 4.2%). Institutional affiliations for all authors represented 67 countries including 40 developing countries (seven Least Developed Countries; LDC); one country with an Economy in Transition and 26 developed countries (figure 4). The majority of articles were written by authors from the same country (141; 66.2%) and a single institution (68; 31.9%). Fifty articles (23.2%) were written by authors from multiple countries and multiple institutions, with a maximum representation of 20 countries (with 27 institutions) and 30 institutions (from two countries). The median number of authors was 5 (IQR 3–10). Lead authors from Developing Economies were less likely to publish alone (3/64 (4.7%); OR 2.11 95% CI 0.58 to 7.61, p=0.25) or solely with coauthors from within their country (38/64 (59.4%); OR 1.53 95% CI 0.82 to 2.81, p=0.17), than lead authors from developed economies (14/149; 9.4% and 103/149 articles; 69.1%, respectively).

We identified 186 articles with lead authors from academia or research organisations, 10 from government (at international, national or local levels), 7 from NGOs (including think tanks and foundations), 1 from a professional society and 3 from industry; no lead authors were identified from consultants (online supplemental table 1). We found 150 articles with authorship from a single sector: 144 articles written solely from academic or research organisation sector; 2 articles written solely from government; 1 with authors solely from NGOs, 1 solely from a professional society and 2 written solely by industry. The remaining articles had authors from multiple sectors with a maximum of five. The most common authorship collaboration was between academia or research organisations and government (online supplemental table 1).

OA availability

The majority of articles (199 articles; 93.4%) had at least one form of OA: 99 under green OA (access available through self-archiving of published or prepublication works), 99 gold OA (access available through the publisher), 39 bronze OA (free read, availability from publisher for a time period) and 30 hybrid gold OA (access available through the publisher but some other articles in the same journal are paywalled), see online supplemental figure 2. Most articles (93.1%) published with academic, or research organisation lead authors were under some form of OA, whereas 60.0% of articles with government lead authors and 28.6% of articles with non-governmental organisation, think tank and foundation lead authors were published under some form of OA. Lead authors from developing economies published OA at similar proportions to lead authors from developed economies (92.2% vs 93.6%, respectively), however lead authors from developing economies had higher proportions published in green OA (54.7% vs 43.0) and gold OA (54.7% vs 43.0%) and lower proportions of bronze OA (14.1% vs 20.0%) and hybrid gold OA (12.5% vs 14.8%).

Policy challenges

About 65% of articles (140) did not comment on the success or failure of a policy. Eleven articles, though not stating perceived policy success or failure, still provided an indication of challenges in the policy process. However, the remaining articles stated either policy success 41 (19.2%) or failure 32 (15.0%). In 10 articles stating policy success, the authors identified challenges that were overcome or required attention for policy improvement. Specific challenges were identified in all but one article stating policy failure (online supplemental figure 4). The most frequently identified challenges were failure of process and poor evidence (including failures of epistemic due care).

Residual questions’ analysis

All 213 articles were evaluated to extract residual or unanswered questions, needs or gaps as stated by the authors. Out of the 125 (58.7%) articles that stated residual questions for future/further research, 241 residual questions were extracted and analysed (see online supplemental material 2). The majority of residual questions 141 (58.5%) were general research questions relevant to public health policymaking for clinical, economic, social and technological advancement in response to the pandemic while 119 (49.4%) questions targeted improving or facilitating policymaking processes, including policy needs or gaps (online supplemental figure 5). All residual questions are available in online supplemental material 2.

Purpose of evidence including policy cycle focus

Pandemic surveillance, response and control strategies gained more traction with an abundance of evidence compared with prevention and recovery strategies in our study (figure 2). This has large public health implications on the way evidence is selected and used for policymaking. With WHO announcing on 5 May 2023, that COVID-19 is no longer a public health emergency of international concern (PHEIC),19 20 the global community is gradually transitioning from a pandemic to an endemic stage of COVID-19 with several countries applying various recovery strategies.21 The decision regarding transition to a non-PHEIC was based on declining COVID-19 deaths, hospitalisations and ICU admissions and on high levels of population immunity from vaccination and previous infections.19 Nonetheless, caution should be applied in the recommendations for the long-term management of the pandemic, acknowledging that risks remain high and that there are uncertainties in the evolution of the virus and our responses.

Unfortunately, our findings also revealed a paucity of evidence in support of recovery strategies with only four articles presenting evidence for community impacts, stabilisation and the economic recovery processes.22–25 We do take into consideration that the time constraints on our search (2020–2022) means that more recent research on recovery, drafted and published after the WHO’s announcement, are not included in our dataset. The short and long-term effects of COVID-19 pandemic response measures on health systems, commerce, education and employment are crucial, especially for economically vulnerable countries.24 For a successful recovery from the COVID-19 pandemic, some researchers have advocated for nations to adopt evidence-based frameworks to ensure community stabilisation and sustainability. Additional evidence will provide support for the prevention of the emergence and spread of pandemic-prone diseases including possible animal-vector management. Unfortunately, only a few of the articles reported on the unintended impact of COVID-19 policies and mitigation measures on the quality of life of the general public26–29 or articulated measures to support the improvement, planning and activation of public health control measures for the prevention of future pandemic.30–35 Key behavioural risk factors can be targeted for the development of strategies to reduce the threat of novel zoonotic diseases.35 Preparedness ensures that some thought and resources are devoted to generating the knowledge and materials needed to respond to and recover from a disaster or epidemic that has occurred.

Although all papers mentioned evidence-based or evidence-informed policy, many of the articles only did so peripherally. The purpose of over 60% of papers was to present research findings without a clear discussion of the research in the context of an existing policy, its impact or a policy need (figure 2). This could suggest several things. First, ‘evidence-based’ and ‘evidence-informed’ policies are catchphrases that are used to increase impact but are not often explained or understood. Second, most evidence collected were of the type that is appropriate to be presented as findings only. Other researchers have previously commented on the fact that most research used in EBPM during the pandemic was biomedicine centric.36 37 As such, it is possible that this type of research was explicitly encouraged, and that research on understanding policy implications, discussions of policy effects and the process of EBPM itself was considered less relevant or not given due consideration. This may also be a result of the types of questions that policymakers did or did not ask.37 38 Nevertheless, our current result could simply be a matter of timing, with fast evidence production at the start of the pandemic not being amenable to exploring in-depth implications for policies. Zaki and Wayenberg39 posit that wicked crisis like COVID-19 pandemic can compromise the quality of epistemic policy learning and offered simplified exploratory analytical framework to assist policymakers to manage the integration of scientific knowledge into policy responses within wicked crisis contexts. The framework provided a potential structure to assist policymakers in assessing the inclusiveness of policy learning during wicked crisis and accommodates for cross-case comparisons while ensuring interdisciplinarity and multiplicity of perspectives. This approach has a potential to improve policymakers’ ability to use scientific advice in policy formulation and public communication taking into consideration the uncertainty of emerging knowledge.

Policy-level focus/type of evidence

The majority of the evidence in our study focused on operational policies, including management strategies, regulations and action or implementation plans to tackle the pandemic. Only a few of the articles mentioned legislation as a policy process relevant to the pandemic.28 30 37 40–50 However, experience has shown that targeting legislation with the involvement of members of parliaments, selected citizens and scientists in evidence-based decision-making will ensure sustainability and improve the value acceptability of measures as well as their technical and financial feasibility.51

The high number of systematic reviews was not surprising, as systematic reviews and meta-analysis are considered the pinnacle of evidence hierarchies in EBPM discourse.52 However, despite the number of systematic reviews, their quality and usefulness in policymaking have been questioned.53 Arguably, reconsideration needs to be given to the propagation of systematic reviews for EBPM, and as Yang54 points out, this is particularly so in an emergency where ‘the evidentiary standard of EBPM needs to be adjusted’ to include, at the least, the informed judgement of medical professionals and other seasoned experts in the absence of clinical trials and other types of evidence commonly considered as ‘appropriate’.55 56

Developing/developed economy disparities and OA availability

Evidence continues to be predominantly produced by developed countries as depicted from our mapping (figure 4). This data are comparable with recent statistics that show that the leading producers of scientific research outputs are the USA, China, the UK and Germany, with a noticeable absence of South American and African countries in the top 10.57 This disparity mirrors the Global North/Global South dialogue (with exception of Australia and New Zealand). This is also in line with larger discussions on epistemic injustice and whose knowledge should count in emergencies like the COVID-19 pandemic.36 Equity and equality in the treatment of groups and regions in knowledge-related and communication practices have been a topic of discussion with some authors advocating for action-oriented framework of epistemic justice in healthcare, information science and education for sustainable development.36 58 59 It is clear that, building resilience and preparedness for future crisis would require strategies that embrace greater pluralism and above all addresses the inter-related inequalities through exploration of alternative social structures and facilitated by modern technological solutions.36 In this study, authors from developing countries were apparently more willing to engage in intersectoral and intercountry collaboration than their counterparts from developed economy. Interdisciplinarity and multicountry collaborations in crucial as it promotes credibility within the institutions of science advice, and this ensures robust policy options with greater inclusivity and a clear understanding of the impacts of policies especially on the most disadvantaged in the society.

Most of the evidence (93.4%) included in this study was available by OA and, therefore, accessible to policymakers directly or to knowledge brokers who can then transmit evidence to policymakers. In our research, material from lead authors from developing countries were available at similar OA proportions to lead authors from developed economies, however the type of OA varied. Open accessibility, however, continues to be a subject of debate due to the (lead) author-based funding requirement, with attention being drawn to the need to ensure that the knowledge gap between the developing/developed economies is bridged. A recent white paper examined the challenges of OA publication in lower and middle-income countries (LMIC) and how the playing field can be levelled to permit knowledge/evidence produced in LMICs to find a wider audience.60 The paper identified some practical options/ways of providing support for researchers in LMICs including their involvement in editorial boards and the peer review system; targeted capacity-building programmes for authors, reviewers and editors from LMIC, greater consistency and transparency around article processing charge waivers and support for the publication of research among others.60 These suggestions if adopted collectively will enhance the chances of researchers from LMICs, to achieve their OA publishing ambitions.

Our study also revealed a low level of intersectoral engagement in evidence generation with 150 articles having authorship from a single sector (online supplemental table 1). The overwhelming contribution of academics to publications observed in this study (144) highlights the concerning lack of engagement with knowledge dissemination by other sectors, potentially resulting from professional performance drivers and reward in various sectors or from deep-rooted publication biases. An effective global response to tackle public health emergencies must coordinate efforts across disciplines and backgrounds.

Policy challenges

The most frequent author-identified challenges in pandemic policymaking were ‘process failure’ and ‘poor evidence’ (including failures of epistemic due care). Based on the fast unfolding of the pandemic and the uncertainty associated with the consequences of the pandemic and measures to be developed, stated challenges and problems linked to the quality of evidence61–79 are germane. The quantity and quality of data generated and the evidence available in a crisis like the COVID-19 pandemic are usually limited and decision-makers must make important decisions quickly. Some authors have argued that evidence utilisation in public health emergencies may seem to be reduced especially if there is a lack of administrative accountability.80 This could explain why some of the authors stated a perception of failures in the uptake and consideration of evidence (process failure).25 44 45 49 61 74 81–95 Nevertheless, the majority of articles did not state any challenges concerning evidence uptake by policymakers, while 41 articles stated policy successes with apparent author satisfaction with the process of evidence uptake by policymakers. A small minority, however, raised issues of biases in the acquisition and consideration of evidence.45 63 67 91 96 97 This might have resulted from the fact that the urgency of the pandemic response required policymakers to engage with scientists and so scientists more often felt that their outputs were recognised and used for policy formulation. This can be contrasted with the EBPM process in other public health issues where there is a perceived lack of urgency.98 The risk of biases and errors in policy decision-making processes has the potential to cause widespread societal damages in a crisis context like COVID-19 pandemic. Some of the response strategies adopted during the pandemic were marred by perception of political influence on information management and bias in the selection and utilisation of evidence for decision-making for activities such as lockdown/movement restriction91 and school reopening97 despite available scientific guidance and practical evidence-based advice on how to manage infection risks. Analysis of COVID-19 lockdown strategy in five European nations identified the strengths and weakness of the process and revealed that while some nations reacted quickly and effectively, some others were flawed by their responses being slow, inconsistent and riddled with U-turns.91 Some of the authors identified information processing errors including failure to share updated information and/or revise and update conclusions and policies in the light of new information45; failure to take adequate account of scientific evidence44; lack of attention towards the nuances of sex/gender disparities within minority population61 and lack of transparency and accountability at local, national and international levels.49 It is, therefore, important that policymakers take steps to maximise the quality of the decision-making process to increase the chances of positive outcomes.

Our study extracted the residual questions and areas for future research prioritisation or policy setting, noted by the authors themselves. Understandably, there was a higher focus on questions addressing societal/public health issues around COVID-19 including impact, behavioural and implementation strategies. However, few questions sought to address issues around economic and financial sustainability or technological advancements with the potential to tackle future pandemics. A similar study by Liu et al,11 identified the ‘use of novel technologies and artificial intelligence’ as an area of COVID-19 research that remains underexplored. For instance, the need to provide access to accurate and low-cost tests for the diagnosis of COVID-19 remains a lingering challenge despite evidence that artificial intelligence and deep learning can enhance the detection and diagnosis of COVID-19.99 It could, therefore, be that the researchers are not asking the right questions or are asking questions that are skewed towards clinical management of the pandemic due to the urgency and evidence demand associated with the pandemic. Liu et al11 showed that the majority of the articles available at the onset of COVID-19 pandemic used manual statistical methods to monitor epidemiological trends as opposed to more robust and complex modelling techniques with promising public health applications.

Study limitations

This evidence map provides an overview of COVID-19 pandemic policy-relevant literature for EBPM. The authorship team had two authors independently coding at all stages and comparing the final outputs for consistency. However, we acknowledge several limitations. The search criteria/terms were designed to specifically target policy-relevant literature with ‘evidence-based’, ‘evidence-informed’ or ‘evidence-led’ policy keywords. As such, any article with no clearly stated relevance to policy was not captured irrespective of the outcome of interest. The limited literature contained in the dataset brings to the fore the need for researchers to properly articulate their policy impact statement in the research output to improve/facilitate uptake and use of evidence for relevant policymaking. This study systematically searched and synthesised COVID-19 policy-relevant evidence to identify gaps in knowledge, trends and future research needs and presented findings in a simplified way to facilitate uptake by policymakers. An in-depth qualitative analysis of synthesised evidence is not the focus of this project.

Evidence maps are not designed to assess the effectiveness of strategies or interventions. We, therefore, relied on the authors’ stated assessment/perception of policy success or failure and identified reasons for perceived policy failure. We conducted systematic searches in three databases Scopus, PubMed and LexisNexis. Therefore, some other policy-relevant literature may be available in other databases. Also, we did not consider grey literature. Another major limitation of this study is the exclusion of non-English-language articles, which led to the exclusion of 38 articles that may or may not be relevant. This was due to the limited language capacity of the team, which was mostly English speaking. This exclusion also limited our ability to make a categorical statement on the low contribution of the Global South as this may be a contributory factor.