Tool for knowledge gaps in guidelines

1. Introduction

A guideline is a document with recommendations aimed at improving the quality of care, founded on systematic summaries of scientific research and evaluation of the advantages and disadvantages of the different care options, and complemented by expertise and experiences from health care professionals and health care users (Kwaliteit van Zorg, 2011). Once knowledge has been synthesised in this process, the necessary evaluations of the desired and undesired effects of diagnosis are made along with other considerations. In the context of guideline development, we can only speak of a knowledge gap if one is found after the synthesis stage that hampers the evaluation of desired and undesired effects. This often relates to knowledge about desired and/or undesired treatment effects, but can also relate to a lack of knowledge about diagnosis and patient preferences regarding the different treatment options or about cost and efficiency.

Several different terms are used to refer to knowledge gaps. International organisations such as the National Institute of Health and Clinical Excellence (NICE) and the European CoCanCPG partnership speak of ‘uncertainties’. Ideally, all identified knowledge gaps are included in an international database that is available to researchers and funders. The status of knowledge gaps should be regularly updated in the light of new research results. This new knowledge can be used in the revision of guidelines and recommendations and to identify possible new knowledge gaps. This knowledge cycle is repeated time and again. Figure 1 shows the life cycle of 'uncertainties' developed by the CoCanCPG.

Figuur 1. Complete life-cycle in identifying uncertainties

The uncertainty life-cycle

2. Objective of this tool

This tool helps to identify knowledge gaps during the guideline development process and to promote research that aims to remove important knowledge gaps. This tool supports researchers and subsidy providers in selecting and funding themes that have a clear relevance to patient care.

3. Definition and forms of knowledge gaps

We speak of a ‘knowledge gap’ if, after the knowledge synthesis, a lack of knowledge is found to be hampering the evaluation of desired and undesired effects. The following knowledge gap forms can be distinguished:

  1. No published research has been found that addresses the review question (provided an optimal search has been conducted)
  2. The quality of the study found (one or more studies) was insufficient due to1:
    1. Low evidential value of the research design (for example, observational or non-comparative research on therapeutic interventions)
    2. The estimate of the effect size(s) is not precise (wide confidence intervals), for example, because the scope of the study was too small
    3. The research results are inconsistent and, as a result, no good conclusion can be drawn about the effect or effect size.
    4. The evidence is indirect because a different patient population was used than is addressed in the guideline, because of different outcomes or determinants or because only indirect comparisons were used.
    5. There is a high chance of reporting or publication bias (for example, resulting from strong conflicts of interests)

1 These are all determinants of the quality of the ‘body of evidence’ (one or more studies), according to the GRADE methodology (BMJ 2004;328:1490).

4. Identifying possible gaps of knowledge

Knowledge Gaps in guidelines that could be entered in a database must meet the following criteria:

  1. The gap regards an important clinical question (disease burden, absenteeism, costs, tax burden on relatives). Yes/no
  2. Filling the gap will affect the guidelines. The recommendations in the guideline may change the content of the guideline or turn a weak recommendation into a strong recommendation (or vice versa). Yes/no
  3. Answering the question potentially improves the quality of care and/or patient outcomes (for example, quality of life). Yes/no
  4. The gap is researchable and the planned research is feasible, that is, it has a realistic chance of success in the current research practice and infrastructure. This may related to the cost as well as ethical issues. Yes/no

If all four criteria are answered with a YES this indicates the presence of a substantial knowledge gap. If one or more of the criteria are answered with a NO, then the development group will need to discuss how useful it is to formulate a knowledge gap. To keep the number of knowledge gaps to be developed at a workable level, the development group can make a top five knowledge gaps that they can elaborate on as recommended in this tool. Applying these criteria should lead to the most important topics for new research being selected for each guideline.

5. Characteristics of knowledge gaps

The aim of identifying knowledge gaps is to guide the research agenda (or be involved in guiding the agenda). A database could be set up in which knowledge gaps are described according to a uniform format. Think of the following characteristics:

  • name of the guideline
  • publication date of guideline
  • organisation responsible for the guideline
  • (abbreviated) title of the knowledge gap
  • date of the gap
  • nature of the gap (see options listed under 'Definition')
  • keywords
  • review question to which the gap relates (patient population, intervention, diagnostic technique or marker, outcomes)
  • desired research design (RCT, observational study, further specification desirable) and
  • expected effect of the research results on the guideline or on answering the review question.

6. National database and prioritising

If all guideline makers in the Netherlands were to describe knowledge gaps in an unambiguous manner based on the above characteristics, a good basis for a national, preferably electronic, database for knowledge gaps would be created. Inclusion in the national registry of the Dutch Healthcare Insurance Board (ZorgInstituut Nederland of the CVZ) is desirable. Agreements would need to be made about meaningful search functions (by sector, disease, intervention or outcome measures) and prioritisation of the knowledge gaps (for example, by severity of the disease, incidence, cost or expected effects). This prioritisation may be leading when developing research and awarding research grants. In addition to search features and prioritisation, agreements would also need to be made on mechanisms for providing feedback about the knowledge gaps identified. Connections could also be sought with international initiatives in this area such as the UK Database of Uncertainties about the Effects of Treatments (DUETs).

7. When to use this tool

The 'timing' of the use of this tool is important: It is not advisable to wait to determine the knowledge gaps until after the completion of the guideline or commentary round. It seems worthwhile to use the tool and describe the gaps after specifying the query based on the review questions. The development group could possibly even start using the tool when formulating the review questions and preparing the search query.

Gaps may still be observed throughout the whole guideline development process. To remain alert to this, 'knowledge gaps' could be a standard agenda item in the development group meetings. The development group could then compile all the knowledge gaps at the end of the process.