Project Methodology 5


Health Evidence Bulletins - Wales (logo)

Stages of Methodology




  • Identify subject areas for searching by examining the existing bulletin or other guidance document (e.g. a National Service Framework) and on advice from the  Steering Group.

Information Manager/

Steering Group

  • Decide deadlines, and members of the critical appraisal and Internal Review Groups (Stages 4 and 5).

    [Each Appraisal Team should adopt a multidisciplinary approach and be aware of the information needs of medical, nursing and therapy professions as well as purchasers and consumers]

    [Members of the Internal Review Group should be acknowledged experts in their field. A multidisciplinary approach should be adopted to ensure coverage of all subject areas and viewpoints]

Information Manager/

Steering Group





  • Search for information in the subject areas identified in Stage 1 using appropriate sources from checklist (Appendix 1) and utilising master searches where available (Appendix 2).   Carry out searches for each subject area in a stepped fashion for Types I to V evidence in a strict order (see Appendix 3 for explanation of evidence types). 
Information Officers
  • Carry out searches for a subject from the date given on the relevant chapter of an earlier bulletin if one exists.  For new subject areas, search five-ten years initially depending on the amount of information in a given subject area.  Search earlier dates if no systematic reviews (of randomised controlled trials (Type I) or observational studies (Type IV)) or randomised controlled trials (Type II) are found.
Information Officers


  • Include English and non-English articles. Where the non-English abstracts seem to indicate a systematic review or a randomised controlled trial, arrange the translation of full-text documents.
Information Officers
  • Store reference details, remove duplicates and print abstracts in a suitable format using reference management software.
Information Officers
  • Examine abstracts and, on the basis of inclusion/exclusion criteria agreed by the Steering Group, obtain relevant publications as full text.  If in doubt, order document.
Information Manager/Officers
  • Acquire full-text publications.
Information Officers




  • For each full-text document supplied, attach an Evidence Form (Appendix 4) to provide the bibliographic details and indicate the type of evidence.  Complete a copyright declaration for each document if required, and store in the Project Office.
Information Officers
  • Organise the critical appraisal of each document using the appropriate Quality Appraisal Checklist as a guide (Appendices 5-11).
Information Officers
  • Decide on the evidence to be included in the first draft of the Bulletin. If in doubt about inclusion, retain document and ask for advice from the critical appraisal review team.
Information Manager/Officers
  • Prepare the first draft of the Bulletin which should include a matrix of statements and the evidence which supports them (see Appendix 12).

The statements should include (where appropriate):

  1.  the target population (e.g. age, sex);

  2. quantitative information (e.g. Number Needed to Treat, % improvement);

The evidence should be:

  1. cited according to the Vancouver System

  2. accompanied by an evidence type and any other relevant information about the research

    (See Appendix 13 for a summary)

Information Manager/Officers
  • Identify areas of little or no evidence, conflicting evidence or where evidence is of poor quality in order to:
    either  pursue the search further

    make a recommendation for further research

    Once evidence has been assigned to statements covering a particular subject, no other types of evidence will be sought although topics covered by systematic reviews may include more recent studies.
Information Manager/Officers
  • Store evidence to be included and critical appraisal checklists. Evidence excluded (and appraisal checklists) should also be retained in case of query.
Information Officers



  • Supply first draft and all papers under consideration to critical appraisal review team.

Information Manager/CA team


  • Quality check all critical appraisal work carried out by Information Team and liaise with information team to resolve any discrepancies.

CA Team/Information Manager/Officers




  • Submit the first draft of the Bulletin to the Internal Review Group asking for the following responses:
  1. Do they agree with the statements and the supporting evidence?

  2.  Are they aware of any better evidence than that quoted?

  3.  Are there any relevant topic areas that are not included?

  4.  Any other comments.


Information Manager/Internal Review Group
  • Gather and expedite responses, and liaise with reviewers over queries.
Information Manager
  • Carry out further searching, appraisal and evaluation as necessary.
Information Manager/Officers
  • Liaise with critical appraisal team and internal review group to draw up a final draft for external review.

Information Manager/ CA Team/
Internal Review Group

  • Seek advice from the Review Groups on the Health Gain Notation to be applied to each statement (if used)

Strength of the finding (Health Gain Notation)

‘beneficial’ - effectiveness clearly demonstrated (1)

‘likely to be beneficial’ - effectiveness not so firmly established (2)

‘trade-off between beneficial and adverse effects’ - effects weighed according to individual circumstances(3)

‘unknown’ - insufficient/inadequate for recommendation (4)

‘unlikely to be beneficial’ - ineffectiveness is not as clearly demonstrated as for 6 (5)

‘likely to be ineffective or harmful’ - ineffectiveness or harm clearly demonstrated (6)

(adapted from Enkin M, Kierse MJNC, Renfrew M and Neilson J.  A guide to effective care in pregnancy and childbirth. 2nd ed.  Oxford: Oxford University Press, 1995)

Information Manager/ CA Team/
Internal Review Group





  • Select External Reviewers of high standing (using suggestions from the Steering Group to appraise final Bulletins for currency, accuracy and completeness.

Steering Group/ Information Manager


  • Co-ordinate the External Review process.

Information Manager/External Review Group

  • Give approval for dissemination of the Bulletins.

Steering Group



  • Complete an action plan for the dissemination of each Bulletin, clearly covering:
  1. Distribution list (in primary, secondary, tertiary settings);

  2. World-wide publication;

  3.  Relevant talks and conferences

Steering Group
  • Carry out an ongoing evaluation programme to inform the further development of both text and electronic versions to assess the value and impact of the Bulletin.

Information Manager/Steering Group


next page

Methodology Home Top