Methods

Methods are adapted from:
Contents:
Download examples files, then replace with your content Files needed

Other downloads:

Selecton of studies (PRISMA Item 6)

Meta-analysis of randomized controlled trials:

Meta-analysis of diagnostic test accuracy:

More strict criteria for inclusion may be based on the criteria by American College of Physicians Journal Club (ACPJC).
 
We encourage reviews to have a Table of Reconciliation of Studies (example) that tabulates studies included by prior reviews and their status in the living review (Not part of PRISMA checklist). The has been advocated previously (PMID 25551377).
 
Searching for studies (PRISMA Items 7,8)

1. Start with making a reconciliation table of all studies included in at least one recent meta-analysis. (PMID 27136216)

2. Search for more recent studies with traditional searching using search terms

3. Search for more recent studies with cited reference searches

4. Each time a study is identified at PubMed, look for addiitonal studies using PubMed's Find Related Data portlet

If you find qualifying studies to an existing repository at openMetaAnalysis

Consider https://library.medicine.yale.edu/tutorials/1559
Data abstraction (PRISMA Items 9-11)
Creating the files for PICO and Bias tables
Consider using an online collaborative text editor with a colleague to develop the xml files for the PICO and bias tables.
  • Kobra is very easy to use. Kobra itself is collaboratively developed with Firepad.
Calculation of population characteristics from 'Table 1' of studies for PICO Table in openMetaAnalysis
Group 1Group 2
   
Weighted value:
Calculation of standard deviation and error from
confidence interval, range, or interquartile range
  
Mean: SE: SD: VAR:
Details from: mean to median, confidence interval, interquartile range, and range. See downloadable spreadsheet for calculations by Wan, 2014 PMID 25524443
Converting continuous measures into binary measures
Many calculators for data conversions are at George Mason University. Equations for conversion are available at the Cochrane handbook-5-1.
 
Odds ratio (under construction):
Details from: Chinn, 2000 (PMID 11113947) and da Costa, 2014 (PMID 23045205).
Standardized mean differences may be abstracted. One proposed interpretration of the SMD is: (PMID 19565683)
Assessing quality of individual studies (PRISMA Items 12)
The Cochrane has release Risk of Bias 2, which has been partially implemented here. Links:
Robot Reviewer can help you locate text in articles to make determinations of biases.
Summary judgment is from the Cochrane Handbook, Table 8.7a
Statistical analysis (PRISMA Items 13 - 16)

Analyses are done with online at OpenCPU using R. Editors are available for randomized controlled trials and diagnostic tests accuracy studies.

Assessing quality across a group of studies (PRISMA Items 15)

Factors developed by the GRADE Working Group are below for assessing a group of studies in a meta-analysis (PMID 22542023). (2) Specific criteria for each factor are are based on those used by the Cochrane Back Group with modifications noted below.(PMID: 23362516)

Factors Criteria
Limitations in the design and implementation of available studies Cochrane Back Group (23362516)
  • Serious risk of bias: More than 25% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS-2 (diagnostic tests) Risk of bias tool (see above)
  • Very serious risk of bias: More than 50% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS-2 (diagnostic tests) Risk of bias tool"
Alternative approach: Cochrane Handbook 5.1; Table 8.7 Alternative approach: Cochrane Handbook 6.0; Table 14.2.b
  • Low risk of bias: "Most information is from results at low risk of bias."
  • Some concerns: "Most information is from results at low risk of biasor with some concerns."
  • High risk of bias: "The proportion of information from results at high risk of bias is sufficient to affect the interpretation of results."
Indirectness Cochrane Handbook 5.0
Heterogeneity or inconsistency of results using I2
(defined as "percentage of total variation across studies that is due to heterogeneity rather than chance." Higgins, 2003 PMID 12958120)
From Identifying and measuring heterogeneity), a 'rough guide' to interpretation is:
• >0% to 40%: might not be important
• >30% to 60%: may represent moderate heterogeneity
• 50% to 90%: may represent substantial heterogeneity
• 75% to 100%: considerable heterogeneity
Imprecision of results
(modified from the Cochrane Back Group)
• Serious imprecision: Fewer than 2000 participants for each outcome (PMID: 11158556) or confidence intervals that include clinically unimportant outcomes
• Very serious imprecision: Fewer than 300 participants for each outcome.(PMID: 23362516)
An alternative approach to determining imprecision is to use the "optimal information size" (PMID: 21839614) with an online calculator (http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html).
Probability of publication bias This area of meta-analytic practice is evolving and presently only addresses studies of interventions. See discussion at http://handbook-5-1.cochrane.org/chapter_10/10_4_5_summary.htm.
  • If more than 10 studies are present, test for the small study effect with the Egger test for continuous outcomes or the Rucker test for binary outcomes (CRAN and PMIDs: 17592831,19836925).
  • When less than 10 studies are present, study size of less than 50 or 1000 patients total (PMID: 23616031) or 100 per arm (PMID: 20639294) in most of all available studies may suggest small study effect.
  • Selective reporting risk if trial not registered. Selective reporting is common (23407296,26287998,19724045). Trial registration, may lead to less favorable conclusions.PMID: 26244868,22214754
Rating up the evidence GRADE recommends rating up the evidence if one of the following exist (PMID: 21802902). Note similarity to Bradford-Hill criteria (PMID 14283879):
  • "GRADE suggests considering rating up quality of evidence one level when methodologically rigorous observational studies show at least a two-fold reduction or increase in risk, and rating up two levels for at least a five-fold reduction or increase in risk"
  • Dose-response gradient is present
  • "All plausible confounders or biases would decrease an apparent treatment effect"
  • Rapidity of the response
 
Summary of Findings (Not part of PRISMA checklist)
 
If pooled studies show significant benefit
If pooled studies do not show significant benefit
P values Judicial analogy
P < 0.05 in all studies “Beyond a reasonable doubt”
P < 0.05 in some studies “Clear and convincing evidence”
P < 0.5 in all studies “Preponderance of the evidence”
P < 0.5 in some studies “Reasonable suspicion”
P < 0.5 in no studies “Insufficient evidence”
Based on Diamond, 2009 (PMID 19667308)
 
Reconciliation of Conclusions with prior meta-analyses (Not part of PRISMA checklist)
The has been advocated (PMID 25551377) and implemented (25796995) previously.
Example table.
 

References

  1. Risk of bias tool. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook-5-1.cochrane.org
  2. The GRADE approach. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook-5-1.cochrane.org
  3. Schünemann H, Brożek J, Guyatt G, Oxman A (editors). GRADE Handbook. [updated October 2013]. Available from http://www.guidelinedevelopment.org/handbook/
  4. Identifying and measuring heterogeneity. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook-5-1.cochrane.org
Click here to see the revision history of the source code.