We encourage reviews to have a Table of Reconciliation of Studies (example
) that tabulates studies included by prior reviews and their status in the living review (Not part of PRISMA checklist). The has been advocated previously (PMID 25551377
Searching for studies
(PRISMA Items 7,8)
1. Start with making a reconciliation table of all studies included in at least one recent meta-analysis. (PMID 27136216)
- Identify up to 3-4 recent meta-analyses; place these in the columns a a table
- List each trial in the rows of a spreadsheet.
- In each cell, note whether a study was included or the reason it was not
- Example of reconciliation tables
2. Search for more recent studies with traditional searching using search terms
- Search PubMed with a 'Brief Search Strategy'(PMID 16042789)
- Manual searches of online textbooks
- For clinical trials
3. Search for more recent studies with cited reference searches
- Identify the seminal (most highly cite)d meta-analysis and original study by using Google Scholar or Web of Science to retrieve the number of citations to each study on the reconciliaiton table. This work can be reduced by starting with studies in high impact journals. Example
- Execute cited references searches with these studies alysis using Google Scholar or Web of Science to retrieve the number of citations to each study on the reconciliaiton table. This work can be reduced by starting with studies in high impact journals.
4. Each time a study is identified at PubMed, look for addiitonal studies using PubMed's Find Related Data portlet
If you find qualifying studies to an existing repository at openMetaAnalysis
- Please add the citation to the relevant: 1) data table, 2) PICO table, and 3) risk of bias table.
- Update the relevant forest plot
- Consider publishing the updated meta-analysis in a journal.
(PRISMA Items 9-11)
- 0.2 represents a small effect
- 0.5 a moderate effect
- 0.8 a large effect
Assessing quality of individual studies
(PRISMA Items 12)
The Cochrane has release Risk of Bias 2, which has been partially implemented here. Links:
can help you locate text in articles to make determinations of biases.
- Studies of intervention
- Studies of diagnostic accuracy
- Studies of prognosis and risk factors
- Conflict of interest
- Per ICJME, "Authors should avoid entering in to agreements with study sponsors, both for-profit and nonprofit,
that interfere with authors’ access to all of the study’s data or that interfere with their ability to analyze and interpret the data and to prepare and publish manuscripts
independently when and where they choose.". Thus, adding to selective reporting or publication bias if authors without conflicts did not vouch for the data. However, disclosure on conflicts of interest bu authors may not be complete (25835490
Summary judgment is from the Cochrane Handbook, Table 8.7a
- Low risk of bias: "Low risk of bias for all key domains."
- Unclear risk of bias: "Unclear risk of bias for one or more key domains."
- High risk of bias: "High risk of bias for one or more key domains."
(PRISMA Items 13 - 16)
Analyses are done with online at OpenCPU using R. Editors are available for randomized controlled trials and diagnostic tests accuracy studies.
- Studies of intervention
- We use the random effects model with the inverse variance as implemented in the R package meta.
- The Knapp-Hartung method adjusts test statistics and confidence intervals to gives wider (more conservative) confidence intervals and is one of the methods suggested by Cornell et al.(PMID 24727843)
- The continuity correction of Diamond is used.(PMID: 17679700)
- Summary measures for binary outcomes include odds ratio and relative risk. Measures for continuous outcomes are either mean differences or standardized mean differences.
- Subgroup analyses and meta-regressions are done as needed with the R package meta. This includes investigating correlation of the control rate with the outcome
- Heterogeneity is assessed with I2 (4).
- I2 is "the percentage of the variability in effect estimates that is due to heterogeneity rather than sampling error (chance)".
- τ2, in random effects meta-analysis, is heterogeneity in variance from random-effects meta-analysis (link from Cochrane Handbook).
- τ, in random effects meta-analysis, is "is the estimated standard deviation of underlying effects across studies" (link from Cochrane Handbook).
- Studies of diagnosis
- Hierarchial bivariate model (Reitsma, 2005. PMID 16168343) as implemented in the R package metatron with AUC from the R package mada
Assessing quality across a group of studies
(PRISMA Items 15)
Factors developed by the GRADE Working Group are below for assessing a group of studies in a meta-analysis (PMID 22542023). (2) Specific criteria for each factor are are based on those used by the Cochrane Back Group with modifications noted below.(PMID: 23362516)
|Limitations in the design and implementation of available studies
Cochrane Back Group (23362516)
Alternative approach: Cochrane Handbook 5.1; Table 8.7
Alternative approach: Cochrane Handbook 6.0; Table 14.2.b
- Serious risk of bias: More than 25% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS-2 (diagnostic tests) Risk of bias tool (see above)
- Very serious risk of bias: More than 50% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS-2 (diagnostic tests) Risk of bias tool"
- Low risk of bias: "Most information is from results at low risk of bias."
- Some concerns: "Most information is from results at low risk of biasor with some concerns."
- High risk of bias: "The proportion of information from results at high risk of bias is sufficient to affect the interpretation of results."
|Heterogeneity or inconsistency of results using I2
(defined as "percentage of total variation across studies that is due to heterogeneity rather than chance." Higgins, 2003 PMID 12958120)
|From Identifying and measuring heterogeneity),
a 'rough guide' to interpretation is:
• >0% to 40%: might not be important
• >30% to 60%: may represent moderate heterogeneity
• 50% to 90%: may represent substantial heterogeneity
• 75% to 100%: considerable heterogeneity
|Imprecision of results
(modified from the Cochrane Back Group)
|• Serious imprecision: Fewer than 2000 participants for each outcome (PMID: 11158556) or confidence intervals that include clinically unimportant outcomes
• Very serious imprecision: Fewer than 300 participants for each outcome.(PMID: 23362516)
|Probability of publication bias
||This area of meta-analytic practice is evolving and presently only addresses studies of interventions. See discussion at http://handbook-5-1.cochrane.org/chapter_10/10_4_5_summary.htm.
- If more than 10 studies are present, test for the small study effect with the Egger test for continuous outcomes or the Rucker test for binary outcomes (CRAN and PMIDs: 17592831,19836925).
- When less than 10 studies are present, study size of less than 50 or 1000 patients total (PMID: 23616031) or 100 per arm (PMID: 20639294) in most of all available studies may suggest small study effect.
- Selective reporting risk if trial not registered. Selective reporting is common (23407296,26287998,19724045). Trial registration, may lead to less favorable conclusions.PMID: 26244868,22214754
|Rating up the evidence
||GRADE recommends rating up the evidence if one of the following exist (PMID: 21802902). Note similarity to Bradford-Hill criteria (PMID 14283879):
- "GRADE suggests considering rating up quality of evidence one level when methodologically rigorous observational studies show at least a two-fold reduction or increase in risk, and rating up two levels for at least a five-fold reduction or increase in risk"
- Dose-response gradient is present
- "All plausible confounders or biases would decrease an apparent treatment effect"
- Rapidity of the response
Summary of Findings
(Not part of PRISMA checklist)
If pooled studies show significant benefit
- Summary of Finding Tables (SoF) are constructed as detailed in the Cochrane Handbook (Chapter 11) and by Guyatt et al (PMID: 21195583). SoF are currently produced with GRADEpro.
If pooled studies do not show significant benefit
- Results are summarized based on p-values using judical analogies developed by Diamond (PMID 19506168).
|P < 0.05 in all studies
||“Beyond a reasonable doubt”
|P < 0.05 in some studies
||“Clear and convincing evidence”
|P < 0.5 in all studies
||“Preponderance of the evidence”
|P < 0.5 in some studies
|P < 0.5 in no studies
|Based on Diamond, 2009 (PMID 19667308)
Reconciliation of Conclusions with prior meta-analyses
(Not part of PRISMA checklist)