Methods are adapted from:

Contents:

Download examples files, then replace with your content Files needed

Selecton of studies (PRISMA Item 6)

More strict criteria for inclusion may be based on the criteria by American College of Physicians Journal Club (ACPJC).

 

We encourage reviews to have a Table of Reconciliation of Studies (example) that tabulates studies included by prior reviews and their status in the living review (Not part of PRISMA checklist). The has been advocated previously (PMID 25551377).

1. Start with making a reconciliation table of all studies included in at least one recent meta-analysis. (PMID 27136216)

• Identify up to 3-4 recent meta-analyses; place these in the columns a a table
• List each trial in the rows of a spreadsheet.
• In each cell, note whether a study was included or the reason it was not
• Example of reconciliation tables

2. Search for more recent studies with traditional searching using search terms

• Search PubMed with a 'Brief Search Strategy'(PMID 16042789)
• Manual searches of online textbooks
• For clinical trials
  • Search of the Cochrane Central Register of Controlled Trials (CENTRAL)
  • Clinical Trials registries with methods suggested by Glanville (PMC4076126):
    • ClinicalTrials.gov (for studies with completed results and the search term random*)
    • International Clinical Trials Registry Platform (ICTRP) (WHO)

3. Search for more recent studies with cited reference searches

• Identify the seminal (most highly cited) meta-analysis and original study by using Google Scholar or Web of Science to retrieve the number of citations to each study on the reconciliaiton table. This work can be reduced by starting with studies in high impact journals. Example
• Execute cited references searches to retrieve the number of citations to each study in hand, or better, in your reconciliation table by using:
  • PubMed's "Cited by" feature below each article's abstract.
  • Google Scholar
  • Instructions at SUMSearch

4. Each time a study is identified at PubMed, look for addiitonal studies using PubMed's Find Related Data portlet

If you find qualifying studies to an existing repository at openMetaAnalysis

• Please add the citation to the relevant: 1) data table, 2) PICO table, and 3) risk of bias table.
• Update the relevant forest plot
• Consider publishing the updated meta-analysis in a journal.

Consider https://library.medicine.yale.edu/tutorials/1559

Creating the files for PICO and Bias tables http://openmetaanalysis.github.io/make-table-pico.html will help make the content for the PICO tables. http://openmetaanalysis.github.io/make-table-bias.html will help make the content for the Risk of bias table. Consider using an online collaborative text editor with a colleague to develop the xml files for the PICO and bias tables. Kobra is very easy to use. Kobra itself is collaboratively developed with Firepad.

Calculation of population characteristics from 'Table 1' of studies for PICO Table in openMetaAnalysis Group 1 Group 2 Counts Percentages Means  Calculate weighted value Weighted value: Calculation of standard deviation and error from confidence interval, range, or interquartile range Confidence interval Interquartile range Range Calculate standard deviation Mean: SE: SD: VAR: Details from: mean to median, confidence interval, interquartile range, and range. See downloadable spreadsheet for calculations by Wan, 2014 PMID 25524443 Converting continuous measures into binary measures Many calculators for data conversions are at George Mason University. Equations for conversion are available at the Cochrane handbook-5-1. Two group means Single difference between group means Calculate binary measures Odds ratio (under construction): Details from: Chinn, 2000 (PMID 11113947) and da Costa, 2014 (PMID 23045205).

• 0.2 represents a small effect
• 0.5 a moderate effect
• 0.8 a large effect

The Cochrane has release Risk of Bias 2, which has been partially implemented here. Links:

• Cochrane announcement
• Google Site

• Studies of intervention
  • Individual criteria for judging risk of bias in a trial using the 'Cochrane's Risk of bias tool'.(1)
    • Attrition: consider judgment of 'Unclear risk" it no Consort flow diagram
    • Selective reporting: consider judgment of 'Unclear risk" it no access to original protocol at a online trial registry
      • Video of finding a trial's initial registration at ClinicalTrials.gov
      • Check whether date of first registration was prior to date of first patient randomized
      • Check whether primary and secondary outcomes planned before randomization match those in publications of results
  • Aggregating individual criteria into a summary judgment about a trial (see column "Within a study")
• Studies of diagnostic accuracy
  • Four criteria for judging risk of bias from QUADAS-2 (see Risk of bias row in the Table guided by the detailed criteria in the rows above)
• Studies of prognosis and risk factors
  • Cochrane's Quality In Prognosis Studies (QUIP) tool (PMID 23420236)
• Conflict of interest
  • Per ICJME, "Authors should avoid entering in to agreements with study sponsors, both for-profit and nonprofit, that interfere with authors’ access to all of the study’s data or that interfere with their ability to analyze and interpret the data and to prepare and publish manuscripts independently when and where they choose.". Thus, adding to selective reporting or publication bias if authors without conflicts did not vouch for the data. However, disclosure on conflicts of interest bu authors may not be complete (25835490

Summary judgment is from the Cochrane Handbook, Table 8.7a

• Low risk of bias: "Low risk of bias for all key domains."
• Unclear risk of bias: "Unclear risk of bias for one or more key domains."
• High risk of bias: "High risk of bias for one or more key domains."

Analyses are done with online at OpenCPU using R. Editors are available for randomized controlled trials and diagnostic tests accuracy studies.

• Studies of intervention
  • We use the random effects model with the inverse variance as implemented in the R package meta.
  • The Knapp-Hartung method adjusts test statistics and confidence intervals to gives wider (more conservative) confidence intervals and is one of the methods suggested by Cornell et al.(PMID 24727843)
  • The continuity correction of Diamond is used.(PMID: 17679700)
  • Summary measures for binary outcomes include odds ratio and relative risk. Measures for continuous outcomes are either mean differences or standardized mean differences.
  • Subgroup analyses and meta-regressions are done as needed with the R package meta. This includes investigating correlation of the control rate with the outcome
    • PubMed citations
  • Heterogeneity is assessed with I² (4).
    • I² is "the percentage of the variability in effect estimates that is due to heterogeneity rather than sampling error (chance)".
    • τ², in random effects meta-analysis, is heterogeneity in variance from random-effects meta-analysis (link from Cochrane Handbook).
    • τ, in random effects meta-analysis, is "is the estimated standard deviation of underlying effects across studies" (link from Cochrane Handbook).
• Studies of diagnosis
  • Hierarchial bivariate model (Reitsma, 2005. PMID 16168343) as implemented in the R package metatron with AUC from the R package mada
• The standardized mean difference (SMD) may help combine studies that report conceptually compatible metrics, but vary in the exact method of measurement yielding conflicting numerical results. Many statistical measures used in healthcare can be converted to the SMD (https://training.cochrane.org/handbook/current/chapter-10#section-10-6">Cochrane Handbook) and then pooled (online calculators at the Campbell Collaboration, R package calculators). The SMD can be interpreted as 0.2 represents a small effect; 0.5 a moderate effect; 0.8 a large effect (PMID 19565683). As the SMD may not be familiar to readers and also does not give a result that can be translated into clinical impact, after the SMD is calculated and a statistically significant result is found, you can identify a study whose result is closest to the pooled SMD and then describe that study. For example, in the diagnosis of adult ADHD, the Brevick study had the closest SMD to the pooled SMD of the two studies of the WURS. Thus you could describe the Brevick results in clinical terms using the aROC that they reported and ADHD experts may understand.

Factors developed by the GRADE Working Group are below for assessing a group of studies in a meta-analysis (PMID 22542023). (2) Specific criteria for each factor are are based on those used by the Cochrane Back Group with modifications noted below.(PMID: 23362516)

Factors	Criteria
Limitations in the design and implementation of available studies	Cochrane Back Group (23362516) Serious risk of bias: More than 25% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS-2 (diagnostic tests) Risk of bias tool (see above) Very serious risk of bias: More than 50% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS-2 (diagnostic tests) Risk of bias tool" Alternative approach: Cochrane Handbook 5.1; Table 8.7 Alternative approach: Cochrane Handbook 6.0; Table 14.2.b Low risk of bias: "Most information is from results at low risk of bias." Some concerns: "Most information is from results at low risk of biasor with some concerns." High risk of bias: "The proportion of information from results at high risk of bias is sufficient to affect the interpretation of results."
Indirectness	Cochrane Handbook 5.0
Heterogeneity or inconsistency of results using I2 (defined as "percentage of total variation across studies that is due to heterogeneity rather than chance." Higgins, 2003 PMID 12958120)	From Identifying and measuring heterogeneity), a 'rough guide' to interpretation is: • >0% to 40%: might not be important • >30% to 60%: may represent moderate heterogeneity • 50% to 90%: may represent substantial heterogeneity • 75% to 100%: considerable heterogeneity
Imprecision of results (modified from the Cochrane Back Group)	• Serious imprecision: Fewer than 2000 participants for each outcome (PMID: 11158556) or confidence intervals that include clinically unimportant outcomes • Very serious imprecision: Fewer than 300 participants for each outcome.(PMID: 23362516) An alternative approach to determining imprecision is to use the "optimal information size" (PMID: 21839614) with an online calculator (http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html).
Probability of publication bias	This area of meta-analytic practice is evolving and presently only addresses studies of interventions. See discussion at http://handbook-5-1.cochrane.org/chapter_10/10_4_5_summary.htm. If more than 10 studies are present, test for the small study effect with the Egger test for continuous outcomes or the Rucker test for binary outcomes (CRAN and PMIDs: 17592831,19836925). When less than 10 studies are present, study size of less than 50 or 1000 patients total (PMID: 23616031) or 100 per arm (PMID: 20639294) in most of all available studies may suggest small study effect. Selective reporting risk if trial not registered. Selective reporting is common (23407296,26287998,19724045). Trial registration, may lead to less favorable conclusions.PMID: 26244868,22214754
Rating up the evidence	GRADE recommends rating up the evidence if one of the following exist (PMID: 21802902). Note similarity to Bradford-Hill criteria (PMID 14283879): "GRADE suggests considering rating up quality of evidence one level when methodologically rigorous observational studies show at least a two-fold reduction or increase in risk, and rating up two levels for at least a five-fold reduction or increase in risk" Dose-response gradient is present "All plausible confounders or biases would decrease an apparent treatment effect" Rapidity of the response

 

If pooled studies show significant benefit

• Summary of Finding Tables (SoF) are constructed as detailed in the Cochrane Handbook (Chapter 11) and by Guyatt et al (PMID: 21195583). SoF are currently produced with GRADEpro.

If pooled studies do not show significant benefit

• Results are summarized based on p-values using judical analogies developed by Diamond (PMID 19667308).

P values	Judicial analogy
P < 0.05 in all studies	“Beyond a reasonable doubt”
P < 0.05 in some studies	“Clear and convincing evidence”
P < 0.5 in all studies	“Preponderance of the evidence”
P < 0.5 in some studies	“Reasonable suspicion”
P < 0.5 in no studies	“Insufficient evidence”
Based on Diamond, 2009 (PMID 19667308)

References