We encourage reviews to have a Table of Reconciliation of Studies (
example) that tabulates studies included by prior reviews and their status in the living review (Not part of PRISMA checklist). The has been advocated previously (PMID
25551377).
1. Start with making a reconciliation table of all studies included in at least one recent metaanalysis. (PMID 27136216)
 Identify up to 34 recent metaanalyses; place these in the columns a a table
 List each trial in the rows of a spreadsheet.
 In each cell, note whether a study was included or the reason it was not
 Example of reconciliation tables
2. Search for more recent studies with traditional searching using search terms
 Search PubMed with a 'Brief Search Strategy'(PMID 16042789)
 Manual searches of online textbooks
 For clinical trials
3. Search for more recent studies with cited reference searches
 Identify the seminal (most highly cite)d metaanalysis and original study by using Google Scholar or Web of Science to retrieve the number of citations to each study on the reconciliaiton table. This work can be reduced by starting with studies in high impact journals. Example
 Execute cited references searches with these studies alysis using Google Scholar or Web of Science to retrieve the number of citations to each study on the reconciliaiton table. This work can be reduced by starting with studies in high impact journals.
4. Each time a study is identified at PubMed, look for addiitonal studies using PubMed's Find Related Data portlet
If you find qualifying studies to an existing repository at openMetaAnalysis
 Please add the citation to the relevant: 1) data table, 2) PICO table, and 3) risk of bias table.
 Update the relevant forest plot
 Consider publishing the updated metaanalysis in a journal.
Creating the files for PICO and Bias tables
Consider using an online collaborative text editor with a colleague to develop the xml files for the PICO and bias tables.
 Kobra is very easy to use. Kobra itself is collaboratively developed with Firepad.


 0.2 represents a small effect
 0.5 a moderate effect
 0.8 a large effect
The Cochrane has release Risk of Bias 2, which has been partially implemented here. Links:
Robot Reviewer can help you locate text in articles to make determinations of biases.
 Studies of intervention
 Studies of diagnostic accuracy
 Studies of prognosis and risk factors
 Conflict of interest
 Per ICJME, "Authors should avoid entering in to agreements with study sponsors, both forprofit and nonprofit,
that interfere with authors’ access to all of the study’s data or that interfere with their ability to analyze and interpret the data and to prepare and publish manuscripts
independently when and where they choose.". Thus, adding to selective reporting or publication bias if authors without conflicts did not vouch for the data. However, disclosure on conflicts of interest bu authors may not be complete (25835490
Summary judgment is from the Cochrane Handbook, Table 8.7a
 Low risk of bias: "Low risk of bias for all key domains."
 Unclear risk of bias: "Unclear risk of bias for one or more key domains."
 High risk of bias: "High risk of bias for one or more key domains."
Analyses are done with online at OpenCPU using R. Editors are available for randomized controlled trials and diagnostic tests accuracy studies.
 Studies of intervention
 We use the random effects model with the inverse variance as implemented in the R package meta.
 The KnappHartung method adjusts test statistics and confidence intervals to gives wider (more conservative) confidence intervals and is one of the methods suggested by Cornell et al.(PMID 24727843)
 The continuity correction of Diamond is used.(PMID: 17679700)
 Summary measures for binary outcomes include odds ratio and relative risk. Measures for continuous outcomes are either mean differences or standardized mean differences.
 Subgroup analyses and metaregressions are done as needed with the R package meta. This includes investigating correlation of the control rate with the outcome
 Heterogeneity is assessed with I^{2} (4).
 I^{2} is "the percentage of the variability in effect estimates that is due to heterogeneity rather than sampling error (chance)".
 τ^{2}, in random effects metaanalysis, is heterogeneity in variance from randomeffects metaanalysis (link from Cochrane Handbook).
 τ, in random effects metaanalysis, is "is the estimated standard deviation of underlying effects across studies" (link from Cochrane Handbook).
 Studies of diagnosis
 Hierarchial bivariate model (Reitsma, 2005. PMID 16168343) as implemented in the R package metatron with AUC from the R package mada
Assessing quality across a group of studies
(PRISMA Items 15)
Factors developed by the GRADE Working Group are below for assessing a group of studies in a metaanalysis (PMID 22542023). (2) Specific criteria for each factor are are based on those used by the Cochrane Back Group with modifications noted below.(PMID: 23362516)
Factors 
Criteria 
Limitations in the design and implementation of available studies 
Cochrane Back Group (23362516)
 Serious risk of bias: More than 25% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS2 (diagnostic tests) Risk of bias tool (see above)
 Very serious risk of bias: More than 50% of participants from studies with low methodological quality as measured by the Cochrane's (interventions) or QUADAS2 (diagnostic tests) Risk of bias tool"
Alternative approach: Cochrane Handbook 5.1; Table 8.7
Alternative approach: Cochrane Handbook 6.0; Table 14.2.b
 Low risk of bias: "Most information is from results at low risk of bias."
 Some concerns: "Most information is from results at low risk of biasor with some concerns."
 High risk of bias: "The proportion of information from results at high risk of bias is sufficient to affect the interpretation of results."

Indirectness 

Heterogeneity or inconsistency of results using I^{2} (defined as "percentage of total variation across studies that is due to heterogeneity rather than chance." Higgins, 2003 PMID 12958120) 
From Identifying and measuring heterogeneity),
a 'rough guide' to interpretation is:
• >0% to 40%: might not be important
• >30% to 60%: may represent moderate heterogeneity
• 50% to 90%: may represent substantial heterogeneity
• 75% to 100%: considerable heterogeneity

Imprecision of results (modified from the Cochrane Back Group) 
• Serious imprecision: Fewer than 2000 participants for each outcome (PMID: 11158556) or confidence intervals that include clinically unimportant outcomes • Very serious imprecision: Fewer than 300 participants for each outcome.(PMID: 23362516)

Probability of publication bias 
This area of metaanalytic practice is evolving and presently only addresses studies of interventions. See discussion at http://handbook51.cochrane.org/chapter_10/10_4_5_summary.htm.
 If more than 10 studies are present, test for the small study effect with the Egger test for continuous outcomes or the Rucker test for binary outcomes (CRAN and PMIDs: 17592831,19836925).
 When less than 10 studies are present, study size of less than 50 or 1000 patients total (PMID: 23616031) or 100 per arm (PMID: 20639294) in most of all available studies may suggest small study effect.
 Selective reporting risk if trial not registered. Selective reporting is common (23407296,26287998,19724045). Trial registration, may lead to less favorable conclusions.PMID: 26244868,22214754

Rating up the evidence 
GRADE recommends rating up the evidence if one of the following exist (PMID: 21802902). Note similarity to BradfordHill criteria (PMID 14283879):
 "GRADE suggests considering rating up quality of evidence one level when methodologically rigorous observational studies show at least a twofold reduction or increase in risk, and rating up two levels for at least a fivefold reduction or increase in risk"
 Doseresponse gradient is present
 "All plausible confounders or biases would decrease an apparent treatment effect"
 Rapidity of the response

If pooled studies show significant benefit
If pooled studies do not show significant benefit
 Results are summarized based on pvalues using judical analogies developed by Diamond (PMID 19506168).
P values 
Judicial analogy 
P < 0.05 in all studies 
“Beyond a reasonable doubt” 
P < 0.05 in some studies 
“Clear and convincing evidence” 
P < 0.5 in all studies 
“Preponderance of the evidence” 
P < 0.5 in some studies 
“Reasonable suspicion” 
P < 0.5 in no studies 
“Insufficient evidence” 
Based on Diamond, 2009 (PMID 19667308) 
References