Methodological quality assessment in systematic reviews in health sciences that included observational studies: a cross-sectional study/ Avaliação da qualidade metodológica em revisões sistemáticas na área das ciências da saúde que incluíram estudos observacionais: um estudo transversal
DOI:
https://doi.org/10.34119/bjhrv4n5-382Keywords:
research design, systematic review, methodological qualityAbstract
There is great variability in methodological quality assessment (MQ) systematic reviews (SRs). We identified how MQ assessment in SRs that included observational studies in Health Science is applied and explored associated characteristics. A search was conducted in PubMed and five trained reviewers randomly selected 1,025 references after sample size calculation. Only SR published in English (September 2019/2020) were included. Selection and data extraction were conducted in two phases. Data were analyzed descriptively and using logistic regressions. After eligibility criteria application, 205 SRs were included. Only 27.8% informed the protocol registration and 80.0% described having followed a reporting guideline. Proportion’ SRs did not seem to present MQ assessment (OR 4.22; 95% CI: 1.38-12.87; P =0.01). SRs that did not register the study protocol (9.70; 95% CI: 1.95-48.27; P <0.001), those that did not inform the included study design (5.96; 95% CI: 1.63-21.77; P <0.001) and those without MA (OR 8.90; 95% CI: 2.79-28.43; p<0.001) increased the odds of not evaluating MQ. Newcastle-Ottawa (47.5%), Joanna Briggs Institute (8.2%) and National Institute of Health (7.1%) were the most commonly used MQ tools. Lack of protocol registration, absence of information about the design of included studies, absence of MA, and proportion’ SRs were associated with lack of MQ assessment.
References
Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. The Milbank quarterly. 2016;94(3):485-514.
Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257-266.
Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. BMJ (Clinical research ed). 2004;328(7454):1490.
4. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane, 2020. Available from www.training.cochrane.org/handbook.
Santaguida PL, Riley CM, Matchar DB. Chapter 5: assessing risk of bias as a domain of quality in medical test studies. J Gen Intern Med. 2012;27 Suppl 1(Suppl 1):S33-38.
Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: a systematic review. Physical therapy. 2008;88(2):156-175.
Porritt K, Gomersall J, Lockwood C. JBI's Systematic Reviews: Study Selection and Critical Appraisal. AJN The American Journal of Nursing. 2014;114(6):47-52.
Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. International journal of epidemiology. 2007;36(3):666-676.
Ma L-L, Wang Y-Y, Yang Z-H, Huang D, Weng H, Zeng X-T. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil med res. 2020;7(1):7.
Migliavaca CB, Stein C, Colpani V, Munn Z, Falavigna M. Quality assessment of prevalence studies: a systematic review. J Clin Epidemiol. 2020;127:59-68.
Page MJ, Altman DG, Shamseer L, McKenzie JE, Ahmadzai N, Wolfe D, et al. Reproducible research practices are underused in systematic reviews of biomedical interventions. J Clin Epidemiol. 2018;94:8-18.
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., Moher, D. (2020, September 14). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. https://doi.org/10.31222/osf.io/v7gm2
Massignan, C., Oliveira, L. B., Honnef, L. R., de Oliveira, J. M. D., Stefani, C. M., & De Luca Canto, G. (2020, October 29). Current critical appraisal tools for observational studies’ methodological quality assessment to use in systematic reviews: a methodological review. https://doi.org/10.17605/OSF.IO/QYCJZ
Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC medical research methodology. 2018;18(1):5.
Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Medical Research Methodology. 2019;19(1):203.
Dos Santos MBF, Agostini BA, Bassani R, Pereira GKR, Sarkis-Onofre R. Protocol registration improves reporting quality of systematic reviews in dentistry. BMC medical research methodology. 2020;20(1):57.
Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. Lancet. 2011;377(9760):108-109.
Johansen M, Thomsen SF. Guidelines for Reporting Medical Research: A Critical Appraisal. International scholarly research notices. 2016;2016:1346026.
Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. Jama. 2000;283(15):2008-2012.
Viswanathan M, Patnode CD, Berkman ND, Bass EB, Chang S, Hartling L, et al. Recommendations for assessing the risk of bias in systematic reviews of health-care interventions. J Clin Epidemiol. 2018;97:26-34.
Tawfik GM, Dila KAS, Mohamed MYF, Tam DNH, Kien ND, Ahmed AM, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Tropical medicine and health. 2019;47:46.
Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://synthesismanual.jbi.global. https://doi.org/10.46658/JBIMES-20-01
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358:j4008.
Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66(9):973-981.
Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality if nonrandomized studies in meta-analyses. Available from: URL: http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm [cited 2020 asessed December 2020].
Sterne JA, Hernán MA, Reeves BC, Savovi? J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Clinical research ed). 2016;355:i4919.
Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25(9):603-605.
Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health technology assessment (Winchester, England). 2003;7(27):iii-x, 1-173.
Li W, Ma D, Liu M, Liu H, Feng S, Hao Z, et al. Association between metabolic syndrome and risk of stroke: a meta-analysis of cohort studies. Cerebrovascular diseases (Basel, Switzerland). 2008;25(6):539-547.
Lo CK, Mertz D, Loeb M. Newcastle-Ottawa Scale: comparing reviewers' to authors' assessments. BMC medical research methodology. 2014;14:45.
Moskalewicz A, Oremus M. No clear choice between Newcastle-Ottawa Scale and Appraisal Tool for Cross-Sectional Studies to assess methodological quality in cross-sectional studies of health-related quality of life and breast cancer. J Clin Epidemiol. 2020;120:94-103.
von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. International journal of surgery (London, England). 2014;12(12):1495-1499.
