![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
In several subdomains of the social, behavioral, health, and human sciences, research questions are increasingly answered through mixed methods studies, combining qualitative and quantitative evidence and research elements. Accordingly, the importance of including those primary mixed methods research articles in systematic reviews grows. It is generally known that the critical appraisal of articles is an essential step in the development of a methodologically sound review. This article provides an overview of the available critical appraisal frameworks developed to evaluate primary mixed methods research articles. In addition, we critically compare and evaluate these frameworks and the quality criteria they include.
References
Section:
|
*Alborz, A., McNally, R. (2004). Developing methods for systematic reviewing in health services delivery and organization: An example from a review of access to health care for people with learning disabilities. Part 2. Evaluation of the literature—A practical guide. Health Information and Libraries Journal, 21, 227-236. Google Scholar | Crossref | Medline | |
|
Alise, M. A., Teddlie, C. (2010). A continuation of the paradigm wars? Prevalence rates of methodological approaches across the social/behavioral sciences. Journal of Mixed Methods Research, 4(2), 103-126. Google Scholar | SAGE Journals | ISI | |
|
Biesta, G. J. J. (2010). Pragmatism and the philosophical foundations of mixed methods research. In Tashakkori, A., Teddlie, C. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 95-118). Thousand Oaks, CA: Sage. Google Scholar | Crossref | |
|
Bryman, A. (2006a). Paradigm peace and the implications for quality. International Journal of Social Research Methodology, 9, 111-126. Google Scholar | Crossref | |
|
Bryman, A. (2006b). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6, 97-113. Google Scholar | SAGE Journals | |
|
*Bryman, A., Becker, S., Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11, 261-276. Google Scholar | Crossref | ISI | |
|
*Caracelli, V., Riggin, L. (1994). Mixed-method evaluation: Developing quality criteria through concept mapping. Evaluation Practice, 15, 139-152. Google Scholar | Crossref | |
|
Collins, K. M. T., Onwuegbuzie, A. J., Sutton, I. L. (2006). A model incorporating the rationale and purpose for conducting mixed-methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67-100. Google Scholar | |
|
Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). London, England: Sage. Google Scholar | |
|
Cooper, H. M., Hedges, L. V. (1994). The handbook of research synthesis. New York, NY: Russell Sage Foundation. Google Scholar | |
|
Cooper, H. M., Hedges, L. V., Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation. Google Scholar | |
|
Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage. Google Scholar | |
|
Creswell, J. W. (2008, July 21). How mixed methods has developed. Keynote address for the Fourth Annual Mixed Methods Conference, Fitzwilliam College, Cambridge University, England. Google Scholar | |
|
Creswell, J. W. (2009). Mapping the field of mixed methods research. Journal of Mixed Methods Research, 3(2), 95-108. Google Scholar | SAGE Journals | ISI | |
|
Creswell, J. W. (2010). Mapping the developing landscape of mixed methods research. In Tashakkori, A., Teddlie, C. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 45-68). Thousand Oaks, CA: Sage. Google Scholar | Crossref | |
|
*Creswell, J. W., Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage. Google Scholar | |
|
Creswell, J. W., Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage. Google Scholar | |
|
Day, C., Sammons, P., Gu, Q. (2008). Combining qualitative and quantitative methodologies in research on teachers’ lives, work, and effectiveness: From integration to synergy. Educational Researcher, 37, 330-342. Google Scholar | SAGE Journals | ISI | |
|
Dellinger, A. B. (2005). Validity and the review of literature. Research in the Schools, 12(2), 41-54. Google Scholar | |
|
*Dellinger, A. B., Leech, N. L. (2007). Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1(4), 309-332. Google Scholar | SAGE Journals | ISI | |
|
*Dybå, T., Dingsøyr, T., Hanssen, G. K. (2007). Applying systematic reviews to diverse study types: An experience report. In Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (pp. 225-234). Washington, DC: IEEE Computer Society. Google Scholar | |
|
Feilzer, M. Y. (2010). Doing mixed methods research pragmatically: Implications for the rediscovery of pragmatism as a research paradigm. Journal of Mixed Methods Research, 4(1), 6-16. Google Scholar | SAGE Journals | ISI | |
|
Fidel, R. (2008). Are we there yet? Mixed methods research in library and information science. Library & Information Science Research, 30, 265-272. Google Scholar | Crossref | ISI | |
|
Gaber, J. (2000). Meta-needs assessment. Evaluation and Program Planning, 23, 139-147. Google Scholar | Crossref | ISI | |
|
Gliner, J. A., Morgan, G. A. (2000). Research methods in applied settings: An integrated approach to design and analysis. Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
*Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass. Google Scholar | |
|
Greene, J. C. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2(1), 7-22. Google Scholar | SAGE Journals | ISI | |
|
Greene, J. C., Benjamin, L., Goodyear, L. (2001). The merits of mixing methods in evaluation. Evaluation, 7, 25-44. Google Scholar | SAGE Journals | |
|
Greene, J. C., Caracelli, V. J. (2003). Making paradigmatic sense of mixed methods practice. In Tashakkori, A., Teddlie, C. (Eds.), Handbook of mixed methods in social and behavioral research (pp. 91-110). Thousand Oaks, CA: Sage. Google Scholar | |
|
Greene, J. C., Caracelli, V. J., Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274. Google Scholar | SAGE Journals | |
|
Greene, J. C., Hall, J. N. (2010). Dialectics and pragmatism: Being of consequence. In Tashakkori, A., Teddlie, C. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 119-143). Thousand Oaks, CA: Sage. Google Scholar | Crossref | |
|
Hall, B., Howard, K. (2008). A synergistic approach: Conducting mixed methods research with typological and systemic design consideration. Journal of Mixed Methods Research, 2(3), 248-269. Google Scholar | SAGE Journals | ISI | |
|
Hannes, K., Lockwood, C. (2011). Pragmatism as the philosophical underpinning of the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. Journal of Advanced Nursing, 67, 1632-1642. Google Scholar | Crossref | Medline | ISI | |
|
Harden, A., Thomas, J. (2005). Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methodology, 8, 257-271. Google Scholar | Crossref | |
|
Harden, A., Thomas, J. (2010). Mixed methods and systematic reviews: Examples and emerging issues. In Tashakkori, A., Teddlie, C. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 749-774). Thousand Oaks, CA: Sage. Google Scholar | Crossref | |
|
Hart, L. C., Smith, S. Z., Swars, S. L., Smith, M. E. (2009). An examination of research methods in mathematics education (1995-2005). Journal of Mixed Methods Research, 3(1), 26-41. Google Scholar | SAGE Journals | ISI | |
|
Heyvaert, M., Maes, B., Onghena, P. (2011). Applying mixed methods research at the synthesis level: An overview. Research in the Schools, 18(1), 12-24. Google Scholar | |
|
Heyvaert, M., Maes, B., Onghena, P. (2013). Mixed methods research synthesis: Definition, framework, and potential. Quality & Quantity, 47, 659-676. Google Scholar | Crossref | ISI | |
|
Higgins, J. P. T., Altman, D. G. (2008). Assessing risk of bias in included studies. In Higgins, J. P. T., Green, S. (Eds.), Cochrane handbook for systematic reviews of interventions (Version 5.0.1, pp. 187-241). Retrieved from http://www.cochrane-handbook.org Google Scholar | |
|
Higgins, J. P. T., Green, S. (Eds.). (2009). Cochrane handbook for systematic reviews of interventions (Version 5.0.2). Retrieved from http://www.cochrane-handbook.org Google Scholar | |
|
Hurmerinta-Peltomaki, L., Nummela, N. (2006). Mixed methods in international business research: A value-added perspective. Management International Review, 46, 439-459. Google Scholar | Crossref | |
|
Hutchinson, S. R., Lovell, C. D. (2004). A review of methodological characteristics of research published in key journals in higher education: Implications for graduate research teaching. Research in Higher Education, 45, 383-403. Google Scholar | Crossref | ISI | |
|
International Centre for Allied Health Evidence, University of South Australia . (n.d.). Critical appraisal tools. Retrieved from http://www.unisa.edu.au/Research/Sansom-Institute-for-Health-Research/Research-at-the-Sansom/Research-Concentrations/Allied-Health-Evidence/Resources/CAT/ Google Scholar | |
|
Johnson, R. B., Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33, 14-26. Google Scholar | SAGE Journals | |
|
Johnson, R. B., Onwuegbuzie, A. J., Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112-133. Google Scholar | SAGE Journals | ISI | |
|
Katrak, P., Bialocerkowski, A. E., Massy-Westropp, N., Kumar, V. S. S., Grimmer, K. A. (2004). A systematic review of the content of critical appraisal tools. BMC Medical Research Methodology, 4, 22-33. Google Scholar | Crossref | Medline | |
|
Khan, K. S., Riet, G., Popay, J., Nixon, J., Kleijnen, J. (2001). Undertaking systematic reviews of research on effectiveness (Centre for Reviews and Dissemination Report No. 4, 2nd ed., pp. 1-20). York, England: University of York. Google Scholar | |
|
**Leech, N. L., Dellinger, A. B., Brannagan, K. B., Tanaka, H. (2010). Evaluating mixed research studies: A mixed methods approach. Journal of Mixed Methods Research, 4(1), 17-31. Google Scholar | SAGE Journals | ISI | |
|
Letts, L., Wilkins, S., Law, M., Stewart, D., Bosch, J., Westmorland, M. (2007). Critical Review Form–Qualitative Studies (Version 2.0). Retrieved from http://www.srs-mcmaster.ca/Portals/20/pdf/ebp/qualreview_version2.0.pdf Google Scholar | |
|
Lincoln, Y. S., Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage. Google Scholar | |
|
Lincoln, Y. S., Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. In Williams, D. D. (Ed.), Naturalistic evaluation. New directions for program evaluation, No. 90 (pp. 78-84). San Francisco, CA: Jossey-Bass. Google Scholar | Crossref | |
|
Major, C. H., Savin-Baden, M. (2010). An introduction to qualitative research synthesis: Managing the information explosion in social science research. London, England: Routledge. Google Scholar | |
|
Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62, 279-300. Google Scholar | Crossref | ISI | |
|
Mertens, D. M. (2010). Transformative mixed methods research. Qualitative Inquiry, 16, 469-474. Google Scholar | SAGE Journals | ISI | |
|
Mertens, D. M. (2012). Transformative mixed methods: Addressing inequities. American Behavioral Scientist, 56, 802-813. Google Scholar | SAGE Journals | ISI | |
|
Mertens, D. M., Bledsoe, K., Sullivan, M., Wilson, A. (2010). Utilization of mixed methods for transformative purposes. In Tashakkori, A., Teddlie, C. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 193-214). Thousand Oaks, CA: Sage. Google Scholar | Crossref | |
|
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749. Google Scholar | Crossref | ISI | |
|
Moffatt, S., White, M., Mackintosh, J., Howel, D. (2006). Using quantitative and qualitative data in health services research: What happens when mixed method findings conflict? BMC Health Services Research, 6, 28-38. Google Scholar | Crossref | Medline | ISI | |
|
Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48-76. Google Scholar | SAGE Journals | ISI | |
|
Newman, I., Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale: Southern Illinois University Press. Google Scholar | |
|
O’Cathain, A. (2010). Assessing the quality of mixed methods research: Towards a comprehensive framework. In Tashakkori, A., Teddlie, C. (Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 531-555). Thousand Oaks, CA: Sage. Google Scholar | Crossref | |
|
*O’Cathain, A., Murphy, E., Nicholl, J. (2008). The quality of mixed methods studies in health services research. Journal of Health Services Research & Policy, 13, 92-98. Google Scholar | SAGE Journals | ISI | |
|
Onwuegbuzie, A. J., Collins, K. M. T., Leech, N. L., Dellinger, A. B., Jiao, Q. G. (2010). A meta-framework for conducting mixed research syntheses for stress and coping and beyond. In Gates, G. S., Gmelch, W. H., Wolverton, M. (Series Eds.) and Collins, K. M. T., Onwuegbuzie, A. J., Jiao, Q. G. (Vol. Eds.), The Research on Stress and Coping in Education Series: Vol. 5. Toward a broader understanding of stress and coping: Mixed methods approaches (pp. 169-212). Charlotte, NC: Information Age. Google Scholar | |
|
*Onwuegbuzie, A. J., Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 13(1), 48-63. Google Scholar | |
|
**Onwuegbuzie, A. J., Johnson, R. B., Collins, K. M. T. (2011). Assessing legitimation in mixed research: A new framework. Quality & Quantity, 45, 1253-1271. Google Scholar | Crossref | ISI | |
|
Onwuegbuzie, A. J., Leech, N. L. (2005a). Taking the “Q” out of research: Teaching research methodology courses without the divide between quantitative and qualitative paradigms. Quality & Quantity, 39, 267-296. Google Scholar | Crossref | ISI | |
|
Onwuegbuzie, A. J., Leech, N. L. (2005b). On becoming a pragmatic researcher: The importance of combining quantitative and qualitative research methodologies. International Journal of Social Research Methodology, 8, 375-387. Google Scholar | Crossref | |
|
Pawson, R., Greenhalgh, T., Harvey, G., Walshe, K. (2005). Realist review: A new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10(Suppl. 1), 21-34. Google Scholar | SAGE Journals | |
|
Petticrew, M., Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell. Google Scholar | Crossref | |
|
*Pluye, P., Gagnon, M. P., Griffiths, F., Johnson-Lafleur, J. (2009a, November). A proposal for concomitantly appraising qualitative, quantitative and mixed methods primary studies included in systematic mixed studies reviews. Paper presented at the 37th Annual Meeting of the North American Primary Care Research Group, Montreal, Quebec, Canada. Google Scholar | |
|
**Pluye, P., Gagnon, M. P., Griffiths, F., Johnson-Lafleur, J. (2009b). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative, and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46, 529-546. Google Scholar | Crossref | Medline | ISI | |
|
*Pluye, P., Grad, R. M., Dunikowski, L., Stephenson, R. (2005). Impact of clinical information-retrieval technology on physicians: A literature review of quantitative, qualitative and mixed methods studies. International Journal of Medical Informatics, 74, 745-768. Google Scholar | Crossref | Medline | ISI | |
|
Public Health Resource Unit . (2006a). Critical Appraisal Skills Programme (CASP). 10 questions to help you make sense of qualitative research. Retrieved from http://www.sph.nhs.uk/sph-files/casp-appraisal-tools/Qualitative%20Appraisal%20Tool.pdf Google Scholar | |
|
Public Health Resource Unit . (2006b). Critical Appraisal Skills Programme (CASP). 11 questions to help you make sense of a case control study. Retrieved from http://www.sph.nhs.uk/sph-files/casp-appraisal-tools/Case%20Control%2011%20Questions.pdf Google Scholar | |
|
Saini, M., Shlonsky, A. (2012). Systematic synthesis of qualitative research. Oxford, England: Oxford University Press. Google Scholar | Crossref | |
|
*Sale, J. E. M., Brazil, K. (2004). A strategy to identify critical appraisal criteria for primary mixed-method studies. Quality & Quantity, 38, 351-365. Google Scholar | Crossref | Medline | ISI | |
|
Sandelowski, M., Voils, C. I., Barroso, J. (2006). Defining and designing mixed research synthesis studies. Research in the Schools, 13(1), 29-40. Google Scholar | Medline | |
|
Sweetman, D., Badiee, M., Creswell, J. W. (2010). Use of the transformative framework in mixed methods studies. Qualitative Inquiry, 16, 441-454. Google Scholar | SAGE Journals | ISI | |
|
Tashakkori, A., Creswell, J. W. (2007). The new era of mixed methods. Journal of Mixed Methods Research, 1(1), 3-7. Google Scholar | SAGE Journals | ISI | |
|
Tashakkori, A., Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage. Google Scholar | |
|
Tashakkori, A., Teddlie, C. (Eds.). (2003a). Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage. Google Scholar | |
|
Tashakkori, A., Teddlie, C. (2003b). The past and future of mixed methods research: From data triangulation to mixed model designs. In Tashakkori, A., Teddlie, C. (Eds.), Handbook of mixed methods in social & behavioral research (pp. 671-701). Thousand Oaks, CA: Sage. Google Scholar | |
|
Tashakkori, A., Teddlie, C. (2006, April). Validity issues in mixed methods research: Calling for an integrative framework. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA. Google Scholar | |
|
**Tashakkori, A., Teddlie, C. (2008). Quality of inferences in mixed methods research: Calling for an integrative framework. In Bergman, M. M. (Ed.), Advances in mixed methods research: Theories and applications (pp. 101-119). London, England: Sage. Google Scholar | |
|
**Teddlie, C., Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In Tashakkori, A., Teddlie, C. (Eds.), Handbook of mixed methods in social & behavioral research (pp. 3-50). Thousand Oaks, CA: Sage. Google Scholar | |
|
*Teddlie, C., Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: Sage. Google Scholar | |
|
Truscott, D., Swars, S., Smith, S., Thornton-Reid, F., Zhao, Y., Dooley, C. (2010). A cross-disciplinary examination of the prevalence of mixed methods in educational research: 1995-2005. International Journal of Social Research Methodology, 13, 317-328. Google Scholar | Crossref | ISI | |
|
Whittemore, R., Chase, S. K., Mandle, C. L. (2001). Validity in qualitative research. Qualitative Health Research, 11, 522-537. Google Scholar | SAGE Journals | ISI | |
|
Whittemore, R., Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52, 546-553. Google Scholar | Crossref | Medline | ISI | |
|
Young, J. M., Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice Gastroenterology & Hepatology, 6, 82-91. Google Scholar | Crossref | Medline |
