![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Although combining methods is nothing new, more contributions about why and how to mix methods for validation purposes are needed. This article presents a case of validating the inferences drawn from the Participatory Evaluation Measurement Instrument, an instrument that purports to measure stakeholder participation in evaluation. Although the process was intended to be almost exclusively quantitative, one of its components unexpectedly turned into a mixed methods study. This, in turn, spurred on a cycle of instrument revision and further quantitative validation. Whereas the validation evidence is modest and tentative, it suggests that the revised version of the Participatory Evaluation Measurement Instrument offers a better fit with the respondents’ opinions regarding the participation level of selected evaluation cases. The article concludes with a brief discussion on the added value of mixed methods for validation purposes.
References
Section:
|
Adcock, R., Collier, D. (2001). Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review, 95, 529-546. Google Scholar | Crossref | ISI | |
|
American Association for Public Opinion Research . (2011). Standard definitions: Final dispositions of case codes and outcome rates for surveys. Retrieved from http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions2&Template=/CM/ContentDisplay.cfm&ContentID=3156 Google Scholar | |
|
Arnon, S., Reichel, N. (2009). Closed and open-ended question tools in a telephone survey about “the good teacher.” Journal of Mixed Methods Research, 3(2), 172-196. Google Scholar | SAGE Journals | ISI | |
|
Braun, V., Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101. Google Scholar | Crossref | |
|
Burford, G., Velasco, I., Janouskova, S., Zahradnik, M., Hak, T., Podger, D., Piggot, G., Harder, M. K. (2013). Field trials of a novel toolkit for evaluating ‘intangible’ values related dimensions of projects. Evaluation and Program Planning, 36(1), 1-14. doi: 10.1016/j.evalprogplan.2012.04.005. Google Scholar | Crossref | Medline | ISI | |
|
Collins, K. M. T., Onwuegbuzie, A. J., Sutton, I. L. (2006). Model incorporating the rationale and purpose for conducting mixed-methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67-100. Google Scholar | |
|
Connors, S. C., Magilvy, J. K. (2011). Assessing vital signs: Applying two participatory evaluation frameworks to the evaluation of a college of nursing. Evaluation and Program Planning, 34, 79-86. Google Scholar | Crossref | Medline | ISI | |
|
Cousins, J. B. (2001). Do evaluator and program practitioner perspective converge in collaborative evaluation? Canadian Journal of Program Evaluation, 16(2), 113-133. Google Scholar | |
|
Cousins, J. B., Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 1998(80), 5-23. Google Scholar | Crossref | |
|
Creswell, J. W. (2011). Controversies in mixed methods research. In Denzin, N. K., Lincoln, Y. S. (Eds.), The Sage handbook of qualitative research (4th ed., pp. 269-283). Thousand Oaks, CA: Sage. Google Scholar | |
|
Creswell, J. W., Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage. Google Scholar | |
|
Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., Hanson, W. E. (2003). Advanced mixed methods research designs. In Tashakkori, A., Teddlie, C. (Eds.), Handbook of mixed methods in social & behavioral research (pp. 209-240). Thousand Oaks, CA: Sage. Google Scholar | |
|
Curry, L. A., Nembhard, I. M., Bradley, E. H. (2009). Qualitative and mixed methods provide unique contributions to outcomes research. Circulation, 119, 1442-1452. Google Scholar | Crossref | Medline | ISI | |
|
Daigneault, P.-M. (2012). La participation á l’évaluation: du concept á la mesure [Stakeholder participation: From the concept to measurement] (doctoral dissertation). Université Laval: Quebec City (QC, Canada). Retrieved from: http://www.theses.ulaval.ca/2012/29096/29096.pdf Google Scholar | |
|
Daigneault, P.-M., Jacob, S. (2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation.. American Journal of Evaluation, 30(3), 330–48. Google Scholar | SAGE Journals | ISI | |
|
Daigneault, P.-M., Jacob, S., Tremblay, J. (2012), Measuring stakeholder participation in evaluation: An empirical validation of the Participatory Evaluation Measurement Instrument. Evaluation Review, 36(4), 243-270. Google Scholar | SAGE Journals | ISI | |
|
Dellinger, A. B., Leech, N. L. (2007). Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1(4), 309-332. Google Scholar | SAGE Journals | ISI | |
|
Durham, J., Tan, B.-K., White, R. (2011). Utilizing mixed research methods to develop a quantitative assessment tool. Journal of Mixed Methods Research, 5(3), 212-226. Google Scholar | SAGE Journals | ISI | |
|
Fenton, A. (2006). Weft QDA (Version 1.0.1) [Computer software]. Retrieved from http://www.pressure.to/qda/ Google Scholar | |
|
Firmin, M. W. (2008). Themes. In Given, L. M. (Ed.), The Sage encyclopedia of qualitative research methods (pp. 869-870). Thousand Oaks, CA: Sage. Google Scholar | |
|
Gerring, J. (1999). What makes a concept good? A criterial framework for understanding concept formation in the social sciences. Polity, 31, 357-393. Google Scholar | Crossref | ISI | |
|
Goertz, G. (2006). Social science concepts: A user’s guide. Princeton, NJ: Princeton University Press. Google Scholar | |
|
Greene, J. C., Caracelli, V. J., Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274. Google Scholar | SAGE Journals | |
|
Gregory, A. (2000). Problematizing participation: A critical review of approaches to participation in evaluation theory. Evaluation, 6, 179-199. Google Scholar | SAGE Journals | |
|
Jacob, S., Desautels, G. (2013). Evaluation of aboriginal programs: What place is given to participation and cultural sensitivity? The International Indigenous Policy Journal, 4(2). Retrieved from: http://ir.lib.uwo.ca/iipj/vol4/iss2/1 Google Scholar | |
|
Jacob, S., Ouvrard, L., Bélanger, J.-F. (2011). Participatory evaluation and process use within a social aid organization for at-risk families and youth. Evaluation and Program Planning, 34(2), 113-23. Google Scholar | Crossref | Medline | ISI | |
|
Johnson, R. B., Onwuegbuzie, A. J., Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112-133. Google Scholar | SAGE Journals | ISI | |
|
Latcheva, R. (2009). Cognitive interviewing and factor-analytic techniques: A mixed method approach to validity of survey items measuring national identity. Quality & Quantity, 45, 1175-1199. Google Scholar | Crossref | ISI | |
|
Laudon, J. M. D. (2010). Participatory to the end: Planning and implementation of a participatory evaluation strategy (FES Outstanding Graduate Student Paper Series). Toronto, Ontario, Canada: York University. Google Scholar | |
|
Littell, J. H., Corcoran, J., Pillai, V. (2008). Systematic reviews and meta-analysis. Oxford, England: Oxford University Press. Google Scholar | Crossref | |
|
Luyt, R. (2011). A framework for mixing methods in quantitative measurement development, validation, and revision: A case study. Journal of Mixed Methods Research, 6(4), 294-316. Google Scholar | SAGE Journals | ISI | |
|
Mark, M. M. (2001). Evaluation’s future: Furor, futile, or fertile? American Journal of Evaluation, 22, 457-479. Google Scholar | SAGE Journals | ISI | |
|
O’Neill Green, D. (2008). Categories. In Given, L. M. (Ed.), The Sage encyclopedia of qualitative research methods (pp. 72-73). Thousand Oaks, CA: Sage. Google Scholar | |
|
Onwuegbuzie, A. J., Bustamante, R. M., Nelson, J. A. (2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56-78. Google Scholar | SAGE Journals | ISI | |
|
Pietiläinen, V. (2012). Testing the participatory education evaluation concept in a national context. Studies in Educational Evaluation, 38, 9-14. Google Scholar | Crossref | |
|
Secolsky, C., Wentland, E., Dennison, B. (2011). The need for documenting validation transactions: A qualitative component of the testing validation process. Quality & Quantity, 45, 1303-1311. Google Scholar | Crossref | ISI | |
|
Small, M. L. (2011). How to conduct a mixed methods study: Recent trends in a rapidly growing literature. Annual Review of Sociology, 37, 57-86. Google Scholar | Crossref | ISI | |
|
Spillane, J. P., Pareja, A. S., Dorner, L., Barnes, C., May, H., Huff, J., Camburn, E. (2010). Mixing methods in randomized controlled trials (RCTs): Validation, contextualization, triangulation, and control. Educational Assessment, Evaluation and Accountability, 22, 5-28. Google Scholar | Crossref | ISI | |
|
Toal, S. A. (2009). The validation of the evaluation involvement scale for use in multisite settings. American Journal of Evaluation, 30, 349-362. Google Scholar | SAGE Journals | ISI | |
|
Ver Ploeg, M., Moffit, R. A., Citro, C. F. (Eds.). (2002). Studies of welfare populations: Data collection and research issues. Washington, DC: National Academies Press. Google Scholar |
