Are There Assessment Criteria for Qualitative Findings? A Challenge Facing Mixed Methods Research
If findings from qualitative and quantitative components in mixed methods research are to be synthesised, the quality of each must be assessed. But an obvious problem is that there are no generally agreed criteria for assessing qualitative findings. The question of criteria has long been debated in the methodological literature. I argue that some important distinctions need to be made if progress is to be achieved on this issue. Perhaps the most important one is between the standards in terms of which assessment is carried out and the indicators used to evaluate findings in relation to those standards. I go on to outline what I believe is involved in such evaluations, rejecting the possibility of a detailed and explicit set of indicators that can immediately be used to determine the validity of knowledge claims. My approach broadly fits the framework of mixed methods research, since I deny that there is any fundamental philosophical difference between quantitative and qualitative methods. But it is at odds with widespread views, even within the realm of mixed methods, whose advocates seek radically to redefine the ontological, epistemological, and/or axiological assumptions of social scientific research, for example in the name of a transformative approach.
Attree, Pamela & Milton, Beth (2006). Critically appraising qualitative research for systematic reviews: Defusing the methodological cluster bombs. Evidence & Policy, 2(1), 109-126.
Barbour, Rosaline & Barbour, Michael (2002). Evaluating and synthesizing qualitative research: The need to develop a distinctive approach. Journal of Evaluation in Clinical Practice, 9(2), 179-186.
Bergman, Max & Coxon, Anthony (2005). The quality in qualitative methods. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(2), Art. 34, https://doi.org/10.17169/fqs-6.2.457 [Accessed: March 11, 2022].
Bochner, Arthur (2000). Criteria against ourselves. Qualitative Inquiry, 6(2), 266-272.
Boyle, David (2020). Tickbox. London: Little, Brown.
Breuer, Franz & Reichertz, Jo (2001). Standards of social research. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 2(3), Art. 24, https://doi.org/10.17169/fqs-2.3.919 [Accessed: March 11, 2022].
Bryman, Alan; Becker, Saul & Sempik, Joe (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11(4), 261-276.
Calderón Gómez, Carlos (2009). Assessing the quality of qualitative health research: Criteria, process and writing. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 10(2), Art. 17, https://doi.org/10.17169/fqs-10.2.1294 [Accessed: March 11, 2022].
Christie, Richard & Jahoda, Marie (Eds.) (1954). Studies in the scope and method of "The authoritarian personality". Glencoe, IL: Free Press.
Collins, Harry (2014). Are we all scientific experts now?. Cambridge: Polity.
Dellinger, Amy & Leech, Nancy (2007). Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1(4), 309-332.
Elashoff, Janet & Snow, Richard (Eds.) (1971). Pygmalion reconsidered. Worthington, OH: Charles A. Jones.
Elliott, Robert; Fisher, Constance & Rennie, David (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38(3), 215-229.
Emden, Carolyn & Sandelowski, Margarete (1999). The good, the bad, and the relative, Part 2: Goodness and the criterion problem in qualitative research. International Journal of Nursing Practice, 5(1), 2-7.
Flick, Uwe (2007). Managing quality in qualitative research. London: Sage.
Flick, Uwe (2018). An introduction to qualitative research (8th ed.). London: Sage.
Geertz, Clifford (1973). The interpretation of cultures. New York, NY: Basic Books.
Gobo, Giampietro; Fielding, Nigel G.; La Rocca, Gevisa & van der Vaart, Wander (2021). Merged methods. London: Sage.
Greene, Judith (2008). Is mixed methods social inquiry a distinctive methodology?. Journal of Mixed Methods Research, 2(1), 7-22.
Hammersley, Martyn (1997). Qualitative data archiving: Some reflections on its prospects and problems. Sociology, 31(1), 131-42.
Hammersley, Martyn (2005). Countering the "new orthodoxy" in educational research: A response to Phil Hodkinson. British Educational Research Journal, 31(2), 139-155.
Hammersley, Martyn (2007). The issue of quality in qualitative research. International Journal of Research and Method in Education, 30(3), 287-306.
Hammersley, Martyn (2008a). Troubling criteria: A critical commentary on Furlong and Oancea’s framework for assessing educational research. British Educational Research Journal, 34(6), 747-762.
Hammersley, Martyn (2008b). Assessing validity in social research. In Pertti Alasuutari, Leonard Bickman & Julia Brannen (Eds.), The Sage handbook of social research methods (pp.42-52). London: Sage.
Hammersley, Martyn (2008c). Troubles with triangulation. In Max Bergman (Ed.), Advances in mixed methods research (pp.22-36). London: Sage.
Hammersley, Martyn (2008d). Questioning qualitative inquiry. London: Sage.
Hammersley, Martyn (2009). Challenging relativism: The problem of assessment criteria. Qualitative Inquiry, 15(1), 3-29.
Hammersley, Martyn (2011). Methodology, who needs it?. London: Sage.
Hammersley, Martyn (2013a). The myth of research-based policy and practice. London: Sage.
Hammersley, Martyn (2013b). What is qualitative research?. London: Bloomsbury.
Hammersley, Martyn (2016 ). Reading ethnographic research. London: Longmans.
Hammersley, Martyn (2017a). On the role of values in social research: Weber vindicated?. Sociological Research Online, 22(1), 7, https://www.socresonline.org.uk/22/1/7.html [Accessed: March 11, 2022].
Hammersley, Martyn (2017b). Interview data: A qualified defence against the radical critique. Qualitative Research, 17(2), 173-186.
Hammersley, Martyn (2018a ). What’s wrong with ethnography?. London: Routledge.
Hammersley, Martyn (2018b). What is ethnography? Can it survive? Should it?. Ethnography and Education, 13(1) 1-17.
Hammersley, Martyn & Atkinson, Paul (2019). Ethnography: Principles in practice (4th ed.). London: Routledge.
Hammersley, Martyn & Traianou, Anna (2012). Ethics in qualitative research. London: Sage.
Harrits, Gitte Sommer (2011). More than method? A discussion of paradigm differences within mixed methods research. Journal of Mixed Methods Research, 5(2),150-166.
Hatch, Amos (2007). Trustworthiness. In George Ritzer (Ed.), The Blackwell encyclopedia of sociology (pp.5083-5085). New York, NY: Wiley.
Heyvaert, Mieke; Hannes, Karin; Maes, Bea & Onghena, Patrick (2013). Critical appraisal of mixed methods studies. Journal of Mixed Methods Research, 7, 302-327.
Kuntz, Aaron (2020). Standing at one's post: Post-qualitative inquiry as ethical enactment. Qualitative Inquiry, 27(2), 215-8.
Lather, Patti (2013). Methodology-21: What do we do in the afterward?. International Journal of Qualitative Studies in Education, 26(6), 634-645.
Lather, Patti & St. Pierre, Elizabeth (2013). Post-qualitative research. International Journal of Qualitative Studies in Education, 26(6), 629-633.
Leavy, Patricia (Ed.) (2019). Handbook of arts-based research. New York, NY: The Guilford Press.
Lee, Justin (2014). Genre-appropriate judgments of qualitative research. Philosophy of the Social Sciences, 44(3), 316-348.
Lincoln, Yvonna S. (1995). Emerging criteria for quality in qualitative and interpretive research. Qualitative Inquiry, 1(3), 275-289.
Luft, Sebastian (2002). Husserl's notion of the natural attitude and the shift to transcendental phenomenology. In Anna-Teresa Tymieniecka (Ed.), Analecta Husserliana The yearbook of phenomenological research: Phenomenology worldwide, 80 (pp.114-119). Dordrecht: Springer.
Mays, Nick & Pope, Catherine (1995). Qualitative research: Rigour and qualitative research. British Medical Journal, 311(6997), 109-112.
Merton, Robert & Lazarsfeld, Paul (Eds.) (1950). Continuities in social research: Studies in the scope and method of "The American soldier". Glencoe, IL: Free Press.
Murphy, Elizabeth; Dingwall, Robert; Greatbatch, David; Parker, Susan & Watson, Pamela (1998). Qualitative research methods in health technology assessment: A review of the literature. Health Technology Assessment, 2(16), 1-260, https://doi.org/10.3310/hta2160 [Accessed: March 11, 2022].
O'Cathain, Alicia (2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In Abbas Tashakkori & Charles Teddlie (Eds.), The Sage handbook of mixed methods in social and behavioral research (pp.531-556). Thousand Oaks, CA: Sage.
Potter, Jonathan (1996). Representing reality discourse, rhetoric and social construction. London: Sage.
Potter, Jonathan & Hepburn, Alexa (2005). Qualitative interviews in psychology: Problems and possibilities. Qualitative Research in Psychology, 2(4), 281-307.
Power, Michael (1997). The audit society. Oxford: Oxford University Press.
Reicher, Stephen (2000). Against methodolatry: Some comments on Elliott, Fischer, and Rennie. British Journal of Clinical Psychology, 39(1), 1-6.
Reichertz, Jo (2019). Method police or quality assurance? Two patterns of interpretation in the struggle for supremacy in qualitative social research. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 20(1), Art. 3, https://doi.org/10.17169/fqs-20.1.3205 [Accessed: March 11, 2022].
Sandelowski, Margarete (1978). The problem of rigor in qualitative research. Advances in Nursing Science, 8(3), 27-37.
Sandelowski, Margarete (1993). Rigor or rigor mortis: The problem of rigor in qualitative research revisited. Advances in Nursing Science, 16(2), 1-8.
Schoonenboom, Judith & Johnson, R. Burke (2017). How to construct a mixed methods research design. Kölner Zeitschrift für Soziologie, 69, 107-131.
Seale, Clive (1999). The quality of qualitative research. London: Sage.
Shadish, William; Cook, Thomas & Leviton, Laura (1991). Foundations of program evaluation. Newbury Park, CA: Sage.
Smith, John K. (1990). Alternative research paradigms and the problem of criteria. In Egon Guba (Ed.), The paradigm dialog (pp.167-87). Newbury Park, CA: Sage.
Smith, John K. (2004). Learning to live with relativism. In Heather Piper & Ian Stronach (Eds.), Educational research: Difference and diversity (pp.45-58). Aldershot: Ashgate.
Smith, John K. & Hodkinson, Phil (2005). Relativism, criteria, and politics. In Norman K. Denzin & Yvonna S. Lincoln (Eds.), Handbook of qualitative research (3rd ed., pp.915-32). Thousand Oaks, CA: Sage.
Smith, John K. & Hodkinson, Phil (2009). Challenging neorealism. Qualitative Inquiry, 15, 30-39.
Spencer, Liz; Ritchie, Jane; Lewis, Jane & Dillon, Lucy (2003). Quality in qualitative evaluation. London: Cabinet Office, https://www.gov.uk/government/publications/government-social-research-framework-for-assessing-research-evidence [Accessed: March 11, 2022].
Symonds, Jennifer & Gorard, Stephen (2010). Death of mixed methods? Or the rebirth of research as a craft. Evaluation and Research in Education, 23(2),121-136.
Tashakkori, Abbas & Teddlie, Charles (Eds.) (2010). The Sage handbook of mixed methods in social & behavioral research (2nd ed.). Thousand Oaks, CA: Sage.
Copyright (c) 2023 Martyn Hammersley
This work is licensed under a Creative Commons Attribution 4.0 International License.