Criteria for Evaluating Papers Using Qualitative Research Methods
The aim of this set of criteria is to enable the reviewers, authors and readers of the Journal of Occupational and Organizational Psychology to judge the quality of research papers using qualitative methods. Given the considerable diversity in research strategies that use qualitative methods, these criteria need to be seen as general guidelines to aid reviewers rather than specific recommendations for all papers.
1. General Features
Contribution to understanding: Does the paper contribute to our understanding of people and organizations at work?
Appropriateness of method: Are the methods chosen appropriate to the research questions? The focus of the study should lend itself to qualitative analysis (e.g. the subjective understanding or sense-making processes of individuals in relation to their work situation).
Literature: Is the paper located within an appropriate literature?
Theoretical considerations: Is the paper of interest theoretically (e.g. does it develop theory in a particular direction)?
Epistemological integrity: Does the paper take a consistent approach towards epistemology, ontology and methods?
2. Outline of methods
Sampling: In this context a sample of participants or cases does not necessarily need to be representative, or random, but a clear rationale is needed for the inclusion of some cases, or individuals, rather than others. Sample size can be justified on the basis of the aims of the study, the specific methodological approach and the claims the authors wish to make about their findings.
Choice of data collection technique: An explanation is required for why a particular method was chosen to access data rather than another method (e.g. why a semi-structured interview rather than a diary study).
Researcher-situation interface: Issues regarding the dynamics of the data collection situation should be explored. For example group dynamics within a focus group situation could be commented upon; or interviewees' responses to the interview situation should be recognised and discussed.
Data collection and management: How were data stored, managed and used? Examples here are did the researcher keep detailed notes of field visits? Were all interviews transcribed? etc.
Contextualization: Is the research clearly contextualised? Is all the relevant information about the participants and/or the organisations clearly specified?
3. Data analysis
Description of analytic framework: An analytic framework needs to be clearly outlined. In some cases there maybe a pre-existing framework for data analysis. In this case the derivation of this framework should be explained. If there is no pre-existing framework then the author/s need to explain the rationale for the analysis strategy.
Auditability of analysis procedures and processes: The processes and procedures for analysis should be detailed. A content analysis will look very different from a discourse analysis for example. The reader needs to be able to understand the processes or procedures or steps through which the data analysis evolved, that is the analysis trail should be auditable.
Derivation of analysis categories: Adequate discussion needs to be provided of how themes, categories or concepts were derived from the data or from the literature.
Sources of raw data: Where more than one method of data collection has been used, authors should refer to the analysis process for each method and any subsequent integration of analysis from the different methods.
Use of transcript excerpts: When using quotes to highlight findings the basis for the selection of particular excerpts should be explained (e.g. are they representative, illustrative, etc.) The source of the excerpt should be referenced.
Confirmability: In some qualitative studies, where appropriate, researchers may use reliability checks to see whether others would categorise the data in the same way. Are these checks, if used, reported?
Credibility: In some qualitative studies researchers may feed back their interpretations of the data collected to research participants in order to gain their views of the coherence or representation of the researcher's analysis. In these cases the paper should outline how this feedback was received and how participants comments on the interpretation were dealt with. This may be particularly appropriate where studies are based on interventions into a given organisational situation.
Alternative explanations: Are mechanisms outlined through which the researcher has sought disconfirmatory evidence or alternative explanations for the results?
4. Findings and discussion
Researcher reflexivity: The researcher should clearly outline their role in the research process: they should be reflexive about their methodological approach and how this relates to the findings they present.
Consistency: The results should seem credible and appropriate, relating back to the research questions and the literature review. Where the paper has been of an exploratory nature with the aim of using data analysis to inductively produce theory, then the pertinent literature can be outlined in the discussion section.
Theoretical considerations: The claims made for theory development need to be consistent with the data presented.
Transferability: If appropriate the authors should comment on the transferability of their findings from one context to another.
Utilization: Ideally the findings will be useful and applicable. The author will provide added value by commenting on how this is the case.
Acknowledgements
These guidelines were compiled by Catherine Cassell and derived from a number of sources including a paper entitled 'Evolving guidelines for publication of qualitative research studies (6/94)' by Robert Elliot, Constance Fischer and David Rennie from a steering group of the US Society for Psychotherapy research and 'Criteria for the evaluation of qualitative research methods: agreed criteria produced by consultation, convened by Mildred Blaxter and adopted by the Medical Sociology Group of the BSA'. Other sources include Lincoln, Y.S. and Guba, E.G. (1985), Naturalistic Inquiry, London: Sage; Hammersley, M. (1992) What's wrong with ethnography? London: Routledge; Flick, U. (1998) An introduction to qualitative research, London: Sage; and Symon, G. and Cassell, C.M. (1998) Qualitative methods and analysis in organizational research: a practical guide, London: Sage, plus comments from Gillian Symon, Nigel King, Phil Johnson and Joanne Duberley.