Speaking of research
Guidelines for evaluating research articles
Phillip Rumrill∗, Shawn Fitzgerald and Megen Ware Kent State University, Department of Educational Foundations and Special Services Center for Disability Studies, 405 White Hall, P.O. Box 5190, Kent, OH 44242-0001, USA
The article describes the components and composition of journal articles that report empirical research findings in the field of rehabilitation. The authors delineate technical writing strategies and discuss the contents of research manuscripts, including the Title, Abstract, Introduction, Method, Results, Discussion, and References. The article concludes with a scale that practitioners, manuscript reviewers, educators, and students can use in critically analyzing the content and scien- tific merits of published rehabilitation research.
Keywords: Evaluation, research articles, guidelines for cri- tique
1. Introduction
The purpose of this article is to examine the com- ponents of a research article and provide guidelines for conducting critical analyses of published works. Distilled from the American Psychological Associa- tion’s [1] Publication Manual and related descriptions in several research design texts [4,8,9,12,15], descrip- tions of how authors in rehabilitation and disability studies address each section of a research article are featured. The article concludes with a framework that rehabilitation educators, graduate students, practition- ers, and other Work readers can use in critiquing re- search articles on the basis of their scientific merits and practical utility.
∗Corresponding author: Tel.: +1 330 672 2294; Fax: +1 330 672 2512; E-mail: prumrill@educ.kent.edu.
2. Anatomy of a research article
For nearly 50 years, the American Psychological As- sociation has presented guidelines for authors to follow in composing manuscripts for publication in profes- sional journals [1]. Most journals in disability studies and rehabilitation adhere to those style and formatting guidelines. In the paragraphs to follow, descriptions of each section of a standard research article are pre- sented: Title, Abstract, Introduction, Method, Results, Discussion, and References.
2.1. Title
As with other kinds of literature, the title of a scien- tific or scholarly journal article is a very important fea- ture. At the risk of contravening the age-old adage “You can’t judge a book by its cover,” Bellini and Rumrill [4] speculated that most articles in rehabilitation journals are either read or not read based upon the prospective reader’s perusal of the title. Therefore, developing a clear, concise title that conveys the article’s key con- cepts, hypotheses, methods, and variables under study is critical for researchers wishing to share their findings with a large, professional audience. A standard-length title for a journal article in the social sciences is 12–15 words, including a sub-title if appropriate. Because so- cial science and medical indexing systems rely heavily on titles in their codification schemes to track and cat- egorize journal articles by topic, providing a title that clearly delineates a general research domain or topic area is of utmost importance. If the title is vague or ambiguous, chances are that the prospective reader will not continue to read through the document to establish where it might fit in terms of a specific research domain or topic area. Examples of clearly descriptive titles that can be found in the contemporary rehabilitation literature include:
“Rehabilitation Counselors’ Assessments of Appli- cants’ Functional Limitations as Predictors of Rehabil- itation Services Provided” [3].
Work 14 (2000) 257–263 ISSN 1051-9815 / $8.00 2000, IOS Press. All rights reserved
258 P. Rumrill et al. / Guidelines for evaluating research articles
“Employer Concerns About Hiring Persons with Psychiatric Disabilities: Results of the Employer Atti- tude Questionnaire” [6].
“Self-Perceived Reasons for Unemployment Cited by Persons with Spinal Cord Injury: Relationship to Gender, Race, Age, and Level of Injury” [13].
“Vocational Rehabilitation Counselors’ Attitudes Toward Self-Employment Outcomes” [18].
“Surveying the Employment Concerns of People with Multiple Sclerosis: A Participatory Action Re- search Approach” [20].
“Effect of Graduate Research Instruction on Per- ceived Research Anxiety, Research Utility, and Confi- dence in Research Skills” [21].
Before we move into descriptions of the content sec- tions of a research article, we want to briefly address the concept of technical writing as it applies to the com- position of academic manuscripts. Journals adhering to the American Psychological Association’s [1] pub- lication guidelines favor manuscripts that are written in direct, uncomplicated sentences. Editors prefer that text be written in the “active voice”; whenever possible, sentences should begin with their subjects and follow with verbs and objects (e.g., “The researcher conducted an experiment” rather than “An experiment was con- ducted by the researcher”). Technical writing is marked by the “less is more” maxim; extraneous phrases and clauses that add words to the sentence without enhanc- ing the overall statement should be avoided (e.g., “In order to. . . ”, “For purposes of. . . ”, “As far as. . . is concerned. . .”). Another element of sound technical writing is the sparing use of adverbs (e.g., very, some- what, strikingly) and adjectives that do not serve to fur- ther define or specify the terms that they are modifying (e.g., interesting, important, good, noteworthy).
In addition to the American Psychological Associa- tion’s guidelines for technical writing, authors should consider these six criteria for effective composition provided by George Orwell (1946) in Politics and the English Language:
1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print.
2. Never use a long word where a short one will do. 3. If it is possible to cut a word out, always cut it
out. 4. Never use the passive (voice) where you can use
the active. 5. Never use a foreign phrase, a scientific word, or
jargon word if you can think of an everyday En- glish equivalent.
6. Break any of these rules sooner than say anything outright barbarous (p. 170).
Organization is also key in preparing an effectively composed journal manuscript, with multi-level head- ings serving to guide the flow of text and keep the reader on track. For authoritative information regard- ing the style and formatting guidelines for submitting manuscripts to most journals in social science fields, readers should consult the American Psychological As- sociation’s [1] Publication Manual. For information concerning the style and formatting requirements of Work and other journals published by IOS Press, see the Guidelines for Authors section included in the be- ginning of this edition.
2.2. Abstract
Next to the title, the abstract is the most widely read section of a journal article. In an empirical article, the abstract should be a succinct, 100–150 word summary of the investigation’s key features, including purpose, objectives, research questions/hypotheses, sample, sci- entific procedures, independent and dependent vari- ables, and salient results. Results of the study should be summarized in full in the abstract; authors should describe both significant and non-significant findings, not only those which upheld their hypotheses or expec- tations. The abstract serves as an advance organizer for the article, and it should include every important premise, method, and result of the investigation. Like the Preface that commonly orients readers to full-length textbooks, the abstract provides a complete, albeit sum- mary, preview of the article. Some journals, includ- ing Work and the Journal of Vocational Rehabilitation, ask authors to list key descriptors on the abstract page, which are then used for purposes of indexing. In most cases, the title is what determines whether a reader will read the abstract; the abstract determines whether the reader will read the body of the article.
2.3. Introduction
Immediately following the abstract, the introductory section of the article sets the stage for the study upon which the article was based. It orients the reader to the problem or issue being addressed, develops the logic and rationale for conducting the investigation, and al- most always expresses the empirical hypotheses or re- search questions. Heppner et al. [9] suggested that the introduction should answer questions such as why the topic is an important one to study, what previous
P. Rumrill et al. / Guidelines for evaluating research articles 259
work bears on the topic, how existing work logically connects to the author’s research questions and/or hy- potheses, how the question will be researched,and what predictions can be made.
To answer these questions, authors typically address three major elements in the introductory section of an article: (1) The Research Problem, (2) The Framework for the Study, and (3) The Research Questions and Hy- potheses [8,15]. We will describe each of these intro- ductory elements in linear fashion, but we do not mean to imply an order in terms of how they should be ad- dressed. Many (if not most) authors blend these con- siderations to fit the flow and logic of their respective manuscripts.
The research problem. Usually in the very first sen- tences of an empirical journal article, the author draws the reader’s attention to the scope, impact, and current status of the problem or issue being investigated. This orientation is most effectively achieved by applying the broadest-possible perspective to the concern. A study of success rates among participants in a stress inocula- tion program for people with diabetes mellitus might be introduced by citing national statistics concerning the incidence and prevalence of this very common disease. An article describing the effects of a model job place- ment program for women with breast cancer might be- gin with a review of existing literature concerning em- ployment and breast cancer, with a particular focus on the difficulties that women have in re-entering the la- bor force following diagnosis and treatment. Authors reporting a longitudinal study of the post- school em- ployment outcomes of secondary students with devel- opmental disabilities would likely introduce their arti- cle with a review of the disappointing adult outcomes which that population has experienced since the incep- tion of formalized transition services in the mid–1980s.
The framework for the study. The specific theoret- ical and empirical framework for the particular inves- tigation is another important part of the Introduction. Authors summarize existing literature related to the identified problem, then build a logical rationale for a study that addresses gaps or inconsistencies in the lit- erature. The author should present the theoretical or conceptual model that informs the inquiry and provides enough background to enable the reader to appreciate the rationale of the current study. This framework elu- cidates the purpose of the current study (e.g., to eval- uate the effectiveness of a job placement program for women with breast cancer), which is then operational- ized in the research questions or hypotheses. Social scientific theories which have figured pominently in
the frameworks of recent rehabilitation investigations include Hershenson’s [10] model of work adjustment, Bandura’s [2] concept of situational self-efficacy, and Bolton and Brookings’ [5] integrated model of empow- erment.
The research questions and hypotheses. The Intro- duction section of a research article typically includes a statement of the research questions and/or hypothe- ses that served to guide the study. A more specula- tive research question tends to be used in descriptive research designs (e.g., surveys, program evaluations, empirical literature reviews) or in qualitative studies. Examples of research questions could include: “What concerns do college students with disabilities have re- garding their future career prospects?”; “What themes are evident in the psycholinguistic development of deaf women?”; and “What steps are Fortune 500 employ- ers taking to provide on-the-job accommodations for workers with disabilities?”.
The hypothesis, on the other hand, is predictive by design. Its specificity is dependent upon the theory un- derlying it or previous, relevant research, but it should include the direction of the expected results when- ever possible. Independent and dependent variables need not be operationalized in theory-based hypotheses (because this is done in the Method section), but the expected relationship among study variables must be clearly articulated. Examples of directional hypotheses could include: “Participation in a cognitive-behavioral stress inoculation program will decrease symptom on- set and magnification”; “Anxiety, depression, and low self-esteem will be collectively, positively, and signif- icantly related to work interference”; and “Rehabilita- tion counselors will rate people with severe disabili- ties as less favorable candidates for employment than similarly qualified people with mild or no disabilities”.
2.4. Method
The Method section delineates how the research questions were addressed and/or how the hypotheses were tested. It should provide the reader with sufficient information so that one could replicate the investiga- tion, and it should leave no question as to what was “done” to the participants. Because the Method section is the primary source for determining the validity of the study [4], the quality and clarity of this section are gen- erally regarded as the strongest determinants of whether an empirically-based manuscript will be accepted for publication [9,16].
260 P. Rumrill et al. / Guidelines for evaluating research articles
Although the type and order of sub-sections found in the Method section of a research article vary de- pending upon the design of the study and the author’s judgement related to the flow of text, most articles in- clude descriptions of the study’s subjects/participants, instruments/measures/variables, materials, design, and procedures.
Subjects/participants. According to Heppner et al. [8,9], the Method section should include (a) the total number of subjects and numbers assigned to groups, if applicable; (b) how subjects were selected and/or as- signed; and (c) demographic and other characteristics of the sample relevant to the study’s purpose. Some au- thors also include a description of the population from which the study sample was drawn, a description of the specific sampling procedure used (e.g., simple random, stratified, cluster; [4]), an indication of the represen- tativeness of the sample vis a vis the broader popula- tion, the circumstances under which subjects partici- pated (e.g., whether they were compensated, what risks they assumed), statistical power analyses, and response rates (if applicable).
Instruments/measures/variables. The Method sec- tion must include a detailed description of how all study variables were operationalized, measured, scored, and interpreted. All instruments or measures that were used in sampling, conducting the study, and evaluating re- sults must be specified in terms of content (e.g., num- ber of items, response sets), how measures were ad- ministered, scoring procedures, relationship to study variables, and psychometric properties (e.g., standard- ization, reliability, validity). Authors should also in- clude a rationale for selecting each instrument, that is, why that instrument was the best choice for measuring a particular construct.
Materials. Researchers should also include a de- scription of any materials that were used to carry out the investigation. Written guides for participants, in- structional manuals, media or technology, and scien- tific apparatus or equipment should be described in de- tail. Some authors include a description of the setting in which the study was executed or data were collected.
Design. One of the most important features of the Method section is a clear description of the design of the study. This is essential because the design serves as the link between (a) the research questions/hypotheses and the scientific procedures used in carrying out the study and (b) the findings of the study and how these are interpreted. Authors typically label their designs in terms of how variables were manipulated, observed, and analyzed. Thereby, the design is the unifying force
in connecting the research objectives to both the results and the knowledge claim that is made. To every extent possible, a direct reference to the hypotheses should be made when authors identify the design of a particu- lar investigation. For example, Rumrill, Roessler, and Denny [19] described their design as follows: “The re- searchers selected a three-group, posttest-only (exper- imental) design to assess the intervention’s univariate and multivariate effects on (a) self-reported attitudes (situational self-efficacy and acceptance of disability) and (b) participation in the accommodation request pro- cess.”
Procedures. The most important component of the Method section is the easiest to describe. In chrono- logical order, authors simply list every step they took in developing, administering, and evaluating the study. Beginning with the recruitment of participants, follow- ing the study through collection of the last datum, and including everything in-between – the Procedures sub- section should provide the reader with a step-by-step protocol that could serve as a guide for replicating the study. Descriptions of any interventions should be pro- vided in detail, along with summaries of the qualifi- cations of project personnel who were instrumental in executing the investigation. Procedures should also in- clude how the investigation ended, along with a state- ment of any debriefing or follow-up services provided to participants.
2.5. Results
The Results section of a research article should in- clude a complete inventory of all relevant findings ob- tained by the investigators. In articles that report quan- titative studies, results are typically presented in two parts – (a) summary, or descriptive, statistics related to participants’ performance on the measures that were taken (e.g., means, standard deviations, frequencies, percentages) and (b) statistical analyses related to the specific hypotheses of the study (e.g., analysis of vari- ance, multiple regression, factor analysis). We believe that all analyses conducted as part of the investigation should be reported in full, not only those which yielded statistically significant results. The Publication Man- ual of the American Psychological Association [1] pro- vides considerable guidance related to how statistics should be presented in the Results section, but it does not always provide adequate guidelines regarding what statistical information should be included. Heppner et al. [9] identified a pattern in recent social science lit- erature whereby researchers tend to err on the side of
262 P. Rumrill et al. / Guidelines for evaluating research articles
providing too little statistical information: “The trend has been to report less; for example, one rarely sees analysis of variance source tables anymore. More dis- turbing is the tendency not to report important informa- tion (such as size of test statistic and probability levels) when results are non-significant. This minimalist point of view puts the emphasis on statistical significance and ignores concepts such as effect size, estimation, and power.”
In recent years, the “minimalist” perspective (in terms of reporting statisitical findings) has been chal- lenged by numerous researchers and statisticians [11, 14,22]. The most serious argument against this per- spective relates to the influence that sample size has in determining the significance of any statistical test. Hayes [7], for example, pointed out that virtually any study can be made to yield statistically significant re- sults if the researcher includes enough subjects. To avoid the possibility of misleading research consumers, the latest edition of the Publication Manual [1] suggests that all authors provide estimates of practical or clinical significance along with all statistical significance tests reported in the Results section.
A quantitative Results section should be limited to the findings obtained by the researcher(s) in the cur- rent investigation. Speculation concerning what those findings mean in a larger context is reserved for the Discussion section.
The Results sections of qualitatively oriented articles display much more variety in the content and manner of presentation than is found in quantitative studies. Be- cause the researcher’s subjective interpretations help to shape the processes and outcomes of qualitative inves- tigations, results are often framed in broad, interpretive contexts. In that regard, the lines between the Results and Discussion sections are often blurred in qualitative research.
Researchers (qualitative and quantitative) commonly use tables and figures to summarize and/or graphically present their results. There is wide variability in the content and presentation of tables and figures, with the most important universal requirement being easy interpretability for the reader.
2.6. Discussion
The Discussion section serves as the researcher’s fo- rum to go beyond the current investigation and discuss the contributions of study findings to existing litera- ture, theory, and professional practices. The first part of a thoughtful Discussion is often an analysis of the
study’s results vis a vis the research questions and hy- potheses. Researchers should begin with a discussion of whether the hypotheses were upheld, posit possible explanations for those outcomes,and draw implications from the findings back to the research problem that was identified in the Introduction. If the results provide a warrant for modifying or re-testing the conceptual framework upon which the investigation was based, the Discussion section is the place to suggest a reformula- tion of the underlying theory. Researchers should also include a statement of the scientific limitations of the current study, along with specific recommendations for future research. Finally, the researcher ends the arti- cle with a cogent summary of the conclusions, in the most general sense, that can be drawn from the methods and findings of the current study. Some authors use a separate Conclusion section for this purpose.
2.7. References
The final section of a research article is always a listing of the references that were cited in the body of the text. References are listed in alphabetical order, according to authors’ last names. Most rehabilitation journals require adherence to the American Psycholog- ical Association’s [1] guidelines regarding the compo- sition of the References section.
3. A scale for critiquing research manuscript and articles
Understanding the components, organization, and composition of a research article will help make Work subscribers better informed consumers as they read em- pirically based publications. As readers digest the con- tents of research articles and apply them to their prac- tices, the “anatomy” of research reports can serve as a useful rubric for critically analyzing the quality, con- tent, and practical significance of published articles. Table 1 presents specific questions for conducting a section-by-section critique of a rehabilitation research article.
4. Conclusion
This article examined the components of a research article and provided guidelines for conducting a critical analysis of published research. Although the descrip- tions of the components of a research article provide
P. Rumrill et al. / Guidelines for evaluating research articles 263
only a skeletal summary of what should be included in a published research article, they should provide the reader enough information to both prepare manuscripts for publication and evaluate the empirical research that appears in Work and other rehabilitation journals.
References
[1] Washington, D.C., American Psychological Association, Pub- lication manual of the American Psychological Association, (Fourth Edition), 1994.
[2] Bandura, A., Social foundations of thought and action: A social cognitive theory, Englewood Cliffs, NJ: Prentice-Hall, 1986.
[3] Bellini, J., Bolton, B. and Neath, J., Rehabilitation counselors assessments of applicants functional limitations as predictors of rehabilitation services provided, Rehabilitation Counseling Bulletin 41(4) (1998), 242–258.
[4] Bellini, J. and Rumrill, P., Research in rehabilitation counsel- ing: A guide to design, methodology, and utilization, Spring- field, IL: Charles C. Thomas Publishers, 1999.
[5] Bolton, B. and Brookings, J., Development of a multifaceted definition of empowerment, Rehabilitation Counseling Bul- letin 39(4) (1996), 256–264.
[6] Diksa, E. and Rogers, E., Employer concerns about hiring per- sons with psychiatric disability: Results of the employer atti- tude questionnaire, Rehabilitation Counseling Bulletin 40(1) (1996), 31–44.
[7] Hayes, W., Statistics for psychologists, New York: Holt, Rine- hart, and Winston, 1981.
[8] Heppner, P., Kivlighan, D. and Wampold, B., Research design in counseling, Pacific Grove, CA: Brooks/Cole, 1992.
[9] Heppner, P., Kivlighan, D. and Wampold, B., Research design in counseling, (2nd Edition), Pacific Grove, CA: Brooks/Cole, 1999.
[10] Hershenson, D., A systems reformulation of a developmental model of work adjustment, Rehabilitation Counseling Bulletin
40(1) (1996), 2–10. [11] Hunter, J., Needed: A ban on the significance test, Psycholog-
ical Science 8 (1997), 3–7. [12] Kazdin, A., Research design in clinical psychology, (2nd Edi-
tion), Needham Heights, MA: Allyn and Bacon, 1998. [13] Krause, J. and Anson, C., Self-perceived reasons for unem-
ployment cited by persons with spinal cord injury: Relation- ship to gender, race, age, and level of injury, Rehabilitation Counseling Bulletin 39(3) (1996), 217–227.
[14] McClure, P., Determining the significance of significance: P- values, effect size, and clinical judgement, Journal of Hand Therapy 12 (1999), 40–41.
[15] McMillan, J. and Schumacher, S., Research in education: A conceptual introduction, (Fourth Edition), New York: Long- man, 1997.
[16] Munley, P., Sharkin, B. and Gelso, C., Reviewer ratings and agreement on manuscripts reviewed for the Journal of Counseling Psychology, Journal of Counseling Psychology 35 (1988), 198–202.
[17] Orwell, G., Politics and the English language, in: A collection of essays, G. Orwell ed., San Diego: Harcourt, Brace, and Jovanovich, 1946, pp. 156–171.
[18] Ravesloot, C. and Seekins, T., Vocational rehabilitation coun- selors’ attitudes toward self-employment outcomes, Rehabili- tation Counseling Bulletin 39(3) (1996), 189–201.
[19] Rumrill, P., Roessler, R. and Denny, G., Increasing confidence in the accommodation request process among persons with multiple sclerosis: A career maintenance self-efficacy inter- vention, Journal of Job Placement 13(1) (1997), 5–9.
[20] Rumrill, P., Roessler, R. and Koch, L., Surveying the employ- ment concerns of people with multiple sclerosis: A participa- tory action research approach, Journal of Vocational Rehabil- itation 12(2) (1999), 75–82.
[21] Schaller, J. and Parker, R., Effect of graduate research in- struction on perceived research anxiety, research utility, and confidence in research skills, Rehabilitation Education 11(4) (1997), 273–287.
[22] Thompson, B., AERA editorial policies regarding statistical significance testing: Three suggested reforms, Educational Researcher 25(2) (1996), 26–30.
P. Rumrill et al. / Guidelines for evaluating research articles 261
Table 1 A scale for critiquing research articles
Instructions: Answer the following questions regarding the article, “ ”. Use examples from the article to support your analyses.
A. Title
1. Does the title describe the study? 2. Do the key words of the title serve as key elements of the article? 3. Is the title concise, i.e., free of distracting or extraneous phrases?
B. Abstract
4. Does the abstract summarize the study’s purpose, methods, and findings? 5. Does the abstract reveal the independent and dependent variables under study? 6. Are there any major premises or findings presented in the article that are not mentioned in the abstract? 7. Does the abstract provide you with sufficient information to determine whether you would be interested in reading
the entire article?
C. Introduction
8. Is the research problem clearly identified? 9. Is the problem significant enough to warrant the study that was conducted?
10. Do the authors present a theoretical rationale for the study? 11. Is the conceptual framework of the study appropriate in light of the research problem? 12. Do the author’s hypotheses and/or research questions seem logical in light of the conceptual framework and research
problem? 13. Are hypotheses and research questions clearly stated? Are they directional? 14. Overall, does the literature review lead logically into the Method section?
D. Method
15. Is the sample clearly described, in terms of size, relevant characteristics, selection and assignment procedures, and whether any inducements were used to solicit subjects?
16. Do the instruments described seem appropriate as meausres of the variables under study? 17. Have the authors included sufficient information about the psychometric properties (e.g., reliability and validity) of
the instruments? 18. Are the materials used in conducting the study or in collecting data clearly described? 19. Are the study’s scientific procedures thoroughly described in chronological order? 20. Is the design of the study identified (or made evident)? 21. Do the design and procedures seem appropriate in light of the research problem, conceptual framework, and research
questions/hypotheses? 22. Overall, does the method section provide sufficient information to replicate the study?
E. Results
23. Is the Results section clearly written and well organized? 24. Are data coding and analysis appropriate in light of the study’s design and hypotheses? 25. Are salient results connected directly to hypotheses? 26. Are tables and figures clearly labeled? Well organized? Necessary (non-duplicative of text)?
F. Discussion and Conclusion
27. Are the limitations of the study delineated? 28. Are findings discussed in terms of the research problem, conceptual framework, and hypotheses? 29. Are implications for future research and/or rehabilitation counseling practice identified? 30. Are the author’s general conclusions warranted in light of the results?
G. References
31. Is the reference list sufficiently current? 32. Do works cited reflect the breadth of existing literature regarding the topic of the study? 33. Are bibliographic citations used appropriately in the text?
H. General Impressions
34. Is the article well written and organized? 35. Does the study address an important problem in the lives of people with disabilities? 36. What are the most important things you learned from this article? 37. What do you see as the most compelling strengths of this study? 38. How might this study be improved?
Copyright of Work is the property of IOS Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s express written permission. However, users may print, download, or email articles for individual use.
