Guidelines for evaluating research articles
Speaking of research
Guidelines for evaluating research articles
Phillip Rumrill∗, Shawn Fitzgerald and Megen Ware Kent State University, Department of Educational Foundations and Special Services Center for Disability Studies, 405 White Hall, P.O. Box 5190, Kent, OH 44242-0001, USA
The article describes the components and composition of journal articles that report empirical research findings in the field of rehabilitation. The authors delineate technical writing strategies and discuss the contents of research manuscripts, including the Title, Abstract, Introduction, Method, Results, Discussion, and References. The article concludes with a scale that practitioners, manuscript reviewers, educators, and students can use in critically analyzing the content and scien- tific merits of published rehabilitation research.
Keywords: Evaluation, research articles, guidelines for cri- tique
1. Introduction
The purpose of this article is to examine the com- ponents of a research article and provide guidelines for conducting critical analyses of published works. Distilled from the American Psychological Associa- tion’s [1] Publication Manual and related descriptions in several research design texts [4,8,9,12,15], descrip- tions of how authors in rehabilitation and disability studies address each section of a research article are featured. The article concludes with a framework that rehabilitation educators, graduate students, practition- ers, and other Work readers can use in critiquing re- search articles on the basis of their scientific merits and practical utility.
∗Corresponding author: Tel.: +1 330 672 2294; Fax: +1 330 672 2512; E-mail: prumrill@educ.kent.edu.
2. Anatomy of a research article
For nearly 50 years, the American Psychological As- sociation has presented guidelines for authors to follow in composing manuscripts for publication in profes- sional journals [1]. Most journals in disability studies and rehabilitation adhere to those style and formatting guidelines. In the paragraphs to follow, descriptions of each section of a standard research article are pre- sented: Title, Abstract, Introduction, Method, Results, Discussion, and References.
2.1. Title
As with other kinds of literature, the title of a scien- tific or scholarly journal article is a very important fea- ture. At the risk of contravening the age-old adage “You can’t judge a book by its cover,” Bellini and Rumrill [4] speculated that most articles in rehabilitation journals are either read or not read based upon the prospective reader’s perusal of the title. Therefore, developing a clear, concise title that conveys the article’s key con- cepts, hypotheses, methods, and variables under study is critical for researchers wishing to share their findings with a large, professional audience. A standard-length title for a journal article in the social sciences is 12–15 words, including a sub-title if appropriate. Because so- cial science and medical indexing systems rely heavily on titles in their codification schemes to track and cat- egorize journal articles by topic, providing a title that clearly delineates a general research domain or topic area is of utmost importance. If the title is vague or ambiguous, chances are that the prospective reader will not continue to read through the document to establish where it might fit in terms of a specific research domain or topic area. Examples of clearly descriptive titles that can be found in the contemporary rehabilitation literature include:
“Rehabilitation Counselors’ Assessments of Appli- cants’ Functional Limitations as Predictors of Rehabil- itation Services Provided” [3].
Work 14 (2000) 257–263 ISSN 1051-9815 / $8.00 2000, IOS Press. All rights reserved
258 P. Rumrill et al. / Guidelines for evaluating research articles
“Employer Concerns About Hiring Persons with Psychiatric Disabilities: Results of the Employer Atti- tude Questionnaire” [6].
“Self-Perceived Reasons for Unemployment Cited by Persons with Spinal Cord Injury: Relationship to Gender, Race, Age, and Level of Injury” [13].
“Vocational Rehabilitation Counselors’ Attitudes Toward Self-Employment Outcomes” [18].
“Surveying the Employment Concerns of People with Multiple Sclerosis: A Participatory Action Re- search Approach” [20].
“Effect of Graduate Research Instruction on Per- ceived Research Anxiety, Research Utility, and Confi- dence in Research Skills” [21].
Before we move into descriptions of the content sec- tions of a research article, we want to briefly address the concept of technical writing as it applies to the com- position of academic manuscripts. Journals adhering to the American Psychological Association’s [1] pub- lication guidelines favor manuscripts that are written in direct, uncomplicated sentences. Editors prefer that text be written in the “active voice”; whenever possible, sentences should begin with their subjects and follow with verbs and objects (e.g., “The researcher conducted an experiment” rather than “An experiment was con- ducted by the researcher”). Technical writing is marked by the “less is more” maxim; extraneous phrases and clauses that add words to the sentence without enhanc- ing the overall statement should be avoided (e.g., “In order to. . . ”, “For purposes of. . . ”, “As far as. . . is concerned. . . ”). Another element of sound technical writing is the sparing use of adverbs (e.g., very, some- what, strikingly) and adjectives that do not serve to fur- ther define or specify the terms that they are modifying (e.g., interesting, important, good, noteworthy).
In addition to the American Psychological Associa- tion’s guidelines for technical writing, authors should consider these six criteria for effective composition provided by George Orwell (1946) in Politics and the English Language:
1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print.
2. Never use a long word where a short one will do. 3. If it is possible to cut a word out, always cut it
out. 4. Never use the passive (voice) where you can use
the active. 5. Never use a foreign phrase, a scientific word, or
jargon word if you can think of an everyday En- glish equivalent.
6. Break any of these rules sooner than say anything outright barbarous (p. 170).
Organization is also key in preparing an effectively composed journal manuscript, with multi-level head- ings serving to guide the flow of text and keep the reader on track. For authoritative information regard- ing the style and formatting guidelines for submitting manuscripts to most journals in social science fields, readers should consult the American Psychological As- sociation’s [1] Publication Manual. For information concerning the style and formatting requirements of Work and other journals published by IOS Press, see the Guidelines for Authors section included in the be- ginning of this edition.
2.2. Abstract
Next to the title, the abstract is the most widely read section of a journal article. In an empirical article, the abstract should be a succinct, 100–150 word summary of the investigation’s key features, including purpose, objectives, research questions/hypotheses, sample, sci- entific procedures, independent and dependent vari- ables, and salient results. Results of the study should be summarized in full in the abstract; authors should describe both significant and non-significant findings, not only those which upheld their hypotheses or expec- tations. The abstract serves as an advance organizer for the article, and it should include every important premise, method, and result of the investigation. Like the Preface that commonly orients readers to full-length textbooks, the abstract provides a complete, albeit sum- mary, preview of the article. Some journals, includ- ing Work and the Journal of Vocational Rehabilitation, ask authors to list key descriptors on the abstract page, which are then used for purposes of indexing. In most cases, the title is what determines whether a reader will read the abstract; the abstract determines whether the reader will read the body of the article.
2.3. Introduction
Immediately following the abstract, the introductory section of the article sets the stage for the study upon which the article was based. It orients the reader to the problem or issue being addressed, develops the logic and rationale for conducting the investigation, and al- most always expresses the empirical hypotheses or re- search questions. Heppner et al. [9] suggested that the introduction should answer questions such as why the topic is an important one to study, what previous
P. Rumrill et al. / Guidelines for evaluating research articles 259
work bears on the topic, how existing work logically connects to the author’s research questions and/or hy- potheses, how the question will be researched, and what predictions can be made.
To answer these questions, authors typically address three major elements in the introductory section of an article: (1) The Research Problem, (2) The Framework for the Study, and (3) The Research Questions and Hy- potheses [8,15]. We will describe each of these intro- ductory elements in linear fashion, but we do not mean to imply an order in terms of how they should be ad- dressed. Many (if not most) authors blend these con- siderations to fit the flow and logic of their respective manuscripts.
The research problem. Usually in the very first sen- tences of an empirical journal article, the author draws the reader’s attention to the scope, impact, and current status of the problem or issue being investigated. This orientation is most effectively achieved by applying the broadest-possible perspective to the concern. A study of success rates among participants in a stress inocula- tion program for people with diabetes mellitus might be introduced by citing national statistics concerning the incidence and prevalence of this very common disease. An article describing the effects of a model job place- ment program for women with breast cancer might be- gin with a review of existing literature concerning em- ployment and breast cancer, with a particular focus on the difficulties that women have in re-entering the la- bor force following diagnosis and treatment. Authors reporting a longitudinal study of the post- school em- ployment outcomes of secondary students with devel- opmental disabilities would likely introduce their arti- cle with a review of the disappointing adult outcomes which that population has experienced since the incep- tion of formalized transition services in the mid–1980s.
The framework for the study. The specific theoret- ical and empirical framework for the particular inves- tigation is another important part of the Introduction. Authors summarize existing literature related to the identified problem, then build a logical rationale for a study that addresses gaps or inconsistencies in the lit- erature. The author should present the theoretical or conceptual model that informs the inquiry and provides enough background to enable the reader to appreciate the rationale of the current study. This framework elu- cidates the purpose of the current study (e.g., to eval- uate the effectiveness of a job placement program for women with breast cancer), which is then operational- ized in the research questions or hypotheses. Social scientific theories which have figured pominently in
the frameworks of recent rehabilitation investigations include Hershenson’s [10] model of work adjustment, Bandura’s [2] concept of situational self-efficacy, and Bolton and Brookings’ [5] integrated model of empow- erment.
The research questions and hypotheses. The Intro- duction section of a research article typically includes a statement of the research questions and/or hypothe- ses that served to guide the study. A more specula- tive research question tends to be used in descriptive research designs (e.g., surveys, program evaluations, empirical literature reviews) or in qualitative studies. Examples of research questions could include: “What concerns do college students with disabilities have re- garding their future career prospects?”; “What themes are evident in the psycholinguistic development of deaf women?”; and “What steps are Fortune 500 employ- ers taking to provide on-the-job accommodations for workers with disabilities?”.
The hypothesis, on the other hand, is predictive by design. Its specificity is dependent upon the theory un- derlying it or previous, relevant research, but it should include the direction of the expected results when- ever possible. Independent and dependent variables need not be operationalized in theory-based hypotheses (because this is done in the Method section), but the expected relationship among study variables must be clearly articulated. Examples of directional hypotheses could include: “Participation in a cognitive-behavioral stress inoculation program will decrease symptom on- set and magnification”; “Anxiety, depression, and low self-esteem will be collectively, positively, and signif- icantly related to work interference”; and “Rehabilita- tion counselors will rate people with severe disabili- ties as less favorable candidates for employment than similarly qualified people with mild or no disabilities”.
2.4. Method
The Method section delineates how the research questions were addressed and/or how the hypotheses were tested. It should provide the reader with sufficient information so that one could replicate the investiga- tion, and it should leave no question as to what was “done” to the participants. Because the Method section is the primary source for determining the validity of the study [4], the quality and clarity of this section are gen- erally regarded as the strongest determinants of whether an empirically-based manuscript will be accepted for publication [9,16].
260 P. Rumrill et al. / Guidelines for evaluating research articles
Although the type and order of sub-sections found in the Method section of a research article vary de- pending upon the design of the study and the author’s judgement related to the flow of text, most articles in- clude descriptions of the study’s subjects/participants, instruments/measures/variables, materials, design, and procedures.
Subjects/participants. According to Heppner et al. [8,9], the Method section should include (a) the total number of subjects and numbers assigned to groups, if applicable; (b) how subjects were selected and/or as- signed; and (c) demographic and other characteristics of the sample relevant to the study’s purpose. Some au- thors also include a description of the population from which the study sample was drawn, a description of the specific sampling procedure used (e.g., simple random, stratified, cluster; [4]), an indication of the represen- tativeness of the sample vis a vis the broader popula- tion, the circumstances under which subjects partici- pated (e.g., whether they were compensated, what risks they assumed), statistical power analyses, and response rates (if applicable).
Instruments/measures/variables. The Method sec- tion must include a detailed description of how all study variables were operationalized, measured, scored, and interpreted. All instruments or measures that were used in sampling, conducting the study, and evaluating re- sults must be specified in terms of content (e.g., num- ber of items, response sets), how measures were ad- ministered, scoring procedures, relationship to study variables, and psychometric properties (e.g., standard- ization, reliability, validity). Authors should also in- clude a rationale for selecting each instrument, that is, why that instrument was the best choice for measuring a particular construct.
Materials. Researchers should also include a de- scription of any materials that were used to carry out the investigation. Written guides for participants, in- structional manuals, media or technology, and scien- tific apparatus or equipment should be described in de- tail. Some authors include a description of the setting in which the study was executed or data were collected.
Design. One of the most important features of the Method section is a clear description of the design of the study. This is essential because the design serves as the link between (a) the research questions/hypotheses and the scientific procedures used in carrying out the study and (b) the findings of the study and how these are interpreted. Authors typically label their designs in terms of how variables were manipulated, observed, and analyzed. Thereby, the design is the unifying force
in connecting the research objectives to both the results and the knowledge claim that is made. To every extent possible, a direct reference to the hypotheses should be made when authors identify the design of a particu- lar investigation. For example, Rumrill, Roessler, and Denny [19] described their design as follows: “The re- searchers selected a three-group, posttest-only (exper- imental) design to assess the intervention’s univariate and multivariate effects on (a) self-reported attitudes (situational self-efficacy and acceptance of disability) and (b) participation in the accommodation request pro- cess.”
