Evidence level and quality rating
Rich narrative documents are used for uncovering themes; describes a problem or condition from the point of view of those experiencing it. Common methods are focus groups, individual interviews (unstructured or semi structured), and participation/observations. Sample sizes are small and are determined when data saturation is achieved. Data saturation is reached when the researcher identifies that no new themes emerge and redundancy is occurring. Synthesis is used in data analysis. Often a starting point for studies when little research exists; may use results to design empirical studies. The researcher describes, analyzes, and interprets reports, descriptions, and observations from participants.
Go to Section II: QuaLitative
Mixed methods (results reported both numerically and narratively)
Both quaNtitative and quaLitative methods are used in the study design. Using both approaches, in combination, provides a better understanding of research problems than using either approach alone. Sample sizes vary based on methods used. Data collection involves collecting and analyzing both quaNtitative and quaLitative data in a single study or series of studies. Interpretation is continual and can influence stages in the research process.
Go to Section III: Mixed Methods
Section I: QuaNtitative
Level of Evidence (Study Design)
Is this a report of a single research study?
Yes No
Go to B
1. Was there manipulation of an independent variable? Yes No
2. Was there a control group? Yes No
3. Were study participants randomly assigned to the intervention and control groups? Yes No
If Yes to questions 1, 2, and 3, this is a randomized controlled trial (RCT) or experimental study. LEVEL I
If Yes to questions 1 and 2 and No to question 3 or Yes to question 1 and No to questions 2 and 3, this is quasi-experimental.
(Some degree of investigator control, some manipulation of an independent variable, lacks random assignment to groups, and may have a control group). LEVEL II
If No to questions 1, 2, and 3, this is nonexperimental.
(No manipulation of independent variable; can be descriptive, comparative, or correlational; often uses secondary data). LEVEL III
Study Findings That Help Answer the EBP Question
Skip to the Appraisal of Quantitative Research Studies section
Section I: Quantitative (continued)
Is this a summary of multiple sources of research evidence? Yes
Continue No
Use Appendix F
1. Does it employ a comprehensive search strategy and rigorous appraisal method?
If this study includes research, non research, and experiential evidence, it is an integrative review (see Appendix F). Yes
Continue No
Use Appendix F
2. For systematic reviews and systematic reviews with meta-analysis
(see descriptions below):
a. Are all studies included RCTs? LEVEL I
b. Are the studies a combination of RCTs and quasi-experimental, or quasi-experimental only? LEVEL II
c. Are the studies a combination of RCTs, quasi-experimental, and nonexperimental, or non- experimental only? LEVEL III
A systematic review employs a search strategy and a rigorous appraisal method, but does not generate an effect size.
A meta-analysis, or systematic review with meta-analysis, combines and analyzes results from studies to generate a new statistic: the effect size.
Study Findings That Help Answer the EBP Question
Skip to the Appraisal of Systematic Review (With or Without a Meta-Analysis) section
Appraisal of Quantitative Research Studies
Does the researcher identify what is known and not known about the problem and how the study will address any gaps in knowledge? Yes No
Was the purpose of the study clearly presented? Yes No
Was the literature review current (most sources within the past five years or a seminal study)? Yes No
Was sample size sufficient based on study design and rationale? Yes No
If there is a control group:
• Were the characteristics and/or demographics similar in both the control and intervention groups? Yes No
N/A
• If multiple settings were used, were the settings similar? Yes No N/A
• Were all groups equally treated except for the intervention group(s)? Yes No N/A
Are data collection methods described clearly? Yes No
Were the instruments reliable (Cronbach’s [alpha] > 0.70)? Yes No N/A
Was instrument validity discussed? Yes No N/A
If surveys or questionnaires were used, was the response
rate > 25%? Yes No N/A
Were the results presented clearly? Yes No
If tables were presented, was the narrative consistent with the table content? Yes No N/A
Were study limitations identified and addressed? Yes No
Were conclusions based on results? Yes No
Complete the Quality Rating for Quantitative Studies section
Appraisal of Systematic Review (With or Without Meta-Analysis)
Were the variables of interest clearly identified? Yes No
Was the search comprehensive and reproducible?
• Key search terms stated Yes No
• Multiple databases searched and identified Yes No
• Inclusion and exclusion criteria stated Yes No
Was there a flow diagram that included the number of studies eliminated at each level of review? Yes No
Were details of included studies presented (design, sample, methods, results, outcomes, strengths, and limitations)? Yes No
Were methods for appraising the strength of evidence (level and quality) described? Yes No
Were conclusions based on results? Yes No
• Results were interpreted Yes No
• Conclusions flowed logically from the interpretation and systematic review question Yes No
Did the systematic review include a section addressing limitations and how they were addressed? Yes No
Complete the Quality Rating for Quantitative Studies section (below)
Quality Rating for Quantitative Studies
Circle the appropriate quality rating below:
A High quality: Consistent, generalizable results; sufficient sample size for the study design; adequate control; definitive conclusions; consistent recommendations based on comprehensive literature review that includes thorough reference to scientific evidence.
B Good quality: Reasonably consistent results; sufficient sample size for the study design; some control, and fairly definitive conclusions; reasonably consistent recommendations based on fairly comprehensive literature review that includes some reference to scientific evidence.
C Low quality or major flaws: Little evidence with inconsistent results; insufficient sample size for the study design; conclusions cannot be drawn.
Section II: QuaLitative
Level of Evidence (Study Design)
Is this a report of a single research study?
Yes
this is
Level III
No
go to II B
Study Findings That Help Answer the EBP Question
Complete the Appraisal of Single Qualitative Research Study section (below)
Appraisal of a Single Qualitative Research Study
Was there a clearly identifiable and articulated:
• Purpose? ❑ Yes ❑ No
• Research question? ❑ Yes ❑ No
• Justification for method(s) used? ❑ Yes ❑ No
• Phenomenon that is the focus of the research? ❑ Yes ❑ No
Were study sample participants representative? ❑ Yes ❑ No
Did they have knowledge of or experience with the research area? ❑ Yes ❑ No
Were participant characteristics described? ❑ Yes ❑ No
Was sampling adequate, as evidenced by achieving saturation of data? ❑ Yes ❑ No
Data analysis:
• Was a verification process used in every step by checking and confirming with participants the trustworthiness of analysis and interpretation?
❑ Yes
❑ No
• Was there a description of how data were analyzed (i.e., method), by computer or manually? ❑ Yes ❑ No
Do findings support the narrative data (quotes)? ❑ Yes ❑ No
Do findings flow from research question to data collected to analysis undertaken? ❑ Yes ❑ No
Are conclusions clearly explained? ❑ Yes ❑ No
Skip to the Quality Rating for Qualitative Studies section
For summaries of multiple qualitative research studies (meta-synthesis), was a comprehensive search strategy and rigorous appraisal method used? Yes
Level III No
go to Appendix F
Study Findings That Help Answer the EBP Question
Complete the Appraisal of Meta-Synthesis Studies section (below)
Appraisal of Meta-Synthesis Studies
Were the search strategy and criteria for selecting primary studies clearly defined? ❑ Yes ❑ No
Were findings appropriate and convincing? ❑ Yes ❑ No
Was a description of methods used to:
• Compare findings from each study? ❑ Yes ❑ No
• Interpret data? ❑ Yes ❑ No
Did synthesis reflect: ❑ Yes ❑ No
• New insights? ❑ Yes ❑ No
• Discovery of essential features of phenomena? ❑ Yes ❑ No
• A fuller understanding of the phenomena? ❑ Yes ❑ No
Was sufficient data presented to support the interpretations? ❑ Yes ❑ No
Complete the Quality Rating for Qualitative Studies section (below)
Quality Rating for Qualitative Studies
Circle the appropriate quality rating below:
No commonly agreed-on principles exist for judging the quality of qualitative studies. It is a subjective process based on the extent to which study data contributes to synthesis and how much information is known about the researchers’ efforts to meet the appraisal criteria.
For meta-synthesis, there is preliminary agreement that quality assessments should be made before synthesis to screen out poor-quality studies1.
A/B High/Good quality is used for single studies and meta-syntheses2.
The report discusses efforts to enhance or evaluate the quality of the data and the overall inquiry in sufficient detail; and it describes the specific techniques used to enhance the quality of the inquiry.
Evidence of some or all of the following is found in the report:
• Transparency: Describes how information was documented to justify decisions, how data were reviewed by others, and how themes and categories were formulated.
• Diligence: Reads and rereads data to check interpretations; seeks opportunity to find multiple sources to corroborate evidence.
• Verification: The process of checking, confirming, and ensuring methodologic coherence.
• Self-reflection and self-scrutiny: Being continuously aware of how a researcher’s experiences, background, or prejudices might shape and bias analysis and interpretations.
• Participant-driven inquiry: Participants shape the scope and breadth of questions; analysis and interpretation give voice to those who participated.
• Insightful interpretation: Data and knowledge are linked in meaningful ways to relevant literature.
C Lower-quality studies contribute little to the overall review of findings and have few, if any, of the features listed for High/Good quality.
1 https://www.york.ac.uk/crd/SysRev/!SSL!/WebHelp/6_4_ASSESSMENT_OF_QUALITATIVE_RESEARCH.htm
2 Adapted from Polit & Beck (2017).
