Reservations about survey bias
at
Scenario
Sam, a therapist, wants to get feedback from his clients about how he is helping them. He considers several approaches to collecting the information. He rejects the idea of mailing people surveys because of the expense. He thinks about giving surveys to clients at the end of sessions and asking them to complete them, but he knows some of his clients do not read well. He decides the best approach will be to ask clients a couple of questions at the end of each session. Sam then considers what questions he should ask and finally decides on asking a series of yes-no questions:
1) Today’s session was helpful to me.
2) I feel happier after today’s session.
3) I feel stronger after today’s session.
4) I am glad I am in counseling.
5) I have gained from counseling.
Sam is surprised to find that his coworkers have some reservations about survey bias. Whose concerns do you think are most valid?
Ava worries that by asking the questions himself, Sam may inadvertently pressure his clients to respond in ways that reflect well on Sam.
Erik wonders if, by having all of the questions worded in positive ways, some people will have a tendency to just say yes to everything, especially when the person asking in in a position of authority.
Pam thinks some people might answer in ways that would make them look good, for example, by saying they feel strong and happy when, in fact, they feel vulnerable and discouraged.
Which concerns about measurement do you agree with the most? Provide a brief explanation and rationale (5-6 sentences).
Sample Solution
agree with Erik’s concern that by having all of the questions worded in positive ways, some people will have a tendency to just say yes to everything. This is because the questions are framed in such a way that implies that any other answer besides “yes” would be wrong or undesirable. Even if Sam is not deliberately trying to pressure his clients into giving particular answers, this type of framing can still have an effect on how they answer.
Furthermore, this type of question also does not allow for nuance or more detailed responses from participants. For example, one might feel better after a session but not necessarily “happier” as the question is phrased; someone might gain insight from counseling but perhaps not feel “stronger” as suggested by another question. By only allowing for either/or answers when it comes to these types of subjective experiences, Sam may be missing out on important details about how his clients are responding and feeling about their progress in counseling.
Ultimately, it would be better for Sam to ask open-ended questions that allow his clients more freedom to accurately express their feelings and experiences rather than yes/no questions which could lead them into answering in ways they think will please him most.
