Appraising a User Study

Key questions to help you make sense of a user study

General comments

Three broad issues need to be considered when appraising a user study:

A/ To what extent is the study a close representation of the "truth" (validity)?

B/ Are the results credible and repeatable (reliability)?

C/ Will the results help me in my own information practice (applicability)?

A/ Is the study a close representation of the truth?

Screening Questions

1. Does the study address a clearly focused issue?
HINT: An issue can be "focused" in terms of : · The population (user group) studied · The intervention (service or facility) provided. Is it clearly defined and is it a useful definition? · The outcomes (quantifiable or qualitative) measured		Yes	Can't Tell	No
2. Is a good case made for the approach that the authors have taken?
HINT: Do the authors state how they identified the problem and provide a justification for why they have chosen to examine it? Do they state in what way their chosen methodology is appropriate to the question? Consider, too, whether the study: · Refers to previous work that has looked at the same user group · Refers to previous work that has looked at the same service or facility · Utilises a methodology or data collection instruments that have been used in previous user studies		Yes	Can't Tell	No
3. Is there a direct comparison that provides an additional frame of reference?
HINT: This may either be external : Or internal :	e.g. contrast with, or similarity to, other studies e.g. contrast with, or similarity to, other user groups within the study e.g. contrast with, or similarity to, the same group at different geographical locations or at a different time period.	Yes	Can't Tell	No
Is it worth continuing?

Detailed Questions
4. Were those involved in collection of data also involved in delivering a service to the user group[1]?
HINT : It may not always be possible to separate researchers from service deliverers but consider has the service deliverers' perspective been acknowledged explicitly and to what extent have the questions in the user study been generated elsewhere (e.g. a previously trialled or validated instrument or from a focus group)	Yes	Can't Tell	No
5. Were the methods used in selecting the users appropriate[2] and clearly described?
HINT: Type of sample: Is it a convenience sample? Were participants self-selecting? Were key informants identified? Is it a randomly selected sample? Is it a comprehensive census or survey? Size of sample: Has a sample size calculation been undertaken? Representativeness of sample: Was the planned sample of users representative of all users (actual and eligible) who might be included in the study? Do the demographics of the sample (e.g. age, sex, staff grade, location) accurately reflect the demographics of the total population? Are any interests or motivations behind participation clearly identified? Are non-users included in the sampling frame?	Yes	Can't Tell	No

B/ Are the results credible and repeatable?
6. Was the data collection instrument/method reliable?
HINT: If there is a questionnaire, survey form or interview schedule do the authors include it in their report? Do they refer to where a full copy might be found? Has the data collection instrument been used before? Have the authors adapted an existing questionnaire and, if so, have they used it appropriately?	Yes	Can't Tell	No
7. What was the response rate and how representative were respondents of the population under study?
HINT: Consider not only the actual percentage of responses but also whether any specific subgroups were either over-represented or under-represented. Are reasons for non-response discussed? Have non-users been included in the analysis of responses?[3]
8. Are the results complete and have they been analysed in an easily interpretable way?
HINT: Consider choices involved in analysis and in presentation. Have all variables identified earlier in the study been analysed[4]? If not, why not?	Yes	Can't Tell	No

9. Are any limitations in the methodology (that might have influenced results) identified and discussed[5]?
HINT: Consider whether the authors give a clear picture of how the study might best be done. Would it be possible for you to replicate the study from the information given? Is there enough detail of any data collection instrument for you to reproduce it?	YES	Can't Tell	No
10. Are the conclusions based on an honest and objective interpretation of the results?
HINT: Do the authors base their conclusions on findings from their experimental data? Can you be sure that they are not presenting their data merely to substantiate some preconceived ideas?	YES	Can't Tell	No

C. Will the results help me in my own information practice
11. Can the results be applied to your local population?
HINT: The burden of proof is on you to identify any ways in which your local population might differ from that in the study.	Yes	Can't Tell	No
If NO, are you able to use the same methodology (if valid) with your local population?
	Yes	Can't Tell	No
12. What are the implications of the study for your practice?
In terms of current deployment of services?:
In terms of cost?:
In terms of the expectations or attitudes of your users?:
13. What additional information do you need to obtain locally to assist you in responding to the findings of this study?

---

[1] Cp. Greenhalgh. What was the researcher's perspective and has this been taken into account?

[2] Cp. Greenhalgh. How were the setting and the subjects selected?

[3] Cp. Intention to treat analysis.

[4] Cp. Accounting for withdrawals and drop outs. Data on how the user group is similar to other groups is also useful in planning services.

[5] Cp. Greenhalgh. Was the qualitative approach appropriate? What methods did the researcher use for collecting data - and are they discussed in enough detail?