What different ways can we assess learners?
This fascinating study assesses a new tool - Script Concordance Test (SCT).
References
Humbert AJ, Besinger B, Miech EJ. Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a Script Concordance Test in an emergency medicine clerkship and residency. Acad Emerg Med. 2011;18(6):627-34. .
Assessing clinical reasoning skills in scenarios of uncertainty: Convergent validity for a Script Concordance Test in an Emergency Medicine clerkship and residency
What are Scripts?
Scripts are organized networks of knowledge. Integrating them improves decision making. Using scripts, experts see associations while novices struggle with causality. In ambiguous cases, experts process multiple scripts with influx of new information.
What is the format of a Script Concordance Test?
The learners are presented with a short clinical vignette with a series of proposed diagnoses and/or plans. The learners are then presented one new piece of information and asked what effect this information has on the proposed diagnoses and/or plans. They score their decisions on a Likert scale, ranging from -2 to +2.
What did this paper study?
An observational study comparing the scores of 4th year med students (n=314) , residents (n=40) and faculty (n=12) on a SCT with scenarios in Emergency Medicine. The student score was compared to USMLE Step 2 score, and resident score with their ABEM in-training exam score.
What were the results?
The SCT scores were able to differentiate students from residents and residents from faculty.
- Students vs residents: 60% +/- 6.2 vs 70% +/- 5.4
- Residents vs faculty: 70% +/- 5.4 vs 79% +/- 2.9
There was a significant correlation between resident score and ABEM exam score and a modest correlation between student score and USMLE Step 2 score.
What were the limitations?
It is a single centre study. The internal reliability of the assessment tool was suboptimal.
What were the conclusions?
The SCT may be useful in assessing clinical reasoning in uncertain scenarios.
What do I think?
I enjoy the examples given in the paper. While it is different and likely will take some getting used to, it could be a useful assessment tool.
References
Humbert AJ, Besinger B, Miech EJ. Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a Script Concordance Test in an emergency medicine clerkship and residency. Acad Emerg Med. 2011;18(6):627-34. .
Seems like it adds some subjectivity into the objective portion of someone's eval. Plus, the scores they reported are kinda funky, don't you think? I mean, barely any difference between residents and student? And Faculty only with 80% correct? Seems like they asked the wrong questions, which I would think would have an effect on the external validity of the study.
ReplyDelete@BAF: Yes, you make excellent and insightful points. Script concordance tests (SCT) are slowly creeping into the medical education literature for EM and this is a starting point. Next the authors can tweak the questions for improved validity. This article seems to be more a proof-of-concept rather than a proof-of-external-validity paper. Assessing clinical reasoning is an ambiguous science and tools that measure them are a bit subjective, I find. I'm not sure whether there is a better alternative. Ideas?
ReplyDeleteI think that this format for testing skill in making a diagnosis or therapeutic decision is certainly worthy for continued investigation. The question will always be what is the "correct" decision and how can the "skill" be established. Obviously, it is being judged by comparing with "seasoned" clinicians. And perhaps that is the best that can be done. Whether clinicians who are "seasoned" represent the best flavor (the most likely to make the best decisions) that, I think, can be open to question. As one goes on within the clinical medicine career, judgment may be altered by the various practice pressures. On the other hand, if one looks at these tests as measures of "standards of practice" (how the average clinician does his or her job and is expected of others)then certainly the "seasoned" clinician would be a worthwhile standard. ..Maurice.
ReplyDelete@Maurice: Well stated. "Seasoned" clinicians are likely the best measure of "standards of practice" given the art of medicine, intermixed with the science of medicine. Thanks for commenting.
ReplyDeleteI like the format especially pertaining to medicine quick points which should provoke a quick response..at least it seems very applicable to how we function in the emergency room.
ReplyDeleteGood point. It's also much more realistic about how we practice. How would we change our ddx based on each individual piece of diagnostic data? I always teach that if the answer is "it doesn't change my management or diagnosis", then don't order the test!
DeleteThanks for all your comments. I really like how the questions get at the meat of how ED clinicians make decisions. I was unsure of the scripts in my head until they were spelt out like that.
ReplyDeletePerhaps only a sprinkle of these questions along with the traditional fact-retaining and 'what approach should I take' questions would be useful to evaluate junior learners. As residents progress, these 'decision' questions could be increasingly useful for discussion and evaluation.