Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?

Introduction: Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use i...

Full description

Bibliographic Details
Main Authors: David Diller, Shannon Cooper, Aarti Jain, Chun Nok Lam, Jeff Riddell
Format: Article
Language:English
Published: eScholarship Publishing, University of California 2019-12-01
Series:Western Journal of Emergency Medicine
Online Access:https://escholarship.org/uc/item/1ds5s744
id doaj-177e97c87e3c48519412c8ca4d230f05
record_format Article
spelling doaj-177e97c87e3c48519412c8ca4d230f052020-11-25T02:05:11ZengeScholarship Publishing, University of CaliforniaWestern Journal of Emergency Medicine1936-90182019-12-0121110.5811/westjem.2019.12.44468wjem-21-173Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?David Diller0Shannon Cooper1Aarti Jain2Chun Nok Lam3Jeff Riddell4LAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, CaliforniaHenry Ford Allegiance Health, Department of Emergency Medicine, Jackson, MichiganLAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, CaliforniaLAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, CaliforniaLAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, CaliforniaIntroduction: Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use in assessing the milestone sub-competencies is lacking. This study aimed to determine the frequency of sub-competencies assessed through WBNAs in an emergency medicine (EM) residency program. Methods: We performed a retrospective analysis of WBNAs of postgraduate year (PGY) 2–4 residents. A shared mental model was established by reading and discussing the milestones framework, and we created a guide for coding WBNAs to the milestone sub-competencies in an iterative process. Once inter-rater reliability was satisfactory, raters coded each WBNA to the 23 EM milestone sub-competencies. Results: We analyzed 2517 WBNAs. An average of 2.04 sub-competencies were assessed per WBNA. The sub-competencies most frequently identified were multitasking, medical knowledge, practice-based performance improvement, patient-centered communication, and team management. The sub-competencies least frequently identified were pharmacotherapy, airway management, anesthesia and acute pain management, goal-directed focused ultrasound, wound management, and vascular access. Overall, the frequency with which WBNAs assessed individual sub-competencies was low, with 14 of the 23 sub-competencies being assessed in less than 5% of WBNAs. Conclusion: WBNAs identify few milestone sub-competencies. Faculty assessed similar sub-competencies related to interpersonal and communication skills, practice-based learning and improvement, and medical knowledge, while neglecting sub-competencies related to patient care and procedural skills. These findings can help shape faculty development programs designed to improve assessments of specific workplace behaviors and provide more robust data for the summative assessment of residents.https://escholarship.org/uc/item/1ds5s744
collection DOAJ
language English
format Article
sources DOAJ
author David Diller
Shannon Cooper
Aarti Jain
Chun Nok Lam
Jeff Riddell
spellingShingle David Diller
Shannon Cooper
Aarti Jain
Chun Nok Lam
Jeff Riddell
Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?
Western Journal of Emergency Medicine
author_facet David Diller
Shannon Cooper
Aarti Jain
Chun Nok Lam
Jeff Riddell
author_sort David Diller
title Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?
title_short Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?
title_full Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?
title_fullStr Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?
title_full_unstemmed Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?
title_sort which emergency medicine milestone sub-competencies are identified through narrative assessments?
publisher eScholarship Publishing, University of California
series Western Journal of Emergency Medicine
issn 1936-9018
publishDate 2019-12-01
description Introduction: Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use in assessing the milestone sub-competencies is lacking. This study aimed to determine the frequency of sub-competencies assessed through WBNAs in an emergency medicine (EM) residency program. Methods: We performed a retrospective analysis of WBNAs of postgraduate year (PGY) 2–4 residents. A shared mental model was established by reading and discussing the milestones framework, and we created a guide for coding WBNAs to the milestone sub-competencies in an iterative process. Once inter-rater reliability was satisfactory, raters coded each WBNA to the 23 EM milestone sub-competencies. Results: We analyzed 2517 WBNAs. An average of 2.04 sub-competencies were assessed per WBNA. The sub-competencies most frequently identified were multitasking, medical knowledge, practice-based performance improvement, patient-centered communication, and team management. The sub-competencies least frequently identified were pharmacotherapy, airway management, anesthesia and acute pain management, goal-directed focused ultrasound, wound management, and vascular access. Overall, the frequency with which WBNAs assessed individual sub-competencies was low, with 14 of the 23 sub-competencies being assessed in less than 5% of WBNAs. Conclusion: WBNAs identify few milestone sub-competencies. Faculty assessed similar sub-competencies related to interpersonal and communication skills, practice-based learning and improvement, and medical knowledge, while neglecting sub-competencies related to patient care and procedural skills. These findings can help shape faculty development programs designed to improve assessments of specific workplace behaviors and provide more robust data for the summative assessment of residents.
url https://escholarship.org/uc/item/1ds5s744
work_keys_str_mv AT daviddiller whichemergencymedicinemilestonesubcompetenciesareidentifiedthroughnarrativeassessments
AT shannoncooper whichemergencymedicinemilestonesubcompetenciesareidentifiedthroughnarrativeassessments
AT aartijain whichemergencymedicinemilestonesubcompetenciesareidentifiedthroughnarrativeassessments
AT chunnoklam whichemergencymedicinemilestonesubcompetenciesareidentifiedthroughnarrativeassessments
AT jeffriddell whichemergencymedicinemilestonesubcompetenciesareidentifiedthroughnarrativeassessments
_version_ 1724939427903438848