Ongoing evaluation of curriculum brings changes and, ultimately, better-prepared physicians
Welcome to Ongoing evaluation of curriculum brings changes and, ultimately, better-prepared physicians
- Faculty physicians as figure skating judges: simulation demands rater reliability
- MCV Campus’ comprehensive complement of schools is fertile ground for training
- Medical students’ role in patient safety grows into international collaboration
- Walk the Walk: transferring educational reform to housestaff training
When the James W. and Frances G. McGlothlin Medical Education Center opens in the spring, its design will make possible the most significant renovation to the medical school's curriculum in more than 30 years.
Paul E. Mazmanian, Ph.D.
But how do school leaders know that these changes will be for the better? That they will produce better-prepared physicians?
A team in the medical school is charged with regularly asking that question. “We can’t answer it overnight,” says Paul E. Mazmanian, Ph.D., associate dean for Assessment and Evaluation Studies. “But we don’t have to wait a generation to see whether we did a good job preparing physicians.”
As faculty members apply massive changes to the curriculum, they want to predict as much as possible what differences those changes will make for students and, ultimately, patients. For help with that, they call upon the Office of Assessment and Evaluation Studies (OAES), a team with the skills to track learning activities, identify needed improvements and disseminate the most successful approaches to other schools and universities.
Many of the anticipated curricular changes are in response to research showing adults learn best in small groups where they can apply new knowledge to real-world problems. Not in medicine’s traditional lecture. In fact, studies show audiences retain only about 5 percent of material in lecture settings.
“You don’t learn to play piano from a lecture,” says Mazmanian. “We are asking students to master skills, not just knowledge, and there is an increasing body of science to guide the preparation of our instructional designs.”
While the medical school won’t throw out the lecture, it will shorten the time spent in large groups. Building on the idea that retention improves as students increase their immersion in material, educators are considering a 10-minute mini lecture followed by small group activities that ask students to apply the knowledge they’ve gained to clinically based scenarios. They’ve asked for the McGlothlin Medical Education Center’s large group spaces to be built so that they break down easily for small teams.
Allison A. Vanderbilt, Ed.D.
One way educators will keep an eye on the effectiveness of the curriculum is through the exams students take in the first two years of the M.D. program. Allison A. Vanderbilt, Ed.D., the director of Assessment and Evaluation at VCU’s Center on Health Disparities, also works with the medical school’s curriculum office.
Just as the curriculum’s transformation will rely on clinical scenarios to help students master concepts and competencies, the exams will be restructured to test that mastery in new ways. “Problem-solving questions describing patient vignettes are more useful in assessing students’ application of knowledge than questions that demand a simple recall of facts,” said Vanderbilt.
By assessing students’ attitudes, skills and knowledge in this way, the revised exams will also better prepare students for the national licensing exams that are given at the end of the second year. With Senior Associate Dean Isaac Wood, M.D., Vanderbilt will lead an effort to prepare in-house faculty experts who’ll help develop high quality exam questions for the nearly four dozen exams that are given medical students over the first two years of study.
Faculty physicians as figure skating judges: simulation demands rater reliability
Many of the most significant changes to the medical school’s curriculum in recent years have been developed with input from OAES.
Ellen L. Brock, M.D., M.P.H.
The expansion of the Center for Human Simulation and Patient Safety is a prime example. Across the country, the use of simulation for education, assessment and research is taking off as mannequins and virtual reality simulators are used increasingly to simulate clinical conditions and provide opportunities for realistic practice in a safe environment.
There’s a skill to interacting with patients that goes beyond technical knowledge and proficiency. It’s a skill that Ellen L. Brock, M.D., M.P.H., says some of today’s physicians weren’t taught. “In the past we were taught the science of medicine, and we were tested with multiple choice questions,” said Brock, who is medical director of the Center for Human Simulation and Patient Safety.” Learning competencies for medical students now encompass other skills that are necessary for effectiveness as a physician — things like appropriate use of the resources available to them as healthcare providers and understanding the roles of other members of the healthcare team. And students can show us that they possess those skills before they interact with real patients.”
The MCV Campus’ comprehensive complement of health science schools is fertile ground for interdisciplinary training in the Center for Human Simulation and Patient Safety.
Photo courtesy of Charlie Archambault
She’s looking to formalize how those skills are taught in the medical school’s simulation center. As she develops scenarios for student training, she always considers — from the very beginning — how faculty will assess the students’ performance.
“If 10 faculty look at the same performance, they need to give it pretty much the same evaluation; otherwise, the assessment loses meaning for the learner. We’ve all had to train ourselves to look more closely at teamwork and how we evaluate performance,” she said. OAES staff have helped faculty to use strategies such as training raters to ensure high standards and provide reliable feedback to students. Brock wants the faculty to be even more accurate and consistent than Olympic figure skating judges.
MCV Campus’ comprehensive complement of schools is fertile ground for training
Moshe Feldman, Ph.D.
On the MCV Campus, simulation also allows educators to move away from traditional school-based silos to create opportunities for interdisciplinary training. This reflects modern healthcare’s reliance on teamwork among doctors, nurses, pharmacists and therapists.
Though healthcare doesn’t have an extensive history of collaboration, other disciplines do. OAES’ expertise in organizational science provides knowledge of other fields’ approaches, bringing those best practices to help train leaders and healthcare teams.
Moshe Feldman, Ph.D., for example, has introduced an approach for training, assessing and debriefing a team after a simulation session. Originally developed for naval aviation teams, the approach aims to help all team members agree on what teamwork should look like.
Alan Dow, M.D.
The MCV Campus’ comprehensive complement of health science schools is fertile ground for Alan Dow, M.D., assistant dean for medical education and assistant vice president of health sciences for interprofessional education. “Although physicians are usually the formal leader of the team, healthcare today demands shared leadership and responsibility from all team members,” says Dow, who uses the support he receives as one of the country’s five Macy Foundation Scholars to create a variety of simulation-based training scenarios that bring together students from different schools. “A community pharmacist takes the lead on medication management. A nurse in the hospital might lead bedside care.”
Deborah DiazGranados, Ph.D.
Dow is not only providing unique experiences for the students, he is also collaborating with OAES to evaluate students’ attitudes and knowledge about working in interdisciplinary teams. Ongoing studies assess students’ small group learning and collaboration skills, like whether they lead and follow as appropriate and if they are comfortable depending on others’ knowledge. In one particular study, Deborah DiazGranados, Ph.D., an organizational psychologist in the OAES, has collaborated with Dow to develop a survey to assess students’ interprofessional collaborative competencies. Results from the study drew attention at a high-visibility meeting organized by AAMC for medical educators last summer, and the survey has been adopted by other institutions.
Medical students’ role in patient safety grows into international collaboration
“With all of these efforts, we’re looking to demonstrate that the way we train our students changes outcomes for patients,” says Dow.
A recent project will do just that. Eliminating errors and improving patient safety has become a priority for all hospitals. At an academic medical center, students are part of the equation. OAES worked with educators to survey students asking “How comfortable are you reporting errors?” “Who has responsibility for errors when they happen?” and “Who did you talk to the last time you witnessed an error?”
As expected, almost all students reported that they had witnessed an error. The surprising finding was that only a third talked to faculty physicians about what they’d witnessed. Instead, two-thirds discussed it with members of the housestaff, the physicians in the hospital who are training in various specialties.
“We found barriers that prevent students from going to the faculty physicians,” says Dow. As a result, housestaff are unexpectedly in the important role of teaching students how to handle errors. “We’d like the faculty physicians to be involved in those conversations, and we also need to reconsider the role of residents and students in improving safety.”
The findings caught the attention of medical educators in France, and DiazGranados has been collaborating closely with a physician from Université Claude Bernard to translate the survey. Data collection is now underway to determine whether the findings hold true in another culture.
With more than two dozen projects ongoing at any one time, OAES generates a tremendous amount of data. Research Assistant Jessica Evans manages its collection and management, from formatting questionnaires and conducting interviews to assuring that data are clean and managed for analysis and reporting.
Walk the Walk: transferring educational reform to housestaff training
Educational reform has stretched beyond the M.D. program to the training the housestaff physicians receive. While up to 30 percent of VCU medical students remain at VCU Medical Center for specialty training, about a hundred other newly minted physicians are recruited from more than 50 medical schools spread over the U.S. To make sure all these new trainees have clear expectations about the kinds of skills that they’re expected to master, they participate in Walk the Walk as part of orientation.
To judge the orientation’s impact on training, OAES will track the residents’ performance as they make a transition from their relatively passive roles as medical students to having more direct responsibilities for patient care. Through small group case-based discussions and simulation, the two-day conference hones core behaviors that are fundamental for safe medical practice, like professionalism and informed consent as well as functioning as part of a team of health care providers.
In addition, DiazGranados worked closely with the Department of Emergency Medicine’s Shawna Perry, M.D., to develop curriculum for the Lead the Walk conference, which teaches incoming residents about being effective organizational leaders. Specifically, residents learn that leadership is dynamic and can change based on the situation and those involved on the healthcare team.