School Shooting Simulation Studies Should Not Inform Policymakers’ Safety Responses

Publication Date:

February 25, 2021

Topic:

Education

In response to school shooting events, some policymakers and school-hardening advocates have called for increasing the presence of law enforcement—known as school resource officers (SROs)—in schools, despite little evidence that SROs are effective at preventing or mitigating acts of school violence. Further, there is growing evidence that the presence of SROs may contribute to disparate and negative outcomes for students of color and students with disabilities. A recent study by the Department of Homeland Security’s (DHS) Cybersecurity and Infrastructure Security Agency (CISA) used a virtual reality simulation to attempt to demonstrate that SROs improve safety during a school shooting incident. However, we believe that the study’s design has critical flaws in both its generalizability and its applicability to policy recommendations. As a result, policymakers should use caution when relying on this study to inform policy about school safety.

The design of the CISA study builds on a body of literature that uses virtual reality for behavioral research: Virtual reality studies allow researchers to create lifelike and controlled environments, study situations that would be too dangerous to study in real life, and repeat experiments many times at lower cost. This technique has been used to research topics as diverse as pedestrian safety, earthquake hazards, fire evacuation, and school shootings. The CISA-funded study on SROs uses virtual reality experiments to test the effects of different security conditions on simulated shooting outcomes: In this case, study participants (teachers, SROs, and university students playing the role of K-12 students) used virtual reality equipment to simulate actions during a school shooting. The virtual reality simulation was designed to mimic the conditions of a K-12 school, with one university student playing the role of school shooter. The simulation (similar to a virtual reality video game) was then run six times per day during the nine days of the study, while the conditions (SRO presence, door locking policy, and notification policy) were varied to test their effects on the outcome. (In the CISA study, outcomes were measured as the number of participants whose characters died in the simulation—assessed by event logging and post-simulation surveys—and the percentage of the simulated shooters’ ammunition that was discharged.)

However, this study is not generalizable because it draws conclusions about what would happen in real school shooter situations in K-12 schools based on how adults and college students act in a video game-like simulation, rather than how children, shooters, and SROs would act in a real, life-threatening situation. Neither the context nor the participants are representative of an actual school shooting. We cannot assume that college students in the safety of a lab will behave in the same manner as younger, terrified K-12 students, nor can we assume that the SROs will take the same actions as their simulation stand-ins. Indeed, in some of history’s most deadly school shootings—including those at Columbine High School, Virginia Tech, and Marjorie Stoneman Douglas High School—SROs either did not directly confront the perpetrators or were ineffective at doing so. It is therefore misleading to use this simulation to draw sweeping conclusions about SROs’ role in preventing school shootings. Moreover, other simulation studies about school shootings with the same design would be similarly flawed, and it would be dangerous for policymakers to draw policy recommendations from their findings.

Furthermore, this kind of simulation focuses on the potential role of SROs in the specific scenario of an active shooter incident, but the methodology cannot assess other potential impacts of having SROs on campus. A recent study by Gottfredson and colleagues found that the presence of an SRO significantly increased rates of exclusionary discipline relative to a matched set of schools that did not increase the presence of SROs. Black students and students with disabilities are disproportionately subject to exclusionary discipline, including arrest, and the presence of an SRO on campus may exacerbate these disparities. Additionally, the presence of law enforcement and other forms of school hardening (e.g., cameras, metal detectors) on campus may actually decrease students’ sense of safety and increase their fear. A simulation in which participants have no long-term relationships with each other cannot hope to capture these broader outcomes.

In assessing the validity of the CISA study’s conclusions, and of findings from similar simulation studies, we recommend that policymakers consider the following:

  1. How does the simulation differ from the real-life conditions it intends to simulate?
  2. Is the research question directly addressed by the study a sufficient basis for policy?

We hope that, by enumerating the challenges with school shooting simulations, we can prevent their misuse in informing future policies. While school shooter simulations may help policymakers understand how to prevent school shootings, they are not sufficient to answer whether school resource officers improve school safety.

Topic: