WO2018200692A1

WO2018200692A1 - Methods and systems for virtual and augmented reality training for responding to emergency conditions

Info

Publication number: WO2018200692A1
Application number: PCT/US2018/029387
Authority: WO
Inventors: Marion LEARY
Original assignee: University of Pennsylvania Penn
Current assignee: University of Pennsylvania Penn
Priority date: 2017-04-26
Filing date: 2018-04-25
Publication date: 2018-11-01
Anticipated expiration: 2019-10-26

Abstract

Method for training a trainee to respond to an emergency response condition such as a sudden cardiac arrest (SCA) including placing the trainee in a virtual reality environment comprising at least one virtual victim having a virtual victim avatar. The method can include causing the virtual victim to manifest symptoms of an SCA. The method can include allowing the trainee to perform, in the virtual reality environment, one or more of cardiopulmonary resuscitation (CPR) on the virtual victim, retrieving an automated external defibrillator (AED), using an AED on the virtual victim, or contacting emergency responders. Additionally, a method of using augmented reality for emergency response training such as CPR. The method allows the integration of a CPR feedback manikin with the AR device such that the holographic blood flow is based on the actual CPR quality being performed on the manikin.

Description

METHODS AND SYSTEMS FOR VIRTUAL REALITY AND AUGMENTED REALITY TRAINING FOR RESPONDING TO EMERGENCY CONDITIONS

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to United States Provisional Application Serial No. 62/490,306, filed on April 26, 2017, which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

The disclosed subject matter is directed to methods and systems for virtual reality and augmented reality training for responding to emergency conditions such as sudden cardiac arrest.

BACKGROUND

Emergency situations requiring medical intervention occur countless times per day. Sudden cardiac arrest (SCA) is an example of an emergency situation where cardiopulmonary resuscitation (CPR) can potentially prevent a fatal outcome. As of 2017, over 350,000 people manifest symptoms of an SCA event annually in the United States, with only a 12% survival rate. Once a victim suffers an SCA event, with each passing minute, the rate of survival can decrease by 10%. CPR administered by a bystander can improve survival rates, for example by double; however, only three out of ten victims will receive this intervention. Only 2% of the US population is CPR certified and 17% are trained. Bystanders often don't intervene due to either lack of knowledge or lack of confidence in their ability to perform CPR. Furthermore, traditional classroom training with plastic manikins can be ineffective because the training does not simulate the stressful environment of a real SCA emergency. In addition there are over 1.5 million myocardial infarction (heart attack) and stroke emergencies annually in the US. Furthermore, many people can suffer from severe, life-threatening allergies, also known as anaphylaxis. These emergency situations, as well as other emergency response conditions such as mass casualty, fire-safety, and open shooter etc., are common events that require emergency preparedness training.

Virtual reality (VR) is a computer programmed 3-dimensional (3D) environment where users interact with alternate realities. VR can simulate rare-occurring events in a safe and dynamic environment. Using VR immersion can change behavior in real-life situations. Augmented reality (AR) is similar to VR except that users can interact with the real-world. Using AR, holographic images can be overlaid into the real environment and users can interact with both simultaneously.

There remains a need for methods and systems to better train people to provide care to victims during emergency situations such as SCA.

SUMMARY

The purpose and advantages of the disclosed subject matter with be set forth in and apparent from the description that follows, as well as will be learned by practice of the disclosed subject matter. Additional advantages of the disclosed subject matter will be realized and attained by the methods and system particularly pointed out in the written description and claims thereof, as well as from the appended drawings.

To achieve these and other advantages in accordance with the purpose of the disclosed subject matter, as embodied and broadly described, the disclosed subject matter is directed to systems and methods for virtual reality and augmented reality training for responding to emergency conditions. As an example, a trainee can be immersed in a virtual reality environment and can be provided an opportunity to respond to an SCA event. In an exemplary embodiment, a method for training a trainee or an individual to respond to an SCA is provided. The method can include placing the trainee in a virtual reality environment including at least one virtual victim having a virtual victim avatar, causing the virtual victim to manifest symptoms of an SCA, and allowing the trainee to perform, in the virtual reality environment, and real environment, one or more of cardiopulmonary resuscitation (CPR) on the virtual victim, interact with virtual bystanders via verbal and/or forced computer commands, retrieve an automated external defibrillator (AED), use an AED on the virtual victim, and contact emergency responders.

The method can include instructing the trainee to explore the VR environment prior to causing the virtual victim to manifest symptoms of the SCA. The method can include causing the virtual victim avatar to align with a training manikin after the virtual victim manifest symptoms of the SCA. The training manikin can provide feedback.

The VR environment can include one or more virtual bystander avatars. The trainee can interact with the one or more virtual bystander avatars. Contacting emergency responders can include instructing at least one virtual bystander avatar to call 911. The

VR environment can be one of a city sidewalk, a park, a professional sports venue, a music venue, and a shopping center.

In an exemplary embodiment, a system for training a trainee to respond to an SCA is provided. The system can include a VR system including a headset, motion sensors and tracking devices, and a training manikin. The VR system can be configured to immerse the trainee in a virtual reality environment including at least one virtual victim having a virtual victim avatar. The virtual victim avatar can align with a training manikin after the virtual victim manifest symptoms of the SCA.

The AR system can be programed such that a CPR feedback manikin is integrated with an AR device, which can produce a holographic image of human anatomy that responds to the quality of CPR being provided. As the trainee performs CPR, the blood flow to vital organs can increase or decrease based on actual human physiology produced by the quality of CPR being performed on the CPR feedback manikin.

BRIEF DESCRIPTION OF THE FIGURES

Figure 1 provides a method for training a trainee to respond to an SCA using a VR system in accordance with the disclosed subject matter.

Figure 2 provides a system for training a trainee to respond to emergency conditions using a VR system in accordance with the disclosed subject matter.

Figure 3 provides images of a system for training a trainee to respond to emergency conditions using a VR system in accordance with the disclosed subject matter.

Figure 4 provides a system for training a trainee to respond to emergency conditions using an AR system in accordance with the disclosed subject matter.

Figure 5 provides images of a system for training a trainee to respond to emergency conditions using an AR system in accordance with the disclosed subject matter.

Figure 6 provides a chart of potential application of the disclosed subject matter. Figure 7 provides images of a mobile VR system in accordance with the disclosed subject matter.

Figure 8 provides exemplary images of a VR environment for training a trainee to repond to emergency conditions in accordance with the disclosed subject matter.

DETAILED DESCRIPTION

The presently disclosed subject matter provides systems and methods for virtual reality training for responding to SCA and systems and methods for augmented reality CPR training. The systems and methods disclosed herein can be used to train lay people; in-hospital medical professionals such as medical students, physicians, nurses, or respiratory therapists; emergency responders such as first responders, basic life support, and advanced life support; police; firefights; or military personnel. The systems and methods can utilize VR technology to simulate an SCA event or any number of emergency and/or health related conditions including, but not limited to, stroke, myocardial infarction, first aid, mass casualty/loss of limb, fire safety, water safety, anaphylaxis, active shooter etc. (see Figure 6). The VR application can set up a scenario in which a trainee views a crowded cityscape and is walking around the cityscape. After a period of time (for example one minute), one of the avatars can experience an SCA event and fall. The victim can fall to the ground such that its head and body are positioned with respect to the trainee in the same configuration as a physical manikin situated in the training space. The trainee can then interact with virtual bystander avatars, for example, to direct them to call 911 and/or retrieve an AED from a nearby building. The trainee can perform CPR on the victim or use the AED on the victim. As such, the trainee can perform CPR or use the AED on the manikin while immersed in the VR environment. Virtual Emergency Responders, for example, Emergency Medical Services (EMS), can arrive in the environment.

Sensor can be used to allow the trainee to walk through the VR environment as well as determine the technical success of the resuscitations. Although described herein for use with CPR training, the disclosed subject matter can be used for other training modules, for example, for other emergency response situations, such as first aid, fire safety, live shooter scenarios, water safety, loss of limb, anaphylaxis, stroke, or other mass casualties.

In particular embodiments, augmented reality can be used, rather than virtual reality. For example, an Augmented Reality wearable integrated with a CPR feedback manikin that produces a holographic image of human anatomy that responds to the quality of CPR being provided can be used. As the trainee performs CPR, the blood flow to vital organs increases or decreases based on the quality of CPR being performed on the CPR feedback manikin. Although described herein for use with CPR training, the disclosed AR subject matter can be used for other training modules, for example, for other emergency response situations, such as first aid, other mass casualties etc.

With reference to Figure 1, for the purpose of illustration and not limitation, a method (100) for training a trainee to respond to a SCA is provided. The method 100 can include placing the trainee in a virtual reality environment 101. The virtual reality environment can include at least one virtual victim having a virtual victim avatar. The method can include causing the virtual victim to manifest symptoms of an SCA 102. The method can include allowing the trainee to perform, in the virtual reality environment, one or more of cardiopulmonary resuscitation (CPR) on the virtual victim, retrieving an automated external defibrillator (AED), using an AED on the virtual victim, or contacting emergency responders (103).

The VR environment can be created to illustrate any number of environments including a cityscape, a corporate office, a baseball field etc. Real -world objects can be integrated into the system similar to the CPR feedback manikin for tactile and haptic response described hereinabove. The trainee can be instructed to explore the VR environment prior to causing the virtual victim to manifest symptoms of the SCA.

The virtual victim can fall into alignment with a training manikin after the victim manifest symptoms of the SCA. For example, this can be arranged by choosing an avatar who is walking across the trainee's path, e.g., from right-to-left, so that the head ends up on the right side of the subject. The manikin is aligned during the set-up of the VR system. The victim avatar can be within a meter of the trainee. This arrangement can place the trainee in a direct situation of having to act; the trainee will not have to locate the victim by search or accident. Depending on the demographic make-up of the trainees and/or the training required (i.e., child CPR versus adult CPR), the operator of the training session can choose the race, age, and gender of the victim or it can be generated randomly.

The VR environment can be a cityscape, for example a city sidewalk, a park, a professional sports venue, a music venue, a shopping center, or other similarly crowed, public locations. The cityscape can include one or more virtual bystander avatars. In particular embodiments, the virtual bystander avatars can have realistic reactions to the emergency response events, for example, crying, begging for help, screaming, or running away. The trainee can interact with one or more of the virtual bystander avatars. For example, the trainee can interact with virtual bystander avatars via voice communication. The trainee can ask the virtual bystanders to perform an action such as call 911, retrieve an AED or fire extinguisher from a nearby building, or to perform actions such as CPR or the Heimlich maneuver if they know how. If a virtual bystander avatar returns with the

AED in the virtual reality, for example, the trainee can apply the AED to the actual manikin (i.e., the virtual victim). In some embodiments, the trainees can search for the AED and then use the AED. The AED in the virtual environment is programmed with the same instructions as an actual AED. In the AR environment a real AED could be integrated with the AR device. Once EMS arrives, the simulation can end.

The VR environment can include an emergency response instructor avatar, which can be controlled by a human instructor also immersed in the VR environment or which can be a virtual instructor programmed to provide instruction. The instructor can follow the trainee throughout the scenario and can be interactive. The instructor can answer questions and alert the trainee when he or she is not performing correctly or using an emergency device such as an AED or fire extinguisher correctly. The instruction can provide information on the proper techniques for emergency response skills such as CPR. In particular embodiments, multiple trainees can be immersed in the VR environment at the same time, and can observe one another or work together during the VR training. Remote trainers can monitor the multiple trainees and provide debriefing after the training session. A training paradigm can be used where: 1) trainees are immersed in the virtual environment and receive an emergency response training (such as CPR) with in the VR system 2) trainees' actual VR sessions are recorded and replayed in 2D during a debriefing session 3) trainees are taught the skills that they performed correctly and instructed on skills they can improve upon 4) trainees complete the session with a quiz to assess knowledge acquisition.

Referring to Figures 2-5 for the purpose of illustration and not limitation, systems for training a trainee to respond to an emergency situation, such as a sudden cardiac arrest (SCA), are provided. The system 300 can include a virtual reality system including a headset 301, motion sensors 302, and a tracking device 303. The system can also include a training manikin 300. The VR system can be configured to immerse the trainee in a VR environment including at least one virtual victim having a virtual victim avatar and the virtual victim avatar can align with the training manikin after the virtual victim manifest symptoms of the SCA. The trainee can practice responding to the SCA event.

VR headsets, motion sensors and tracking devices can be used. For example, the

Vive system by HTC or the Oculus Rift can be used as VR headsets. The HTC Vive can allow trainees to walk in the simulated space. Hand controllers or data gloves, for example by Manus VR, can be worn by the trainee 304 and tracked by tracking devices set up in the training room to determine walking speed and direction. The trainee's hands can be tracked by the tracking device so that the trainee 304 can see his or her hands and the positioning on the manikin in the virtual reality scape. For example, leap motion can be used to track and analyze hand movement. As another example, a Razor Hydra Motion Device can be used to track hand motion or haptic wearables could be integrated to allow for high-realism for tactile response.

The training manikin can be, for example, a Laerdal CPR feed-back manikin which can provide CPR quality feedback, as the trainee performs CPR on the manikin.

A laptop gaming computer can be used to run the VR, for example, the GPU can be NVIDIA GeForce GTX 970 / AMD Radeon R9 290 equivalence or greater. The CPU can be an Intel Ϊ5-4590 / AMD FX 8350 equivalent or greater. Four or more GB of RAM can be used. The system can operate on Windows 7 SP1 or newer. As an example, an Alienware gaming laptop with GPU amplifier can be used.

For the purpose of illustration and not limitation, a system for training a trainee in CPR using the AR system is provided. An augmented reality wearable device, such as the Microsoft HoloLens, integrated with a CPR feedback manikin 501 can be used to run the AR system. The training manikin can be, for example, a Laerdal CPR feed-back manikin which can provide CPR quality feedback, as the trainee 502 performs CPR on the manikin. The disclosed AR system with a CPR feedback manikin can produce a holographic image 503 of human anatomy that responds to the quality of CPR being provided. As the trainee 502 performs CPR, the blood flow 504 to vital organs increases or decreases based on the quality of CPR being performed on the CPR feedback manikin. The holographic image can include a heart, a lung, a blood vessel, a brain, or any other organs. The holographic image can dynamically interact with the trainee and can include a plurality of organs. In certain embodiments, each organ can be artificially painted with different colors (Figure 5). In certain embodiments, the disclosed subject matter provides mobile systems for virtual reality training for responding to SCA. As shown in Figure 7C, an exemplary mobile system can include a house 700 (Figure 7 A), coupled to a mobile VR system 701 (Figure 7B). The house can include a cardboard, a plastic box, a metal box, a gear VR, a head set, a VR viewer, and any mobile work stations that can be coupled to the mobile

VR system and provide VR images. The mobile VR system can include a smartphone, a tablet PC, a PDA, a mobile PC, a camera, or similar hardware systems. As shown in Figure 7D, the mobile VR system 703 can be secured to a user's head 702 through a system binding device. The system binding device can include a strap, buckle, elastic band, tape, Velcro, or any other suitable securing features.

In certain embodiments, the disclosed system can provide a mobile application. The mobile application can include an out-of hospital SCA scenario allowing users to be immersed in the SCA experience without test subjects (e.g., manikin). In non-limiting embodiments, as shown in Figure 8, the mobile application can include a 5 -minute CPR training video followed by an SCA scenario on the VR viewer integrated with a mobile device. Users can interact with other simulated lay bystanders in the scenario, perform CPR on the simulated victim, and acquire an AED using a VR cardboard viewer click button. Chest compression (CC) rate can be captured using a click button, located on the top of the VR Cardboard viewer. The click button can be integrated with the smartphone on/off button via the mobile VR app.

The presently disclosed subject matter is not to be limited in scope by the specific embodiments herein. Indeed various modifications of the disclosed subject matter in addition to those described herein will become apparent to those skilled in the art from the foregoing description and the accompanying figures. Such modifications are intended to fall within the scope of the appended claims.

Claims

1. A method for training a trainee to respond to a sudden cardiac arrest (SCA), comprising:

placing the trainee in a virtual reality environment comprising at least one virtual victim having a virtual victim avatar;

causing the virtual victim to manifest symptoms of an SCA; and

allowing the trainee to perform, in the virtual reality environment, one or more of cardiopulmonary resuscitation (CPR) on the virtual victim, retrieving an automated external defibrillator (AED), using an AED on the virtual victim, or contacting emergency responders.

2. The method of claim 1, further comprising instructing the trainee to explore the virtual reality environment prior to causing the virtual victim to manifest symptoms of the SCA.

3. The method of claim 1, further comprising causing the virtual victim avatar to align with a training manikin after the virtual victim manifest symptoms of the SCA.

4. the method of claim 3, wherein the training manikin can provide feedback.

5. The method of claim 1, wherein the virtual reality environment comprises one or more virtual bystander avatars.

6. The method of claim 5, wherein the trainee can interact with the one or more virtual bystander avatars.

7. The method of claim 5, wherein contacting emergency responders comprises instructing at least one virtual bystander avatar to call 911.

8. The method of claim 1, wherein the virtual reality environment is one of a city sidewalk, a park, a professional sports venue, a music venue, and a shopping center.

9. A system for training a trainee to respond to a sudden cardiac arrest (SCA), comprising: a virtual reality system comprising a headset, motions sensors and tracking devices; and

a training manikin;

wherein the virtual reality system is configured to immerse the trainee in a virtual reality environment comprising at least one virtual victim having a virtual victim avatar; and wherein the virtual victim avatar aligns with a training manikin after the virtual victim manifest symptoms of the SCA.

10. A system for training a trainee to perform CPR, comprising:

an augmented reality system configured to immerse the trainee in an augmented reality environment; and

a training manikin;

wherein the augmented reality system is configured to produce a holographic overlay of a human body onto the training manikin; and

wherein the augmented reality system causes the holographic overlay to include blood flow to essential organs based at least in part on human physiology and a quality of cardiopulmonary resuscitation (CPR) performed by the trainee on the training manikin.