US11893906B2 - Extended reality simulator with real-world interaction capabilities - Google Patents

Extended reality simulator with real-world interaction capabilities Download PDF

Info

Publication number
US11893906B2
US11893906B2 US17/983,158 US202217983158A US11893906B2 US 11893906 B2 US11893906 B2 US 11893906B2 US 202217983158 A US202217983158 A US 202217983158A US 11893906 B2 US11893906 B2 US 11893906B2
Authority
US
United States
Prior art keywords
injury
living body
treatment
simulation
sensors
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US17/983,158
Other languages
English (en)
Other versions
US20230306865A1 (en
Inventor
Sören Björn Gutekunst
Peter Schneider
Frank Klemmer
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ESG Elektroniksystem und Logistik GmbH
Original Assignee
ESG Elektroniksystem und Logistik GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ESG Elektroniksystem und Logistik GmbH filed Critical ESG Elektroniksystem und Logistik GmbH
Assigned to ESG ELECKTRONIKSYSTEM- UND LOGISTIK-GMBH reassignment ESG ELECKTRONIKSYSTEM- UND LOGISTIK-GMBH ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KLEMMER, Frank, GUTEKUNST, SOEREN BJOERN, SCHNEIDER, PETER
Publication of US20230306865A1 publication Critical patent/US20230306865A1/en
Application granted granted Critical
Publication of US11893906B2 publication Critical patent/US11893906B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B9/00—Simulators for teaching or training purposes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
    • G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
    • G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
    • G09B23/30—Anatomical models
    • G09B23/303—Anatomical models specially adapted to simulate circulation of bodily fluids
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00—Electrically-operated educational appliances
    • G09B5/06—Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
    • G09B5/067—Combinations of audio and projected visual presentation, e.g. film, slides
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/41—Medical

Definitions

  • Simulations are based on situations and events from real life and enable learning in a safe, controlled environment. They help by reducing the gap between theory and practice. Scenarios can form the basis of training through simulations. Scenarios are structured descriptions of events or problems from the real world. They allow learners to deepen into a specific real-life problem and to find solutions to this problem. They must use their already acquired knowledge, cognitive and social skills, to successfully solve the problem or task. It is a method that allows learning in an authentic context and tries to put learners into a more active role. However, designing and preparing scenarios are no simple tasks. It is necessary to define how the process will proceed, describe the activity of the participants in the training, determine the roles they will take and the tools they will need. It is important to provide the most realistic environment for students to get accustomed to such situations.
  • mannequins are usually used for injury simulation. However, these are unable to mimic human behavior realistically.
  • actors are used as injured persons. However, depending on the scenario, these actors must be hired and require significant preparation time. Moreover, they may also have difficulty reacting appropriately to treatment by the first responder if they do not receive adequate feedback information.
  • An underlying problem of the present disclosure can therefore be regarded as how to modify a given injury simulation system that is based on mannequins so that it can be applied to a living body.
  • said problem is solved by a system and a method for simulating injury treatment on a living body as described herein.
  • the present disclosure is directed to provide a system for simulating injury treatment on a living human or animal acting as an injured living body in a training scenario and thereby eliminating the need for mannequins for injury simulation. Furthermore, the disclosure aims at providing a method for simulating injury treatment on the injured living body.
  • a system for simulating injury treatment on a living body includes an embedded system configured to be placed on a living body and to simulate an injury of the living body in response to instructions received from a simulation system.
  • the system further includes one or more sensors configured to be placed on the living body and to detect a parameter corresponding to a treatment of said injury, said one or more sensors being further configured to transmit said detected parameter to said simulation system.
  • the simulation system is connected to said one or more sensors and configured to transmit the instructions to said embedded system to cause said embedded system to simulate said injury of the living body, and to receive sensor data representing said parameter detected by said one or more sensor and representing said treatment of said injury, and to execute a simulation scenario, which associates said detected parameter representing said treatment of said injury with a corresponding effect on said injury. Furthermore, the simulation system is configured to send instructions to said embedded system, which cause said embedded system to simulate said injury such that the simulation reflects said treatment.
  • the system may eliminate the need for practice mannequins or enlarge their scope of training. Furthermore, the system may enable a sensor feedback system by simulating injuries on the living body and collecting data on how the first responder treats the simulated injuries.
  • a control unit may analyze the sensor data, generate, and transmit new instructions to the embedded system or injury simulation module. The instructions may cause the simulated injuries and the actor (living body) to respond as realistically and authentically as possible to the applied treatment. This can make simulated injury treatment very authentic while reducing the need for preparing actors and the time spent preparing the injury scenarios. Next to the possibility to direct the actor of the injured body.
  • the embedded system may include one or more actuators configured to be placed on the living body and to simulate said injury of the living body in response to the instructions sent from said simulation system, wherein said simulation system is connected to said one or more actuators.
  • the one or more actuators may be computer-controlled and may correspond to at least one of the following: a pump for spilling a liquid to simulate a bleeding injury or other liquid losses, an indicator for visual effects of the injury or status of the injury, a speaker for acoustic effects related to the injury, a box for packing a bandage of a simulated wound or an intraosseous infusion (TO), intravenous therapy (IV), a motor for simulating an injury comprising dislocations and/or open broken bones.
  • a pump for spilling a liquid to simulate a bleeding injury or other liquid losses an indicator for visual effects of the injury or status of the injury
  • a speaker for acoustic effects related to the injury
  • TO intraosseous infusion
  • IV intravenous therapy
  • said one or more sensors may correspond to at least one of the following: a pressure sensor to detect a location and amount of an exertion of pressure of the treatment, a gyroscopic sensor or an accelerometer to detect a movement and acceleration of the living body based on the treatment, a resistance sensor to detect a location and amount of an exertion of pressure and/or a cut of the executed treatment, and a temperature sensor to detect an increase or decrease in temperature of the living body based on the treatment.
  • a pressure sensor to detect a location and amount of an exertion of pressure of the treatment
  • a gyroscopic sensor or an accelerometer to detect a movement and acceleration of the living body based on the treatment
  • a resistance sensor to detect a location and amount of an exertion of pressure and/or a cut of the executed treatment
  • a temperature sensor to detect an increase or decrease in temperature of the living body based on the treatment.
  • the simulation scenario may associate the detected parameter caused by said treatment with a corresponding control signal to said embedded system to simulate a change in the simulated injury in response to said detected parameter. This is advantageous for simulating the injury in a realistic manner.
  • the embedded system may further include one or more markers to be placed on said living body, and wherein said simulation system is adapted to replace said one or more markers in an image taken by a camera of an XR system by a simulation image, which simulates said injury at the location of said one or more markers, wherein said injury simulated at the location of said one or more markers includes at least one of the following: an open wound, a pneumothorax, a frostbite, a combustion, a cut, a pleural effusion, a hemothorax, a bleeding; and wherein the XR system further includes an extended reality headset including a camera and a display. This allows for a realistic injury simulation and overall injury simulation scenario and leads to improved first responder training results.
  • the embedded system may include one or more environment markers to be placed in an environment of the living body, wherein the simulation image simulates changes, parts, items, devices, wearables or displays in the environment of the living body at the location of said one or more environment markers, wherein said changes simulated in said environment of the living body at the location of said one or more environment markers includes at least one of the following: a shot, an explosion, a leakage, a fire or an electric shock.
  • a shot an explosion
  • a leakage a fire or an electric shock.
  • the treatment detected by said one or more sensors may include at least one of the following: a tourniquet, a compression bandage, a tracheotomy, a thoracocentesis, a chest tube, a puncture, an infusion, a wound packing, a fixing of broken bone, a wound sealing, an intraosseous infusion (TO), intravenous therapy (IV).
  • a tourniquet a compression bandage, a tracheotomy, a thoracocentesis, a chest tube, a puncture, an infusion, a wound packing, a fixing of broken bone, a wound sealing, an intraosseous infusion (TO), intravenous therapy (IV).
  • the simulation system may include a console or station and the simulation system may be further configured to receive user inputs provided by a user over the console or station, and to generate user instructions based on the received user inputs, and/or said system may further include an earphone configured to be placed in the ear of the living body to be treated, and said simulation system may be further adapted to transmit instructions to said earphone and the embedded system.
  • the system may include a virtual environment generated and controlled by the simulation system such that it simulates by acoustic, physical, and visual stimulus to a user conducting the treatment in the virtual environment comprising one of the following: a battlefield scenario, a traffic accident scenario, a natural disaster scenario, a household accident scenario, a riot scenario, a marine scenario, a casualty evacuation, a medical evacuation, a casualty collection point (CCP), or a firefighting scenario.
  • a battlefield scenario a traffic accident scenario, a natural disaster scenario, a household accident scenario, a riot scenario, a marine scenario, a casualty evacuation, a medical evacuation, a casualty collection point (CCP), or a firefighting scenario.
  • the system may include one or more user sensors configured to detect a user parameter of the user conducting the treatment, wherein the one or more user sensors correspond to at least one of the following: a pulse oximeter, a pulse meter, a heart rate variability sensor, a breathing sensor, a video-oculography (VOG) or electrooculography (EOG) sensor.
  • VOG video-oculography
  • EOG electrooculography
  • the acoustic and physical stimulus to the user conducting the treatment may be generated by one or more environment actuators placed in the virtual environment, and controlled by the simulation system, and wherein the one or more environment actuators correspond to at least one of the following: a motor, a pump, a gas exhaust, or a flame and smoke generator.
  • the embedded system may be incorporated into at least one of the following: a garment, an item, a device, a wearable, or a bag.
  • a garment an item
  • a device a wearable
  • a bag a bag
  • a method for simulating injury treatment on a living body includes, simulating by an embedded system placed on a living body an injury of the living body in response to instructions received from a simulation system, detecting by one or more sensors placed on a living body a parameter corresponding to a treatment of said injury, said one or more sensors being configured to transmit said detected parameter to said simulation system.
  • the simulation system performing the following steps by a simulation system connected to said one or more sensors, transmitting the instructions to said embedded system to cause said embedded system to simulate an injury of the living body, receiving sensor data representing said parameter detected by said one or more sensors and representing a treatment of said simulated injury, executing a simulation scenario, which associates said detected parameter representing a treatment of said injury with a corresponding effect on said injury, sending instructions to said embedded system, which cause said embedded system to simulate said injury such that the simulation reflects said treatment.
  • FIG. 1 is an example diagram of the overall system.
  • the present disclosure relates to a system and method for simulating injuries and the treatment of the simulated injuries on a living body based on a specific simulated injury scenario.
  • Simulated injury scenarios in this context are structured simulations of events or problems from the real world involving a living body like a human or an animal being injured.
  • the injuries on the living body are simulated in accordance with the selected scenario and for example include an open wound, a pneumothorax, a frostbite, a combustion, a cut, a pleural effusion, a hemothorax, or a bleeding.
  • FIG. 1 shows an overall diagram of the system.
  • the central part of FIG. 1 shows the training area 110 , that is the area in which the training takes place, and the injury scenario is simulated.
  • the training area 110 includes at least one client 210 and at least one executor 220 , wherein both are connected to a simulation system or central control unit 340 via an embedded system connection 410 and an extended reality (XR) system connection 430 .
  • the embedded system connection 410 and the XR system connection 430 enable bidirectional data exchange and may be wireless or wired.
  • the client 210 is the injured living body to whom an action is performed by the executor 220 , the action being a treatment of the simulated one or more injuries of the living body.
  • the executor 220 as a firsttician, for example, executes the actions. It is also conceivable that multiple executors participate in the scenario.
  • the system includes sensors that are configured to be placed on the living body or the client 210 .
  • the sensors are configured to detect and measure one or more parameters that correspond to a treatment of a simulated injury of the client 210 .
  • the executor 220 conducts such treatment of the living body.
  • the sensors, and the parameters they measure include a pressure sensor to detect a location and an amount of an exertion of pressure of the treatment, a gyroscopic sensor or an accelerometer to detect a movement and acceleration of the living body based on the treatment, a resistance sensor to detect a location and an amount of an exertion of pressure of the treatment and/or a cut or incision made as part of the treatment, or a temperature sensor to detect an increase or decrease in temperature of the living body based on the treatment.
  • a pressure sensor to detect a location and an amount of an exertion of pressure of the treatment
  • a gyroscopic sensor or an accelerometer to detect a movement and acceleration of the living body based on the treatment
  • a resistance sensor to detect a location and an amount of an exertion of pressure of the treatment and/or a cut or incision made as part of the treatment
  • a temperature sensor to detect an increase or decrease in temperature of the living body based on the treatment.
  • the treatments detectable by the sensors or being determined by the simulation system 340 based on the measured sensor parameters include, among others, a tourniquet, a compression bandage, a tracheotomy, a thoracocentesis, a chest tube, a puncture, an infusion, a wound packing, a vacuum splint for fixing of broken bones, and a wound sealing.
  • the sensors Upon detection of a treatment and a measurement of the corresponding parameters resulting from the treatment, the sensors transmit the detected or measured parameters via embedded system connection 410 to the simulation system 340 for further processing and analysis of the data.
  • the simulation system 340 generates instructions or control signals.
  • These instructions are sent to the embedded system 310 and cause the actuators to simulate the effects of the treatment on the simulated injury, that is, a change in the simulated injury in response to the detected parameter.
  • a sensor may detect a pressure applied in proximity of or on the wound and transmits the sensor measurements to the simulation system 340 .
  • the simulation system 340 determines that the pressure must be induced by a compression bandage, for example, because the pressure detected by the sensors is constant over a certain area.
  • the simulation system 340 instructs the pump to reduce fluid flow based on the location, the amount, the area, and the duration of the pressure applied. Next to the real fluid flow, a corresponding virtual visual and audio effect is possible to increase the immersion of the said wounding.
  • the client 210 is equipped with the embedded system 310 .
  • the embedded system 310 is a combination of hardware and software that includes one or more actuators.
  • the software may in one implementation include a computer program for controlling the actuators in response to signals received from sensors and/or a simulation system 340 .
  • the embedded system 310 may further include an embedded computer.
  • the actuators are configured to be placed on the living body and to simulate an injury of the client 210 .
  • the embedded system 310 may be integrated into a garment to be worn by the client 210 or alternatively or in addition thereto in a bag, an item, a wearable, or a device such as but not limited to defibrillator, patient monitor, stretcher, first aid equipment, first aid box, bag and environmental items like tree stump, road marking, helmet, cables, tubes, cable or tube reel, handrail, chair, sofa, pouches, bottles, radio set, intercoms, magazine bag, field harness, binoculars, equipment bag, backpack, bag, or mines.
  • the embedded system 310 and the actuators are controlled by the simulation system 340 and receive the corresponding control instructions via embedded system connection 410 .
  • the actuators may correspond to a pump for spilling a liquid to simulate a bleeding injury or other liquid losses, an indicator for visual effects of the injury or status of the injury such as but not limited to light bulbs, LEDs, diodes, glow lamps, displays, analog displays, or servo mechanical displays, a speaker for acoustic effects related to the injury such as but not limited to screaming, breathing, fading/gaining/steady physiological noises (heart noises, lung noises [crackling, stridor, rhonchi, whooping, pleural friction rub, mediastinal rub], colon noises), suffocation, running liquids, a box for packing a bandage of a simulated wound, an intraosseous infusion (TO) or intravenous therapy (IV), a compartment to set an access or infusion, a motor for simulating an injury comprising dislocations and
  • FIG. 1 further shows a simulation system 340 .
  • the simulation system 340 is the central computing unit including the necessary hardware such as switches/routers, control units (fog, air conditioning, light, sound, vibration, pump, etc.).
  • This system serves as the central connection of all components but can also take over subsystems, e.g., smaller simulators, such as the embedded system 310 , the surveillance system 320 , the computing unit of the XR System 330 , the instructor/debriefing System 350 and/or the observer system 360 .
  • video-sound recordings of the training can be stored here and distributed for later use.
  • the simulation system 340 may provide the processed sensor data to a console or station that is controlled by a user or an instructor. Based on user inputs received over the console or station, the simulation system 340 generates user instructions that are transmitted to the embedded system 310 or to the client 210 .
  • the user instructions transmitted to the embedded system 310 are control signals for controlling the injury simulation performed by the embedded system 310 .
  • the user instructions when transmitted to the client 210 instruct the client 210 to behave in a certain way.
  • the instructions are received by headphones in the client's 210 ear or by glasses worn by the client 210 , to instruct the client 210 how to respond to specific treatments performed by the executor 220 .
  • the simulation system 340 may also provide instructions that are generated based on the measured sensor data directly to the client 210 without requiring user inputs on the console or station.
  • the instructions include feedback information for the client 210 that instructs the client 210 to respond to an injury treatment in a specific way. For example, suppose the simulation system 340 registers that the executor 220 is improperly treating a wound, for example, by applying a compression bandage too tightly. If the simulation system 340 determines that the executor 220 is causing significant pressure and thus pain, it relays this information to the client 210 . The client 210 may then respond to the injury treatment by screaming or twitching the limbs.
  • FIG. 1 further shows that the executor 220 is equipped with an XR system 330 .
  • the XR system 330 comprising an XR or mixed reality headset with a computing unit (mobile, stationary, or embedded) for generating the view.
  • the computing unit can be embedded in the glasses, attached to the person or, e.g., placed externally as a high-end workstation (as a backpack or similar).
  • the XR headset can be connected to an independent computing unit or, for example, to a central simulation host like the simulation system 340 (centralized or decentralized implementation).
  • the connection between XR system 330 and simulation system 340 via XR system connection 430 is bidirectional.
  • the simulation system 340 controls the simulation and generates the mix of virtual and real environment/world. Also, the live video of XR vision or eye cameras etc. is transmitted from the headset and made available for further use.
  • the training area 110 may additionally include one or more markers to be placed on the living body.
  • a marker in this context is a visual marker, for instance, a binary marker or an image or an object that can be recognized by the XR system 330 and is used to trigger augmented reality features.
  • the markers are used to simulate additional injuries, which are only visible in virtual or extended reality, the additional injuries comprising, for example, an open wound, a pneumothorax, a frostbite, a combustion, a cut, a pleural effusion, a hemothorax, or a bleeding.
  • the executor 220 or any other participant will be able to see the injuries when wearing the XR headset.
  • the training area 110 may also include environment markers to be placed within the training area in the environment of the living body.
  • the environment markers are used as fixed point of reference for augmented reality features, for example, for simulating environmental changes, parts, items, devices, wearables or displays.
  • medical devices can be provided to the executor 220 in virtual and extended reality, displaying simulated data on the condition of the injured living body or feedback data from the sensors placed on the living body.
  • the environment markers can also be used to simulate changes in the living body's environment. These changes may depend on the scenario and include, for example, a shot, an explosion, a leakage, a fire, or an electric shock.
  • the markers may be used to simulate a leak and an ingress of water into the submarine or a damage to the submarine engine.
  • a virtual environment may be generated in the training area 110 that is controlled by the simulation system 340 .
  • the virtual environment is generated in a chromakey room, in a room with screens or projectors, or by object-based masking, meaning that only digital content without an environment can be shown, for example, through depth detection with LIDAR and everything further than 2 to 5 meters away is displayed digitally.
  • the virtual environment makes the simulated injury scenario more realistic by simulating an acoustic, physical, and visual stimulus to the executor 220 .
  • a stimulus is anything that can trigger a physical or behavioral change of the executor 220 , for example, sights, sounds, smells, and temperature changes.
  • a respective virtual environment is generated, for example, a battlefield, a traffic accident, a natural disaster, a household accident, a riot, a marine accident, a casualty evacuation, a medical evacuation, a casualty collection point (CCP), or a firefighting accident.
  • a battlefield for example, a battlefield, a traffic accident, a natural disaster, a household accident, a riot, a marine accident, a casualty evacuation, a medical evacuation, a casualty collection point (CCP), or a firefighting accident.
  • the training area 110 may further include environmental actuators placed in the virtual environment and controlled by the simulation system 340 for generating further stimuli to the participants of the training scenario based on optical, physical, chemical, and/or mechanical technology.
  • the actuators for instance, include a motor, a pump, a gas exhaust, a headlight, a light reflector, a light source, a light shutter, a flame and smoke generator, or actuators based on self-contained and self-sustained exothermic chemical reactions to make heat, light, gas, smoke and/or sound.
  • liquid, and smoke generators may be used to add more realism to the overall setting.
  • the executor 220 may also be equipped with sensors, as well. These sensors are configured to detect parameters of the executor 220 conducting the treatment and, for example, include a temperature sensor, a pulse oximeter, a heart rate variability sensor, a breathing sensor, or a video-oculography or electrooculography sensor.
  • the video-oculography or electrooculography sensor detect eye direction, an eye movement, blinking rate, and saccadic eye movement.
  • the sensors transmit the sensor data to the simulation system 340 for further processing.
  • the training area 110 also includes a surveillance system 320 .
  • This typically includes cameras (e.g., IR, electro-optical, thermal, magnetic) or webcams to capture and transmit live footage from the training area 110 in the instructor area 120 or observer area 130 .
  • cameras e.g., IR, electro-optical, thermal, magnetic
  • webcams to capture and transmit live footage from the training area 110 in the instructor area 120 or observer area 130 .
  • a wired or wireless system can be used.
  • the system shown in FIG. 1 in one implementation may also include an instructor or debriefing area 120 .
  • This is the area in which training is controlled, observed, and evaluated by instructors 230 . Control and evaluation can be split in the same area/room or in separate areas. Separate areas allow parallel training and debriefing and thus increase the efficiency/performance of the training.
  • the instructor 230 is an expert who controls the simulation, instructs the client 210 , in addition to the simulation system 340 and executor 220 , and evaluates the action of the same.
  • the instructor 230 controls an instructor or debriefing system 350 , which is a system for control and observation of the simulation. Control and evaluation can be implemented on one system or several separate systems.
  • the instructor or debriefing system 350 communicates with the simulation system 340 through a wireless or wired communication 440 , for example, for transmitting commands to control the simulation such as start, stop pause, record, replay, mission selection, weather, day-/nighttime and events. Also transmitted is information to display events and sequence of actions in the simulation such as videos, readings, wounded condition, or objects.
  • the system shown in FIG. 1 may also include an observer area 130 .
  • This is the area or room in which people observe the current simulation.
  • the observer area 130 is where other participants, instructors, and/or spectators can participate in the current training remotely or on-site. It can be one area/room or multiple areas/rooms that can be in multiple locations. This area/room is not necessary for the simulation itself but adds value as an extension.
  • the observers 240 are training participants who observe the current training to learn from others' mistakes. Instructors/trainers can also be included here. These people are not necessary for the training, however, if they are included it increases the training success.
  • the observer area 130 includes an observer system 360 , which is a system for observing the simulation without the possibility to intervene.
  • the observer system 360 is connected to the simulation system 340 via wireless or wired communication 450 .
  • Information for displaying events in the simulation, such as videos and measured values, is transmitted. Audio-visual exchange with other systems is enabled.
  • a method for simulating an injury treatment of a living body may include the following steps.
  • Executing a simulation scenario by the simulation system 340 which associates the detected parameter representing a treatment with a corresponding effect on the injury and sending instructions to the embedded system 310 causing it to simulate the injury such that the simulation reflects the treatment.
  • detecting by the sensors location and amount of an exerted pressure and the simulation system 340 instructing a pump to reduce the spilling of liquid depending on the detected location and amount of the exerted pressure.
  • instructing by the simulation system 340 the actuators to simulate the injury instructing by the simulation system 340 the actuators to simulate the injury.
  • the method includes simulating injuries in virtual and extended reality by placing markers on the living body. Furthermore, simulating changes, parts, items, devices, wearables or displays in virtual and extended reality by placing environment markers in the environment of the living body, the training area 110 .
  • receiving by the simulation system 340 instructor inputs via a console or station of the instructor debriefing system 350 .
  • generating by the simulation system 340 a virtual environment in the training area 110 generating by the simulation system 340 a virtual environment in the training area 110 .
  • the exemplary embodiments described herein may be implemented by software (a computer program), hardware, or a combination of both.
  • any functions with respect to the elements described herein such as the transmission of data, the reception of data, or the controlling of any device such as sensors or actuators or a simulation engine, these functions may be implemented by a combination of suitable hardware (e.g., communication links, transmission and/or reception modules, sensors, actuators, processors) and one or more computer programs controlling the hardware by executing the computer program.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Mathematical Optimization (AREA)
  • Algebra (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Physics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Medicinal Chemistry (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Instructional Devices (AREA)
US17/983,158 2022-03-24 2022-11-08 Extended reality simulator with real-world interaction capabilities Active US11893906B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
EP22164239.0A EP4250273B1 (de) 2022-03-24 2022-03-24 Erweiterter realitätssimulator mit interaktionsmöglichkeiten in der realen welt
EP22164239.0 2022-03-24
EP22164239 2022-03-24

Publications (2)

Publication Number Publication Date
US20230306865A1 US20230306865A1 (en) 2023-09-28
US11893906B2 true US11893906B2 (en) 2024-02-06

Family

ID=80933143

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/983,158 Active US11893906B2 (en) 2022-03-24 2022-11-08 Extended reality simulator with real-world interaction capabilities

Country Status (2)

Country Link
US (1) US11893906B2 (de)
EP (1) EP4250273B1 (de)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070243512A1 (en) 2004-12-02 2007-10-18 King Lynn R Trauma Training System
WO2008018889A2 (en) 2005-09-29 2008-02-14 The General Hospital Corporation Medical training system for casualty simulation
WO2009097045A1 (en) * 2008-01-28 2009-08-06 Paul Lecat Device and methods for medical training using live subjects
US20130295538A1 (en) * 2008-02-20 2013-11-07 Philip Ambrose Hazardous material detector simulator and training system
US20180293802A1 (en) 2017-04-07 2018-10-11 Unveil, LLC Systems and methods for mixed reality medical training
US10854098B1 (en) * 2018-08-13 2020-12-01 University Of Central Florida Research Foundation, Inc. Adaptive visual overlay wound simulation

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070243512A1 (en) 2004-12-02 2007-10-18 King Lynn R Trauma Training System
WO2008018889A2 (en) 2005-09-29 2008-02-14 The General Hospital Corporation Medical training system for casualty simulation
WO2009097045A1 (en) * 2008-01-28 2009-08-06 Paul Lecat Device and methods for medical training using live subjects
US20130295538A1 (en) * 2008-02-20 2013-11-07 Philip Ambrose Hazardous material detector simulator and training system
US20180293802A1 (en) 2017-04-07 2018-10-11 Unveil, LLC Systems and methods for mixed reality medical training
US10854098B1 (en) * 2018-08-13 2020-12-01 University Of Central Florida Research Foundation, Inc. Adaptive visual overlay wound simulation

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
D. K.J.E. von Lubitz et al. Simulation-based medical training: the Medical Readiness Trainer concept and the preparation for civilian and military medical field operations, VRIC, Virtual Reality International Conference, Laval Virtual, May 16, 18, 2001.
W. L. van Meurs et al. Functional Anatomy of Full-Scale Patient Simulators, Journal of Clinical Monitoring 13, 317 324, 1997

Also Published As

Publication number Publication date
EP4250273C0 (de) 2025-04-09
US20230306865A1 (en) 2023-09-28
EP4250273A1 (de) 2023-09-27
EP4250273B1 (de) 2025-04-09

Similar Documents

Publication Publication Date Title
KR102045260B1 (ko) 증강현실과 가상현실을 이용한 훈련용 응급처치 시뮬레이션 방법
US10438415B2 (en) Systems and methods for mixed reality medical training
Stansfield et al. Design and implementation of a virtual reality system and its application to training medical first responders
US12287911B2 (en) Virtual reality de-escalation tool for delivering electronic impulses to targets
Baus et al. Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: a review
US8382485B2 (en) Methods and apparatus for providing realistic medical training
US20240099934A1 (en) Wearable devices, systems, methods and architectures for sensory stimulation and manipulation, and physiological data acquisition and wearable haptic navigation system for use in navigating a user and or positioning a user's body along a safe egress path in obscured visibility environments
US20170193858A1 (en) Hemorrhage control trainer
US9067132B1 (en) Systems and methods for indirect control of processor enabled devices
CN103226390A (zh) 全景式火灾紧急逃生的虚拟现实系统
US12183215B2 (en) Simulated reality technologies for enhanced medical protocol training
US20190340943A1 (en) Simulated reality technologies for enhanced medical protocol training
US20230237921A1 (en) Mixed Reality Content Generation
CN111870918A (zh) 一种用于仿真格斗训练、娱乐及安防性质的假人
Taupiac et al. Ad-hoc study on soldiers calibration procedure in virtual reality
Militello et al. Handbook of augmented reality training design principles
Echeverria et al. KUMITRON: Artificial intelligence system to monitor karate fights that synchronize aerial images with physiological and inertial signals
US11893906B2 (en) Extended reality simulator with real-world interaction capabilities
Shi Design and realization of shooting training system for police force
Liu et al. The wide area virtual environment: a new paradigm for medical team training
WO2025094182A1 (en) Xr based evaluation system and method
KR102447460B1 (ko) Xr 기반 바늘 감압술 응급처치 교육훈련시스템
Akramjon et al. STUDY OF AN INTERACTIVE TRAINING SIMULATOR OF FIRE FIGHTERS-RESCUERS
CN120375678A (zh) 空降作战典型场景伤员救治模拟训练系统及方法
CN118098037A (zh) 一种心肺复苏智能培训系统及智能培训屋

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: ESG ELECKTRONIKSYSTEM- UND LOGISTIK-GMBH, GERMANY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GUTEKUNST, SOEREN BJOERN;KLEMMER, FRANK;SCHNEIDER, PETER;SIGNING DATES FROM 20221221 TO 20230116;REEL/FRAME:062501/0636

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: AWAITING TC RESP, ISSUE FEE PAYMENT VERIFIED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE