KR20230008406A - Hand gesture recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training system - Google Patents

Abstract

The present invention relates to a hand gesture and motion recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training system. The cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training system, according to the present invention, comprises: smart glasses; a module; a storage device (DB); and a control unit. According to the present invention, the overall AED usage rate can be increased.

Description

Hand gesture recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training system}

The present invention relates to hand shape and motion recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training systems. A system using image analysis solves the obstacles that make witnesses, including first-time witnesses, reluctant to use AEDs by simultaneously checking the AED usage guide data, so that first aid can be administered to the patient without time lag while receiving guidance. This increases the overall AED usage rate, thereby increasing the patient's survival rate, and also recognizes the user's hand shape on the smart glasses to accurately and conveniently evaluate the performance result. Hand shape and motion recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) is about an AR education system.

If a cardiac arrest patient occurs in a place currently classified as a multi-use facility (a large-scale building used by an unspecified number of people), it is possible to conduct emergency rescue using an AED installed so that it can be easily found inside the facility. However, when a cardiac arrest patient occurs, the first witness does not use an AED. Instead, the witness informs the facility's trained staff of the situation, who then proceed with the first aid. It is confirmed that the use of the AED, which must be used on the patient in the shortest time by the first witness, is hindered by several factors.

Currently, in our society, safety education including CPR (cardiopulmonary resuscitation) is conducted to teach emergency rescue methods from elementary school students to adults. In the case of patients with cardiac arrest, the importance of emergency measures is very high because the survival rate changes greatly with or without emergency measures for a short time of about 5 minutes from the onset of symptoms. Therefore, in the case of multi-use facilities used by an unspecified number of people, AEDs are installed in case of cardiac arrest patients, and since this is legally mandated, there are very few buildings without them. However, the problem lies in the usage rate, not the installation rate of AEDs.

As a factor that makes people hesitant to actively use AEDs for first aid, 'don't know how to use' was the highest at 65% of the total. Even if they want to use the AED, they do not know exactly how to do it, so the main reason is the fear of legal responsibility that will arise later and adversely affecting the patient due to the wrong method of use.

For patients with cardiac arrest, the survival rate after transfer and the severity of the aftereffects are determined to a large extent depending on how quickly and accurately the initial response was provided by the discoverers. Therefore, for cardiac arrest patients, it is the most desirable scenario with the highest survival rate if the correct AED use measures are executed in the shortest time by the first witness. However, when analyzing the cases in which AEDs are used in real situations, the early discoverers were actively involved in patient lifesaving activities, but showed a tendency to be somewhat hesitant to use the most important AEDs, and waited for facility staff who had received training to use them. It was not that he took the initiative to take emergency measures.

There were two major factors that prevented discoverers from using AEDs easily. These factors are problems that need to be recognized and resolved as factors hindering the use of AED, with 'not knowing how to use' and 'burden of responsibility due to misuse', respectively, and even trained personnel remember the contents but did not dare to use them. This is the main reason why there is an intention to use it, but not actually using it.

And, as a conventional patented technology, Registered Patent Publication Registration No. 10-1255088 includes a central processing unit 10 installed in an AED storage box and performing overall control functions, and a user's fingerprint information sent to the central processing unit 10 to provide a fingerprint A fingerprint recognition device 20 for recognizing and storing a face, a face recognition device 30 for sending facial information of a user to the central processing unit 10 to recognize and store a face, and the face recognition device 30 ) facial information recognition and illumination device 40 for providing lighting to guide the location of the AED storage box, and when the user's fingerprint and facial information are stored through the central processing unit 10, the door of the AED storage box A system for using an automated external defibrillator storage box using user recognition, characterized in that it consists of a locking device (50) for releasing the lock and a communication device (70) for notifying a manager and emergency personnel when the automated external defibrillator storage box is opened. this is disclosed.

In addition, Patent Publication No. 10-2019-0011480 discloses an electrocardiogram measuring unit, a defibrillation electric shock unit, a battery, a control unit, a wireless communication unit, a defibrillator body including a defibrillation electrode, and a defibrillator body and wireless A smart device including a wireless communication unit for connection, a display, and a control unit for controlling operation of the defibrillator by analyzing the heart waveform transmitted from the defibrillator main body;

The defibrillator body is configured to perform a defibrillation function according to a control command of the smart device when wireless communication is connected with the smart device, and the smart device is configured to selectively transmit a simulation command and a defibrillation execution command to the defibrillator body. When a simulation command is received, the defibrillator body transmits an electrocardiogram measurement signal to the smart device, and the smart device analyzes the received measurement signal to display a message for determining whether or not the attachment state of the defibrillation electrodes is good or bad. configured to display, and when the defibrillation execution command is received, the defibrillator body transmits an electrocardiogram measurement signal to a smart device, and the smart device analyzes the received measurement signal to determine whether or not the attachment state of the defibrillation electrodes is good or bad A message is displayed on the display, the measured signal is analyzed, and a command to apply an electric shock to the defibrillation electrode is transmitted to the defibrillator main body, and the defibrillator main body is configured to receive the electric shock application command to the defibrillation electrode and apply the electric shock. A defibrillator control device using a smart device is disclosed.

In addition, Registered Patent Publication No. 10-1780214 discloses an emergency rescue system using an automated external defibrillator. However, since the above conventional technologies do not show configurations for measuring a patient's condition in real time while the defibrillator is in use and feeding back appropriate measures accordingly, the risk of misuse by the user may arise.

In addition, conventional technologies wear smart glasses and use 'Image Tracking' technology or 'object tracking' to recognize the location of a dummy, AED, etc., and to detect 3D objects related to education seen through smart glasses The learning guide animation is superimposed on the actual dummy and the AED in the correct position, and the learner proceeds with learning by following this three-dimensional guide and voice guide, but if the practice is not accurately implemented, the sensor mounted on the dummy and AED operates. There was a disadvantage of not doing it and repeating learning until it was properly practiced.

Therefore, the present invention has been devised to solve the above problems, and in the mixed reality environment using smart glasses, the user is guided by simultaneously checking the patient and the AED usage guide data within his field of view, and at the same time Since first aid can be performed on the patient without time lag, the system using image analysis solves the obstacle that makes witnesses, including first witnesses, reluctant to use AEDs. It works. In addition, it is intended to provide an AR training system for CPR and AED based on hand shape and motion recognition that recognizes the user's hand shape through smart glasses and evaluates the result accurately and conveniently.

The present invention relates to hand shape and motion recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training systems, smart glasses that can collect on-site image data by following the user's gaze through a camera module;

A module for generating information required by the system using the image data collected through the camera module of the smart glasses through the image processing module;

a storage device (DB) for preserving the patient's bio information, personal information, and medical history information entered in advance;

And a control unit; characterized in that it consists of.

In addition, the control unit may recognize a user's hand shape through the smart glasses and evaluate a performance result.

Therefore, the present invention utilizes an image analysis technique to recognize the site, informs and instructs the user on the process of using the AED procedurally through an audio-visual UI, induces the user to use the AED correctly even if he or she has no knowledge of the AED, and provides instructions By using the use of Joe and the reliability of the system, it reduces the pressure on users to take responsibility and wrong actions.

In a mixed reality environment using smart glasses, the user is guided by checking the patient and AED usage guide materials at the same time within his or her field of view, and at the same time, it is possible to administer first aid to the patient without time lag, so witnesses including the first witness There is a remarkable effect of increasing the survival rate of patients by increasing the overall AED use rate by solving the obstacles that make people reluctant to use AEDs with a system using image analysis. In addition, there is a remarkable effect of accurately and conveniently evaluating the performance result by recognizing the user's hand shape in the smart glasses.

1 is a motion picture of hand shape and motion recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training systems
Figure 2 is a photograph confirming the response of the patient of the present invention
3 is a detailed photograph of the use of the present invention

The present invention relates to hand shape and motion recognition-based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training systems, smart glasses that can collect on-site image data by following the user's gaze through a camera module;

An OpenCV module that generates information required by the system by using the image processing module from the image data collected through the camera module of the smart glasses;

a storage device (DB) for preserving the patient's bio information, personal information, and medical history information entered in advance;

And a control unit; characterized in that it consists of.

In addition, the control unit is characterized in that by using the camera module of the smart glasses worn by the user to obtain the image data of the site.

In addition, the control unit may recognize a user's hand shape through the smart glasses and evaluate a performance result.

In addition, the usage guide uses digital data composed of audio-visual data that mixes visual data such as 3D (AR, MR) images and video images with audio data including voice guidance, and the control unit uses digital data so that users can quickly recognize the guidance data. The materials are characterized in that the user can imitate the motion to be acted upon by outputting virtual hand graphics to the output means.

The present invention will be described in detail with reference to the accompanying drawings.

The system control unit provides and guides the user on how to use the AED in real time so that even inexperienced users who do not know how to use the AED can use the AED correctly. It promotes the use of AED by reducing the burden and tension by guiding that hesitating to use the AED will only follow the guidance of the system.

Since the system needs to inform the user of how to use the AED in real time, it needs to be equally aware of the environment the user is viewing. Therefore, it is solved by integrating the real world where patients are located and digital visual data for AED guidance using a mixed reality (MR) environment to guide users on the use of AED intuitively and effectively. At this time, the user wears the smart glasses (MR device) and uses the camera module of the device to allow the system to acquire the image data of the site. The system recognizes the patient's face, identifies the patient's medical history and identity registered in the DB, and at the same time recognizes surrounding objects (items used for emergency measures such as AEDs) through the OpenCV image analysis process.

After the system recognizes the scene, it guides the user on how to use the AED. Instructions for use use digital materials composed of audio-visual materials that combine visual materials such as videos and photos with audio materials including audio instructions. Due to the nature of first aid, speed and accuracy are required, so digital data outputs graphics such as virtual hands so that users can quickly recognize the guidance materials so that users can follow them. By analyzing the patient's face, it is possible to check the personal information and medical history registered in the DB, saving time spent on identifying the patient after transfer.

In a mixed reality environment using smart glasses, the user can receive guidance by simultaneously checking the patient and AED usage guide materials within his field of view, and at the same time, he can perform first aid to the patient without time lag.

Referring to the main components of the present invention, the smart glasses (MR devices) can collect field image data along the user's gaze through the camera module of the smart glasses. It recognizes the patient and compares it with the DB to find out his identity and medical history information. At the same time, through the mixed reality display, the user is instructed to use the AED according to the procedure using digital data.

The OpenCV module uses the image data collected through the camera module of the smart glasses and the image processing module written in C# code to create the information required by the system. The result obtained through the image processing is to specify the coordinates of the patient's face, the patient's thoracic pressure area (CPR application area), and two AED electrical stimulation pad attachment locations (under the right collarbone, left side of the 6th rib). Provides digital materials for guidance on the use of AEDs.

The storage device (DB) stores the patient's bio information, personal information, and medical history information entered in advance, and allows the smart glasses to identify the patient's identity by comparing the face recognition information of the patient in the field with the information stored in the storage. which is stored.

As another embodiment, the control unit of the system that recognizes the situation through the camera module of the smart glasses activates an alarm unit to inform the user of an alarm sound when the user misuses it. Or correction of wrong actions is announced through a speaker, and in the case of using a defibrillator, the heart rate connected to the patient is input to the system by the heart rate sensor, camera module, etc., which are the input means of the AED system, and input If the value differs greatly from the set value of the system, the control unit issues a warning or presents the correct answer by voice so that the user can correct it immediately. At this time, the patient's real-time condition analysis and voice instructions use AI, not humans.

The present invention is a configuration for recognizing the location of a dummy, AED, etc. by wearing smart glasses and using 'Image Tracking' technology or 'Object Tracking', so that learners can learn and train (practice). As a method of determining whether the performance is performed correctly, a method in which the smart glasses recognize the user's hand shape and evaluate the performance result can be applied. In the past, 'Magic Leap One' was used as a smart glass, and as a sensor You can also use Arduino. On the other hand, as a preliminary configuration, the present invention connects Arduino sensor devices to a dummy, and the operating value of the sensor generated when the learner touches and manipulates the dummy is transferred to 'Magic Leap One' through the Bluetooth (one of the Arduino modules) module. It is passed on. Through this, learners can know if they are learning properly, and if they have practiced appropriately, they can move on to the next one, and they can keep repeating it until the practice is not done properly.

The method of evaluating the performance result by recognizing the hand shape of the user by the smart glasses of the present invention is as follows.

Step 1 is

After checking the patient's condition

This is a patient awareness screen.

'Hello, hello. Are you okay? Can you hear me?” and asks if he is conscious.

'Hello Hello. Are you okay? Can you hear me?', an animation of a 3D hand tapping on the dummy's shoulder is shown at the same time.

3D hand position matching by recognizing dummy by HoloLens object recognition technology

Learning proceeds properly by recognizing the learner's hand? make sure

Step 2 is

Expose the patient's chest.

The video shows the patient taking off the top.

Learning proceeds properly by recognizing the learner's hand? make sure

Step 3 is

Place the heel of one palm halfway below the patient's breastbone.

Place your other palm on the back of that hand.

Interlock the fingers so that only the elbow touches the patient's chest.

According to subtitles and voice guidance, the hand shape is sequentially implemented in 3D.

By recognizing the learner's hand, it is possible to confirm that the learning has proceeded properly.

Therefore, the present invention utilizes an image analysis technique to recognize the site, informs and instructs the user on the process of using the AED procedurally through an audio-visual UI, induces the user to use the AED correctly even if he or she has no knowledge of the AED, and provides instructions Using Joe and the reliability of the system, the pressure on responsibility and wrong actions felt by the user is alleviated. Since first aid can be administered to the patient without time lag, the system using image analysis can solve the obstacles that make witnesses, including the first witness, reluctant to use AEDs, with a system using image analysis, thereby increasing the overall AED use rate. There is a significant effect of increasing the survival rate of

Claims (3)

Smart glasses that can collect on-site image data along the user's line of sight through the camera module;
A module for generating information required by the system using the image data collected through the camera module of the smart glasses through the image processing module;
a storage device (DB) for preserving the patient's bio information, personal information, and medical history information entered in advance;
And a controller; hand shape, motion recognition based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR training system characterized by consisting of The method of claim 1, wherein the control unit obtains on-site image data using a camera module of smart glasses worn by the user. AR training system The hand shape and motion recognition based cardiopulmonary resuscitation (CPR) and automated external defibrillator (AED) AR education system according to claim 1, wherein the control unit recognizes the user's hand shape through the smart glasses and evaluates the performance result.

Images