KR20090070325A - Emergency calling system and method based on multimodal information - Google Patents

Abstract

A multi-modal information base emergency recognition system and a method thereof for grasping the action of the old people or the independent person and generating the emergency call signal are provided to synthesize the video information, the sound information, and the operation information by the gravity sensor. A video processor(110) recognizes an operation by extracting and tracing a user from inputted image. A gravity sensor processor(120) recognizes swoon operation by sensing the operation of the user through a gravity sensor(121). A sound processor recognizes clearly the kind of the sound and speech word. A multimodal integrated recognizer determines emergency by integrating the information.

Description

Emergency calling system and method based on multimodal information}

The present invention relates to a multi-modal information-based emergency situation recognition system and method, and more particularly comprises an image processing unit, gravity sensor processing unit, sound processing unit, voice output unit, multi-modal integrated recognition unit and emergency situation processing unit for the elderly or single person The present invention relates to a multi-modal information-based emergency situation recognition system and method for making an emergency call to the outside in case of emergency.

The present invention relates to an emergency situation recognition system based on multimodal information using video, audio, and gravity sensors, and automatically recognizes an emergency situation that may occur when a resident lives in the house and performs processing appropriate to the situation. It is about a method.

As the elderly population increases due to the advancement of medical science and the increase of human lifespan, the number of the elderly and the handicapped people living alone are increasing. Accordingly, the silver industry considering the health of the elderly is developing, and technologies are developed to effectively identify the condition of the wards and cope with emergencies. Recently, a system has been developed that can be attached to the body of a ward to automatically measure the pulse rate and to automatically notify first aid to a medical center or relatives in case of an abnormality.

Conventional emergency recognition system employs a photoelectric system that detects the flow of blood flow from reflected light or transmitted light by irradiating light to a peripheral artery flowing through the finger or the neck of the human body, and measuring the pulse in proportion to the heart rate. However, this method has a problem that it is difficult to accurately measure the pulse rate because the blood flow decreases due to temperature decrease, and the pulse rate is easily changed by the light intensity difference due to sunlight.

In addition, the device that measures the pulse rate by detecting the heart rate of the human body with a vibration sensor is sensitive to extreme physical activity or external slight vibration, and it is necessary to apply pressure on the wrist or chest for accurate measurement. Difficult and inconvenient In addition, in the bathroom or toilet having a high probability that the elderly are faint or slip, there is a problem that can not detect the status of the elderly at a critical point.

In addition, there is a method of providing emergency contact buttons or emergency buttons in the main location of the home so that a single person can easily press in case of emergency, but there is a problem that the single person cannot grasp the emergency situation if the button is not operated directly. . In addition, by attaching a motion sensor to a location where a single person lives, the method of defining an emergency when a motion is not detected for a certain period of time is a simple function of the motion sensor, so that only the person who is active at home can be identified. And, there was a problem that can not distinguish the pause state of activities due to sleep and fainting.

In addition, a technology for comprehensively detecting images, sounds, and body movements to identify an accurate situation and to detect an emergency situation through an alternative device even if one device does not detect the situation, so as not to miss an emergency situation of a single person or an elderly person. There was no.

The present invention has been made to solve the problems of the prior art as described above, by accurately determining whether the elderly or the single person is missing by combining the image information, sound information, the operation information by the gravity sensor, to prevent malfunction due to the determination error An object of the present invention to provide a multi-modal information-based emergency situation recognition system and method capable of making an emergency rescue request quickly.

The object of the present invention is an image processing unit for receiving an image and detects and tracks the user from the image and recognize the fainting motion to deliver to the multi-modal integrated recognition unit; A gravity sensor processor for detecting a user's motion through a gravity sensor attached to a user's body, recognizing a faint motion, and transmitting the multimodal integrated recognition unit; A sound processor that receives a sound and distinguishes a voice from a non-voice, recognizes a spoken word for a voice, and recognizes a type of sound for a non-voice and transmits the multi-modal integrated recognition unit in case of an emergency; Integrate the information detected by the image processing unit, gravity sensor processing unit, sound processing unit to determine the emergency situation, the multi-modal integrated recognition to determine the emergency situation to the user through the voice output unit and to request an emergency call to the emergency situation processing unit in case of an emergency situation part; A voice output unit receiving the signal of the multi-modal integrated recognition unit and outputting a voice signal; And it is achieved by a multi-modal information-based emergency situation recognition system including an emergency situation processing unit for performing an emergency call to a medical institution, a rescue organization, etc. in response to the emergency call request of the multi-modal integrated recognition unit.

In addition, another object of the present invention is a first step of detecting the user's syncope motion in the image processing unit, gravity sensor processing unit or sound processing unit and transmits an emergency signal to the multi-modal integrated recognition unit; A second step of synchronizing the time at which the emergency occurred by processing the image, audio, and gravity information to judge the emergency situation; A third step of determining whether a fainting operation is performed based on the synchronized image information, voice information, and gravity information; A fourth step of re-confirming an emergency by generating a voice or warning sound; And if it is finally determined that the emergency situation through the re-confirmation process is achieved by a multi-modal information-based emergency situation recognition method comprising a fifth step of performing an emergency call to the outside.

Therefore, the multi-modal information-based emergency situation recognition system and method of the present invention automatically detects and accurately determines the emergency situation in the home that can occur at any time by combining image information, voice information, and operation information by a gravity sensor. It is effective to allow home-only resident in an emergency to receive quick emergency treatment or rescue without being left unattended. In addition, since three pieces of information are judged together, any one or two of the three detection devices do not miss an emergency situation, and thus, there is an advantage of accurately identifying an emergency situation without being missed.

The terms or words used in this specification and claims are not to be construed as being limited to their ordinary or dictionary meanings, and the inventors may appropriately define the concept of terms in order to best describe their invention. It should be interpreted as meaning and concept corresponding to the technical idea of the present invention based on the principle that the present invention.

Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiment of the present invention and do not represent all of the technical idea of the present invention, various modifications that can be replaced at the time of the present application It should be understood that there may be equivalents and variations.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an entire system according to the present invention. Referring to FIG. 1, the configuration of the present invention includes an image processing unit, a gravity sensor processing unit, an audio processing unit, an audio output unit, a multi-modal integrated recognition unit, and an emergency situation processing unit.

The image processor 100 detects and tracks a person from the image input unit 111 that receives an external image, the image preprocessor 112 that removes noise of the image, and corrects the change of the image according to the illumination change, and the person from the preprocessed image. It is configured to include an image recognition unit 113 for recognizing the operation.

The image input unit 111 receives a fisheye image of an RGB color model having an angle of view of 180 ° from a camera equipped with a fish-eye lens positioned in the center of a ceiling of a living room or a room.

The image preprocessing unit 112 first converts the RGB color model to the LUV color model to separate the illumination from the color to reduce the change of the image due to the change in illumination. After that, the average brightness of the input image is obtained, and when the average brightness changes from the previous average brightness above the threshold value, the image pixel value is corrected so that the average brightness does not change rapidly.

The image recognition unit 113 first detects a moving object by finding a pixel having a large difference between the preprocessed input image and the background image. In order to generate a background image, there should be no moving object in the input image at the beginning of the system operation. Since the background image changes continuously according to lighting changes or object movement, the background image is dynamically updated using an adaptive background modeling method based on a Gaussian mixture model.

Detecting a moving object extracts the size change, position change, and movement speed of the object. This information is used to determine if a moving object is a person and to recognize whether the person is moving, stationary, standing or walking and taking a fall. If there is no movement for a certain time after standing or walking down, the multi-modal integrated recognition unit 100 is notified of the emergency situation.

The gravity sensor processing unit 120 is composed of a gravity sensor unit 121 attached to the waist of the lone to detect the movement information of the body and the motion recognition unit 122 to recognize the body motion using the movement information of the gravity sensor.

Gravity sensor processing unit 120 is attached to the waist of the user (alone) measures the three-axis tilt information of x, y, z through the gravity sensor unit 121 for measuring the gravity, and using the measured tilt information The motion recognition unit 122 detects a user's movement to recognize an emergency situation.

The gravity sensor unit 121 receives inclination information of three axes x, y, and z using a gravity sensor. In order to correct the tilted axis of the gravity sensor based on the tilted information, the 3-axis tilted information is converted into the same value and the tilting degree of each of the x, y, and z axes is measured.

The motion recognition unit 122 recognizes motions such as walking, lying down, and fainting through the change of the inclination of each axis. The walking motion recognizes a case in which the tilt information of the gravity sensor is shaken by a user by a user, and the lying motion recognizes when the tilt information of the gravity sensor is kept constant after being turned on. In the case of the fainting operation, the shock wave is detected on three axes of the tilt information, and when the tilt information of the gravity sensor is kept in an inverted state, it is determined as an emergency situation and transmitted to the multimodal integrated recognition unit 100.

The sound processor 130 may include a sound input unit 131 that receives a voice and ambient sound of a lone, a sound preprocessor 132 that removes noise, detects an end point, and extracts features for identification and recognition; When the voice discriminating unit 133 and the voice discriminating unit 133 that distinguish the voice from the non-voice using the extracted feature are recognized as the voice, the word spoken using the extracted feature is recognized and not the voice. The voice / sound recognition unit 134 may recognize the type of sound using the extracted feature.

The sound input unit 131 receives sound information from a microphone attached to the ceiling, a wireless microphone attached to the lone, and transmits the sound information to the sound preprocessor 132. The acoustic preprocessor 132 first removes channel noise and background noise from the input sound, and performs endpoint detection to detect only a portion of the sound that is not silent. The detected sound portion is extracted feature for each frame for subsequent speech discrimination, speech recognition and sound recognition.

The voice discriminating unit 133 determines whether the input sound is a voice of a lone or a non-voice sound such as a music sound or a broken glass window by using the extracted feature.

The voice / sound recognition unit 134 recognizes whether the voice of the lone person is a request for processing an emergency situation when the determination result of the voice discriminating unit 133 is voice. If the voice of the lone person in the case of an emergency request, such as "contact the hospital quickly" to inform the multi-modal integrated recognition unit 100 that an emergency has occurred. However, if the voice of the lone person is a general conversation rather than an emergency request, the multimodal integrated recognition unit 100 is notified that the lone person's activity is normal.

When the determination result of the voice discriminating unit 133 is determined to be non-negative, an emergency situation sound is generated in the multi-modal integrated recognition unit 100 by grasping a sound related to an emergency situation such as a sound of broken glass or a cup being broken. Inform them. However, in the case of everyday noise informs the multimodal integrated recognition unit 100 that the activity of the lone is normal.

The multi-modal integrated recognition unit 100 generates a final result by combining the image recognition result, the current acoustic recognition state, and the recognition result of the gravity sensor. When the multimodal integrated recognition unit 100 determines that the emergency situation, the voice output unit 140 outputs a voice for reconfirming the abnormality to the lone person. When the lone responds, the sound processor 130 recognizes the voice and transmits the result to the multi-modal integrated recognition unit 100. When there is no response from the lone person or the lone person is notified of the emergency situation, the emergency call processing unit 160 sends a message informing that the emergency situation is to operate the emergency call system.

The voice output unit 140 receives a voice signal from the multi-modal integrated recognition unit 100 and outputs a voice message. The voice message is a message to reconfirm the presence of an abnormality in the behavior of a single person. In addition, when it is finally determined that the emergency situation, it is necessary to generate a rescue request message or a warning sound to know from the outside.

The emergency situation processing unit 150 receives an emergency signal from the multi-modal integrated recognition unit 100 when an emergency occurs, and contacts a medical institution such as a hospital, an emergency rescue organization, or a guardian. In addition, the patient's input, such as the presence or absence of the disease, blood pressure, pulse, etc. received from the medical institution can be automatically sent to the emergency rescue organization to facilitate the rescue.

2 is a flowchart illustrating an image recognition process according to the present invention. Referring to FIG. 2, an image is received through an image input unit to detect a background (S210). Background detection is to adaptively generate and maintain a background model using a Gaussian mixture model for each pixel. At the beginning of the system operation, a background image of at least 300 frames is input and compared to generate a background model.

After generating the background model, the pixel determined to be a shadow in the input image is found and removed (S220). In the shadow area, the brightness becomes dark and the original color is maintained, so it is determined whether it is the shadow area by using the brightness change and the color change amount.

 In addition, the isolation point is removed by finding a very small area among the connected areas of the foreground pixels (S230). Even in areas not included in the moving object, pixels can be classified as foreground pixels by noise. In this step, the areas caused by the noise are removed.

If the activity object appears in the input image after the background model is generated, the pixel is different from the background and classified as a foreground pixel. This step leaves only the foreground pixel and removes the remaining pixels.

When the active object is detected through the image, a shape approximation operation is performed to merge regions separated from each other (S240). If a part of the moving object has a color similar to the background, the part of the object may be classified as a background and the object may be separated. By applying this operation, the parts of the separated object may be merged. The contour is detected by tracking the outer points of the connected foreground pixel areas in order to detect the movement of the active object (S250).

If the contour is detected, the shape of the region is simplified by finding an ellipse that best surrounds the contour and mapping the shape of the contour into an ellipse (S260). After mapping the activity object to an ellipse (S260), by tracking the ellipse, a person is detected and tracked (S270). The tracking of the ellipse is performed by extracting the change of the size and position of the ellipse and the moving speed information.

From this information, an ellipse corresponding to a person is detected and tracked to determine whether a person exists in the image, whether it is moving or stationary, and whether the movement is similar to the fainting motion. In addition, the location of the living alone is determined by setting an area such as a toilet, a room, a front door, etc. in the input image.

Judging the movement of the lonelier (S280), if there is no abnormality of the lonelier continues to detect and track the person (S270), and in case of an emergency, the emergency situation is delivered to the multi-modal integrated recognition unit to instruct an emergency call. (S290).

3 is a flowchart illustrating a process of recognizing a gravity sensor processor according to the present invention. Referring to FIG. 3, voltages (tilt information) of each axis are measured in three axes x, y, and z (S310). The tilt of the gravity sensor with respect to the measured x, y, and z axes is measured and corrected by measuring the distortion of each axis in the microprocessor (S320), so that the sensor can measure the tilt even if it starts at an arbitrary tilt. The tilt of the x, y, and z axes is detected based on the tilt information of each axis after the correction (S330). By sequentially calculating the detected slopes, the motion is recognized and the tilt change and the impact information of each axis are extracted (S340).

From this information, it is determined whether the user is currently active, stationary, moving or lying down. In addition, when falling or hitting, it is determined whether the impact information through the gravity sensor for each axis is similar to the fainting motion (S350). If it is determined that the emergency situation is transmitted to the multi-modal emergency situation recognition unit that the emergency occurred (S360).

4 is a flowchart illustrating a voice / sound recognition process according to the present invention. Referring to FIG. 4, channel noise and background noise are removed from the sound input through the sound input unit (S410), and end point detection is performed to detect a start point and an end point of a portion having a non-silent sound (S420).

From the detected sound, features for each frame are extracted for speech discrimination, speech recognition, and sound recognition (S430), and Mel-Frequency Cepstral Coefficients (MFCC) is a feature for speech recognition and sound classification. Use

The melprecency capsular coefficients are obtained by taking Fourier transforms on the audio signals of the analysis section.Then, the sum of the magnitudes in each band is obtained by matching the triangular filter banks according to the mel scale with the obtained spectrums. It is known to represent human auditory characteristics based on spectrum as a feature vector obtained by taking a logarithm to, and performing discrete cosine transform.

Features for discriminating voice and non-voice include Modulation Energy (ME), Cepstral Flux (CF) and Mel-Frequency Cepstrum Modulation Energy (MCME) Find feature vectors, etc. The remaining feature vectors are the feature vectors used to represent the changing aspect of the acoustic feature over a wider time interval, provided that the melprincency capsular coefficients reflect the acoustic characteristics of the audio signal present at any time. .

The modulation energy is a feature vector that represents the degree of change over time of the spectrum by performing a Fourier transform on the vector sequence of output values of the filter banks that reflect the spectrum. The spectral flux represents the amount of change in the cepstrum over time by calculating the mean spectral distance (the square of the difference between the spectral components) of several frames from an adjacent frame to a slightly distant frame. .

The melprecipitation capsular modulation energy performs the Fourier transform in the cepstral region, whereas the modulation energy performs the Fourier transform based on the spectrum. The spectrum of the audio signal is sensitive to the change in the detailed spectrum due to the change of pitch such as pitch harmonic component, and thus there is a problem that the discrimination performance is lower than that of the spectrum-based method. Therefore, the Fourier transform is performed using a cepstrum with less cross-correlation than the spectrum to measure the change pattern over time, thereby showing more reliable discrimination performance.

If the extracted voice / non-voice discrimination feature is used, it may be determined whether the input sound is voice or non-voice sound (S440).

If it is determined that the determined result (S440) is speech, speech recognition is performed (S450). If the voice of the lone person is a stored emergency occurrence message such as “please contact the hospital”, it is determined as an emergency (S470) and the multimodal integrated recognition unit transmits that an emergency occurred (S480). However, when the voice of the lone is a normal situation instead of an emergency request, the sound is continuously input through the sound input unit (S410), and the data is processed. Speech determination result (S440) If it is determined that the voice is non-voice is recognized (S460). In the case of a sound related to an emergency situation such as the sound of a broken glass or a broken glass (S470), the multimodal integrated recognition unit notifies that an emergency has occurred (S480). However, if the general sound is repeated (S410).

5 is a flowchart illustrating a multimodal integrated recognition process according to the present invention. Referring to FIG. 5, the multi-modal integrated recognition unit receives an emergency occurrence from one or more of an image processor, a gravity sensor processor, or a sound processor (S510) and stores it in a database. The multi-modal integrated recognition unit processes the image, voice, and gravity information to judge the emergency situation and performs a preprocessing process to synchronize the time of occurrence of the emergency situation (S520).

Based on the synchronized image information, voice information and gravity information at the same time, the determination of the lone lone is a real fainting motion or a normal motion (S530). Determining the synchro motion includes state information such as the position, movement, and posture of the lone, from motion information, motion information with or without motion from gravity information, and state information such as voice and noise from sound. The recognition rate is increased by extracting. If it is determined as a fainting operation, the emergency situation is reconfirmed with a voice or a warning sound that can ventilate the surrounding person through the voice output unit (S540).

If there is no response for a certain period of time even after the warning or the emergency signal is confirmed from the sound processor, the emergency situation processor is called immediately to notify the emergency (S550).

Although the present invention has been shown and described with reference to the preferred embodiments as described above, it is not limited to the above embodiments and those skilled in the art without departing from the spirit of the present invention. Various changes and modifications will be possible.

The present invention can be used for an emergency call system for quickly determining an emergency situation including the fainting by combining image information, audio information, and motion information.

  1 is a block diagram showing an overall system according to the present invention;

  2 is a flowchart illustrating an image recognition process according to the present invention;

  3 is a flowchart illustrating a process of recognizing a gravity sensor processor according to the present invention;

  4 is a flowchart illustrating a voice / sound recognition process according to the present invention;

  5 is a flowchart illustrating a multimodal integrated recognition process according to the present invention.

<Explanation of symbols for the main parts of the drawings>

100: multi-modal integrated recognition unit 110: image processing unit

120: gravity sensor processing unit 130: sound processing unit

140: voice output unit 150: emergency situation processing unit

Claims (6)

An image processing unit which receives an image and detects and tracks a user from the image, recognizes the fainting motion, and delivers the multimodal integrated recognition unit; A gravity sensor processor for detecting a user's motion through a gravity sensor attached to a user's body, recognizing a faint motion, and transmitting the multimodal integrated recognition unit; A sound processor that receives a sound and distinguishes a voice from a non-voice, recognizes a spoken word for a voice, and recognizes a type of sound for a non-voice and transmits the multi-modal integrated recognition unit in case of an emergency; And Integrate the information detected by any one or more of the image processing unit, gravity sensor processing unit, sound processing unit to determine the emergency situation, reconfirm the emergency situation to the user through the audio output unit, and when an emergency occurs emergency call request signal to the emergency situation processing unit Multimodal integrated recognition unit for transmission Multi-modal information-based emergency situation recognition system comprising a. The method of claim 1, A voice output unit receiving the signal of the multi-modal integrated recognition unit and outputting a voice signal; And An emergency situation processing unit for receiving an emergency call request signal from the multi-modal integrated recognition unit and transmits an emergency call signal to an external organization. Multi-modal information-based emergency situation recognition system further comprises. The image processing apparatus of claim 1, wherein the image processor comprises: Multi-modal information-based emergency situation recognition system that detects syncope and recognizes emergency situations by mapping moving people to ellipses from fish-eye images acquired using fish-eye lenses and measuring size changes, position changes, and movement speeds of ellipses. The method of claim 1, wherein the gravity sensor processing unit, Multi-modal information-based emergency situation recognition system that can measure the three-axis information of x, y, z by gravitational sensor attached to the user, and detect the posture and faint motion of the user by grasping the tilt information of each axis according to the movement. The method of claim 1, wherein the sound processing unit, Multi-modal information-based emergency situation recognition system that can distinguish emergency from speech by distinguishing between speech and non-voice and recognizing speech words for voice and sound type for non-voice. A first step of detecting a fainting motion of a user in at least one of an image processor, a gravity sensor processor, and an audio processor and transmitting an emergency signal to a multi-modal integrated recognition unit; A second step of synchronizing the time at which the emergency occurred by processing the image, audio, and gravity information to judge the emergency situation; A third step of determining whether a fainting operation is performed based on the synchronized image information, sound information, and gravity information; A fourth step of re-confirming an emergency by generating a voice or warning sound; And A fifth step of performing an emergency call to the outside when the emergency situation is finally determined through the reconfirmation process; Multi-modal information-based emergency situation recognition method comprising a.

Images