KR102303633B1 - Smart Home Care System Using Emotional and Status Analysing of Compansion Animal - Google Patents

Abstract

The smart home care system using the emotion and condition analysis of companion animals checks the sound, behavior, and heartbeat of companion animals to monitor the status of the elderly or intruders in the house. It has the effect of monitoring intruders.
The present invention can check the abnormal behavior of the companion animal and notify the guardian, so that the elderly can arrive to the hospital in a short time, thereby saving lives. It can be reported directly, which has the effect of improving the security effect of the house.

Description

Smart Home Care System Using Emotional and Status Analysis of Compansion Animal

The present invention relates to a smart home care system, and more particularly, to a smart home care system using emotion and state analysis of companion animals that detects abnormal behavior of companion animals and monitors the elderly or intruders in the house.

Recently, as the number of single-person households increases, pets are changing to companion animals, which is the concept of family, and the number of households raising companion animals is also increasing.

These companion animals are often raised as a concept to replace the loneliness of the elderly due to aging and nuclear families.

Due to the aging population, the elderly spend more time alone at home, and accordingly, when accidents due to cerebrovascular disease, heart attack, or falls occur, it is often difficult to respond quickly.

In particular, it is widely known that the golden time of arriving at the hospital after onset of serious diseases such as cerebrovascular disease and heart attack is very important for saving lives.

However, the elderly and infirm are often left alone for a long time, so it takes a lot of time to get to the hospital even if a serious disease occurs.

Korean Patent No. 10-1907383

In order to solve such a problem, the present invention provides a smart home care system using emotion and state analysis of companion animals that monitors the state of the elderly or intruders in the house by checking the sound, behavior, and heartbeat of the companion animal. There is a purpose.

A smart home care system using the emotion and state analysis of companion animals according to the features of the present invention for achieving the above object,

A microphone attached to the companion animal and receiving the cry of the companion animal as an acoustic signal; a sensor device having a detection unit; and

The type of the cry is received by receiving an acoustic signal, a motion detection signal, and a biosignal from the sensor device, extracting an acoustic characteristic vector of the received acoustic signal, and performing machine learning using the extracted acoustic characteristic vector as input data. In the pattern state table stored in the memory unit, a combination condition of the emotional state information, the motion detection signal, and the biosignal is generated by generating a sound pattern for classifying and a smart home care server that searches and extracts monitoring information of a target monitored by the companion animal matching the combination condition.

The pattern state table may form a plurality of combination conditions and set a plurality of matching monitoring information corresponding to each combination condition.

According to the above-described configuration, the present invention has an effect that the companion animal can monitor the state of the elderly or infirm in response to a person or environment or monitor an intruder.

The present invention has a home care effect that can save lives by checking the abnormal behavior of the companion animal and notifying the guardian, so that the elderly and the infirm can arrive to the hospital in a short time.

The present invention has the effect of improving the crime prevention effect in the house by checking the abnormal behavior of the companion animal and directly reporting the intrusion to the police.

1 is a diagram showing the configuration of a smart home care system using the emotion and state analysis of a companion animal according to a first embodiment of the present invention.
2 is a diagram showing the configuration of a smart home care system using the emotion and state analysis of a companion animal according to a second embodiment of the present invention.
3 is a view showing the internal configuration of the sensor device according to the first and second embodiments of the present invention.
4 is a view showing the internal configuration of the smart home care server according to the first and second embodiments of the present invention.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

1 is a diagram showing the configuration of a smart home care system using an analysis of emotions and states of companion animals according to a first embodiment of the present invention, and FIG. 2 is a diagram showing emotions and states of companion animals according to a second embodiment of the present invention. It is a diagram showing the configuration of a smart home care system using analysis, FIG. 3 is a diagram showing the internal configuration of the sensor device according to the first and second embodiments of the present invention, and FIG. 4 is the first and second embodiments of the present invention. It is a diagram showing the internal configuration of the smart home care server according to the

The smart home care system 100 using the emotion and state analysis of companion animals according to the first and second embodiments of the present invention includes a sensor device 110 worn on a companion animal, a smart home care server 120, and the elderly ( 10) includes a guardian terminal 130 possessed by the guardian.

The sensor device 110 is composed of clothing worn by the companion animal, and includes a microphone 111 , a motion sensor unit 112 , a GPS module 113 , a sensor control unit 114 , a biosignal detection unit 115 , and a wireless communication unit 116 . ), a speaker 117 and a camera unit 118 .

The microphone 111 receives an acoustic signal such as a cry of a companion animal or an ambient sound.

The motion sensor unit 112 includes a gyro sensor and an acceleration sensor for detecting the movement of the companion animal, and transmits a motion detection signal detected by the gyro sensor and the acceleration sensor to the sensor controller 114 .

The GPS module 113 periodically receives and calculates the current location information of the companion animal from the GPS satellite, and accordingly measures the current location information of the companion animal to determine the moving path of the companion animal.

The biosignal detector 115 is mounted on one of the neck, back, and thigh of the companion animal to measure the biosignal of the companion animal's pulse.

The sensor controller 114 transmits the sound signal input from the microphone 111 to the smart home care server 120 through the wireless communication unit 116 .

The sensor controller 114 transmits the location information of the companion animal measured by the GPS module 113, the biosignal measured by the biosignal detector 115, and the motion detection signal measured by the motion sensor 112 to the wireless communication unit ( 116) through the smart home care server 120.

The sensor controller 114 may be connected to the guardian terminal 130 or the smart home care server 120 through the communication network 101 , and may transmit/receive data.

The speaker 117 may be connected to the guardian terminal 130 under the control of the sensor controller 114 to output a guardian voice provided from the guardian terminal 130 .

The camera unit 118 captures an image of the space in which the companion animal is active, and acquires image information by photographing the elderly person 10 or the intruder 11 in the view of the companion animal.

The guardian terminal 130 is exemplified by, but not limited to, a smart phone, a feature phone, a tablet PC, a slate PC, a desktop PC, a notebook PC Computers (notebook computers), digital broadcasting terminals, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation (Navigation), MP3P (MPEG layer 3 Player), etc. may apply, and all terminals capable of mobile communication may be included.

The guardian terminal 130 installs the smart home care application, communicates with the sensor device 110 or the smart home care server 120 , controls the sensor device 110 or provides a smart care service from the smart home care server 120 . can be provided.

The communication network 101 may include a wired or wireless communication network such as Wi-Fi, UMTS/GPRS, or Ethernet. Communication network 101 is magnetic secure transmission (Magnetic Secure Transmission), RFID (Radio Frequency IDentification), NFC (Near Field Communication), ZigBee (ZigBee), Z-Wave, Bluetooth (Bluetooth), low-power Bluetooth (Bluetooth Low Energy), It may include a local area network, such as infrared communication (InfraRed communication).

The guardian terminal 130 may be connected to the video call terminal 102 through the communication network 101 . When the guardian terminal 130 is connected to the video call terminal 102 , data may be exchanged with each other through the communication network 101 . Here, the video call terminal 102 may be installed in a specific place in the house. Of course, the video call terminal 102 may be installed in a companion animal.

The video call terminal 102 may receive a video call brokerage request from the guardian terminal 130 .

The smart home care server 120 according to the first and second embodiments of the present invention includes a signal receiving unit 121 , an acoustic feature extracting unit 122 , an emotional state analyzing unit 123 , a control unit 124 , and a movement pattern database unit. 125 , a signal transmission unit 126 , a memory unit 127 , and a video call processing unit 128 .

The signal receiver 121 receives the location information of the companion animal, the biosignal, and the motion detection signal.

The signal receiving unit 121 receives an acoustic signal of the cry of a companion animal to be detected.

The acoustic feature extraction unit 122 is a process of converting a one-dimensional acoustic signal into a log-mel spectrogram, which is a two-dimensional acoustic feature having a frequency axis and a time axis.

The acoustic feature extraction unit 122 receives a one-dimensional sound signal from the signal receiving unit 121 and passes the one-dimensional sound signal through a Short Time Fourier Transform (STFT) transform unit and a Mel Filter Bank unit. After that, a logarithmic operation is performed and converted into a log-mel spectrogram, which is a two-dimensional acoustic characteristic.

The STFT conversion module 122a may analyze the frequency of the sound signal after segmenting the sound signal using a window of a certain length.

The Mel filter bank unit 122b may extract a characteristic value of an input sound signal by using the frequency analyzed by the STFT conversion module 122a by a Mel Frequency Cepstral Coefficient (MFCC) method.

The acoustic feature extraction unit 122 calculates a specific section of the acoustic signal sampled at a specific frequency through STFT transformation, and passes the calculated specific section of the acoustic signal through a Mel Filter Bank to obtain the Mel of a specific frequency band. A filter bank value (Mel Filterbank) is obtained, and a logarithmic operation is performed on the obtained Mel filter bank value, which is converted into a log-mel spectrogram, which is a two-dimensional acoustic characteristic.

The emotional state analyzer 123 extracts emotional state information of the companion animal by performing machine learning using the acoustic feature vector extracted by the acoustic feature extracting unit 122 as input data.

The emotional state analyzer 123 is a technique for finding various acoustic features using an acoustic feature vector using a pattern recognition technique, and is configured with a Long Short-Term Memory (LSTM) of a Recurrent Neural Network (RNN) model.

The pattern recognition technique is a prediction method using an artificial neural network, and when the result value of the output layer is predicted from the input layer, the input value can be predicted from the result values in the learning process. Since the artificial neural network does not have a one-to-one correspondence between the input value and the output value, it is impossible to restore the input layer as an output layer as it is, but the output data calculated from the result value by the Backpropagation (Backpropa) algorithm in consideration of the prediction algorithm If is different from the original input data, it can be considered that the prediction of the artificial neural network is inaccurate. Therefore, learning is trained by changing the prediction coefficients so that the output data calculated under the constraint is similar to the original input data.

A deep neural network refers to a neural network composed of several layers among neural network algorithms. A layer consists of several nodes, and actual calculations are performed at the nodes, and this calculation process is designed to mimic the processes occurring in the neurons constituting the human neural network. A typical artificial neural network is divided into an input layer, a hidden layer, and an output layer. becomes the input of the output layer, and the output of the output layer becomes the final output.

The emotional state analysis unit 123 uses a prediction deep neural network that receives input data from an input layer and outputs a predicted value to a buffer of an output layer, and the structure or shape of the prediction deep neural network is not limited, and a representative method is a deep neural network (DNN). Network), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), etc. A deep neural network of various structures can be constructed by constructing a predictive deep neural network by combining each neural network.

The emotional state analyzer 123 processes the cry signal of the companion animal, generates a sound pattern for classifying the type of cry by machine learning, and sets the emotional state information corresponding to the generated sound pattern to Peace. , Curiosity, Warning, Woof, Fear, etc.

The emotional state analyzer 123 may infer emotion by comparing the pattern of the acoustic feature vector extracted by the acoustic feature extractor 122 with the pattern of the reference acoustic feature vector. To this end, the emotional state analyzer 123 recognizes emotional state information about the companion animal by determining the most similar emotion using a pattern comparison algorithm such as DNN, CNN, or RNN.

The emotional state analyzer 123 may selectively infer the emotion of the companion animal from among emotional state information such as peace, curiosity, warning, anger, fear, and the like.

The movement pattern database unit 125 pre-stores movement patterns in response to movement detection signals detected from the gyro sensor and the acceleration sensor.

The motion pattern database unit 125 generates and stores a pattern by signal processing the motion detection signal of the companion animal.

The control unit 124 analyzes the received motion detection signal in conjunction with the motion pattern database unit 125 and extracts which type of motion pattern it is from among the previously stored motion patterns.

The control unit 124 searches the pattern state table stored in the memory unit 127 for a combination condition including the emotional state information received from the emotional state analysis unit 123, the extracted movement pattern, and the bio-signal received from the sensor input unit, and matches it. Extract monitoring information.

Here, the target of the monitoring information includes the elderly 10 or the intruder 11 monitored by the companion animal.

The pattern state table forms a plurality of combination conditions, and sets a plurality of matching monitoring information corresponding to each combination condition.

Here, the emotional state information generates a sound pattern for classifying the type of cry by processing the cry signal of a companion animal, and sets the state information corresponding to the generated sound pattern to Peace, Curiosity, It is expressed as Warning, Woof, and Fear.

The movement pattern consists of the movement size, change, movement speed, and repetition period of the companion animal to represent the movement of the companion animal. The vital signal represents the heart rate per minute.

As shown in Table 1 below, the pattern state table sets the monitoring information to the guardian notification state when the emotional state information is a warning, the movement pattern is '1', and the biosignal is heart rate 150 per minute, and the emotional state information is angry and movement pattern. If '2', biosignal is set to 200 heart rate per minute, monitoring information is set to outsider intrusion, emotional state information is set to serenity, movement pattern is set to '3', and biosignal is set to 100 or less heart rate per minute, monitoring information is converted to daily life. If the emotional state information is set to curiosity, the movement pattern is '4', and the biosignal is set to 120 or less heart rate per minute, the monitoring information is set to daily life.

The movement pattern is set to codes such as '1', '2', '3', and '4', and each code represents the movement of the companion animal. Code 1 is a state in which the companion animal roams around the elderly and weak person 10 in a state in which the elderly person 10 is collapsed, and code 2 is a movement state in which the companion animal reciprocates in a straight direction when the thief 11 breaks in. , Codes 3 and 4 indicate that the companion animal does not move in one place or moves less than a certain distance.

When the monitoring information is a guardian notification or an outsider intrusion, since the companion animal is in an excited state, it indicates a high heart rate that deviates from the normal heart rate of the companion animal of 80 to 120.

The guardian notification indicates the state in which the elderly (10) collapsed due to an accident due to vascular disease, heart attack, or a fall in the house, and external intrusion indicates the state in which a thief (11) broke into the house, and daily life is the elderly (10) It may indicate a state that is stationary in one place, or it may indicate a state in which the elderly and infirm 10 walk in the house.

The pattern state table may set various monitoring information by matching emotional state information, movement patterns, and biosignals, and may increase or decrease as necessary.

When it is determined that the extracted monitoring information is 'guardian notification', the control unit 124 generates an emergency notification signal and transmits it to the guardian terminal 130 through the communication network 101 using the signal transmission unit 126, or the guardian terminal 130 and the emergency medical server 140 may be simultaneously transmitted.

When it is determined that the extracted monitoring information is 'intrusion of an outsider', the control unit 124 generates a thief intrusion notification signal and transmits it to the guardian terminal 130 through the communication network 101 using the signal transmission unit 126, or It can be transmitted to the guardian terminal 130 and the police server 150 at the same time.

When the control unit 124 determines that the extracted monitoring information is 'guardian notification' or 'external intrusion', the control unit 124 transmits the current location information of the companion animal received from the GPS module 113 of the sensor device 110 to the guardian terminal 130. , control to be transmitted to the emergency medical server 140 and the police server 150 .

As another embodiment, when it is determined that the extracted monitoring information is 'parental notification', the control unit 124 receives image information photographed using the camera unit 118 from the sensor device 110 through the signal receiving unit 121 . and control to generate an emergency notification signal and transmit the emergency notification signal and image information to the guardian terminal 130 through the communication network 101 using the signal transmission unit 126 .

When receiving an emergency notification signal or a thief intrusion notification signal from the control unit 124 , the guardian terminal 130 is connected to the sensor control unit 114 of the sensor device 110 and controls to output a voice signal through the speaker 117 . You can do it (such as a thief warning or a call to the elderly).

The video call processing unit 128 may include connection information for the video call terminal 102 , for example, an IP address and port number of the video call terminal 102 .

The video call processing unit 128 may receive a video call mediation request from the guardian terminal 130 . When receiving a video call brokerage request signal from the guardian terminal 130, the video call processing unit 128 may try to connect to the video call terminal 102 by using connection information on the video call terminal 102. . Accordingly, the video call processing unit 128 establishes a video call session between the guardian terminal 130 and the video call terminal 102 as the connection attempt with the video call terminal 102 is successful.

The guardian terminal 130 may transmit a video or sound to the video call terminal 102 through a video call session.

The guardian terminal 130 may encode a video or sound into a digital signal, and transmit the encoded data to the video call terminal 102 .

In addition, the video call terminal 102 may receive a video or sound from the guardian terminal 130 through a video call session. The video call terminal 102 may receive an encoded image or sound as a digital signal, and may decode the received encoded image or sound.

Through this process, the guardian terminal 130 can take a follow-up action by determining whether the state of the elderly person 10 in the house or whether the thief 11 has invaded.

In the above, the embodiment of the present invention is not implemented only through an apparatus and/or method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention, a recording medium in which the program is recorded, etc. And, such an implementation can be easily implemented by an expert in the technical field to which the present invention belongs from the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

10: old and weak 11: thief, intruder
20: companion animal 100: smart home care system
101: communication network 102: video call terminal
110: sensor device 111: microphone
112: motion sensor unit 113: GPS module
114: sensor control unit 115: biosignal detection unit
116: wireless communication unit 117: speaker
118: camera unit 120: smart home care server
121: signal receiving unit 122: acoustic feature extraction unit
123: emotional state analysis unit 124: control unit
125: motion pattern database unit 126: signal transmission unit
127: memory unit 128: video call processing unit
130: guardian terminal 140: emergency medical server
150: police server

Claims (5)

A microphone attached to the companion animal and receiving the cry of the companion animal as an acoustic signal; a sensor device having a detection unit; and
The type of the cry is received by receiving an acoustic signal, a motion detection signal, and a biosignal from the sensor device, extracting an acoustic characteristic vector of the received acoustic signal, and performing machine learning using the extracted acoustic characteristic vector as input data. In the pattern state table stored in the memory unit, a combination condition of the emotional state information, the motion detection signal, and the biosignal is generated by generating a sound pattern for classifying A smart home care server that searches and extracts monitoring information of a target monitored by the companion animal matching the combination condition,
The pattern state table forms a plurality of the combination conditions and sets a plurality of matching monitoring information corresponding to the respective combination conditions,
The motion detection signal is provided to indicate a motion pattern,
The movement pattern is
Code 1, indicating the state in which the companion animal roams around the elderly and infirm when the elderly are collapsed;
Code 2 indicating the movement status of the companion animal moving back and forth in a straight line when a thief breaks in;
Includes code 3 and code 4 indicating that the companion animal is stationary in one place or moving less than a certain distance,
The pattern state table is,
When the emotional state information is a warning, the movement pattern is code 1, and the biosignal is a heart rate of 150 per minute, the monitoring information is set as a guardian notification;
When the emotional state information is angry, the movement pattern is code 2, and the biosignal is 200 heart rate per minute, set the monitoring information to outsider intrusion;
When the emotional state information is calm, the movement pattern is code 3, and the biosignal is less than or equal to 100 heart rate per minute, set the monitoring information to daily life;
Smart home using emotion and state analysis of companion animals, characterized in that when the emotional state information is curiosity, the movement pattern is code 4, and the biosignal is less than or equal to 120 heart rate per minute, the monitoring information is set to daily life care system. delete According to claim 1,
The smart home care server generates an emergency notification signal when it is determined that the monitoring information is a 'guardian notification' and controls it to be transmitted to the guardian's terminal using the signal transmitter. Smart home care system. According to claim 1,
Companion animal, characterized in that the smart home care server generates a notification signal of a thief intrusion when it is determined that the monitoring information is 'intrusion by an outsider' and controls it to be simultaneously transmitted to the guardian terminal and the police server using the signal transmitter. Smart home care system using emotion and state analysis of According to claim 1,
When it is determined that the monitoring information is 'guardian notification', the smart home care server receives image information photographed using the camera unit from the sensor device, generates an emergency notification signal, and provides the image information and the emergency notification signal. A smart home care system using the emotion and state analysis of companion animals, characterized in that the control is transmitted to the guardian terminal using a signal transmitter.

Images