JP2020160608A - Abnormality detection system - Google Patents

Abstract

To provide an abnormality detection system for reducing a load on a watching object person.SOLUTION: An abnormality detection system includes: sensors NS for detecting a behavior of a person; a behavior prediction section 12 for predicting a behavior of a person by a mechanical learning model based on detection information of the sensors; and an alert mode operation control section 16 for setting the sensors in an alert mode for minute detection when a difference between an average behavior label indicating a behavior in normal time of a person and a prediction behavior label obtained by behavior prediction section exceeds a threshold, and determining presence/absence of an abnormality in a person. The sensors include one of a human sensor, an illuminance sensor, and a temperature and humidity sensor. The behavior prediction section calculates a prediction behavior label by an LSTM network model based on detection values and detection times of the sensors.SELECTED DRAWING: Figure 2

Description

The present invention relates to an abnormality detection system.

In recent years, the number of elderly households living alone has increased due to the declining birthrate and aging population. For the family of the elderly living alone who live apart, it is desirable to be able to easily understand the content of the elderly's daily behavior, their health condition, and their surroundings.
For example, in Patent Document 1, a large number of sensors for detecting the health condition and surrounding conditions of the monitoring target person, and preset management of the health condition and surrounding conditions of the monitoring target person based on the sensor signals. A monitoring system provided with a management means for managing by item and a monitoring means for notifying the fact when an abnormality occurs in the management item is described.

JP-A-2017-168098

According to the above-mentioned prior art, it is possible to accurately grasp the health condition and surrounding conditions of an elderly person living alone and to monitor daily behavior well. However, the output condition of the management item of the behavior pattern alert is that the average value of the sleep data and the activity detection data for the past month is calculated, and there is a difference of 6 to 12 hours between this and the current value ( (See Fig. 3, paragraph 0026), there is a problem that it can be detected only when there is a large change of state from the normal state. Further, when the room temperature signal is 5 ° C. or lower or 30 ° C. or higher for 60 minutes, the room temperature abnormality is detected (FIG. 4, paragraph 0028), and there is a problem that the abnormal state continues for a long time.

An object of the present invention is to solve the above-mentioned problems and to provide an abnormality detection system having a small burden on a person to be watched over.

In order to solve the above problems, the abnormality detection system of the present invention includes a sensor that detects human behavior, a behavior prediction unit that predicts human behavior by a machine learning model based on the detection information of the sensor, and a normal human behavior. When the difference between the average behavior label indicating the behavior of the person and the predicted behavior label obtained by the behavior prediction unit exceeds the threshold value, the sensor is set to the alert mode for detailed detection to determine the presence or absence of a human abnormality. The alert mode operation control unit is provided.

According to the present invention, since detailed information can be acquired and an abnormality determination can be made at the stage when the possibility of an abnormal state is detected, it is possible to quickly respond to the abnormal state of the person to be watched over, and the object to be watched over. The burden on the person can be reduced.

It is a figure which shows an example of a dwelling which monitors a person to be watched over and detects an abnormality. It is a figure which shows the structure of an abnormality detection system. It is a flow figure explaining the processing content of the action recognition server. It is a figure explaining an example of a sensor information. It is a figure explaining the determination method of the necessity of an alert. It is a figure explaining the alert mode operation setting information. It is a figure explaining the sensor definition information.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing an example of a residence in which an abnormality detection system of the embodiment is applied to monitor a resident watching target person and detect an abnormality.

In a house, there are multiple sensors (NS1 to NS7) such as an illuminance sensor that detects the brightness of a room, a motion sensor that detects the presence or absence of a person, and a temperature / humidity sensor that detects the temperature / humidity of a room. It is installed at a predetermined position (NR1 to NR7).

For example, if a motion sensor is installed near the entrance / exit of a room, it is possible to detect the behavior of a person entering / exiting the room.
Further, if the detection value of the illuminance sensor installed in the bedroom changes from light to dark at night, the sleeping behavior of a person can be detected.
Furthermore, bathing behavior and face-washing behavior can be distinguished by the humidity value of the temperature / humidity sensor installed in the bathroom or washroom.

As described above, human behavior can be estimated from one sensor detection value, and human behavior can also be estimated from a combination of a plurality of sensor detection values. For example, the movement behavior of a person can be estimated from the time change of a plurality of motion sensors.

Further, in FIG. 1, each of the plurality of sensors NS1 to NS7 is installed in each room, but the present invention is not limited to this, and a plurality of sensors may be installed in one room. As a result, a plurality of behavior estimates can be performed.

The robot vacuum cleaner 3 of FIG. 1 is a vacuum cleaner that stores map information of a room and runs on its own for cleaning. At the time of standby, it is waiting at the charging station 31. The solid line in FIG. 1 shows an example of the self-propelled path of the robot vacuum cleaner 3. In the abnormality detection system of the embodiment, a speaker, a camera, a microphone, a biological information monitor, and the like mounted on the robot vacuum cleaner 3 enable the state of the person to be watched over to be grasped at the time of abnormality detection.

The abnormality detection system of the embodiment may acquire sensor information from a home electric appliance equipped with a sensor such as an air conditioner as a sensor NS and detect an action. For example, an air conditioner is equipped with an indoor temperature sensor and an infrared camera that acquires the temperature and position of a person, and these are used as sensor NS to utilize detection information.

Further, as the sensor NS, a microphone for collecting sound in the room and a camera such as a Web camera are provided, and when the abnormality detection system of the embodiment detects a behavioral abnormality of the target person, the sound collecting microphone or the camera is used. The state of the person to be watched over may be grasped. As a result, detailed abnormality determination can be performed while managing the privacy of the person to be watched over.

The details will be described later, but when the abnormality detection system of the embodiment detects an abnormality of the monitoring target person, the microphone, camera, and speaker installed in the room are attached to the family of the monitoring target person or the terminal of the care service provider. It connects and enables family members and care service providers to "call out" to those who are being watched over.

FIG. 2 is a configuration diagram of the abnormality detection system of the embodiment.
The abnormality detection system notifies the sensor NS and the robot vacuum cleaner 3 described with reference to FIG. 1, an action recognition server 1 that determines an abnormality of the person being watched over, an operation terminal 51 that sets the operation of the action recognition server 1, and an abnormality. It is composed of a family terminal 52 and an alarm terminal 53 (for a care service provider), and is connected by wireless or wired networks 61 and 62.

The action recognition server 1 is an information processing device that realizes a desired function by executing a program of a processor, and realizes a function described later by executing the program. Further, the action recognition server 1 is configured to include a storage device such as a hard disk drive for storing information described later.

The sensor NS is a communication unit that controls communication between the detection unit 21, which is a detection element for detecting the state in the room, the control unit 22 that controls the detection unit 21, and the sensor NS and the action recognition server 1 via the network 61. 23 and.

Specifically, the detection unit 21 is an illuminance detection element such as a photodiode that detects the brightness of a room, a temperature detection element such as a thermistor, and a resistance change type humidity sensor in which the resistance value of the sensor element changes in response to a change in humidity. , It is a photodiode type infrared sensor.

The control unit 22 acquires the detected value from the detection unit 21 at a predetermined cycle, and notifies the action recognition server 1 of the detected value via the communication unit 23. At this time, the detected value may be normalized in the detection range and notified. Further, when the detected value of the detection unit 21 changes by a predetermined amount, the detected value or the normalized value of the detected value may be notified together with the change time.
Further, the control unit 22 changes the detection cycle by the detection unit 21 according to the setting from the action recognition server 1 and controls whether or not the detection unit 21 can operate.

The robot vacuum cleaner 3 communicates with the detection unit 32, which is a detection element for detecting the state of the person to be watched over, the control unit 33 for controlling the detection unit 32, the robot vacuum cleaner 3 and the action recognition server 1 via the network 61. It is composed of a communication unit 34 that controls the device, a drive unit 35 that runs on its own in the house, a storage unit that stores travel map information 36 for calculating the self-propelled road, and a dust suction unit (not shown). To.

Specifically, the detection unit 32 includes a camera that captures the state of the watching subject, a microphone that collects the voice of the watching target, and a biological information monitor that detects biological information such as the body temperature, heart rate, and respiratory rate of the watching target. Is. In addition, a speaker will be installed to "call out" to the person being watched over.

The control unit 33 controls the drive unit 35 with reference to the travel map information 36 according to the setting from the action recognition server 1, and self-propells to a set position in the house. Then, the detection unit 32 detects the state of the monitoring target person and notifies the action recognition server 1 of the detected value.

Next, the configuration of the action recognition server 1 will be described. The action recognition server 1 has a storage unit that stores sensor information 41, sensor definition information 42, average action label 43, and alert mode operation setting 44 as information for performing abnormality detection processing. ..

The sensor communication unit 11 is a communication unit that connects to the sensor NS via the network 61 and acquires the state (hereinafter referred to as sensor information) of the house or the person to be watched detected by the detection unit 21.
The behavior prediction unit 12 stores the sensor information acquired by the sensor communication unit 11 in the sensor information 41, and predicts the behavior of the person to be watched from the history of the sensor information.

Specifically, the behavior prediction unit 12 is, for example, an LSTM (Long Short Term Memory) network which is a kind of recurrent neural network (RNN) based on time series data of normalized sensor information. Use to calculate the predictive behavior label.

The behavior learning model by the LSTM network of the behavior prediction unit 12 learns and constructs the sensor information of the normal state (the state without abnormal behavior) in the predetermined analysis period (2 weeks to 1 month) of the monitoring target person. The label of the behavior learning model during this analysis period indicates the normal behavior of the person and is stored as the average behavior label 43.

The above average behavior label is not limited to the behavior learning model by the LSTM network, but is a deep layer that ensemble, for example, the K-nearest neighbor method, decision tree, Random forest, SVM, etc. It may be obtained based on a non-learning machine learning model.
Further, the average behavior label in another abnormality detection system that monitors the behavior pattern of a similar person may be diverted and set as the average behavior label of the abnormality detection system.

The alert determination unit 13 compares the predicted action label calculated by the action prediction unit 12 with the average action label 43, and determines whether or not the difference is larger than a predetermined value. The details will be described later, but when the difference is larger than the predetermined value, the action recognition server 1 shifts to the alert mode.

This is because the behavior learning model of the behavior prediction unit 12 learns a normal state (a state that is not abnormal), but cannot learn all normal states. Therefore, the behavior recognition server 1 shifts to the alert mode, performs detailed analysis of the behavior of the watching target person, and detects the presence or absence of an abnormality.

The alert mode operation selection unit 14 refers to the alert mode operation setting 44 based on the sensor information 41 and the current sensor information, and selects the operation setting of the sensor NS for detailed analysis.

The alert mode operation setting unit 15 changes the operation setting of the sensor NS or the robot vacuum cleaner 3 according to the operation setting of the sensor NS selected by the alert mode operation selection unit 14. Specifically, the measurement interval of the sensor NS is set short, or other sensor information of the sensor NS is acquired. Alternatively, the alert mode operation setting unit 15 sets the operation of the robot vacuum cleaner 3 so as to move to the area where detailed analysis is performed and acquire the sensor information with reference to the sensor definition information.

The alert mode operation control unit 16 performs detailed analysis based on the sensor NS set in the alert mode or the sensor information of the robot vacuum cleaner 3, and determines whether or not there is an abnormality in the watch target person.
Further, when the alert mode operation control unit 16 detects an abnormality of the monitoring target person, the alert mode operation control unit 16 notifies the family terminal 52 of the family of the monitoring target person or the alarm terminal 53 of the care service provider (of the monitoring target person). Abnormality notification) and sensor information when an abnormality is detected.

The behavior learning model update unit 18 notifies the behavior recognition server 1 that the monitoring target person has no abnormality in response to the alarm notification by the family of the monitoring target person or the care service provider, and the alert determination unit 13 The sensor information determined to shift to the alert mode in is used as the sensor information in the normal state, additional learning of the behavior learning model is performed, and the behavior learning model and the average behavior label are updated. As a result, the robustness of the behavior learning model of the behavior prediction unit 12 is improved.

The external communication unit 17 connects the action recognition server 1, the family terminal 52, and the alarm terminal 53 via the network 62, transmits an abnormality notification of the person to be watched, transmits sensor information, and receives a confirmation result for the abnormality notification. It is a communication unit that performs.
The operation terminal 51 is a terminal that sets the sensor definition information 42, the initial information of the alert mode operation setting 44, the notification destination address of the family terminal 52 and the alarm terminal 53 for abnormal notification, and the like in the action recognition server 1. ..

The behavior recognition server 1 may be configured to function as a network server without being installed in the residence and to be connected to the sensor NS via an edge server installed in the residence.

In the above, an example in which the behavior learning model of the behavior prediction unit 12 is configured by the LSTM network has been described, but the present invention is not limited to this, and any other algorithm may be used as long as it is a machine learning model corresponding to time series information. ..

Next, the processing flow of the action recognition server 1 will be described with reference to FIG.
The processing flow of FIG. 3 is based on the premise that the sensor information is learned in advance in a predetermined analysis period, the behavior learning model of the behavior prediction unit 12 is constructed, and the average behavior label 43 is stored.

In step S31, the action recognition server 1 acquires sensor information from the sensor NS by the sensor communication unit 11, stores it as time-series information in the sensor information 41, and sets the acquired sensor information in the action prediction unit 12.

FIG. 4 is a diagram showing an example of time-series information of sensor information. As for the sensor information, the behavior recognition server 1 is notified of the data obtained by normalizing the detection values of the detection units 21 of the sensors NS1 to NS7 and the time (event time) when the detection value of the detection units 21 changes, and the behavior recognition server. Based on the time when the detected value changes in 1, the sensor information is complemented with the periodic sensor information and set in the accumulation and behavior prediction unit 12 as the sensor information.
Of course, the acquired data may be the raw detection values of the sensors NS1 to NS7 at a predetermined cycle, and may be accumulated and notified to the action recognition server 1. In this case, the behavior recognition server 1 normalizes the data.

Returning to FIG. 3, in step S32, the behavior prediction unit 12 (behavior learning model) calculates the prediction behavior label based on the history of the plurality of sensor information set sequentially.
In step S33, the alert determination unit 13 compares the average behavior label 43 with the predicted behavior label, and whether or not the behavior recognition server 1 shifts to the alert mode and performs detailed analysis of the behavior of the watch target person. Determine if an alert is required.

FIG. 5 is a diagram showing a time change of the action label value of the action prediction unit 12 and explaining a method of determining whether or not an alert is necessary in step S33.
The broken line in FIG. 5 shows the time change of the action label value of the average action label 43. The action prediction unit 12 compares the predicted action label (a plurality of predicted action labels) having a predetermined time width with the label value of the corresponding time zone of the average action label, and the area of the difference (shaded area in FIG. 4). When is equal to or greater than a predetermined threshold, it is determined that the transition to the alert mode is required.

Returning to FIG. 3, in step S33 (alert determination), if the difference area between the predicted action label and the average action label of the predetermined time width is smaller than the predetermined value, the alert determination unit 13 is normal in the behavior of the watching target person. It is determined that the transition to the alert mode is unnecessary (S33 is unnecessary), and the process returns to step S31. If the difference area between the predicted behavior label and the average behavior label is equal to or greater than a predetermined value, it is considered that the behavior of the person to be watched is not normal, it is determined that the transition to the alert mode is necessary (essential of S33), and the process proceeds to step S34. ..

Here, the average behavior label 43 will be described.
In the abnormality detection system of the embodiment, as shown in FIG. 5, the time change of the action label for one day is set as the average action label 43.
If the behavior pattern of the person to be watched differs between weekdays and holidays, or if there is a regular outing schedule such as going to the hospital, it is desirable to select the average behavior label 43 corresponding to each behavior pattern. ..
Specifically, in carrying out the processing flow of FIG. 3, the day of the week and the action schedule of the person to be watched are acquired, and the average action label 43 corresponding to this is selected. As a result, the alert determination accuracy can be improved.

By the way, when it is determined in step S33 of FIG. 3 that the transition to the alert mode is unnecessary, the average action label may be updated by the predicted action label generated in step S32. As a result, the determination accuracy of step S33 can be improved.

Returning to FIG. 3, in step S34, the alert mode operation selection unit 14 refers to the alert mode operation setting 44 based on the sensor information 41 and the current sensor information, and selects the sensor operation setting for detailed analysis.

More specifically, as shown in FIG. 6, as the alert mode operation setting 44, the operation setting in the alert mode is defined in advance for each sensor NS, and when the transition to the alert mode is determined, the alert mode operation is performed. The operation setting of the sensor NS related to the abnormal state is acquired by referring to the setting 44.

Specifically, the operation setting in the alert mode for each sensor NS is set by selecting one from a plurality of setting options such as the detection cycle for each sensor NS at the start of operation of the abnormality detection system. In addition, since the camera activation and microphone sound collection settings have a large effect on the privacy of the person being watched over, it is desirable to set them with the approval of the person being watched over or their family members.

As shown in FIG. 6, as the operation setting in the alert mode, in addition to changing the operation setting of the sensor NS such as "increase in detection cycle", a camera or a biological information monitor to be mounted by dispatching a robot vacuum cleaner You may set to operate a sensor other than the sensor NS that acquires sensor information such as an instruction to operate the sensor, a camera activation instruction installed in a house, a microphone sound collection instruction, and a speaker output instruction.

Further, the alert mode operation selection unit 14 may select an alert mode operation in which all the sensor NSs are set to the alert mode, or when the transition to the alert mode is determined, the sensor NS combination is used. , The operation content of the sensor NS may be changed.

In step S35, the alert mode operation setting unit 15 changes the operation setting of the sensor NS according to the setting content of the alert mode operation setting information of FIG. 6 selected in step S34.
Here, a case where the robot vacuum cleaner is dispatched as in NS2 of the alert mode operation setting 44 shown in FIG. 6 will be described in detail.

In the abnormality detection system of the embodiment, as shown in FIG. 7, sensor definition information 42 indicating the sensor type, sensor identification information (ID), and installation position of each sensor NS is defined.
The alert mode operation setting unit 15 acquires the installation position of the sensor NS2 with reference to the sensor definition information 42 of FIG. 7 for the sensor NS2 in which the operation of dispatching the robot vacuum cleaner is set in the alert mode operation setting 44. .. The alert mode operation setting unit 15 sets the robot vacuum cleaner 3 so as to move to the acquired installation position. At this time, the operation instructions of the mounted camera and biometric information monitor are also set.

In step S36, the alert mode operation control unit 16 acquires sensor information for performing detailed analysis of the state of the watch target person from the sensor NS and the robot vacuum cleaner 3 for which the alert mode operation is set in step S35.

Then, in step S37 (abnormality determination), the alert mode operation control unit 16 determines the abnormality in the alert mode based on the sensor information, and in the case of normal determination, proceeds to step S30 (normality of S37) and abnormalities. In the case of determination, the process proceeds to step S38 (abnormality in S37).

More specifically, the alert mode operation control unit 16 performs image processing based on the captured image (sensor information) of the camera to detect the fall of the watching target person, or requests rescue based on the sensor information of the microphone. By recognizing the voice of, the abnormal state of the watching target person is determined.
In addition, the abnormal state of the person to be watched is determined from the sensor information of the biological information monitor mounted on the robot vacuum cleaner 3.

In step S38, the alert mode operation control unit 16 notifies the family terminal 52 or the alarm terminal 53 that the monitoring target person is in an abnormal state, and requests confirmation of the presence or absence of the abnormality of the monitoring target person. ..
At this time, by sending the captured image (sensor information) of the camera acquired in step S36 together, it becomes easy to grasp the situation of the family and the caregiver of the person to be watched over.

In step S39 (confirmation result determination), the alert mode operation control unit 16 acquires the confirmation result of the presence or absence of abnormality of the monitoring target person requested in step S38 from the family terminal 52 or the alarm terminal 53.
Then, in step S39, the alert mode operation control unit 16 determines whether or not there is an abnormality in the confirmation result, proceeds to step S40 if it is in an abnormal state, and proceeds to step S30 if it is in a normal state. move on.

In step S40, the alert mode operation control unit 16 supports the rescue activity by connecting the family terminal 52 or the alarm terminal 53 to the speaker of the home appliance installed in the house, the microphone or the camera as the sensor NS. As a result, the family of the person to be watched over and the long-term care service provider can carry out care activities such as "calling out" to the person to be watched from a remote location.
When the relief activity support in step S40 is completed, the process returns to step S31.

In step S30, the behavior learning model update unit 18 additionally learns the behavior learning model by using the sensor information that the alert determination unit 13 has determined to shift to the alert mode in step S33 as the normal state sensor information, and performs behavior learning. Update the model and average behavior label. As a result, the robustness of the behavior learning model of the behavior prediction unit 12 is improved.
Then, the process returns to step S31 and is repeatedly executed at a predetermined cycle.

In the anomaly detection system of the above embodiment, an example of estimating an abnormal behavior by constructing a behavior learning model in which a plurality of sensor information is input to calculate one predicted behavior label has been described.
If the behavior learning model is considered to calculate the predicted behavior label of the crowd, it can be used as an anomaly detection system of the crowd behavior in the trademark facility.

Further, in the abnormality detection system of the above embodiment, an example of estimating an abnormal behavior by constructing a behavior learning model in which a plurality of sensor information is input to calculate one predicted behavior label has been described. A behavior learning model may be constructed in which a plurality of sensor information including a sensor is input to calculate a predicted behavior label of a plurality of people, and an abnormal behavior of a plurality of people may be estimated.
This abnormality detection system can be applied to facilities such as elderly homes, and can have the effect of promptly responding to an abnormal state of a person to be watched over.

Further, the present invention is not limited to the above-described examples, and includes various modifications. The above-described embodiment has been described in detail for the sake of easy understanding in the present invention, and is not necessarily limited to the one including all the configurations described.

11 Sensor communication unit 12 Action prediction unit 13 Alert judgment unit 14 Alert mode operation selection unit 15 Alert mode operation setting unit 16 Alert mode operation control unit 17 External communication unit 18 Behavior learning model update unit 41 Sensor information 42 Sensor definition information 43 Average behavior Label 44 Alert mode operation setting 51 Operation terminal 52 Family terminal 53 Alarm terminal NS sensor 21 Detection unit 22 Control unit 23 Communication unit 3 Robot vacuum cleaner 32 Detection unit 33 Control unit 34 Communication unit 35 Drive unit 36 Travel map information

Claims (7)

Sensors that detect human behavior and
A behavior prediction unit that predicts human behavior using a machine learning model based on the detection information of the sensor,
When the difference between the average behavior label indicating the normal behavior of a person and the predicted behavior label obtained by the behavior prediction unit exceeds a threshold value, the sensor is set to an alert mode for detailed detection, and the person is abnormal. Alert mode operation control unit that determines the presence or absence of
Anomaly detection system characterized by being equipped with. In the abnormality detection system according to claim 1,
The sensor includes any of a motion sensor, an illuminance sensor, and a temperature / humidity sensor.
The behavior prediction unit is an abnormality detection system characterized in that a predicted behavior label is calculated by an LSTM network model based on a detection value and a detection time of the sensor. In the abnormality detection system according to claim 1,
The alert mode operation control unit is an abnormality detection system characterized in that the measurement interval of the sensor is shortened and the alert mode is set. In the abnormality detection system according to claim 1, further
Equipped with a camera that captures the state of a person
The alert mode operation control unit is an abnormality detection system characterized in that, in the alert mode of the sensor, the presence or absence of an abnormality in a person is determined from an image captured by the camera. In the abnormality detection system according to claim 1, further
Equipped with a microphone that captures human voice
The alert mode operation control unit is an abnormality detection system characterized in that, in the alert mode of the sensor, the presence or absence of an abnormality in a person is determined by the voice acquired by the microphone. In the abnormality detection system according to claim 1, further
Equipped with a biometric information monitor that detects human biometric information
The alert mode operation control unit is an abnormality detection system characterized in that, in the alert mode of the sensor, the presence or absence of an abnormality in a person is determined based on the detection information of the biological information monitor. In the abnormality detection system according to claim 1, further
It is equipped with a self-propelled home appliance equipped with at least one of a camera that captures a person's condition, a microphone that acquires a person's voice, or a biometric information monitor that detects a person's biometric information as the sensor.
The alert mode operation control unit is an abnormality detection system characterized in that, in the alert mode of the sensor, the sensor mounted on the self-propelled home appliance determines the presence or absence of an abnormality in a person.

Images