WO2018235628A1

WO2018235628A1 - Monitoring assistance system, control method therefor, and program

Info

Publication number: WO2018235628A1
Application number: PCT/JP2018/021983
Authority: WO
Inventors: 純平松永; 田中　清明; 信二高橋; 達哉村上
Original assignee: オムロン株式会社
Priority date: 2017-06-23
Filing date: 2018-06-08
Publication date: 2018-12-27
Also published as: JP6822326B2; JP2019008515A

Abstract

A monitoring assistance system for assisting in monitoring a subject in a bed, said system having: a regression device into which an image that includes a bed and a person is input, and in which mechanical learning is carried out so as to output a score indicating the state of the person with respect to the bed; an image acquisition unit for acquiring an image from an imaging device arranged so as to capture an image of a monitoring region that includes the subject's bed; a determination unit for determining a behavior of the subject on the basis of a score obtained by inputting an image of the monitoring region acquired by the image acquisition unit to the regression device; and an output unit for providing a notification based on the determination result from the determination unit.

Description

Oversight support system, control method therefor, and program

TECHNICAL FIELD The present invention relates to a technique for supporting watching of a subject on a bed.

In order to prevent a fall accident from a bed and the like, a system that supports watching and listening of patients in hospitals and nursing facilities is known. In Patent Document 1, the head of a patient is detected from an image captured by a camera installed obliquely above the bed, and it is determined that the head moves up when the head exceeds a boundary set on the bed, and a nurse A system has been proposed to provide notification etc.

Unexamined-Japanese-Patent No. 2012-071004

The conventional system generally detects a patient's head from an image and estimates a patient's behavior (such as getting up or leaving) from the position and movement of the head. However, such a conventional method has a problem that the behavior of the patient can not be correctly estimated when the head can not be detected from the image or when it is detected erroneously. For example, if the detection accuracy varies depending on the disturbance or environment, such as when the patient is covered with a futon, a person other than the patient or a confusing object is present, or the lighting environment changes, dangerous behavior may occur. And false alarms (unnecessary alarms) may occur, which may significantly reduce the reliability of the system.

The present invention has been made in view of the above-described circumstances, and an object thereof is to provide a technique for detecting the state or behavior of a subject on a bed with high accuracy and high reliability.

In order to achieve the above object, according to the present invention, the state of the watching target is estimated from the monitoring image using a regressor that machine-learns the pattern of the bed and the state of the person, and the action of the target is detected. Adopt the method.

Specifically, a first aspect of the present invention is a watching support system for supporting watching of a subject on a bed, and an image showing a bed and a person is input, and a score indicating the state of the person relative to the bed is input. A regressor that is machine-learned to output the image, an image acquisition unit that acquires an image from an imaging device installed to image the monitoring region including the bed of the subject, and the image acquisition unit A determination unit that determines the action of the subject based on a score obtained by inputting the image of the monitoring area to the regressor; an output unit that performs notification based on the determination result of the determination unit; To provide a watching support system characterized by having:

According to this configuration, the regressor is used to estimate the state of the subject in the input image. Therefore, highly accurate state estimation can be performed on an unknown input image. Further, since the score of the regressor is output as a continuous value, it is possible to obtain a reasonable estimation result even if the condition of the subject can not be clearly classified. Furthermore, even if the target person is covered with a futon, there are strange people or objects around the target person, or the lighting environment is different from normal, etc., the image is difficult to detect the head. There is also the advantage that you can get

The regressor may be a neural network. By using a neural network, highly accurate and highly robust state estimation can be performed.

The state of the person with respect to the bed is classified in advance into a plurality of types, and different scores are assigned to each of the plurality of types, and the regressor is configured to determine that the state of the person is between two types. If so, it may be configured to output a value between the two types of scores. Such a design makes it possible to express various human-readable states as a one-dimensional score, making it easy to handle "human states" mathematically or in programs, for example, The processing of (judgment logic by the judgment unit etc.) can be constructed extremely easily.

For example, the plurality of types may include state 0 in which the person is sleeping in the bed, state 1 in which the person is rising on the bed, and state 2 in which the person is away from the bed . This is because if it is possible to distinguish at least three types of states of state 0 to state 2, it is possible to detect “wake up” and “getting out of bed” which have a high need for watching.

The determination unit may determine an action of the subject based on a temporal change of the state of the subject estimated by the score. The accuracy of the action determination can be improved by determining the action of the subject based on the temporal change (state transition) of the state of the subject rather than the temporary state of the subject.

For example, when the state of the subject transits from the state 0 to the state 1, or the state of the subject transits from the state 0 to the state 1, the determination unit further determines that the state 1 is When continuing for a predetermined time, it may be determined that the subject has got up.

In addition, when the state of the subject changes from the state 0 to the state 1 and further changes from the state 1 to the state 2, the determination unit determines the state of the subject from the state 0 The state 1 is transitioned to, the state 1 is changed to the state 2, and the state 2 continues for a predetermined time, or the state of the subject is changed from the state 0 to the state 2 Alternatively, if the state of the subject changes from the state 0 to the state 2 and the state 2 continues for a predetermined time, it may be determined that the subject has left the bed.

The present invention can be understood as a watching support system having at least a part of the above configuration or function. The present invention also provides a watching support method or a watching support system control method including at least a part of the above-described processing, a program for causing a computer to execute these methods, or non-temporarily such a program. It can also be regarded as a recorded computer readable recording medium. Each of the above configurations and processes can be combined with each other as long as there is no technical contradiction.

According to the present invention, the state or behavior of the subject on the bed can be detected with high accuracy and high reliability.

FIG. 1 is a block diagram schematically showing a hardware configuration and a functional configuration of the watching support system of the first embodiment. FIG. 2 is a view showing an installation example of the imaging device. FIG. 3A and FIG. 3B are examples of the monitoring area set for the image. FIG. 4 shows types of human states and examples of images. FIG. 5 is a diagram schematically showing machine learning of the regressor. FIG. 6 is a diagram schematically showing the ability of the regressor. FIG. 7 is a flowchart of the state monitoring process. FIG. 8 is a flowchart of action determination. FIG. 9 is an example of the determination logic.

TECHNICAL FIELD The present invention relates to a technique for supporting watching of a subject on a bed. This technology can be applied to a system that automatically detects the getting-up and getting-up behavior of patients and care recipients in hospitals and nursing facilities, etc., and performs necessary notification when a dangerous state occurs. This system can be preferably used, for example, for watching and supporting elderly people, patients with dementia, children and the like.

Hereinafter, an example of a preferred embodiment for carrying out the present invention will be described with reference to the drawings. However, the configurations and operations of the devices described in the following embodiments are merely examples, and the scope of the present invention is not limited thereto.

(System configuration)
The configuration of a watching support system according to an embodiment of the present invention will be described with reference to FIGS. 1 and 2. FIG. 1 is a block diagram schematically showing a hardware configuration and a functional configuration of the watching support system 1, and FIG. 2 is a diagram showing an installation example of an imaging device.

The watching support system 1 includes an imaging device 10 and an information processing device 11 as main hardware configurations. The imaging device 10 and the information processing device 11 are connected by wire or wirelessly. Although only one imaging device 10 is shown in FIG. 1, a plurality of imaging devices 10 may be connected to the information processing device 11.

The imaging device 10 is a device for capturing a subject on a bed and capturing image data. As the imaging device 10, a monochrome or color visible light camera, an infrared camera, a three-dimensional camera or the like can be used. In the present embodiment, the imaging device 10 configured by the infrared LED illumination 100 and the near infrared camera 101 is adopted in order to enable watching of the target person even at night (even when the room is dark). As shown in FIG. 2, the imaging device 10 is installed to look over the entire bed 20 from the top of the bed 20 to the foot. The imaging device 10 captures an image at a predetermined time interval (for example, 30 fps), and the image data is sequentially captured by the information processing device 11.

The information processing apparatus 11 has a function of analyzing image data taken in from the imaging apparatus 10 in real time, automatically detecting wakeup behavior and leaving behavior of the subject 21 on the bed 20, and notifying when necessary. It is. The information processing apparatus 11 includes, as specific functional modules, an image acquisition unit 110, an area setting unit 111, a preprocessing unit 112, a regression unit 113, a score stabilization unit 114, a determination unit 115, an output unit 116, and a storage unit 117. Have.

The information processing apparatus 11 according to this embodiment includes a CPU (processor), memory, storage (HDD, SSD, etc.), input device (keyboard, mouse, touch panel, etc.), output device (display, speaker, etc.), communication interface, etc. Each module of the information processing apparatus 11 described above is realized by the CPU executing a program stored in the storage or the memory. However, the configuration of the information processing apparatus 11 is not limited to this example. For example, distributed computing may be performed by a plurality of computers, a part of the module may be executed by a cloud server, or a part of the module may be configured by a circuit such as an ASIC or an FPGA. It is also good.

The image acquisition unit 110 is a module for acquiring an image captured by the imaging device 10. The image data input from the image acquisition unit 110 is temporarily stored in a memory or storage, and is used for area setting processing and status monitoring processing described later.

The area setting unit 111 is a module for setting a monitoring area for an image captured by the imaging device 10. The monitoring area is a range (in other words, an image range used as an input of the regressor 113) to be subjected to the state monitoring process in the field of view of the imaging device 10. Details of the area setting process will be described later.

The preprocessing unit 112 is a module for performing necessary preprocessing on an image (hereinafter referred to as an “original image”) input from the image acquisition unit 110 in the state monitoring process. For example, the preprocessing unit 112 performs processing of clipping an image within the monitoring area from the original image (hereinafter, the clipped image is referred to as a “monitoring area image”). In addition, the preprocessing unit 112 may perform processing such as resizing (reduction), affine transformation, and luminance correction on the monitoring area image. Resizing (reduction) has the effect of shortening the calculation time of the regressor 113. Although any existing method may be used for resizing, it is preferable to use a bilinear method that has a good balance of calculation cost and quality. The affine transformation can be expected to have the effect of normalizing the input image to the regressor 113 and improving the estimation accuracy by performing necessary distortion correction such as, for example, deforming a bed reflected in a trapezoidal shape in the image into a rectangular shape . The luminance correction can be expected to have an effect of improving the estimation accuracy, for example, by reducing the influence of the illumination environment. When the original image is input to the regressor 113 as it is, the preprocessing unit 112 may be omitted.

The regressor 113 is a module for outputting a score indicating the state of the subject 21 (for example, bedtime state, wakeup state, leaving state) shown in the monitoring area image when the monitoring area image is given. . The regressor 113 is a machine learning model of a relation model between the features of the input image and the human state so as to output an image showing the bed and the person and output a score indicating the human state with respect to the bed. . It is assumed that the training of the regressor 113 is performed in advance (before shipping or operation of the system) by the learning device 12 using a large number of training images. In addition, as a learning model of the regressor 113, any model such as a neural network, a random forest, a support vector machine, etc. may be used. In this embodiment, a convolutional neural network (CNN) suitable for image recognition is used.

The score stabilization unit 114 is a module for suppressing rapid change and fluttering of the score output from the regressor 113. The score stabilization unit 114, for example, calculates an average of the current score obtained from the image of the current frame and the past score obtained from the images of the immediately preceding two frames, and outputs the average as a stabilization score. This process is equivalent to applying a temporal low-pass filter to the time series data of the score. In addition, if stabilization of a score is unnecessary, the score stabilization part 114 may be abbreviate | omitted.

The determination unit 115 is a module for determining the action of the subject based on the score obtained by the regressor 113. Specifically, the determination unit 115 determines what kind of behavior (for example, wake-up action, leaving-behind action) of the subject based on temporal change in score (that is, transition of the “subject's state” indicated by the score). Etc.) are estimated. Details of the processing of the determination unit 115 will be described later.

The output unit 116 is a module that performs necessary notification based on the determination result of the determination unit 115. For example, the output unit 116 determines whether or not the notification is necessary (for example, notifies only in the dangerous state), the content of the notification (for example, the content of the message), the notification unit (for example, for example). Voice, mail, buzzer, warning light, notification destination (eg, nurse, doctor, carer), frequency of notification, etc. can be switched.

The storage unit 117 is a module for storing various data used by the watching support system 1 for processing. The storage unit 117 stores, for example, setting information of a monitoring area, parameters used in preprocessing, parameters used in score stabilization processing, time series data of scores, parameters used in determination processing, and the like.

(Setting of monitoring area)
Various items other than the bed 20 and the target person 21 are reflected in the angle of view of the imaging device 10. When detecting the state or behavior of the target person 21, it is preferable to exclude as much as possible, since things other than the bed 20 and the target person 21 may act as noise. Further, with regard to the image input to the regressor 113, the estimation accuracy can be easily improved if the image size (width, height), and the position, range, and size of the bed in the image are standardized. Therefore, in the present embodiment, a predetermined range based on the bed 20 is set as a monitoring area, and in a state monitoring process described later, an image in the monitoring area is clipped to be an input image of the regression unit 113.

The setting of the monitoring area may be performed manually or automatically. In the case of manual setting, the area setting unit 111 may provide a user interface for allowing the user to input the area of the bed 20 in the image or the monitoring area itself. In the case of automatic setting, the area setting unit 111 may detect the bed 20 from the image by object recognition processing, and set the monitoring area so as to include the detected area of the bed 20. The area setting process is performed, for example, when the monitoring area needs to be updated along with the movement of the bed 20 or the imaging device 10 when the monitoring area is not set (for example, when the system is installed).

FIG. 3A is an example of the monitoring area set for the original image. In the present embodiment, a monitoring area 30 is set by adding a margin of a predetermined width to the left side, the right side, and the upper side (foot side) of the area of the bed 20. The width of the margin is set so that the whole body of the person (see FIG. 3B) rising on the bed 20 falls within the monitoring area 30.

(Types of states and machine learning)
In this system, in order to treat the human condition by regression, "the human condition with respect to the bed" is classified into three types from 0 to 2 in advance. The "type 0 type" is a state in which a person is sleeping in the bed (referred to as "sleeping state" or "state 0"), the "type 1 type" is a state in which a person is rising on the bed ("wake up state Or “state 1”), “type 2” is a state in which a person is separated from the bed (dismounted) (referred to as “bed leaving state” or “state 2”). FIG. 4 is an example showing correspondence between time-series images representing a series of actions of a person who was sleeping rising and leaving the bed, and three types.

FIG. 5 schematically shows the machine learning of the regressor 113. First, images obtained by photographing an actual patient room and the like are collected, and each image is classified into type 0 to type 2. Then, a portion corresponding to the monitoring area of each image is clipped, and the type number (0, 1, 2) is assigned as a label to generate a set of training images. In order to improve the accuracy of regression, it is preferable to prepare a sufficient number of images, and it is preferable to prepare images of various variations for each type. However, it is preferable to exclude the image which is ambiguous to which type of human condition belongs, because it is not suitable for the training image.

The learning device 12 uses a set of training images and outputs a convolutional neural network so as to output, for each input image, the same score as its label (that is, score = 1 if the input image belongs to the first type) Do training. Then, the learning device 12 incorporates the parameter group of the neural network, which is the learning result, into the regressor 113 of the present system. The specific layer structure of the neural network, the filter, the activation function, the specification of the input image, and the like may be appropriately designed according to the mounting and the required accuracy.

FIG. 6 schematically shows the ability of the regressor 113. The regressor 113 models the correspondence between the “feature amount” of the image and the “score” indicating the human state. The regressor 113 extracts the feature amount from the input image according to the relationship model, and calculates and outputs a score corresponding to the feature amount. Although FIG. 6 shows the relationship model as a two-dimensional linear model for the convenience of description, the actual feature amount space is multidimensional, and the relationship model is non-linear.

The score output from the regressor 113 is a real value (continuous value) in the range of 0-2. For example, when an input image of the first type (wake-up state) is given, the output score becomes 1 or a value very close to 1. The same is true for other types. On the other hand, in the input image, there is also an ambiguous image as to which type it belongs to, such as a state in which the upper body is about to wake up from the sleeping position or a state in which it is about to stand up from the bed. In the case of such an intermediate state image, the extracted feature quantity is a feature quantity between the two types, so that an intermediate score between the two types is output. For example, in the case of an image in a state in which the upper body is about to wake up from the sleeping position, a score of a value larger than 0 and smaller than 1 is obtained because it is an intermediate state between the 0th type and the 1st type.

Thus, in the present system, the regressor 113 is used to estimate the human state in the input image. Therefore, highly accurate state estimation can be performed on an unknown input image. In particular, the following advantages can be mentioned as superior points as compared with the conventional method using head detection.
Even when an intermediate state image is input, a reasonable estimation result can be obtained.
・ Even if the subject is covered with a futon, there are strange people or objects around the subject, or the lighting environment is different from usual, etc., it is an image that is difficult to detect the head. You can get it.

(Status monitoring process)
An example of the state monitoring process of the present system will be described with reference to FIG. The processing flow of FIG. 7 is executed each time an image of one frame is captured from the imaging device 10.

In step S <b> 70, the image acquisition unit 110 captures an image of one frame from the imaging device 10. The acquired original image is temporarily stored in the storage unit 117. Next, the preprocessing unit 112 clips the monitoring area image from the original image, and executes resizing, affine transformation, luminance correction and the like as necessary (step S71). Next, the regressor 113 inputs the monitoring area image and outputs the corresponding score (step S72). Next, the score stabilization unit 114 performs stabilization processing of the score obtained in step S72 (step S73), and delivers the obtained score to the determination unit 115.

First, based on the score (continuous value), the determination unit 115 selects one of 0 type (state 0) / first type (state 1) / second type (state 2) as the current state of the subject 21. Classified into The classification method is not limited, but in the present embodiment, state 0 (steps S74 and S75) in the case of score ≦ threshold th1, state 1 (step S76 and S77) in the case of threshold th1 <score ≦ threshold th2, threshold th2 < In the case of the score, it is classified as state 2 (step S78). The threshold values th1 and th2 are set, for example, as th1 = 0.5 and th2 = 1.5. The detection sensitivity can be adjusted by changing the thresholds th1 and th2.

Subsequently, the determination unit 115 determines the action of the subject 21 based on the temporal change of the state of the subject 21 (step S79). If it is determined in step S79 that the target person 21 is "wake up" or "getting up", the output unit 116 performs necessary notification (step S80). The above steps S70 to S80 are executed for each frame until the system is completed (step S81).

The determination logic by the determination unit 115 will be described with reference to FIGS. 8 and 9. FIG. 8 is a detailed flow of the action determination process in step S79, and FIG. 9 is an example of the determination logic, showing the correspondence between the state transition, the determination result, and the notification setting. Here, the wake-up preparation flag is a flag for confirming transition from the state 0, and the bed-out preparation flag is a flag for confirming transition from the state 1. It is assumed that the wakeup preparation flag and the bed leaving preparation flag are initialized as "OFF" when the state monitoring process is started.

If the current state is "0" (step S790; YES), the determination unit 115 sets the wakeup preparation flag and the bed leaving preparation flag to "ON" (step S791).

If the current state is “1” (step S 792: YES), the determination unit 115 confirms whether the wake-up preparation flag is “ON” (step S 793). If “ON”, the object person 21 is It is determined that the user gets up (step S794), the wake-up preparation flag is set to "OFF", and the bed-up preparation flag is set to "ON" (step S795).

If the current state is "2" (step S796; YES), the determination unit 115 confirms whether the bed leaving preparation flag is "ON" (step S797), and if "ON", the target person 21 is It is determined that the user has got out of bed (step S798), and the wake-up preparation flag and the release flag are set to "OFF" (step S799).

According to the determination logic as described above, as shown in FIG. 9, when transitioning from state 0 to state 1, it is determined to be “wake up”, and when transitioning from state 0 or state 1 to state 2 is “leaving” It is determined that Otherwise, the determination unit 115 does not output the determination result or outputs the result of “no abnormality”.

According to the present embodiment described above, since the state estimation of the target person 21 is performed by the regressor 113, the state or the behavior of the target person 21 can be determined with high accuracy as compared with the conventional method. In addition, since the wakeup behavior and the leaving behavior of the target person 21 are determined based on the temporal change (state transition) of the state of the target person 21 instead of the temporary state of the target person 21, the accuracy of action determination is improved. can do. Moreover, when the wakeup determination and the leave determination are made and the notification is made, the wakeup preparation flag and the stay preparation flag are temporarily set to "OFF", so the wakeup determination and the leave will be performed unless the target person 21 returns to the bed state etc. again. Judgment does not work. Therefore, false notification of wakeup and wakeup notification can be reduced, and the reliability of the system can be improved.

<Others>
The above description of each embodiment merely illustrates the present invention. The present invention is not limited to the above specific embodiments, and various modifications are possible within the scope of the technical idea thereof.

For example, in the above embodiment, when it transits from the state 0 to the state 1, it is determined to be a wake up, but instead, the state 1 continues for a predetermined time (for example 1 second) after transitioning from the state 0 to the state 1 In this case, it may be determined that the user has got up. Thereby, the reliability of the wakeup determination can be further enhanced. Similarly, after transition from the state 0 or the state 1 to the state 2, if the state 2 continues for a predetermined time (for example, 1 second), it may be determined that the bed has left. Thereby, the reliability of the bed leaving determination can be further enhanced.

In the above-described embodiment, an example has been described in which the sleeping state / wake-up state / disengagement state is estimated from the image, and the wake-up behavior and the leaving behavior of the subject are detected. However, the state of the estimation target and the action of the detection target are not limited to these. That is, it is possible to handle various "human states" and "actions" as long as different features appear in the image. For example, it is also possible to detect actions such as eating and reading.

1: Watch-over support system 10: imaging device, 11: information processing device, 12: learning device 100: illumination, 101: near infrared camera, 110: image acquisition unit, 111: area setting unit, 112: preprocessing unit, 113: Regressor, 114: Score stabilization unit, 115: Determination unit, 116: Output unit, 117: Storage unit 20: Bed, 21: Target person 30: Monitoring area

Claims

It is a watching support system that supports watching of the target person on the bed,
A regressor that is machine-learned to input an image of a bed and a person and output a score indicating the state of the person relative to the bed;
An image acquisition unit configured to acquire an image from an imaging device installed to photograph a monitoring area including the bed of the subject;
A determination unit that determines the action of the subject based on the score obtained by inputting the image of the monitoring area obtained by the image acquisition unit to the regressor;
An output unit that makes a notification based on the determination result of the determination unit;
A watching support system characterized by having.
The watching support system according to claim 1, wherein the regressor is a neural network.
The state of the person with respect to the bed is classified in advance into a plurality of types, and different scores are assigned to each of the plurality of types,
The regressor is configured to output a value between the scores of the two types if the state of the person is a state between the two types. The watching support system described in 2.
The plurality of types include a state 0 in which the person sleeps in the bed, a state 1 in which the person stands up on the bed, and a state 2 in which the person leaves the bed. The watching support system according to claim 3.
5. The watching support system according to claim 4, wherein the determination unit determines an action of the subject based on a temporal change of the state of the subject estimated by the score.
The determination unit is
When the state of the subject changes from the state 0 to the state 1, or
When the state of the subject changes from the state 0 to the state 1 and the state 1 continues for a predetermined time,
The watching support system according to claim 5, wherein it is determined that the target person has got up.
The determination unit is
When the state of the subject changes from the state 0 to the state 1, and further from the state 1 to the state 2, or
When the state of the subject changes from the state 0 to the state 1, and further changes from the state 1 to the state 2, and the state 2 continues for a predetermined time, or
When the state of the subject changes from the state 0 to the state 2, or
When the state of the subject changes from the state 0 to the state 2 and the state 2 continues for a predetermined time,
7. The watching support system according to claim 5, wherein it is determined that the target person has left the bed.
A control method of a watching support system for supporting watching of a target person on a bed,
Acquiring an image from an imaging device installed to image a monitoring area including the bed of the subject;
Inputting an image of the monitoring area into a regressor which is machine-learned to input an image including a bed and a person and output a score indicating the state of the person relative to the bed;
Determining the behavior of the subject based on the score obtained from the regressor;
Performing notification based on the determination result;
A control method of a watching support system characterized by having.
A program for causing a computer to execute each step of the control method of the watching support system according to claim 8.