JP6822326B2 - Watching support system and its control method - Google Patents

Description

The present invention relates to a technique for assisting in watching over a subject on a bed.

In order to prevent accidents such as falling from the bed, a system that supports watching over patients in hospitals and long-term care facilities is known. In Patent Document 1, the patient's head is detected from an image taken by a camera installed diagonally above the bed, and when the head crosses the boundary line set on the bed, it is determined that the patient is getting up. A system for notifying such people has been proposed.

Japanese Unexamined Patent Publication No. 2012-071004

In the conventional system, a method of detecting the patient's head from an image and estimating the patient's behavior (getting up, getting out of bed, etc.) from the position and movement of the head has been common. However, such a conventional method has a problem that the behavior of the patient cannot be correctly estimated when the head cannot be detected from the image or when the head is erroneously detected. For example, if the patient is wearing a futon, there is a person other than the patient or a confusing object in the surroundings, the lighting environment changes, etc., and the detection accuracy varies depending on the disturbance or environment, it is a dangerous behavior. There is a risk that the system will be overlooked or false alarms (unnecessary alarms) will occur, significantly reducing the reliability of the system.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for detecting the state or behavior of a subject on a bed with high accuracy and high reliability.

In order to achieve the above object, in the present invention, the state of the monitoring target person is estimated from the monitoring image by using a regression device that machine-learns the pattern of the state of the bed and the person, and the behavior of the target person is detected. The method is adopted.

Specifically, the first aspect of the present invention is a watching support system that supports watching over a subject on a bed, in which an image of the bed and a person is input and a score indicating the state of the person with respect to the bed. An image acquisition unit that acquires an image from an image pickup device installed so as to capture a monitoring area including a bed of the subject, and an image acquisition unit that is machine-learned to output the image. A determination unit that determines the behavior of the target person based on the score obtained by inputting the image of the monitoring area into the regression device, and an output unit that gives a notification based on the determination result of the determination unit. It provides a watching support system characterized by having a bed.

According to this configuration, the state of the subject in the input image is estimated using a regression device. Therefore, it is possible to perform highly accurate state estimation for an unknown input image. Moreover, since the score of the regression device is output as a continuous value, a reasonable estimation result can be obtained even if the state of the subject cannot be clearly classified. Furthermore, even if the image is difficult to detect the head, such as when the subject is wearing a futon, there are confusing people or objects around the subject, or the lighting environment is different from normal, a reasonable estimation result is obtained. There is also an advantage that can be obtained.

The regressor may be a neural network. By using a neural network, it is possible to perform state estimation with high accuracy and high robustness.

The state of the person with respect to the bed is classified into a plurality of types in advance, and different scores are assigned to each of the plurality of types, and the regressionr is a state in which the state of the person is between two types. In the case of, it is preferable that the value between the scores of the two types is output. With such a design, various states that a person can take can be expressed by a one-dimensional score, so that it becomes easy to handle the "human state" mathematically or in a program. For example, the latter stage. Processing (judgment logic by the judgment unit, etc.) can be constructed extremely easily.

For example, the plurality of types may include a state 0 in which the person is sleeping on the bed, a state 1 in which the person is awake on the bed, and a state 2 in which the person is away from the bed. .. This is because if at least three types of states 0 to 2 can be discriminated, it is possible to detect "wake up" and "get out of bed", which are highly needed for watching.

The determination unit may determine the behavior of the subject based on the temporal change in the state of the subject estimated from the score. By determining the behavior of the target person based on the temporal change (state transition) of the target person's state instead of the temporary state of the target person, the accuracy of the behavior determination can be improved.

For example, in the determination unit, when the state of the target person changes from the state 0 to the state 1, or the state of the target person changes from the state 0 to the state 1, the state 1 further changes. If it continues for a predetermined time, it may be determined that the subject has woken up.

Further, in the determination unit, when the state of the target person transitions from the state 0 to the state 1 and further transitions from the state 1 to the state 2, or the state of the target person changes from the state 0. Transition to the state 1, further transition from the state 1 to the state 2, and further, when the state 2 continues for a predetermined time, or the state of the subject transitions from the state 0 to the state 2. In this case, or when the state of the subject transitions from the state 0 to the state 2 and the state 2 continues for a predetermined time, it may be determined that the subject has left the bed.

The present invention can be regarded as a watching support system having at least a part of the above-mentioned configuration or function. Further, the present invention provides a monitoring support method or a control method of a watching support system including at least a part of the above processing, a program for causing a computer to execute these methods, or a program such as such. It can also be regarded as a recorded computer-readable recording medium. Each of the above configurations and processes can be combined with each other to construct the present invention as long as there is no technical contradiction.

According to the present invention, the state or behavior of the subject on the bed can be detected with high accuracy and high reliability.

FIG. 1 is a block diagram schematically showing a hardware configuration and a functional configuration of the monitoring support system of the first embodiment. FIG. 2 is a diagram showing an installation example of an imaging device. FIG. 3 is an example of a monitoring area set for an image. FIG. 4 is an example of a human state type and an image. FIG. 5 is a diagram schematically showing machine learning of the regression device. FIG. 6 is a diagram schematically showing the ability of the regressor. FIG. 7 is a flowchart of the state monitoring process. FIG. 8 is a flowchart of action determination. FIG. 9 is an example of the determination logic.

The present invention relates to a technique for assisting in watching over a subject on a bed. This technology can be applied to a system in hospitals, long-term care facilities, etc. that automatically detects the behavior of patients and people requiring long-term care to get out of bed and wake up, and gives necessary notifications when a dangerous situation occurs. This system can be preferably used, for example, to support watching over elderly people, dementia patients, children and the like.

Hereinafter, an example of a preferred embodiment for carrying out the present invention will be described with reference to the drawings. However, the configurations and operations of the devices described in the following embodiments are examples, and the scope of the present invention is not limited to them.

(System configuration)
The configuration of the monitoring support system according to the embodiment of the present invention will be described with reference to FIGS. 1 and 2. FIG. 1 is a block diagram schematically showing a hardware configuration and a functional configuration of the monitoring support system 1, and FIG. 2 is a diagram showing an installation example of an imaging device.

The monitoring support system 1 has an image pickup device 10 and an information processing device 11 as main hardware configurations. The image pickup device 10 and the information processing device 11 are connected by wire or wirelessly. Although only one image pickup device 10 is shown in FIG. 1, a plurality of image pickup devices 10 may be connected to the information processing device 11.

The image pickup device 10 is a device for photographing a subject on a bed and capturing image data. As the image pickup apparatus 10, a monochrome or color visible light camera, an infrared camera, a three-dimensional camera, or the like can be used. In the present embodiment, in order to enable watching over the subject even at night (even when the room is dark), an image pickup device 10 composed of an infrared LED illumination 100 and a near infrared camera 101 is adopted. As shown in FIG. 2, the image pickup apparatus 10 is installed so as to give a bird's-eye view of the entire bed 20 from above the head side to the foot side of the bed 20. The image pickup apparatus 10 takes pictures at predetermined time intervals (for example, 30 fps), and the image data is sequentially taken into the information processing apparatus 11.

The information processing device 11 has a function of analyzing image data captured from the image pickup device 10 in real time, automatically detecting the wake-up behavior and the wake-up behavior of the subject 21 on the bed 20, and notifying the subject 21 when necessary. Is. The information processing device 11 includes an image acquisition unit 110, an area setting unit 111, a preprocessing unit 112, a regression device 113, a score stabilization unit 114, a determination unit 115, an output unit 116, and a storage unit 117 as specific functional modules. Have.

The information processing device 11 of the present embodiment includes a CPU (processor), a memory, a storage (HDD, SSD, etc.), an input device (keyboard, mouse, touch panel, etc.), an output device (display, speaker, etc.), a communication interface, and the like. Each module of the information processing apparatus 11 described above is realized by executing a program stored in a storage or a memory by a CPU. However, the configuration of the information processing device 11 is not limited to this example. For example, distributed computing by a plurality of computers may be performed, a part of the module may be executed by a cloud server, or a part of the module may be configured by a circuit such as an ASIC or FPGA. May be good.

The image acquisition unit 110 is a module for acquiring an image captured by the image pickup apparatus 10. The image data input from the image acquisition unit 110 is temporarily stored in a memory or a storage, and is used for an area setting process and a state monitoring process described later.

The area setting unit 111 is a module for setting a monitoring area for an image captured by the image pickup apparatus 10. The monitoring area is a range of the field of view of the image pickup apparatus 10 that is the target of the state monitoring process (in other words, the image range used as the input of the regressor 113). The details of the area setting process will be described later.

The pre-processing unit 112 is a module for performing necessary pre-processing on an image (hereinafter referred to as "original image") input from the image acquisition unit 110 in the state monitoring process. For example, the preprocessing unit 112 performs a process of clipping an image in the monitoring area from the original image (the clipped image is hereinafter referred to as a “monitoring area image”). Further, the preprocessing unit 112 may perform processing such as resizing (reduction), affine transformation, and luminance correction on the monitored area image. Resizing (reduction) has the effect of shortening the calculation time of the regressor 113. Any existing method may be used for resizing, but the bilinear method, which has a good balance between calculation cost and quality, is preferable. The affine transformation can be expected to have the effect of standardizing the input image to the retriever 113 and improving the estimation accuracy by performing necessary distortion correction such as deforming the bed appearing in a trapezoidal shape in the image into a rectangular shape. .. Luminance correction can be expected to have the effect of improving estimation accuracy by, for example, reducing the influence of the lighting environment. When the original image is input to the regression unit 113 as it is, the preprocessing unit 112 may be omitted.

The regressor 113 is a module for outputting a score indicating a state (for example, a bed state, a wake-up state, a bed-off state) of the target person 21 reflected in the monitoring area image when the monitoring area image is given. .. The regressor 113 takes an image of a bed and a person as an input, and constructs a relational model between the characteristics of the input image and the state of the person by machine learning so as to output a score indicating the state of the person with respect to the bed. .. It is assumed that the training of the regressor 113 is performed in advance by the learning device 12 (before the system is shipped or before the system is put into operation) using a large number of training images. As the learning model of the regressor 113, any model such as a neural network, a random forest, or a support vector machine may be used. In this embodiment, a convolutional neural network (CNN) suitable for image recognition is used.

The score stabilizing unit 114 is a module for suppressing a sudden change or fluttering of the score output from the regressor 113. The score stabilization unit 114 calculates, for example, the average of the current score obtained from the image of the current frame and the past score obtained from the images of the immediately preceding two frames, and outputs the result as the stabilization score. This process is equivalent to applying a temporal low-pass filter to the time series data of the score. If it is not necessary to stabilize the score, the score stabilization unit 114 may be omitted.

The determination unit 115 is a module for determining the behavior of the subject based on the score obtained by the regression device 113. Specifically, the determination unit 115 determines what kind of behavior (for example, getting up behavior, getting out of bed) of the subject based on the temporal change of the score (that is, the transition of the “subject's state” indicated by the score). Etc.) is estimated. Details of the processing of the determination unit 115 will be described later.

The output unit 116 is a module that gives necessary notifications based on the determination result of the determination unit 115. For example, the output unit 116 determines the necessity of notification (for example, notifying only in the case of a dangerous state), the content of the notification (for example, the content of the message), and the notification means (for example, depending on the state or action of the target person 21). Voice, email, buzzer, warning light), notification destination (eg nurse, doctor, caregiver),
You can switch the frequency of notifications.

The storage unit 117 is a module that stores various data used for processing by the monitoring support system 1. The storage unit 117 stores, for example, monitoring area setting information, parameters used in preprocessing, parameters used in score stabilization processing, time-series data of scores, parameters used in determination processing, and the like.

(Setting of monitoring area)
Various objects other than the bed 20 and the subject 21 are reflected in the angle of view of the image pickup apparatus 10. In detecting the state and behavior of the subject 21, it is preferable to exclude the bed 20 and objects other than the subject 21 as much as possible because they may act as noise. Further, for the image to be input to the regression device 113, it is easier to improve the estimation accuracy if the image size (width, height) and the position / range / size of the bed in the image are standardized. Therefore, in the present embodiment, a predetermined range based on the bed 20 is set in the monitoring area, and in the state monitoring process described later, the image in the monitoring area is clipped and used as the input image of the regressor 113.

The monitoring area may be set manually or automatically. In the case of manual setting, the area setting unit 111 may provide a user interface for allowing the user to input the area of the bed 20 in the image or the monitoring area itself. In the case of automatic setting, the area setting unit 111 may detect the bed 20 from the image by the object recognition process, and set the monitoring area so as to include the detected area of the bed 20. The area setting process is executed when the monitoring area is not set (for example, when the system is installed), or when it is necessary to update the monitoring area due to the movement of the bed 20 or the imaging device 10.

FIG. 3A is an example of a monitoring area set for the original image. In the present embodiment, the monitoring area 30 is set to a range in which a margin of a predetermined width is added to the left side, the right side, and the upper side (foot side) of the area of the bed 20. The width of the margin is set so that the whole body of the person standing up on the bed 20 (see FIG. 3B) is within the monitoring area 30.

(State typology and machine learning)
In this system, in order to handle the human condition by regression, the "human condition with respect to the bed" is classified into the third to second three types in advance. "Type 0" is a state in which a person is sleeping on the bed (referred to as "sleeping state" or "state 0"), and "type 1" is a state in which a person is awake on the bed ("wakening state"). "Or" state 1 ")," type 2 "is a state in which a person is away from (getting off) the bed (referred to as" getting out of bed "or" state 2 "). FIG. 4 is an example showing the correspondence between the three types and a time-series image showing a series of actions of a sleeping person getting up and getting out of bed.

FIG. 5 schematically shows the machine learning of the regressor 113. First, images of actual hospital rooms and the like are collected, and each image is classified into the 0th type to the 2nd type. Then, the portion corresponding to the monitoring area of each image is clipped, the type numbers (0, 1, 2) are assigned as labels, and a set of training images is generated. In order to improve the accuracy of regression, it is preferable to prepare a sufficient number of images, and it is preferable to prepare images of various variations for each type. However, it is preferable to exclude an image in which it is unclear which type the human condition belongs to because it is not suitable for a training image.

The learning device 12 uses a set of training images, and a convolutional neural network so as to output the same score as the label for each input image (that is, score = 1 if the input image belongs to the first type). Training. And the learning device 12
The neural network parameter group, which is the learning result, is incorporated into the regression device 113 of this system. The specific layer structure, filter, activation function, input image specifications, etc. of the neural network may be appropriately designed according to the implementation and required accuracy.

FIG. 6 schematically shows the capability of the regressor 113. The regressor 113 models the correspondence between the “feature amount” of the image and the “score” indicating the state of the person. The regressor 113 extracts a feature amount from the input image according to the relational model, calculates a score corresponding to the feature amount, and outputs the feature amount. In FIG. 6, the relational model is shown as a two-dimensional linear model for convenience of explanation, but the actual feature space is multidimensional, and the relational model is non-linear.

The score output from the regressor 113 is a real value (continuous value) in the range of 0 to 2. For example, when an input image of the first type (wake-up state) is given, the output score is 1 or a value extremely close to 1. The same applies to other types. On the other hand, some input images are ambiguous as to which type they belong to, such as a state in which the upper body is about to rise from a lying posture or a state in which the upper body is about to rise from a bed. In the case of an image in such an intermediate state, the extracted feature amount is a feature amount between the two types, so that an intermediate score between the two types is output. For example, in the case of an image in which the upper body is about to be raised from a lying posture, a score larger than 0 and smaller than 1 can be obtained because of the intermediate state between the 0th type and the 1st type.

As described above, in this system, the regression device 113 is used to estimate the state of a person in the input image. Therefore, it is possible to perform highly accurate state estimation for an unknown input image. In particular, the following advantages can be mentioned as advantages as compared with the conventional method using head detection.
-A reasonable estimation result can be obtained even when an image in an intermediate state is input.
・ Even if the image is difficult to detect the head, such as when the subject is wearing a futon, there are confusing people or objects around the subject, or the lighting environment is different from normal, a reasonable estimation result can be obtained. Obtainable.

(Status monitoring process)
An example of the state monitoring process of this system will be described with reference to FIG. 7. The processing flow of FIG. 7 is executed every time one frame of an image is captured from the image pickup apparatus 10.

In step S70, the image acquisition unit 110 captures an image of one frame from the image pickup apparatus 10. The acquired original image is temporarily stored in the storage unit 117. Next, the preprocessing unit 112 clips the monitoring area image from the original image, and executes resizing, affine transformation, brightness correction, and the like as necessary (step S71). Next, the regressor 113 inputs the monitoring area image and outputs the corresponding score (step S72). Next, the score stabilization unit 114 performs the score stabilization process obtained in step S72 (step S73), and delivers the obtained score to the determination unit 115.

First, the determination unit 115 sets the current state of the subject 21 to any one of the 0th type (state 0) / 1st type (state 1) / 2nd type (state 2) based on the score (continuous value). Classify into. The classification method is not limited, but in the present embodiment, when the score ≤ threshold value th1, the state 0 (steps S74, S75), and when the threshold value th1 <score ≤ threshold value th2, the state 1 (step S76, S77), the threshold value th2 < In the case of a score, it is classified as state 2 (step S78). The threshold values th1 and th2 are set, for example, th1 = 0.5 and th2 = 1.5. The detection sensitivity can be adjusted by changing the threshold values th1 and th2.

Subsequently, the determination unit 115 determines the behavior of the subject 21 based on the temporal change of the state of the subject 21 (step S79). In step S79, when it is determined that the subject 21 has "woke up" or "get out of bed", the output unit 116 gives a necessary notification (step S80). The above steps S70 to S80 are executed frame by frame until the system is terminated (step S81).

The determination logic by the determination unit 115 will be described with reference to FIGS. 8 and 9. FIG. 8 is a detailed flow of the action determination process in step S79, and FIG. 9 is an example of determination logic, showing the correspondence between the state transition, the determination result, and the notification setting. Here, the wake-up preparation flag is a flag for confirming the transition from the state 0, and the wake-up preparation flag is a flag for confirming the transition from the state 1. It is assumed that the wake-up preparation flag and the wake-up preparation flag are initialized by "OFF" when the status monitoring process is started.

When the current state is "0" (step S790; YES), the determination unit 115 sets the wake-up preparation flag and the wake-up preparation flag to "ON" (step S791).

When the current state is "1" (step S792; YES), the determination unit 115 confirms whether the wake-up preparation flag is "ON" (step S793), and if it is "ON", the subject 21 is It is determined that the user has woken up (step S794), the wake-up preparation flag is set to "OFF", and the wake-up preparation flag is set to "ON" (step S795).

When the current state is "2" (step S796; YES), the determination unit 115 confirms whether the bed leaving preparation flag is "ON" (step S797), and if it is "ON", the subject 21 It is determined that the person has left the bed (step S798), and the wake-up preparation flag and the bed-leaving flag are set to "OFF" (step S799).

According to the determination logic as described above, as shown in FIG. 9, it is determined as "wake up" when transitioning from state 0 to state 1, and "getting out" when transitioning from state 0 or state 1 to state 2. Is determined. In other cases, the determination unit 115 does not output the determination result or outputs the result of "no abnormality".

According to the present embodiment described above, since the state of the target person 21 is estimated by the regression device 113, the state or behavior of the target person 21 can be determined more accurately than in the conventional method. Further, since the wake-up behavior and the wake-up behavior of the subject 21 are determined based on the temporal change (state transition) of the condition of the subject 21 instead of the temporary state of the subject 21, the accuracy of the behavior determination is improved. can do. Moreover, when the wake-up judgment or the wake-up judgment is made and the notification is given, the wake-up preparation flag and the wake-up preparation flag are once turned "OFF". Judgment does not work. Therefore, it is possible to reduce false alarms of getting up / leaving notifications and improve the reliability of the system.

<Others>
The description of each of the above embodiments is merely an example of the present invention. The present invention is not limited to the above-mentioned specific form, and various modifications can be made within the scope of its technical idea.

For example, in the above embodiment, it is determined that the person wakes up when the state changes from the state 0 to the state 1, but instead, the state 1 continues for a predetermined time (for example, 1 second) after the transition from the state 0 to the state 1. In some cases, it may be determined to wake up. As a result, the reliability of the wake-up determination can be further improved. Similarly, after transitioning from state 0 or state 1 to state 2, if state 2 continues for a predetermined time (for example, 1 second), it may be determined that the patient has left the bed. As a result, the reliability of the determination of getting out of bed can be further improved.

In the above embodiment, an example of estimating the bedtime / wake-up state / wake-up state from the image and detecting the wake-up behavior and the wake-up behavior of the subject has been described. However, the state of the estimation target and the behavior of the detection target are not limited to these. That is, it is possible to handle various "human states" and "behaviors" as long as different features appear in the image. For example, it is possible to detect behaviors such as eating and reading.

1: Watching support system 10: Imaging device, 11: Information processing device, 12: Learning device 100: Lighting, 101: Near infrared camera, 110: Image acquisition unit, 111: Area setting unit, 112: Preprocessing unit, 113: Retractor, 114: Score stabilization unit, 115: Judgment unit, 116: Output unit, 117: Storage unit 20: Bed, 21: Subject 30: Monitoring area

Claims (7)

It is a watching support system that supports watching the target person on the bed.
A neural network that is machine-learned to output a score indicating the state of a person existing in the monitoring area when a monitoring area image, which is an image of the monitoring area set in a predetermined range including a bed, is given as an input. Regressor composed of
An image acquisition unit that acquires an image from the imaging device installed so as to photograph the front Ki監vision area,
A determination unit that determines the behavior of the target person based on a score obtained by inputting the monitoring area image extracted from the image obtained by the image acquisition unit into the regression unit.
An output unit that gives notification based on the determination result of the determination unit, and
Have a,
The state of the person is classified into a plurality of types in advance, and different scores are assigned to each of the plurality of types.
The regressor outputs the state of the person as a continuous value score, and when the state of the person is a state between two types, the value between the scores of the two types is output. A monitoring support system characterized by being configured to output . The plurality of types are characterized by including a state 0 in which the person is sleeping on the bed, a state 1 in which the person is awake on the bed, and a state 2 in which the person is away from the bed. The monitoring support system according to claim 1 . The monitoring support system according to claim 2 , wherein the determination unit determines the behavior of the subject based on a temporal change in the state of the subject estimated from the score. The determination unit
When the state of the subject transitions from the state 0 to the state 1, or
When the state of the subject transitions from the state 0 to the state 1, and the state 1 continues for a predetermined time,
The monitoring support system according to claim 3 , wherein it is determined that the subject has woken up. The determination unit
When the state of the subject transitions from the state 0 to the state 1, and further transitions from the state 1 to the state 2, or
When the state of the subject transitions from the state 0 to the state 1, further transitions from the state 1 to the state 2, and the state 2 continues for a predetermined time, or.
When the state of the subject transitions from the state 0 to the state 2, or
When the state of the subject transitions from the state 0 to the state 2, and the state 2 continues for a predetermined time,
The monitoring support system according to claim 3 or 4 , wherein it is determined that the subject has left the bed. It is a control method of a watching support system that supports watching over the subject on the bed.
A step of acquiring an image from an imaging device installed so as to capture a monitoring area set in a predetermined range including the subject's bed, and
When a monitoring area image, which is an image of the monitoring area, is given as an input, a regressor composed of a neural network machine-learned to output a score indicating the state of a person existing in the monitoring area is subjected to. inputting the monitoring territory Ikiga image,
A step of determining the behavior of the subject based on the score obtained from the regression device, and
Steps to notify based on the judgment result and
Have a,
The state of the person is classified into a plurality of types in advance, and different scores are assigned to each of the plurality of types.
The regressor outputs the state of the person as a continuous value score, and when the state of the person is a state between two types, the value between the scores of the two types is output. A control method for a monitoring support system that is configured to output . A program for causing a computer to execute each step of the control method of the monitoring support system according to claim 6 .

Images