WO2019003859A1

WO2019003859A1 - Monitoring system, control method therefor, and program

Info

Publication number: WO2019003859A1
Application number: PCT/JP2018/021984
Authority: WO
Inventors: 純平松永; 田中　清明; 信二高橋; 達哉村上
Original assignee: オムロン株式会社
Priority date: 2017-06-27
Filing date: 2018-06-08
Publication date: 2019-01-03
Also published as: JP2019008638A; JP6822328B2

Abstract

This monitoring system for assisting monitoring of a subject on a bed has: an image acquisition unit which acquires an image from an imaging device installed so as to photograph a monitoring area including the bed of the subject; a status quantification unit which, on the basis of the image of the monitoring area obtained by the image acquisition unit, outputs a score obtained by quantifying the status of the subject; and a status display unit which displays, on a display device, a graph indicating the temporal change of the score outputted from the status quantification unit.

Description

Oversight support system, control method therefor, and program

TECHNICAL FIELD The present invention relates to a technique for supporting watching of a subject on a bed.

In order to prevent a fall accident from a bed and the like, a system that supports watching and listening of patients in hospitals and nursing facilities is known. In patent document 1, a human sensor, a respiration sensor, a body movement sensor, a door sensor, etc. are installed in each living room of the facility so that the condition of the resident can be grasped without contact, and a server installed in the facility Sensor data from each sensor is collected, and the present situation display screen of the management PC allows the resident's number of breaths, the amplitude of body movement and the feeling of people to be able to instantly grasp the situation of multiple tenants in the facility An arrangement for graphically displaying the detection results of sensors is disclosed. Patent Document 2 discloses displaying a graph of body motion, respiration and the like. Patent Document 3 discloses that an alert icon is displayed when a fall of a patient occurs.

JP, 2017-016611, A JP, 2009-082511, A JP, 2008-289676, A

As in

Patent Documents

1 and 2, the watching side (nurse, doctor, carer, etc.) can monitor the presence or absence of the movement of the patient and the size by displaying a graph of the body movement measured by the sensor. . However, it can not be determined whether or not the movement is due to the dangerous behavior only by the presence or absence and the size of the movement. For example, it can not be distinguished from the output of the motion sensor alone whether the patient is about to leave or fall from the bed or merely stretches on the bed. In other words, the output of the body movement sensor contains many noises (signals due to movements other than the action desired to be detected), so that it is not possible to accurately detect or predict the state or action of the patient. In addition, it is not possible to prevent an accident such as falling from a bed or falling by simply outputting an alert when a fall or the like occurs as in Patent Document 3.

The present invention has been made in view of the above-described circumstances, and an object thereof is to provide a technique for detecting the state or behavior of a subject on a bed with high accuracy and high reliability.

In order to achieve the above object, the present invention adopts a method of generating a score that quantifies the condition of a subject based on an image and graphically displaying a time change of the score.

Specifically, a first aspect of the present invention is a watching support system that supports watching of a subject on a bed, and an imaging apparatus installed to shoot a monitoring area including the bed of the subject An image acquisition unit for acquiring an image, a state quantification unit for outputting a score obtained by quantifying the condition of the subject based on the image of the monitoring area acquired by the image acquisition unit, and the state quantification unit And a status display unit for displaying on the display device a graph indicating temporal change of the score output from the monitoring device.

According to this configuration, a graph indicating temporal changes in the score that quantified the condition of the subject is output, so the watching side (nurse, carer, etc.) can easily change the condition or trend of the subject. It can be confirmed. Then, if the change or tendency of the state is known, for example, the action of the subject can be predicted to prevent the occurrence of danger, or the daily action pattern of the subject can be grasped and used for watching.

The state quantifying unit has a regressor machine-learned to receive a bed and an image of a person and output a score indicating the state of the person relative to the bed, and the image of the monitoring area is It is good to acquire the score which quantified the said subject's state by inputting into a regressor. Since the state of the subject in the input image is estimated using the regressor, highly accurate state estimation can be performed on the unknown input image. Further, since the score of the regressor is output as a continuous value, it is possible to obtain a reasonable estimation result even if the condition of the subject can not be clearly classified. Furthermore, even if the target person is covered with a futon, there are strange people or objects around the target person, or the lighting environment is different from normal, etc., the image is difficult to detect the head. You can expect to get

The regressor may be a neural network. By using a neural network, highly accurate and highly robust state estimation can be performed.

The state of the person with respect to the bed is classified in advance into a plurality of types, and different scores are assigned to each of the plurality of types, and the regressor is configured to determine that the state of the person is between two types. If so, it may be configured to output a value between the two types of scores. Such a design makes it possible to express various human-readable states as a one-dimensional score, making it easy to handle "human states" mathematically or in programs, for example, The processing of (the graph output by the status display unit etc.) can be constructed extremely easily.

For example, the plurality of types may include state 0 in which the person is sleeping in the bed, state 1 in which the person is rising on the bed, and state 2 in which the person is away from the bed . This is because if it is possible to distinguish at least three types of states of state 0 to state 2, it is possible to detect “wake up” and “getting out of bed” which have a high need for watching.

The score quantifying the condition of the subject may be a score representing the degree of danger of the condition of the subject. By displaying the degree of danger of the condition of the subject graphically, it is possible for the watching side to easily predict the danger of the subject and to prevent the occurrence of an accident or the like.

A determination criteria storage unit in which determination criteria for determining a dangerous state are set in advance for each of a plurality of determination areas set based on the bed area in the image of the monitoring area, the state quantification The unit has a detection unit that detects the head of the subject from the image of the monitoring area, and the judgment unit of the judgment area corresponding to the position at which the head is detected It is good to calculate the score showing the degree of danger of a state. According to this configuration, it is possible to easily and accurately determine from the image whether the subject on the bed is in a safe state or in a dangerous state.

The state quantifying unit determines the degree of danger of the state of the subject based on at least one of the head orientation, head movement speed, head movement direction, and head movement vector. It is good to calculate the score to express.

The present invention can be understood as a watching support system having at least a part of the above configuration or function. The present invention also provides a watching support method or a watching support system control method including at least a part of the above-described processing, a program for causing a computer to execute these methods, or non-temporarily such a program. It can also be regarded as a recorded computer readable recording medium. Each of the above configurations and processes can be combined with each other as long as there is no technical contradiction.

According to the present invention, the state or behavior of the subject on the bed can be detected with high accuracy and high reliability.

FIG. 1 is a block diagram schematically showing a hardware configuration and a functional configuration of the watching support system of the first embodiment. FIG. 2 is a view showing an installation example of the imaging device. FIG. 3A and FIG. 3B are examples of the monitoring area set for the image. FIG. 4 shows types of human states and examples of images. FIG. 5 is a diagram schematically showing machine learning of the regressor. FIG. 6 is a diagram schematically showing the ability of the regressor. FIG. 7 is a flowchart of the state monitoring process. FIG. 8 is an example of the status display. FIG. 9 is a block diagram schematically showing the hardware configuration and the functional configuration of the watching support system of the second embodiment. 10A to 10C are examples of determination regions set for an image. FIG. 11A is an example of a data structure of the determination reference of head orientation for each determination area, and FIG. 11B is a diagram for explaining codes representing eight directions. FIG. 12 is a flowchart of the state monitoring process of the first example of the second embodiment. FIG. 13: is an example of danger degree determination of Example 1 of 2nd Embodiment. FIG. 14 is a state display example of the second embodiment. FIG. 15 is a flowchart of the state monitoring process of the second embodiment of the second embodiment. FIG. 16: is an example of danger degree determination of Example 2 of 2nd Embodiment. FIG. 17 is a flowchart of the state monitoring process of the third example of the second embodiment. FIG. 18 is a flowchart of the state monitoring process of the fourth example of the second embodiment.

TECHNICAL FIELD The present invention relates to a technique for supporting watching of a subject on a bed. This technology can be applied to a system that automatically detects the getting-up and getting-up behavior of patients and care recipients in hospitals and nursing facilities, etc., and performs necessary notification when a dangerous state occurs. This system can be preferably used, for example, for watching and supporting elderly people, patients with dementia, children and the like.

Hereinafter, an example of a preferred embodiment for carrying out the present invention will be described with reference to the drawings. However, the configurations and operations of the devices described in the following embodiments are merely examples, and the scope of the present invention is not limited thereto.

First Embodiment
(System configuration)
The configuration of a watching support system according to an embodiment of the present invention will be described with reference to FIGS. 1 and 2. FIG. 1 is a block diagram schematically showing a hardware configuration and a functional configuration of the watching support system 1, and FIG. 2 is a diagram showing an installation example of an imaging device.

The watching support system 1 includes an imaging device 10, an information processing device 11, and a display device 13 as main hardware configurations. The imaging device 10 and the information processing device 11 are connected by wire or wirelessly. Further, the information processing device 11 and the display device 13 are also connected by wire or wirelessly. For example, the child machine unit including the imaging device 10 and the information processing device 11 is installed in the room of the target person 21 while the parent machine unit including the display device 13 is installed in the nurse center or the central monitoring room. A configuration in which communication is performed by wired LAN or wireless LAN can be adopted between machine units. Alternatively, all or part of the information processing apparatus 11 may be provided on the base unit side. Also, a plurality of slave units may be connected to one master unit.

The imaging device 10 is a device for capturing a subject on a bed and capturing image data. As the imaging device 10, a monochrome or color visible light camera, an infrared camera, a three-dimensional camera or the like can be used. In the present embodiment, the imaging device 10 configured by the infrared LED illumination 100 and the near infrared camera 101 is adopted in order to enable watching of the target person even at night (even when the room is dark). As shown in FIG. 2, the imaging device 10 is installed to look over the entire bed 20 from the top of the bed 20 to the foot. The imaging device 10 captures an image at a predetermined time interval (for example, 30 fps), and the image data is sequentially captured by the information processing device 11.

The information processing apparatus 11 is an apparatus provided with a function of analyzing image data taken in from the imaging apparatus 10 in real time, detecting the state and behavior of the subject 21 on the bed 20, and outputting the detection result to the display apparatus 13. is there. The information processing apparatus 11 includes, as specific functional modules, an image acquisition unit 110, an area setting unit 111, a preprocessing unit 112, a regression unit 113, a score stabilization unit 114, a determination unit 115, a state display unit 116, and a storage unit 117. have. In the present embodiment, the preprocessing unit 112, the regression unit 113, the score stabilizing unit 114, and the determining unit 115 constitute a state quantifying unit that outputs a score that quantifies the condition of the target person 21. .

The information processing apparatus 11 according to the present embodiment includes a general-purpose computer including a CPU (processor), memory, storage (HDD, SSD, etc.), input device (keyboard, mouse, touch panel, etc.), communication interface, etc. Each module of the information processing apparatus 11 is realized by the CPU executing a program stored in the storage or the memory. However, the configuration of the information processing apparatus 11 is not limited to this example. For example, distributed computing may be performed by a plurality of computers, a part of the module may be executed by a cloud server, or a part of the module may be configured by a circuit such as an ASIC or an FPGA. It is also good.

The image acquisition unit 110 is a module for acquiring an image captured by the imaging device 10. The image data input from the image acquisition unit 110 is temporarily stored in a memory or storage, and is used for area setting processing and status monitoring processing described later.

The area setting unit 111 is a module for setting a monitoring area for an image captured by the imaging device 10. The monitoring area is a range (in other words, an image range used as an input of the regressor 113) to be subjected to the state monitoring process in the field of view of the imaging device 10. Details of the area setting process will be described later.

The preprocessing unit 112 is a module for performing necessary preprocessing on an image (hereinafter referred to as an “original image”) input from the image acquisition unit 110 in the state monitoring process. For example, the preprocessing unit 112 performs processing of clipping an image within the monitoring area from the original image (hereinafter, the clipped image is referred to as a “monitoring area image”). In addition, the preprocessing unit 112 may perform processing such as resizing (reduction), affine transformation, and luminance correction on the monitoring area image. Resizing (reduction) has the effect of shortening the calculation time of the regressor 113. Although any existing method may be used for resizing, it is preferable to use a bilinear method that has a good balance of calculation cost and quality. The affine transformation can be expected to have the effect of normalizing the input image to the regressor 113 and improving the estimation accuracy by performing necessary distortion correction such as, for example, deforming a bed reflected in a trapezoidal shape in the image into a rectangular shape . The luminance correction can be expected to have an effect of improving the estimation accuracy, for example, by reducing the influence of the illumination environment. When the original image is input to the regressor 113 as it is, the preprocessing unit 112 may be omitted.

The regressor 113 is a module for outputting a score indicating the state of the subject 21 (for example, bedtime state, wakeup state, leaving state) shown in the monitoring area image when the monitoring area image is given. . Regressor 113 has constructed a relationship model between features of the input image and human state by machine learning so that the image showing the bed and the person is taken as input, and outputs a score quantitatively indicating the human state with respect to the bed. It is a thing. It is assumed that the training of the regressor 113 is performed in advance (before shipping or operation of the system) by the learning device 12 using a large number of training images. In addition, as a learning model of the regressor 113, any model such as a neural network, a random forest, a support vector machine, etc. may be used. In this embodiment, a convolutional neural network (CNN) suitable for image recognition is used. In addition, the score of this embodiment is also called a "state score."

The score stabilization unit 114 is a module for suppressing rapid change and fluttering of the score output from the regressor 113. The score stabilization unit 114, for example, calculates an average of the current score obtained from the image of the current frame and the past score obtained from the images of the immediately preceding two frames, and outputs the average as a stabilization score. This process is equivalent to applying a temporal low-pass filter to the time series data of the score. In addition, if stabilization of a score is unnecessary, the score stabilization part 114 may be abbreviate | omitted.

The determination unit 115 is a module for determining the action of the subject based on the score obtained by the regressor 113. Specifically, the determination unit 115 determines what kind of behavior (for example, wake-up action, leaving-behind action) of the subject based on temporal change in score (that is, transition of the “subject's state” indicated by the score). Etc.) are estimated. Details of the processing of the determination unit 115 will be described later.

The state display unit 116 is a module that displays on the display device 13 a graph (hereinafter referred to as a “state change graph”) indicating temporal change of the score output from the score stabilization unit 114 on a real time basis.

The storage unit 117 is a module for storing various data used by the watching support system 1 for processing. The storage unit 117 stores, for example, setting information of a monitoring area, parameters used in preprocessing, parameters used in score stabilization processing, time series data of scores, parameters used in determination processing, and the like.

(Setting of monitoring area)
Various items other than the bed 20 and the target person 21 are reflected in the angle of view of the imaging device 10. When detecting the state or behavior of the target person 21, it is preferable to exclude as much as possible, since things other than the bed 20 and the target person 21 may act as noise. Further, with regard to the image input to the regressor 113, the estimation accuracy can be easily improved if the image size (width, height), and the position, range, and size of the bed in the image are standardized. Therefore, in the present embodiment, a predetermined range based on the bed 20 is set as a monitoring area, and in a state monitoring process described later, an image in the monitoring area is clipped to be an input image of the regression unit 113.

The setting of the monitoring area may be performed manually or automatically. In the case of manual setting, the area setting unit 111 may provide a user interface for allowing the user to input the area of the bed 20 in the image or the monitoring area itself. In the case of automatic setting, the area setting unit 111 may detect the bed 20 from the image by object recognition processing, and set the monitoring area so as to include the detected area of the bed 20. The area setting process is performed, for example, when the monitoring area needs to be updated along with the movement of the bed 20 or the imaging device 10 when the monitoring area is not set (for example, when the system is installed).

FIG. 3A is an example of the monitoring area set for the original image. In the present embodiment, a monitoring area 30 is set by adding a margin of a predetermined width to the left side, the right side, and the upper side (foot side) of the area of the bed 20. The width of the margin is set so that the whole body of the person (see FIG. 3B) rising on the bed 20 falls within the monitoring area 30.

(Types of states and machine learning)
In this system, in order to treat the human condition by regression, "the human condition with respect to the bed" is classified into three types from 0 to 2 in advance. The "type 0 type" is a state in which a person is sleeping in the bed (referred to as "sleeping state" or "state 0"), the "type 1 type" is a state in which a person is rising on the bed ("wake up state Or “state 1”), “type 2” is a state in which a person is separated from the bed (dismounted) (referred to as “bed leaving state” or “state 2”). FIG. 4 is an example showing correspondence between time-series images representing a series of actions of a person who was sleeping rising and leaving the bed, and three types.

FIG. 5 schematically shows the machine learning of the regressor 113. First, images obtained by photographing an actual patient room and the like are collected, and each image is classified into type 0 to type 2. Then, a portion corresponding to the monitoring area of each image is clipped, and the type number (0, 1, 2) is assigned as a label to generate a set of training images. In order to improve the accuracy of regression, it is preferable to prepare a sufficient number of images, and it is preferable to prepare images of various variations for each type. However, it is preferable to exclude the image which is ambiguous to which type of human condition belongs, because it is not suitable for the training image.

The learning device 12 uses a set of training images and outputs a convolutional neural network so as to output, for each input image, the same score as its label (that is, score = 1 if the input image belongs to the first type) Do training. Then, the learning device 12 incorporates the parameter group of the neural network, which is the learning result, into the regressor 113 of the present system. The specific layer structure of the neural network, the filter, the activation function, the specification of the input image, and the like may be appropriately designed according to the mounting and the required accuracy.

FIG. 6 schematically shows the ability of the regressor 113. The regressor 113 models the correspondence between the “feature amount” of the image and the “score” indicating the human state. The regressor 113 extracts the feature amount from the input image according to the relationship model, and calculates and outputs a score corresponding to the feature amount. Although FIG. 6 shows the relationship model as a two-dimensional linear model for the convenience of description, the actual feature amount space is multidimensional, and the relationship model is non-linear.

The score output from the regressor 113 is a real value (continuous value) in the range of 0-2. For example, when an input image of the first type (wake-up state) is given, the output score becomes 1 or a value very close to 1. The same is true for other types. On the other hand, in the input image, there is also an ambiguous image as to which type it belongs to, such as a state in which the upper body is about to wake up from the sleeping position or a state in which it is about to stand up from the bed. In the case of such an intermediate state image, the extracted feature quantity is a feature quantity between the two types, so that an intermediate score between the two types is output. For example, in the case of an image in a state in which the upper body is about to wake up from the sleeping position, a score of a value larger than 0 and smaller than 1 is obtained because it is an intermediate state between the 0th type and the 1st type.

Thus, in the present system, the regressor 113 is used to estimate the human state in the input image. Therefore, highly accurate state estimation can be performed on an unknown input image. In addition, even when an intermediate state image is input, a reasonable estimation result can be obtained. Furthermore, even if the target person is covered with a futon, there are strange people or objects around the target person, or the lighting environment is different from normal, etc., the image is difficult to detect the head. You can get

(Status monitoring process)
An example of the state monitoring process of the present system will be described with reference to FIG. The processing flow of FIG. 7 is executed each time an image of one frame is captured from the imaging device 10.

In step S 70, the image acquisition unit 110 captures an image of one frame from the imaging device 10. The acquired original image is temporarily stored in the storage unit 117. Next, the preprocessing unit 112 clips the monitoring area image from the original image, and executes resizing, affine transformation, luminance correction and the like as necessary (step S71). Next, the regressor 113 inputs the monitoring area image and outputs the corresponding score (step S72). Next, the score stabilization unit 114 performs stabilization processing of the score obtained in step S72 (step S73), and delivers the obtained score to the determination unit 115.

The determination unit 115 classifies the current state of the subject 21 into any one of a bed-up state / wake-up state / off-bed state based on the score (continuous value). There is no limitation on the classification method, but in the present embodiment, the bed state (steps S74 and S75) in the case of score ≦ threshold th1, the wake state (steps S76 and S77) in the case of threshold th1 <score ≦ th2, the threshold th2 In the case of <score, it is classified as leaving state (step S78). The threshold values th1 and th2 are set, for example, as th1 = 0.5 and th2 = 1.5. The detection sensitivity can be adjusted by changing the thresholds th1 and th2.

Subsequently, the state display unit 116 outputs the score as the detection result and the state of the target person 21 to the display device 13 (step S79). The above steps S70 to S79 are executed for each frame until the system is completed (step S80).

(Display example)
FIG. 8 shows an example of the status display screen output to the display device 13. On the left side of the screen, a state image 80 graphically showing the current state of each of the watchers A to D is displayed. In the example of FIG. 8, it is shown that the subjects A and C are in the bed, the subject B is in the wake, and the subject D is in the bed. Further, the state change graph 81 of each of the subjects A to D is displayed on the right side of the screen. The horizontal axis of the state change graph 81 is time, and the vertical axis is a state score. In addition, a marker 82 indicating the time of the current time is displayed so that the state score of each of the subjects A to D at the current time can be known.

According to this embodiment described above, since the state estimation of the subject 21 is performed by the regressor 113, it is possible to accurately determine the state or the behavior of the subject 21. Further, since the state change graph 81 indicating the time change of the state score which quantified the state of the subject 21 is output, the watching side (nurse, carer etc.) can easily change the state or tendency of the subject 21. It can be confirmed. And if the change and the tendency of the state are known, for example, the action of the target person 21 can be predicted to prevent the occurrence of danger, or the daily action pattern of the target person 21 can be grasped and used for watching .

Second Embodiment
Next, a second embodiment of the present invention will be described. In the first embodiment, the state of the target person 21 is estimated by the regressor, whereas in the present embodiment, the head of the target person 21 is detected, and the state (risk degree) of the target person 2 from the position and state of the head. Estimate

FIG. 9 is a block diagram schematically showing a hardware configuration and a functional configuration of the watching support system 1 of the second embodiment. The difference in configuration from the first embodiment (FIG. 1) is that the state quantifying unit is configured by the detection unit 90, the state recognition unit 91, and the determination unit 92, and the area setting unit 93 and the state The function of the display unit 94 is different. Hereinafter, the configuration specific to the present embodiment will be mainly described, and the description overlapping with the first embodiment will be omitted.

The detection unit 90 is a module that analyzes the image acquired by the image acquisition unit 110 and detects the human body of the watching target person 21 or a part thereof (head, face, upper body, etc.) from the image. Any method may be used as a method of detecting a human body or a part thereof from an image. For example, it is possible to preferably use an object detection algorithm using a classical Haar-like feature or a classifier using HoG feature or a recent Faster R-CNN. The detection unit 90 of the present embodiment detects the head (portion above the neck) 22 of the object person 21 by the classifier using the Haar-like feature amount, and as a detection result, the position (x, y) and the size (the number of vertical and horizontal pixels) are output. The position (x, y) of the head 22 is represented by, for example, image coordinates of a central point of a rectangular frame surrounding the head 22. In addition, although the detection part 90 of this embodiment outputs a detection result by the position and size of an image coordinate system, the detection part 90 converts an image coordinate system into a space coordinate system, The three-dimensional in the space coordinate system of the object person 21 The position or three-dimensional size may be output.

The state recognition unit 91 is a module that recognizes the state of the subject 21 detected by the detection unit 90. In the present embodiment, the state of the head 22 of the target person 21, specifically, (1) the orientation of the head 22, (2) the moving speed of the head 22, and (3) the moving direction of the head 22. Calculate at least one of the

The orientation of the head 22 may be recognized based on, for example, the positional relationship of the facial organs (eyes, nose, mouth, etc.) in the image of the head 22, or multiple types of identification learned for each orientation of the head 22 It may be recognized by using the device, or another algorithm may be used. In addition, for the direction of the head 22, a continuous value (angle) may be calculated, and for example, which of N types of directions (directions) determined in advance such as right direction, front direction, and left direction correspond to You may judge. In addition, the directions around three axes of yaw, pitch, and roll may be calculated, or the directions in the image coordinate system (in the xy plane) may be simply calculated.

The moving speed of the head 22 is the moving amount of the head 22 per predetermined time. For example, the moving speed can be obtained by calculating the distance between the position of the head 22 in the image of a plurality of frames earlier and the position of the head 22 in the latest image. In addition, the moving direction of the head 22 can be calculated, for example, from the direction (angle) of a line connecting the position of the head 22 in the image of a plurality of frames earlier and the position of the head 22 in the latest image. The movement velocity and the movement direction may be combined to obtain the movement vector of the head 22. Also in this case, the movement speed, movement direction, and movement vector in the real space (three-dimensional space) may be calculated by converting the image coordinate system into a space coordinate system.

The determination unit 92 is a module that determines whether the state of the subject 21 is a safe state or a dangerous state based on the results of the detection unit 90 and the state recognition unit 91. Specifically, the determination unit 92 uses the “determination criteria” for evaluating and determining a dangerous condition, the state of the head 22 recognized by the state recognition unit 91 (direction, moving speed, moving direction, A process of determining the "danger degree" of at least one of the movement vectors is performed. The determination of the degree of danger may be a two-stage determination of safety / danger, or may be a multiple-stage determination such as the degree of danger = 0, 1, 2,. This degree of danger is also called a degree of danger score. In the present embodiment, a plurality of determination criteria are set in advance in the determination criteria storage unit in the storage unit 117, and the determination unit 92 uses in accordance with the position at which the target person 21 (head 22) is detected. Switch judgment criteria. This feature will be described in detail later.

The area setting unit 93 is a module for setting a monitoring area and a determination area on an image. The state display unit 94 is a module that displays in real time a graph indicating temporal change of the risk score on the display device 13.

(Setting of judgment area)
An example of setting processing of the monitoring area and the determination area will be described with reference to FIGS. 10A to 10C. 10A to 10C are examples of the monitoring area and the determination area set for the image.

First, the image acquisition unit 110 acquires an image from the imaging device 10 (FIG. 10A). Next, the area setting unit 93 allows the user to input the monitoring area 30 and the points 40 to 43 at the four corners of the bed, and sets a quadrangle surrounded by the four points 40 to 43 in the bed area 44 (FIG. 10B). Next, the area setting unit 93 calculates the range of the four determination areas A1 to A4 based on the bed area 44 (FIG. 4C). Then, the area setting unit 93 stores the information of the monitoring area 30 and the bed area 44 (the coordinates of the four corners of the bed area 44) and the information of the four determination areas A1 to A4 (the coordinates of the four corners of each determination area) in the storage unit 117. Store and complete the setting process.

The determination area A1 is an area set on the head side of the bed 20, and corresponds to the range in which the head 22 may exist when the subject 21 sleeps in the bed 20. The judgment area A2 is an area set at the center of the foot side of the bed 20, and the head 22 when raising the upper body from the state where the subject 21 is sleeping or when getting down or falling from the foot side of the bed 20. Corresponds to the range in which The judgment area A3 is an area set on the left side of the foot side of the bed 20, and when the subject 21 is seated on the left edge of the bed 20, or when the subject 21 descends or falls from the left side of the bed 20, the head 22 corresponds to the range that may exist. The determination area A4 is an area set on the right side of the foot side of the bed 20, and the head is located when the subject 21 is seated on the right edge of the bed 20 or when the subject 21 descends or falls from the right side of the bed 20. 22 corresponds to the range that may exist. As shown in FIG. 10C, the determination areas A2 to A4 extend to the outside of the bed area 44.

The reason for setting a plurality of determination areas in this way is that the evaluation of the safe state or the dangerous state may change depending on where the subject 21 is present on the bed 20. For example, when the head 22 of the subject 21 is present in the determination area A1, it is considered that the subject 21 is lying on the bed 20 in a normal posture, and the head 22 has largely moved or changed its direction. Also, the risk of the subject 21 falling from the bed 20 can be said to be low. When the head 22 of the target person 21 is present in the judgment area A3, if the head 22 faces the left, it is considered that the target person 21 is about to get off the bed 20 by his own intention. Although the degree of danger can be evaluated as low, if the head 22 is facing up, down or to the right, it should be judged that there is any abnormality or there is a risk of falling. The determination area A4 is determined opposite to the left and right of the determination area A3.

FIG. 11A illustrates an example of the data structure of the determination reference set in the storage unit 117. FIG. 11A is an example of a determination criterion of head orientation. Symbols such as “−1, −1”, “−1, 0”,... Indicate head orientation (eight directions) as shown in FIG. 11B, and the value of the judgment criterion indicates the degree of danger . The larger the value, the higher the degree of danger, with 1 being the head with the lowest degree of risk and 5 being the head with the highest degree of risk. As described above, since the evaluation of the safe state or the dangerous state changes in each of the determination areas A1 to A4, the determination criteria storage unit in the storage unit 117 is associated with different determination criteria for each determination area. There is. Although the example of FIG. 11A is the determination criterion of head orientation, determination criteria corresponding to a plurality of items used for evaluation by the determination unit 92 may be set, such as the moving speed and movement direction of the head.

Next, a specific example of the state monitoring process by the watching support system 1 of the second embodiment will be described.

Example 1
An example of the state monitoring process of the first embodiment will be described with reference to FIGS. 12, 13 and 14. FIG. 12 is a flowchart of the state monitoring process of the first embodiment executed by the information processing apparatus 11. FIG. 13 is an example of risk degree determination. FIG. 14 is a state display screen displayed on the display device 13. An example of

In step S 60, the image acquisition unit 110 acquires an image from the imaging device 10. The acquired image is temporarily stored in the storage unit 117. In step S61, the detection unit 90 detects the head 22 of the target person 21 from the monitoring area in the image acquired in step S60. In step S62, the state recognition unit 91 estimates the orientation of the head detected in step S61. FIG. 13 illustrates an example in which the head 22 is detected in the determination area A3 and the direction of the head 22 is estimated to be the direction of the arrow 70.

In step S63, the determination unit 92 reads the determination reference corresponding to the determination area A3 from the storage unit 117. The reference numeral 71 in FIG. 13 schematically shows the determination criterion corresponding to the determination area A3, and the danger degree (1 to 5) is set in each of eight directions (arrows). Then, in step S64, the determination unit 92 determines whether the direction of the head 22 (arrow 70) is a safe direction or a dangerous direction, using the determination criterion corresponding to the determination area A3. In the example of FIG. 13, the determination result of the safe direction (degree of danger = 1) is obtained. That is, since the direction of the head 22 faces the outside of the bed, it is considered that the subject is about to get off the bed by his own intention, and the degree of danger is determined to be low. The determination result (risk score) of the determination unit 92 is stored in the storage unit 117.

Subsequently, the state display unit 94 outputs the degree of danger score and the state of the target person 21 to the display device 13 (step S65). The above steps S60 to S65 are executed for each frame until the system is completed (step S66).

FIG. 14 shows an example of the status display screen output to the display device 13. On the left side of the screen, a state image 83 graphically showing the current state of each of the watchers A to D is displayed. In the example of FIG. 14, it is shown that the risk score of the subject A is 1, the risk score of the subjects B and D is 2, and the risk score of the subject C is 5. For example, an alert may be notified such as blinking display when the degree of risk score is 3 or more. Further, the state change graph 84 of each of the subjects A to D is displayed on the right side of the screen. The horizontal axis of the state change graph 84 is time, and the vertical axis is the risk score. In addition, a marker 85 indicating the time of the current time is displayed so that the state score of each of the subjects A to D at the current time can be known.

According to this embodiment described above, since the state change graph 84 indicating the time change of the risk score which quantified the state (risk degree) of the object person 21 is output, the watching side (nurse, carer etc.) Can easily confirm the change and tendency of the degree of danger of the subject person 21. Therefore, as in the first embodiment, it is possible to easily predict the danger of the object person 1 and to prevent the occurrence of an accident or the like.

(Example 2)
An example of the state monitoring process of Example 2 of the second embodiment will be described with reference to FIGS. 15 and 16. FIG. 15 is a flowchart of the state monitoring process of the second embodiment executed by the information processing apparatus 11, and FIG. 16 is an example of the risk degree determination.

In step S 150, the image acquisition unit 110 acquires an image from the imaging device 10. The acquired image is temporarily stored in the storage unit 117. In step S151, the detection unit 90 detects the head 22 of the target person 21 from the image acquired in step S150. Information on the position of the detected head 22 is stored in the storage unit 117 in association with time information or a frame number of the image. Next, the state recognition unit 91 reads the information of the position of the head 22 in the image of a predetermined time before (for example, one second) from the storage unit 117 (step S152), and the position of the head 22 of the predetermined time and the step Based on the position of the head 22 detected in S151, the movement speed (movement amount per predetermined time) of the head 22 and the movement direction are calculated, and the movement vector of the head 22 is determined (step S153). FIG. 16 shows an example in which the head 22 is detected in the determination area A3 and the movement vector of the head 22 is calculated as indicated by the arrow 150.

In step S154, the determination unit 92 reads the determination reference corresponding to the determination area A3 from the storage unit 117.

Reference numerals

151 and 152 in FIG. 16 schematically show the judgment criteria corresponding to the judgment area A3. The code | symbol 151 is an example of the determination standard regarding a moving direction. In addition, reference numeral 152 is an example of a determination criterion regarding the moving speed (moving amount), and indicates that the longer the arrow, the larger the degree of danger. In this example, four levels of danger of 1 to 4 are assigned according to the moving speed.

In step S 155, the determination unit 92 determines the degree of danger of the movement vector 150 of the head 22 using the

determination criteria

151 and 152. For example, the product (multiplication value) or sum (addition value) or the like of the danger degree in the movement direction and the danger degree in the movement speed can be used as the danger degree of the movement vector 150. In the example of FIG. 16, the movement direction is safe (danger degree = 1), but the movement speed is large (danger degree = 4), so the judgment result that the danger degree of the movement vector 90 is 4 (in the case of multiplication value) is can get. In other words, looking only at the moving direction of the head 22, the moving speed (moving amount) is very high although it is the direction of getting down from the bed, so it is not a normal getting down from the bed but a falling or falling movement from the bed It is considered that the risk is high. The determination result (risk score) of the determination unit 92 is stored in the storage unit 117.

Subsequently, the state display unit 94 outputs the degree of danger score and the state of the subject 21 to the display device 13 (step S156). The above steps S150 to S156 are executed for each frame until the system is completed (step S157). The status display screen is the same as that of the first embodiment (FIG. 14). The same effect as that of the first embodiment can be obtained by the method of the present embodiment.

(Example 3)
An example of the state monitoring process of Example 3 of the second embodiment will be described with reference to FIG. FIG. 17 is a flowchart of the state monitoring process of the third embodiment executed by the information processing apparatus 11.

In step S 100, the image acquisition unit 110 acquires an image from the imaging device 10. The acquired image is temporarily stored in the storage unit 117. In step S101, the detection unit 90 detects the head 22 of the target person 21 from the image acquired in step S100. Information on the position of the detected head 22 is stored in the storage unit 117 in association with time information or a frame number of the image. Next, the state recognition unit 91 reads out the information on the position of the head 22 in the image before a predetermined time (for example, one second) from the storage unit 117 (step S102), and the position and the step of the head 22 before the predetermined time Based on the position of the head 22 detected in S101, the moving speed of the head 22 (moving amount per predetermined time) is calculated (step S103).

In step S104, the determination unit 92 reads from the storage unit 117 the determination criteria corresponding to the determination area in which the head 22 is detected. In the present embodiment, the determination criteria in which the moving speed and the degree of danger are associated are set for each determination area. For example, when the head 22 is detected in the determination area A1, the subject 21 should be in the sleeping state. Therefore, based on the general velocity (for example, 20 cm / sec) of the head 22 in the rising motion (motion for raising the upper body), it is preferable to set the determination criteria for the determination area A1 (for example, the moving velocity is 20 cm / sec or less) Case: degree of danger = 1, in the case of 20 to 40 cm / sec: degree of danger = 2, in the case of greater than 40 cm / sec: degree of danger = 3, etc.). In addition, when the head 22 is detected in the determination area A3 or A4, the rising action is assumed as an action that the target person 21 can take next. Therefore, based on the general velocity (for example, 50 cm / sec) of the head 22 in the rising motion, it is preferable to set the determination criteria for the determination regions A3 and A4 (for example, when the moving velocity is 50 cm / sec or less: degree of danger = 1, in the case of 50 to 80 cm / sec: degree of danger = 2, in the case of greater than 80 cm / sec: degree of danger = 3 etc.)

In step S105, the determination unit 92 determines the degree of danger of the movement speed of the head 22 using the above-described determination criteria. Subsequently, the state display unit 94 outputs the degree of danger score and the state of the subject 21 to the display device 13 (step S106). The above steps S100 to S106 are executed for each frame until the system is completed (step S107). The status display screen is the same as that of the first embodiment (FIG. 14). The same effect as that of the first embodiment can be obtained by the method of the present embodiment.

(Example 4)
An example of the state monitoring process of the fourth example of the second embodiment will be described with reference to FIG. FIG. 18 is a flowchart of the state monitoring process of the fourth embodiment performed by the information processing apparatus 11.

In step S 110, the image acquisition unit 110 acquires an image from the imaging device 10. The acquired image is temporarily stored in the storage unit 117. In step S111, the detection unit 90 detects the head 22 of the target person 21 from the image acquired in step S100. Information on the position of the detected head 22 is stored in the storage unit 117 in association with time information or a frame number of the image. In step S112, the state recognition unit 91 calculates the direction, moving speed, and moving direction of the head 22. The specific calculation method may be the same as that described in the first to third embodiments.

In step S113, the determination unit 92 reads from the storage unit 117 the determination criteria corresponding to the determination area in which the head 22 is detected. Then, the determination unit 92 calculates the degree of danger for the direction of the head 22 (step S114), the degree of danger for the movement vector of the head 22 (step S115), and the degree of danger for the movement speed of the head 22 (step S116). Do. The specific calculation method may be the same as that described in the first to third embodiments. Next, in step S117, the determination unit 92 integrates the three values of the degree of danger obtained in steps S114 to S116 to calculate an integrated score of the degree of danger. For example, the maximum value among the three may be selected as the integrated score, or the average value, the multiplication value, the addition value, etc. of three or two values may be selected as the integrated score.

Subsequently, the state display unit 94 outputs the integrated score and the state of the target person 21 to the display device 13 (step S118). The above steps S110 to S118 are executed for each frame until the system is completed (step S119). The status display screen is the same as that of the first embodiment (FIG. 14). The same effect as that of the first embodiment can be obtained by the method of the present embodiment.

<Others>
The above description of each embodiment merely illustrates the present invention. The present invention is not limited to the above specific embodiments, and various modifications are possible within the scope of the technical idea thereof.

In the above-described embodiment, an example has been described in which the sleeping state / wake-up state / disengagement state is estimated from the image, and the wake-up behavior and the leaving behavior of the subject are detected. However, the state of the estimation target and the action of the detection target are not limited to these. That is, it is possible to handle various "human states" and "actions" as long as different features appear in the image. For example, it is also possible to detect actions such as eating and reading.

1: Watching support system 10: Imaging device, 11: Information processing device, 12: Learning device, 13: Display device 100: Illumination, 101: Near infrared camera, 110: Image acquisition unit, 111: Area setting unit, 112: Previous Processing unit, 113: Regressor, 114: Score stabilization unit, 115: Determination unit, 116: Status display unit, 117: Storage unit 20: Bed, 21: Target person, 22: Head 30: Monitoring region, 40- 43: points at four corners of bed, 44: bed area, A1 to A4: judgment area 70: arrow indicating head orientation, 71: judgment standard 80: state image, 81: state change graph, 82: marker, 83: State image 84: state change graph 85: marker 90: detection unit 91: state recognition unit 92: determination unit 93: area setting unit 94: state display unit 150: head movement vector 151: Criteria for moving direction, 152: determination as to the moving speed reference

Claims

It is a watching support system that supports watching of the target person on the bed,
An image acquisition unit configured to acquire an image from an imaging device installed to photograph a monitoring area including the bed of the subject;
A state quantifying unit that outputs a score obtained by quantifying the condition of the subject based on the image of the monitoring area obtained by the image obtaining unit;
A state display unit for displaying on the display device a graph indicating temporal changes in the score output from the state quantifying unit;
A watching support system characterized by having.
The state quantifying unit has a regressor machine-learned to receive a bed and an image of a person and output a score indicating the state of the person relative to the bed, and the image of the monitoring area is The surveillance support system according to claim 1, wherein a score quantifying the condition of the subject is acquired by inputting into a regressor.
The surveillance support system according to claim 2, wherein the regressor is a neural network.
The state of the person with respect to the bed is classified in advance into a plurality of types, and different scores are assigned to each of the plurality of types,
3. The apparatus according to claim 2, wherein the regressor is configured to output a value between the scores of the two types when the state of the person is a state between the two types. The watching support system described in 3.
The plurality of types include a state 0 in which the person sleeps in the bed, a state 1 in which the person stands up on the bed, and a state 2 in which the person leaves the bed. The watching support system according to claim 4.
The surveillance support system according to claim 1, wherein the score obtained by quantifying the condition of the subject is a score representing the degree of risk of the condition of the subject.
A determination criterion storage unit in which determination criteria for determining a dangerous state are set in advance for each of a plurality of determination regions set based on the bed region in the image of the monitoring region,
The state quantifying unit has a detection unit that detects the head of the subject from the image of the monitoring area, and uses the determination reference of the determination area corresponding to the position at which the head is detected. 7. The watching support system according to claim 6, wherein a score representing the degree of danger of the condition of the subject is calculated.
The state quantifying unit determines the degree of danger of the state of the subject based on at least one of the head orientation, head movement speed, head movement direction, and head movement vector. The watching support system according to claim 7, wherein a score to be represented is calculated.
A control method of a watching support system for supporting watching of a target person on a bed,
Acquiring an image from an imaging device installed to image a monitoring area including the bed of the subject;
Outputting a score quantifying the condition of the subject based on the image of the monitoring area;
Displaying a graph indicating temporal change of the score on a display device;
A control method of a watching support system characterized by having.
A program for causing a computer to execute each step of the control method of the watching support system according to claim 9.