JP2021065617A - Image processing device and image processing program - Google Patents

Abstract

To provide an image processing device and image processing program which improve the detection accuracy of the behavior of an object person.SOLUTION: An image processing device which is used in a care system that monitors the behavior of an object person includes an image acquisition unit 451, a detection unit 452, a storage unit 410, a behavior estimation unit 454 and an output unit 455. The image acquisition unit acquires an image. The detection unit detects a movable body by detecting a detection area including the movable body on the basis of the image. The storage unit stores the detection area detected by the detection unit. The behavior estimation unit estimates the behavior of the movable body in the detection area detected by the detection unit when the detection unit detects the movable body, and estimates the behavior of the movable body in the detection area stored in the storage unit when the detection unit does not detect the movable body. The output unit outputs information about the behavior when the behavior of the movable body estimated by the behavior estimation unit is the prescribed behavior.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing apparatus and an image processing program.

In Japan, the longevity of life has become remarkable due to the improvement of living standards, the improvement of sanitary environment, and the improvement of medical standards due to the high economic growth after the war. For this reason, coupled with the decline in the birth rate, the aging society has a high aging rate. In such an aging society, it is expected that the number of people requiring long-term care and nursing care (hereinafter referred to as "target persons") will increase due to illness, injury, aging, etc. ..

The target person enters a hospital or facility such as a welfare facility for the elderly and receives nursing care and long-term care. In such a facility, the subject may be injured due to, for example, falling from the bed or falling while walking, or may get out of the bed and wander. It is necessary to respond to such a situation as soon as possible. Moreover, if such a situation is left unattended, it may develop into a more serious situation. For this reason, at the facility, nurses, caregivers, etc. (hereinafter referred to as "nurses, etc.") regularly patrol to confirm their safety and condition.

However, the number of nurses and others has not caught up with the increase in the number of subjects, and the nursing industry and the long-term care industry are chronically understaffed. Furthermore, compared to the day shift hours, the number of nurses and the like decreases during the semi-night shift and night shift hours, and the work load per person increases. Therefore, it is required to reduce the work load. In order to reduce such labor shortages and the burden on nurses, there is a need for technology that complements nursing work and long-term care work. Therefore, in recent years, a monitoring device for monitoring a target person has been researched and developed.

However, although the monitoring device is a device that monitors the target person, there is a possibility of erroneously detecting that there is a person who is not the target person in the location of the target person. For example, when a cleaner or the like is in the place of the target person. When a warning is issued to prompt confirmation of safety due to this false detection, for example, a monitor such as a nurse rushes to the location of the subject who has been notified, or resets the notification, which is a waste of time and effort. It may occur.

In relation to this, in Patent Document 1 below, in the monitored person monitoring device that monitors the monitored person, in order to avoid false detection due to the behavior of a person other than the monitored person, the detection is performed by the instruction of a nurse or the like. It is disclosed to provide a function to temporarily stop the function.

International Publication No. 2016/152428

However, the technique of Patent Document 1 can avoid erroneous detection due to the behavior of a person other than the monitored person, but cannot improve the detection accuracy of the behavior of the monitored person itself.

The present invention has been made in view of the above-mentioned problems. Therefore, an object of the present invention is to provide an image processing device and an image processing program that improve the detection accuracy of the behavior of the subject.

The above object of the present invention is solved by the following means.

(1) An image processing device used in a nursing care system that monitors the behavior of a subject, and the movement is performed by detecting an image acquisition unit that acquires an image and an area including a moving body based on the image. A detection unit that detects a body, a storage unit that stores a detection area set corresponding to the moving body detected by the detecting unit, and a storage unit that stores the moving body detected by the detecting unit. The behavior of the moving body is estimated in the detection area detected by the detection unit, but when the moving body is not detected by the detection unit, the behavior of the moving body is estimated in the detection area stored in the storage unit. An image processing device having a behavior estimation unit that estimates a behavior, and an output unit that outputs information about the behavior when the behavior of the moving body estimated by the behavior estimation unit is a predetermined behavior.

(2) In the detection region, the movement estimation unit further includes a posture estimation unit that estimates the position of a feature point of the moving body with respect to the human body and estimates the posture of the moving body, and the behavior estimation unit is the posture estimation unit. The image processing apparatus according to (1) above, which estimates the behavior of the moving body in the detection region based on the posture of the moving body estimated by.

(3) The output unit transmits information about the behavior to the terminal device of the caregiver who cares for the target person when the behavior of the moving body is the predetermined behavior. The image processing apparatus according to 2).

(4) The behavior estimation unit estimates the behavior of the moving body based on the overlap between the detection region and the estimated feature point position, and the output unit is the moving body by the behavior estimation unit. The image processing apparatus according to (2) above, which determines whether or not to output information regarding the behavior of the moving body based on the estimation result of the behavior of the moving body.

(5) The behavior estimation unit is based on the estimation result of the posture estimation unit in the detection region for a predetermined period before the moving body is no longer detected when the moving body is no longer detected by the detection unit. The behavior of the mobile body is estimated, and the output unit determines whether or not to output information regarding the behavior of the mobile body based on the estimation result of the behavior of the mobile body by the behavior estimation unit. The image processing apparatus according to (2).

(6) The behavior estimation unit needs to output information about the behavior of the mobile body when the estimation result of the posture estimation unit in the detection region for the predetermined period is standing. The image processing apparatus according to (5) above, wherein the output unit does not output information regarding the behavior of the moving body when the behavior of the moving body is the specific behavior. ..

(7) An image processing program used in a care system that monitors the behavior of a target person, which is described in the step (a) of acquiring an image and by detecting an area including a moving body based on the image. A step (b) for detecting a moving body, a step (c) for storing a detection area set corresponding to the detected moving body in the storage unit, and a detection when the moving body is detected. The step (d) of estimating the behavior of the moving body in the detection area stored in the storage unit when the moving body is not detected while estimating the behavior of the moving body in the detected detection area. , An image processing program for causing a computer to perform a process including a step (e) of outputting information about the action when the estimated action of the moving body is a predetermined action.

According to the present invention, when the target person is not detected, the behavior of the moving body is estimated in the detection area stored in the storage unit, so that the accuracy of the exit determination of the target person can be improved. Therefore, it is possible to reduce the risk of losing a predetermined action that requires notification of an event such as a fall or a fall when the subject is determined to leave.

It is a figure which shows the schematic structure of the image processing system including the arithmetic unit which concerns on one Embodiment. It is a schematic block diagram which illustrates the structure of the arithmetic unit shown in FIG. It is a block diagram which shows the function of the control part shown in FIG. It is explanatory drawing for demonstrating the example of the photographed image by the photographing apparatus shown in FIG. It is explanatory drawing which shows the example of the characteristic point of a person. It is explanatory drawing which shows the example of the feature point estimated by the posture estimation part shown in FIG. It is a flowchart for demonstrating the main processing procedure of the control method of the arithmetic unit shown in FIG. It is explanatory drawing for demonstrating the example of the photographed image including the subject. 6 is a subroutine / flowchart for explaining the procedure of the process of estimating the behavior of the target person (step S105) in FIG. 7. It is explanatory drawing for demonstrating the application example of the condition about leaving.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same elements are designated by the same reference numerals, and duplicate description will be omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

<Image processing system 10>
FIG. 1 is a diagram showing a schematic configuration of an image processing system 10 including an arithmetic unit according to an embodiment.

The image processing system 10 includes a photographing device 100, a communication network 200, a mobile terminal 300, and an arithmetic unit 400. The arithmetic unit 400 is communicably connected to the photographing device 100 and the mobile terminal 300 by the communication network 200.

<Shooting device 100>
The photographing device 100 is composed of, for example, a near-infrared camera, and is installed at a predetermined position to photograph a photographing area that is overlooked with the predetermined position as a viewpoint. Hereinafter, the image taken by the photographing device 100 is referred to as "photographed image 600" (see FIG. 4). That is, the photographing device 100 irradiates the near-infrared ray toward the photographing area by the LED (Light Emitting Device), and receives the reflected light of the near-infrared ray reflected by the object in the photographing area by the CMOS (Complementary Metal Oxide Semiconductor) sensor. By doing so, the shooting area can be shot. The captured image 600 may be a black-and-white image having the reflectance of near infrared rays as each pixel. The predetermined position is, for example, the ceiling of the living room of the subject 500. The photographing area is, for example, a three-dimensional area including the entire floor of the living room. The photographing device 100 photographs the photographing area as a moving image having a frame rate of, for example, 15 fps to 30 fps. The captured image 600 includes a moving image and a still image. The photographing device 100 transmits the photographed image 600 to the arithmetic unit 400 or the like.

Further, the photographing device 100 may be, for example, a near-infrared camera included in the sensor box. The sensor box is a box-shaped device including a near-infrared camera, a body motion sensor, and the like. In this case, the sensor box may be configured to have a part or all of the functions of the arithmetic unit 400. The body motion sensor is a Doppler shift type sensor that transmits and receives microwaves to and from the bed and detects the Doppler shift of microwaves generated by the body movement (for example, respiratory movement) of the subject 500.

<Communication network 200>
For the communication network 200, a network interface based on a wired communication standard such as Ethernet (registered trademark) can be used. For the communication network 200, a network interface based on a wireless communication standard such as Bluetooth (registered trademark) or IEEE802.11 may be used. An access point 210 is provided in the communication network 200, and the mobile terminal 300, the photographing device 100, and the arithmetic unit 400 are communicably connected by a wireless communication network.

<Mobile terminal 300>
The mobile terminal 300 receives information about the behavior of the target person 500 from the arithmetic unit 400, displays it on the screen, sounds an alarm, and notifies the nurse or the like of the behavior of the target person 500. The mobile terminal 300 may be composed of, for example, a smartphone. The mobile terminal 300 may be configured to receive and display the captured image 600 from the photographing device 100.

<Arithmetic logic unit 400>
The arithmetic unit 400 is an image processing device used in a nursing care system that monitors the behavior of the subject 500. The arithmetic unit 400 receives the captured image 600 captured by the photographing device 100 from the photographing device 100, and transmits information on the behavior of the person (target person 500) included in the captured image 600 to the mobile terminal (caregiver's terminal device). Send to 300. The long-term care system is a system used for care including long-term care, nursing, and support.

FIG. 2 is a schematic block diagram illustrating the configuration of the arithmetic unit 400 shown in FIG. The arithmetic unit 400 includes a storage unit 410, a display unit 420, an input unit 430, a communication unit 440, and a control unit 450. These components are connected to each other via bus 460.

The storage unit 410 may be composed of a RAM (Random Access Memory), a ROM (Read Only Memory), and an SSD (Solid State Drive). The RAM temporarily stores programs and data as a work area of the control unit 450. The ROM stores various programs and various data in advance. The SSD stores various programs including an operating system and various data. In the present embodiment, data for specifying the position and size of the person area 510, which will be described later, is stored.

The display unit 420 is, for example, a liquid crystal display and displays various information. Further, the input unit 430 is composed of, for example, a touch panel and various keys. The input unit 430 is used for various operations and inputs.

The communication unit 440 is an interface for communicating with an external device. For communication, a network interface according to a standard such as Ethernet (registered trademark), SATA, PCI Express, USB, IEEE1394, etc. can be used. In addition, a wireless communication interface such as Bluetooth (registered trademark), 802.11, 4G, etc. can be used for communication. The communication unit 140 receives the captured image 600 from the photographing device 100. In addition, the communication unit 440 transmits information regarding the behavior of the target person 500, which is estimated by the behavior estimation unit 454 of the control unit 450, which will be described later, to the mobile terminal 300.

The control unit 450 is configured by a CPU (Central Processing Unit), and controls and performs arithmetic processing of each unit of the arithmetic unit 400 according to an image processing program. The details of the function of the control unit 450 will be described below.

FIG. 3 is a block diagram showing the functions of the control unit 450 shown in FIG. The control unit 450 includes an image acquisition unit 451, an area detection unit (detection unit) 452, a posture estimation unit 453, a behavior estimation unit 454, and an output unit 455.

The image acquisition unit 451 acquires the captured image 600 from the photographing device 100 through the communication unit 440. The acquired captured image 600 is transmitted to the area detection unit 452.

FIG. 4 is an explanatory diagram for explaining an example of a photographed image 600 by the photographing apparatus 100 shown in FIG. The captured image 600 includes a plurality of objects including the subject 500, the bed 700, and the inner door 720. The inner door 720 is, for example, a sliding door that separates the outside and the inside of the living room. The area detection unit 452 detects the person area 510 and the object area 710 from the captured image 600. The person area 510 is an area within a rectangle (indicated by a thick line frame) including the target person 500 in the captured image 600, and the object area 710 is a rectangle (in a broken line) including a specific object (for example, a chair) other than the person. The area within the display).

The area detection unit 452 detects the area where the object (object) exists on the captured image 600 as the object existence area, and calculates the reliability score for each predetermined category of the object included in the detected object existence area. The confidence score is the likelihood with respect to the target object. The region detection unit 452 can calculate the reliability score by a known technique using a DNN (Deep Neural Network) such as Faster R-CNN.

Predetermined categories can be, for example, people, chairs, and equipment. The area detection unit 452 detects the object existence area having the highest reliability score in the person category as the person area 510. That is, the area detection unit 452 functions as a detection unit that detects the target person 500 as an object (moving body). Similarly, the object existence region having the highest reliability score in a predetermined object category is detected as the object region 710 (for example, the object region of a chair) in the category having the highest reliability score.

Alternatively, as a method for detecting the person area 510, a background subtraction method for extracting the difference between the captured image 600 to be detected and the background image of the captured area previously extracted by the photographing device 100 can be used. Further, an inter-frame (time) difference method or the like for extracting the difference between the captured image 600 to be detected and the average of the captured images in the past can be used.

The person area 510 and the object area 710 detected by the area detection unit 452 are stored in the storage unit 410.

The mobile terminal 300 reproduces a moving image based on the captured image 600 of a plurality of frames received from the photographing device 100, and positions a rectangle (frame) indicating the person area 510 received from the arithmetic unit 400 on the display corresponding to the person area 510. Display on.

The posture estimation unit 453 estimates the posture of the target person 500 from the person area 510. More specifically, the posture estimation unit 453 estimates the feature points 520 (hereinafter, also referred to as "feature points 520") relating to the human body of the subject 500 from the person area 510, and is based on the estimation result of the feature points 520. The posture of the subject 500 is estimated.

FIG. 5 is an explanatory diagram showing an example of human feature points. In FIG. 5, the positions of the feature points 520 are indicated by white circles. The thick black line connecting the feature points 520 indicates the bone. In the example of FIG. 5, the relative position of the feature point 520 allows the subject 500 to recognize the posture in which both arms are spread while standing. Feature points 520 include, for example, feature points 520 of the head, neck, shoulders, elbows, hands, hips, thighs, knees, and feet. The feature point 520 may include a feature point 520 other than the above, and may not include any of the above. The posture estimation unit 453 can estimate the feature points 520 of the subject 500 from the person area 510 by the DNN reflecting the dictionary for detecting the feature points 520 from the person area 510. For example, the posture estimation unit 453 can estimate the feature point 520 by a known technique using DNN such as DeepPose. DeepPose is described in detail in the known literature (Alexander Toshev, et al. "DeepPose: Human Pose Optimization via Deep Neural Networks", in CVPR, 2014).

FIG. 6 is an explanatory diagram showing an example of feature points 520 estimated by the posture estimation unit 453 shown in FIG. In the example of FIG. 6, the feature points 520 are superimposed on the person area 510.

Further, the posture estimation unit 453 estimates the posture of the subject 500 based on the estimation result of the feature point 520. For example, the posture estimation unit 453 calculates the posture of the plausible subject 500 from the estimated feature points 520 using the hidden Markov model.

As other methods for recognizing a person's posture, for example, a method in which the distance between a predetermined posture model and each feature point position is the minimum is set as the posture, and the posture is learned as a feature amount, and the learning is performed. A method of recognizing a posture using the result can be mentioned.

The behavior estimation unit 454 estimates the behavior of the subject 500 based on the posture of the subject 500 estimated by the posture estimation unit 453. More specifically, the behavior estimation unit 454 estimates the behavior of the subject 500 based on the positional relationship between the position of the feature point 520 and the object region 710. The behavior includes, for example, the behavior of getting up from the bed 700 (getting up), the behavior of getting away from the bed 700 (getting out of bed), the behavior of falling from the bed 700 (falling), the behavior of falling down on the floor (falling), and walking. Etc. are included.

Further, in the present embodiment, the action includes an action (exit) of going out of the living room, that is, outside the photographing area. Exiting includes, for example, going out or going to the bathroom. The behavior estimation unit 454 has a function of determining whether or not the subject 500 has left. This feature can be enabled or disabled depending on the settings. The case where this function is enabled will be described below.

When the target person 500 leaves, since the target person 500 is out of the shooting area, the area detection unit 452 cannot detect the person area 510. That is, the area detection unit 452 cannot detect the target person 500. On the other hand, for example, even when the subject 500 is wrapped in a blanket on the bed 700 in the living room, the area detection unit 452 cannot detect the subject 500. Therefore, it is difficult to determine the exit of the target person 500 only by the presence or absence of the detection of the target person 500 by the area detection unit 452.

Therefore, in the present embodiment, when the target person 500 is no longer detected by the area detection unit 452, the posture estimation unit 453 in the person area 510 before the target person 500 is no longer detected, which is stored in the storage unit 410. Based on the estimation result, the behavior of the subject 500 is estimated, and the exit of the subject 500 is determined. A specific method for determining the exit of the target person 500 will be described later.

Further, when the target person 500 is not detected by the area detection unit 452 due to the subject person 500 leaving the living room, the actual image of the target person 500 is not reflected in the captured image 600. Here, the case where the target person 500 is not detected is the case where the person area 510 is not detected by the area detection unit 452 because a part or all of the target person 500 does not exist in the captured image 600. Is. In the embodiment, when the target person 500 is no longer detected by the area detection unit 452, the person area 510 stored in the storage unit 410 before the target person 500 is no longer detected is displayed on the display unit 420 of the arithmetic unit 400 or the display unit 420. , Control to display on the screen of the mobile terminal 300.

The output unit 455 determines whether or not to output information regarding the behavior of the target person 500 based on the estimation result of the behavior of the target person 500 by the behavior estimation unit 454. More specifically, for example, when the action of the target person 500 is a predetermined action, the output unit 455 transmits information about the action of the target person 500 to the mobile terminal 300 (event notification). Predetermined actions include getting up, getting out of bed, falling, and falling. On the other hand, leaving is an action that is not included in the predetermined action and does not need to be transmitted to the mobile terminal 300 (hereinafter, referred to as "specific action"). When the action of the target person 500 is a specific action, the output unit 455 does not transmit the information about the action of the target person 500 to the mobile terminal 300. In this embodiment, exit is illustrated as a specific action, but the specific action is not limited to exit. For example, the specific action includes an action in which the subject 500 enters the toilet in the living room. In this case as well, the person area 510 is not detected by the area detection unit 452 as in the case of exiting.

<Control method of arithmetic unit 400>
FIG. 7 is a flowchart for explaining a main processing procedure of the control method of the arithmetic unit 400 in the present embodiment. Further, FIG. 8 is an explanatory diagram for explaining an example of the photographed image 600 including the subject 500. Further, FIG. 9 is a subroutine / flowchart for explaining the procedure of the process (step S105) of estimating the behavior of the target person in FIG. 7. Each of the processes shown in FIGS. 7 and 9 is realized by the CPU of the control unit 450 executing an image processing program.

In the following, it is assumed that the subject 500 shown in FIG. 4 leaves the bed 700 and walks toward the inner door 720.

First, the image data of the captured image 600 is acquired (step S101). The image acquisition unit 451 sequentially acquires image data captured at the above frame rate by the imaging device 100. For example, when the target person 500 reaches in front of the inner door 720 and opens the inner door 720, the image acquisition unit 451 sets the target person 500 in front of the doorway leading to the outside of the living room as shown in FIG. The captured image 610 including the image is acquired. In FIG. 8, the gray area D1 shows the space outside the living room photographed through the doorway.

Next, the person area is detected based on the image data (step S102). The area detection unit 452 calculates the reliability score of the person for each object existence area for each captured image 610, detects the object existence area having the highest reliability score in the person category as the person area 511, and detects the target person. Detect 500. The person area 511 constitutes a detection area. The detection area may be set in correspondence with the target person 500 (moving body) and may be the same area as the person area 511, or may be an area of a different size including the person area 511. For example, the detection region may be a region larger than the human region 511 estimated by learning the neural network, or may have a shape other than a rectangle.

Next, the person area 511 is stored (step S103). The storage unit 410 stores data for identifying the person area 511 (for example, when the person area 511 is rectangular, the coordinates of the start point and the end point). Next, the posture of the target person 500 is estimated (step S104). .. The posture estimation unit 453 estimates the position of the feature point 520 of the target person 500 in the person area 511 and estimates the posture of the target person 500. The posture estimation unit 453 estimates the posture of the subject 500 for each photographed image 610 acquired by the image acquisition unit 451 and stores the estimation result in the storage unit 410. In the example shown in FIG. 8, the posture estimation unit 453 estimates that the subject 500 is in a standing posture from the positional relationship of each of the feature points 520.

Next, the behavior of the subject 500 is estimated (step S105). As shown in FIG. 9, it is determined whether or not the target person 500 has been detected (step S201). When the target person 500 is detected (step S201), the behavior estimation unit 454 estimates the behavior of the target person 500 based on the posture of the target person 500 estimated by the posture estimation unit 453 (step S202). For example, in the behavior estimation unit 454, the subject 500 sits on the bed 700 as shown in FIG. 4 and reaches the front of the doorway of the living room as shown in FIG. It is estimated that he got out of bed and walked. The control unit 450 ends the process of estimating the behavior of the target person 500, and shifts to the process (S106) of outputting the information regarding the operation in FIG. 7.

On the other hand, when the subject 500 goes out of the photographing area, it cannot be detected by the area detection unit 452. When the target person 500 is no longer detected by the area detection unit 452 (step S201: NO), the behavior estimation unit 454 is based on the estimation result of the posture estimation unit 453 in the person area 511 before the target person 500 is no longer detected. , Estimate the behavior of the subject 500. The specific procedure is as follows.

The condition for leaving is determined (step S203). The condition regarding exit may be, for example, a matter that positively indicates or suggests the exit of the subject 500 with respect to the posture of the subject 500. It is preferable that the conditions for each exit indicate the exit of the subject 500 from different viewpoints. By determining the exit of the target person 500 based on different matters indicating the exit of the target person 500, the accuracy of the exit determination is improved. Specific examples of conditions for leaving will be described later.

Next, it is determined whether or not all the conditions for leaving are satisfied (step S204). When all the conditions for exiting are satisfied (step S204: YES), the behavior estimation unit 454 determines that the subject 500 has exited (step S205). On the other hand, when at least one of the conditions relating to exit is not satisfied (step S204: NO), the behavior estimation unit 454 determines that the subject 500 has not exited (step S206).

Next, the determination result is stored (step S207). The behavior estimation unit 454 stores the result of the exit determination of the subject 500 in the memory 410.

Returning to FIG. 7 again, the information regarding the action is output (step S106). When the action of the target person 500 is the above-mentioned predetermined action, the output unit 455 transmits information about the action of the target person 500 to the mobile terminal 300. Further, even when it is determined in step S206 that the target person 500 has not left, there is a possibility that the target person 500 has fallen or has fallen due to the shadow. Send to 300. On the other hand, when the action of the target person 500 is a specific action such as leaving, the output unit 455 does not transmit information about the action of the target person 500.

For example, when the subject 500 leaves the bed, walks closer to the inner door 720, stands in front of the inner door 720, opens the inner door 720, and exits from the doorway, the output occurs when the subject 500 leaves the bed. The unit 455 transmits to the mobile terminal 300 that the subject 500 has left the bed. On the other hand, the exit of the target person 500 is not transmitted to the mobile terminal 300.

<Conditions for leaving>
When the subject 500 leaves the living room and goes out of the photographing area, the area detection unit 452 cannot detect the subject 500. Therefore, it is a condition for the subject 500 to leave that the subject 500 is not detected. That is, the behavior estimation unit 454 determines that the area detection unit 452 "lost" the subject 500 as a precondition.

When the subject 500 leaves the room, it is considered that he / she will not return to the room immediately because he / she usually has some purpose to leave the room. Therefore, it is expected that the subject 500 will not be detected for a certain period of time after leaving. Therefore, when the subject 500 leaves the living room, the following condition 1 is satisfied.

(Condition 1) The state in which the subject 500 is not detected continues for a predetermined first period. The first period may change depending on the value of the frame rate when converted into the number of frames, but may be, for example, a period corresponding to 20 frames or more. It should be noted that this value is a value when a highly accurate exit determination can be performed in the specific implementation environment assumed in the present embodiment, and can be appropriately changed depending on the implementation environment. The same applies to the values exemplified under each of the following conditions.

Further, in the present embodiment, when the target person 500 leaves the living room and the target person 500 is no longer detected, for example, the human rectangle immediately before the target person 500 is no longer detected is maintained as it is. Therefore, under the above condition 1, it can be said that the human rectangle does not move for the first period after the subject 500 is no longer detected.

Further, in the period from when the subject 500 stands in front of the inner door 720 to open the inner door 720 until immediately before the detection by the area detection unit 452, the subject 500 is in a standing posture. Therefore, when the subject 500 leaves the living room, the following condition 2 is satisfied.

(Condition 2) The subject 500 is not detected by the area detection unit 452, that is, the subject 500 is in a standing posture for a predetermined second period (predetermined period) before the human rectangle becomes immobile. The second period may be a period corresponding to, for example, about 5 frames when converted into the number of frames.

According to the above-mentioned conditions 1 and 2, it can be confirmed that the subject 500 was not detected during the first period and was in a standing posture for the second period before the subject 500 was no longer detected. In addition to these conditions, by confirming the position in the photographing area where the target person 500 is no longer detected, the exit determination of the target person 500 can be performed with higher accuracy. That is, when the subject 500 leaves the living room, the following condition 3 is satisfied.

(Condition 3) The number of feature points 520 outside the living room area is a predetermined number or more. The predetermined number can be, for example, 8 or more on a frame average. The number of feature points 520 outside the living room area can be calculated, for example, as follows. Since the subject 500 is not detected, the position of the feature point 520 of the subject 500 is estimated based on the person area 511 stored in the storage unit 410 before the subject 500 is no longer detected (see FIG. 8). ). The behavior estimation unit 454 analyzes the position of the feature points 520 of the subject 500 in the person area 511, and calculates the number of the feature points 520 outside the living room area (area D1 in the example shown in FIG. 8). That is, the behavior estimation unit 454 determines the condition for leaving the subject 500 based on the overlap between the person area 511 and the estimated position of the feature point 520.

Further, in addition to the above condition regarding the number of feature points 520 outside the living room area, the distance from the center point of the photographed image (position of reference numeral 800 in the example of the photographed image 610 of FIG. 8) to the feature point 520 of the outside room area. Conditions can be imposed. For example, it may be a condition that the minimum value of the distance from the center point to the feature point 520 in the region D1 outside the living room (the distance indicated by the broken line 810 in the example of the captured image 610 of FIG. 8) is equal to or longer than a predetermined length. .. The predetermined length can be 377 or more, for example, when the size of the captured image is 1280 × 960 (full size).

Further, since the subject 500 should be standing when leaving the living room, the following condition 4 is satisfied.

(Condition 4) While the target person 500 is not detected by the area detection unit 452, that is, while the human rectangle does not move, the posture of the target person 500 remains standing.

When the sensor box has a function of detecting a person, which is equivalent to that of the area detection unit 452, the person area 511 detected by the area detection unit 452 and the person area detected by the sensor box (for the target person 500) Hereinafter, the accuracy of the exit determination can be further improved by comparing with the "SB person area").

That is, when the subject 500 leaves the living room, for example, the following condition 5 is satisfied as a condition corresponding to the above condition 1.

(Condition 5) As for the captured image of the sensor box, the state in which the subject 500 is not detected continues for a predetermined third period. The third period, when converted to the number of frames, may change depending on the value of the frame rate, but may be, for example, a period corresponding to 20 frames or more.

The sensor box detects a human silhouette from the captured image and detects the target person 500 based on the human silhouette. The human silhouette can be detected, for example, by extracting a range of pixels having a relatively large difference by the time difference method for extracting the difference between images (frames) whose shooting times are before and after. It is also possible to confirm the degree of coincidence between the person area 511 and the SB person area. For example, it is determined whether the degree of coincidence between the person area 511 and the SB person area is equal to or greater than a predetermined threshold value. The degree of agreement can be evaluated by, for example, IoU (Intersection over Union). IoU is defined by the area of the intersection / the area of the union. In the present embodiment, the IoU is a value obtained by dividing the common area between the person area 511 and the SB person area by the sum area of the person area 511 and the SB person area. That is, when the subject 500 leaves the living room, the following condition 6 is satisfied.

(Condition 6) The average of IoU between the person area 511 and the SB person area is equal to or higher than a predetermined threshold value. The predetermined threshold value can be, for example, about 0.3.

FIG. 10 is an explanatory diagram for explaining an application example of the condition relating to exit. In the example shown in FIG. 10, the case where the determination is made for the conditions 1 to 6 is illustrated, but the case is not limited to such a case, and the condition is appropriately omitted or a new condition is added according to the required determination accuracy. Can be done. For example, from the viewpoint of maintaining the determination accuracy, it is preferable to perform the determination for at least conditions 1 to 3. Condition 4 can be omitted. Regarding conditions 5 and 6, it is possible to select whether or not to carry out the determination depending on the configuration of the sensor box.

As shown in FIG. 10, captured images are acquired by the image acquisition unit 451 from frames 1 to 50, and the subject 500 is not detected by the area detection unit 452 after the frame 27, but also by the sensor box after the frame 29. It is assumed that it will not be done.

Regarding condition 1, it is shown that the subject 500 is not detected in the frames 27 to 50 (frames in which the squares are painted black). In the example shown in FIG. 10, since this period is 24 frames (≧ 20 frames), the condition 1 is satisfied.

Regarding condition 2, it is shown that the frames 22 to 26 (frames whose squares are painted in gray) before the subject 500 is no longer detected are in a standing posture. In the example shown in FIG. 10, since this period is 5 frames, the condition 2 is also satisfied.

Regarding conditions 3, 4, and 6, it is indicated that the determination of each condition is performed in frames 29 to 50 (indicated by diagonal lines, stars, and “●”, respectively).

Regarding condition 5, it is shown that the subject 500 is not detected in frames 29 to 50 (indicated by “◯”). In the example shown in FIG. 10, since this period is 22 frames (≧ 20 frames), the condition 5 is satisfied.

The image processing apparatus 400 of the present embodiment described above has the following effects.

When the person area 511 is stored in the storage unit 410 and the target person 500 is not detected by the area detection unit 452, the behavior of the target person 500 is estimated in the person area 511 stored in the storage unit 410. It is possible to improve the accuracy of the exit determination of the target person 500. Therefore, it is possible to reduce the risk of losing a predetermined action that requires notification of an event such as a fall or a fall when the target person 500 is determined to leave.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims.

For example, the image processing program may be provided by a computer-readable recording medium such as a USB memory, a flexible disk, or a CD-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to a memory, storage, or the like and stored. Further, this image processing program may be provided as a single application software, or may be incorporated into the software of each device as a function of the server.

Further, in the embodiment, a part or all of the processing executed by the image processing program may be replaced with hardware such as a circuit and executed.

10 Image processing system,
100 shooting equipment,
200 communication networks,
210 access points,
300 mobile terminals,
400 arithmetic unit,
410 storage,
420 display section,
430 input section,
440 Communication Department,
450 control unit,
451 Image acquisition unit,
452 area detector,
453 Posture estimation unit,
454 Behavior Estimator,
455 output unit,
500 subjects,
510,511 People area,
520 feature points,
600,610 captured images,
700 beds,
710 Object area.

Claims (7)

An image processing device used in a long-term care system that monitors the behavior of the subject.
The image acquisition unit that acquires images and
A detection unit that detects the moving body by detecting a region including the moving body based on the image, and a detection unit that detects the moving body.
A storage unit that stores a detection area set corresponding to the moving body detected by the detection unit, and a storage unit.
When the moving body is detected by the detecting unit, the behavior of the moving body is estimated in the detection region detected by the detecting unit, but when the detecting unit does not detect the moving body, the moving body is not detected. A behavior estimation unit that estimates the behavior of the moving body in the detection area stored in the storage unit, and a behavior estimation unit.
An image processing device having an output unit that outputs information about the behavior when the behavior of the moving body estimated by the behavior estimation unit is a predetermined behavior. The detection region further includes a posture estimation unit that estimates the feature points of the moving body with respect to the human body and estimates the posture of the moving body.
The image processing device according to claim 1, wherein the behavior estimation unit estimates the behavior of the moving body in the detection region based on the posture of the moving body estimated by the posture estimation unit. The output unit
The image processing device according to claim 1 or 2, wherein when the action of the moving body is the predetermined action, information about the action is transmitted to a terminal device of a caregiver who cares for the target person. The behavior estimation unit
The behavior of the moving body is estimated based on the overlap between the detection region and the estimated feature point position.
The output unit
The image processing apparatus according to claim 2, wherein it is determined whether or not to output information regarding the behavior of the moving body based on the estimation result of the behavior of the moving body by the behavior estimating unit. The behavior estimation unit
When the moving body is no longer detected by the detection unit, the behavior of the moving body is estimated based on the estimation result of the posture estimation unit in the detection region for a predetermined period before the moving body is no longer detected. ,
The output unit
The image processing apparatus according to claim 2, wherein it is determined whether or not to output information regarding the behavior of the moving body based on the estimation result of the behavior of the moving body by the behavior estimating unit. The behavior estimation unit
When the estimation result of the posture estimation unit in the detection region for the predetermined period is standing, it is estimated that the behavior of the moving body is a specific behavior that does not need to output information about the behavior of the moving body. ,
The output unit
The image processing device according to claim 5, which does not output information regarding the behavior of the moving body when the behavior of the moving body is the specific behavior. An image processing program used in a long-term care system that monitors the behavior of the subject.
Step (a) to acquire an image and
The step (b) of detecting the moving body by detecting the region including the moving body based on the image, and
The step (c) of storing the detection area set corresponding to the detected moving object in the storage unit, and
When the moving body is detected, the behavior of the moving body is estimated in the detected detection area, while when the moving body is not detected, the detection area stored in the storage unit is used. Step (d) of estimating the behavior of the moving body and
An image processing program for causing a computer to perform a process including a step (e) of outputting information about the action when the estimated action of the moving body is a predetermined action.

Images