JP6780641B2 - Image analysis device, image analysis method, and image analysis program - Google Patents

Description

The present invention relates to an image analysis apparatus, an image analysis method, and an image analysis program.

In recent years, the number of accidents caused by wandering of inpatients, facility residents, persons requiring long-term care, etc., has been increasing year by year. In order to prevent such accidents, a camera installed indoors takes a picture of the person being watched over, and by analyzing the captured image, the person being watched over is detected to get out of bed, and the person being watched over continues to get out of bed for a certain period of time or longer. In some cases, a watching system has been developed to give an alarm.

For example, Patent Document 1 proposes a patient recognition device that detects a patient getting out of bed by photographing a patient on a bed with a camera and processing the image obtained thereby. Specifically, this patient recognition device binarizes a color captured image captured by a camera with a color of interest such as black and another color, and a linking component extracted with a direct eye color in the binarized image. Label (Area).

Then, the patient recognition device recognizes the existence of the human body by determining whether or not the area corresponds to a part of the human body based on the size and shape of the labeled area. As a result, the patient recognition device can notify that the patient has been out of bed for a predetermined time or longer when the state of not recognizing the existence of the human body continues for a predetermined time.

Japanese Unexamined Patent Publication No. 2014-021619

However, the patient recognition device recognizes the existence of the human body based on the determination of whether or not the area extracted by the labeling process corresponds to a part of the human body. Therefore, if there is a person other than the person to be watched over in the shooting range of the camera, even if the person to be watched gets out of bed, the recognition of the existence of the human body is continued based on the area where the person other than the person to be watched is captured. There was a possibility that it would not be possible to detect the person being watched over getting out of bed.

In addition, when a person other than the person to be watched enters the shooting range of the camera after the person to be watched gets out of bed, the existence of the human body is recognized based on the area in which the person other than the person to be watched is captured. There was a possibility that the measurement of the time when the person to be watched got out of bed would be stopped.

Therefore, in the above-mentioned patient recognition device, there is a possibility that a person other than the watching target person is mistaken for the watching target person, and the time when the watching target person is out of bed cannot be accurately measured. Therefore, the above-mentioned patient recognition device has a problem that an alarm regarding getting out of bed of the person being watched over may not be executed even though the person being watched over has been out of bed for a certain period of time or more.

One aspect of the present invention has been made in consideration of such a point, and an object of the present invention is to provide a technique for accurately executing an alarm regarding getting out of bed of a person to be watched over.

The present invention employs the following configuration in order to solve the above-mentioned problems.

That is, the image analysis device according to one aspect of the present invention is a photographed image taken by a photographing device installed for watching the behavior of the person to be watched in the bed, and the depth of each pixel in the photographed image is determined. The foreground of the acquired image by calculating the difference between the image acquisition unit that acquires the photographed image including the indicated depth data and the acquired image and the background image set as the background of the photographed image. By referring to the foreground extraction unit that extracts the area and the depth of each pixel included in the extracted foreground area, the position where the foreground area appearing on the bed is separated from the bed by a predetermined distance or more in the real space. It is determined whether or not the person has moved to the bed, and when it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more, it is detected that the person to be watched has left the bed. By referring to the depth of each pixel included in the extracted foreground region of the bed leaving detection unit and the photographed image acquired after detecting that the watching target person has left the bed, in real space. Then, when it is determined whether or not the foreground area appearing outside the bed has moved onto the bed, and when it is determined that the foreground area appearing outside the bed has moved onto the bed, the watching target person is said to have moved to the bed. When the bedtime detection unit that detects that the person has gone to bed and the elapsed time since the detection of the person being watched over have left the bed before detecting the person being watched over have exceeded a certain period of time. , It is provided with a notification unit for notifying that the person to be watched has been out of bed for a certain period of time or longer.

According to the above configuration, a photographed image taken by a photographing device installed to watch the behavior of the person to be watched in the bed is acquired, and the acquired photographed image and the background image set as the background of the photographed image are used. The difference is calculated. As a result, the foreground region is extracted from the acquired captured image.

Here, in the above configuration, the captured image includes depth data indicating the depth of each pixel. The depth of each pixel expresses the depth of the subject reflected in each pixel. Therefore, according to this depth data, it is possible to specify the position of the subject in the photographed image in the real space without depending on the installation position and the photographing direction of the photographing device.

In addition, the foreground region is extracted from the background image at the place where the change occurs. Therefore, when the watching target person performs some action, the foreground area corresponding to the watching target person is extracted at the place where the watching target person moves due to the action. Therefore, according to the configuration, by referring to the depth of each pixel included in the extracted foreground area, the positional relationship between the extracted foreground area and the bed in the real space can be specified, and this position can be specified. It is possible to analyze the behavior of the watching subject based on the relationship.

Specifically, when the person to be watched gets out of bed, it is assumed that the foreground area appears on the bed when the person to be watched gets up, and the foreground area appearing in such a place moves in the direction away from the bed. To. On the other hand, it is assumed that a person other than the person to be watched does not perform such an operation even if he / she is in the shooting range of the shooting device. That is, it is assumed that the foreground area corresponding to a person other than the person to be watched only occurs around the bed and does not occur on the bed.

Therefore, the image analysis device according to the above configuration refers to the depth of each pixel included in the extracted foreground region, so that the foreground region appearing on the bed is located at a position separated from the bed by a predetermined distance or more in the real space. Determine if it has moved to. Then, the image analysis device according to the above configuration detects that the person to be watched has left the bed when it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more. As a result, even when a person other than the person to be watched is present in the shooting range of the photographing device, the above-mentioned problems do not occur, and it is possible to appropriately detect the person being watched to get out of bed. Therefore, according to the above configuration, it is possible to accurately start measuring the elapsed time after detecting the leaving of the watched person.

In addition, when the person to be watched goes to bed, the foreground area appears in a place outside the bed such as near the frame of the photographed image when the person to be watched away from the bed returns to the surroundings of the bed. It is assumed that the foreground area that appears in such a place moves onto the bed. On the other hand, it is assumed that a person other than the person to be watched does not perform such an operation. That is, even if a person other than the person to be watched enters the shooting range of the photographing device, the foreground area corresponding to the person other than the person to be watched only moves around the bed and does not move on the bed. Is assumed.

Therefore, the image analysis device according to the above configuration refers to the depth of each pixel included in the extracted foreground region for the photographed image acquired after detecting the leaving of the bed of the person to be watched over, in real space. Determine if the foreground area that appears outside the bed has moved onto the bed. Then, when it is determined that the foreground region appearing outside the bed has moved onto the bed, the image analysis device according to the above configuration detects that the person to be watched has gone to bed. As a result, even if a person other than the watching target enters the shooting range of the photographing device after the watching target has left the bed, the above problems do not occur, and the watching target can properly lie down on the bed. It can be detected. Therefore, according to the above configuration, it is possible to accurately stop the measurement of the elapsed time from the detection of the person being watched over getting out of bed.

Then, in the above configuration, if the elapsed time from detecting the bed leaving of the watching target person exceeds a certain time before detecting the bedtime of the watching target person, the watching target person leaves the bed for a certain period of time or more. You will be notified that you are doing so. That is, in the above configuration, in a situation where it is possible to appropriately detect getting out of bed and going to bed of the person to be watched over, it is related to getting out of bed of the person to be watched over based on the elapsed time from detecting getting out of bed of the person to be watched over. An alarm is executed. Therefore, according to the above configuration, it is possible to prevent erroneous detection of the person being watched over getting out of bed due to the existence of a person other than the person being watched over, thereby accurately executing an alarm regarding getting out of bed of the person being watched over. can do.

The person to be watched over is a person to be watched over, for example, an inpatient, a resident of a facility, a person requiring long-term care, and the like. In addition, the notification destination and notification method of the notification regarding the person to be watched over getting out of bed can be appropriately selected according to the embodiment. For example, the notification may be given to the watcher. A watcher is a person who watches over the behavior of the watched person. When the person to be watched over is an inpatient, a resident of a facility, a person requiring long-term care, etc., the person to be watched over is, for example, a nurse, a facility staff member, a caregiver, or the like.

Further, as another form of the image analysis device according to the one aspect, the bed detection unit is described when the foreground area appearing outside the bed moves into a predetermined area provided on the side of the bed. It may be determined that the foreground area appearing outside the bed has moved onto the bed, and it may be detected that the person to be watched has gone to bed.

When the person to be watched after leaving the bed returns to the bed and sits on the bed, the foreground area that appears outside the bed is assumed to move to the side of the bed. Further, in general, it is assumed that a person other than the person to be watched may enter the shooting range of the photographing device, but may not sit on the bed of the person to be watched.

Therefore, the image analysis device according to the configuration detects the bed of the person to be watched over when the foreground area appearing outside the bed moves into a predetermined area provided on the side of the bed. That is, the image analysis device according to the configuration detects the bed of the watching target person based on the fact that the watching target person sits on the bed.

Therefore, according to the configuration, the bed of the watching target person is detected based on the movement of the end sitting position which is assumed not to be performed by a person other than the watching target person. Therefore, it is possible to improve the accuracy of detecting the bed of the person being watched over, and thereby it is possible to accurately execute the alarm regarding the person being watched over getting out of bed.

Further, as another form of the image analysis device according to the one aspect, the bed detection unit further increases when the foreground area appearing outside the bed moves into a predetermined area provided on the side of the bed. In the foreground region, it may be estimated that the head region in which the head of the watching target person is captured and the shoulder region in which the shoulder portion is captured. Then, when the estimated center of gravity of the head region is located outside the bed from the center of gravity of the shoulder region, the bed detection unit moves the foreground region appearing outside the bed onto the bed. It may be determined that the person has gone to bed, and it may be detected that the person to be watched has gone to bed.

For example, a person other than the person to be watched over, such as a caregiver, may perform work such as preparing a futon on the bed in the vicinity of the side of the bed. In such a case, the body (particularly, the upper body) of the person other than the person to be watched can be located above the bed surface. Therefore, depending on the setting of the predetermined area, the foreground area corresponding to the person other than the person to be watched is used. May move from outside the bed into the predetermined area.

However, in this case, the person other than the person to be watched is facing the inside of the bed. On the other hand, the watching subject who sits on the edge is facing the outside of the bed. That is, since a person other than the person to be watched who works near the bed faces the inside of the bed, the center of gravity of the head of this person is located inside the bed rather than the center of gravity of the shoulder. On the other hand, since the watching subject who sits on the bed faces the outside of the bed, the center of gravity of the head of the watching subject is located outside the bed rather than the center of gravity of the shoulder.

Therefore, in the image analysis device according to the configuration, when the foreground area appearing outside the bed moves into a predetermined area provided on the side of the bed, the head of the person to be watched is further moved in the foreground area. Estimate the head area where the image is captured and the shoulder area where the shoulder is captured. Then, the image analysis device according to the configuration determines the positional relationship between the estimated center of gravity of the head region and the center of gravity of the shoulder region, and the center of gravity of the head region is located outside the bed from the center of gravity of the shoulder region. When doing so, it detects that the person being watched over has gone to bed.

Therefore, according to the configuration, it is possible to prevent a person other than the watching target person who works toward the inside of the bed in the vicinity of the bed from erroneously detecting the bed of the watching target person. Can be done. Therefore, according to the configuration, it is possible to improve the accuracy of detecting the bed of the person to be watched over, and thereby it is possible to accurately execute the alarm regarding the person to be watched over getting out of bed.

Further, as another form of the image analysis device according to the one aspect, the bed detection unit is used when the foreground region appearing outside the bed moves to a position lower than a predetermined height directly above the bed surface. It may be determined that the foreground area appearing outside the bed has moved onto the bed, and it may be detected that the person to be watched has gone to bed.

When the person to be watched after leaving the bed goes to bed again, it is assumed that the foreground area appearing outside the bed moves to a position directly above the bed surface and lower than a predetermined height. Further, in general, it is assumed that a person other than the person to be watched does not go to bed in the bed of the person to be watched, even if he / she may enter the shooting range of the photographing device.

Therefore, the image analysis device according to the configuration detects the bed of the person to be watched when the foreground area appearing outside the bed moves to a position lower than a predetermined height directly above the bed surface. That is, the image analysis device according to the configuration detects the bed of the watching target person based on the fact that the watching target person has fallen asleep on the bed.

Therefore, according to the configuration, the bedtime of the watching target person is detected based on the sleeping behavior that is assumed that the person other than the watching target person does not perform. Therefore, it is possible to improve the accuracy of detecting the bed of the person being watched over, and thereby it is possible to accurately execute the alarm regarding the person being watched over getting out of bed.

As another form of the image analysis device according to each of the above forms, an information processing system that realizes each of the above configurations may be used, an information processing method may be used, or a program may be used. , A computer or other device or machine that records such a program may be a readable storage medium. Here, the recording medium that can be read by a computer or the like is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action. Further, the information processing system may be realized by one or more information processing devices.

For example, the image analysis method according to one aspect of the present invention is a photographed image taken by a photographing device installed by a computer to watch the behavior of the person to be watched in the bed, and each pixel in the photographed image. By calculating the difference between the step of acquiring the captured image including the depth data indicating the depth of the photographed image and the background image set as the background of the captured image and the captured image of the captured image, the captured image of the acquired image is calculated. By referring to the step of extracting the foreground region and the depth of each pixel included in the extracted foreground region, the foreground region appearing on the bed is located at a position separated from the bed by a predetermined distance or more in the real space. It is determined whether or not the person has moved, and when it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more, it is detected that the person to be watched has left the bed. By referring to the step and the depth of each pixel included in the extracted foreground region for the photographed image acquired after detecting that the watching target person has left the bed, the bed is placed in the real space. When it is determined whether or not the foreground area appearing outside has moved onto the bed and it is determined that the foreground area appearing outside the bed has moved onto the bed, the person to be watched over enters the bed. If the step of detecting that the person has floored and the elapsed time from the detection of the person being watched leaving the bed before detecting the bed of the person being watched exceeds a certain period of time, the person being watched over will It is an information processing method that executes a step of notifying that the person has been out of bed for a certain period of time or longer.

Further, for example, the image analysis program according to one aspect of the present invention is a photographed image taken by a photographing device installed on a computer in order to watch the behavior of the person to be watched in the bed, and is included in the photographed image. The captured image obtained by calculating the difference between the step of acquiring a captured image including depth data indicating the depth of each pixel and the acquired photographed image and the background image set as the background of the captured image. By referring to the step of extracting the foreground region of the image and the depth of each pixel included in the extracted foreground region, the foreground region appearing on the bed is separated from the bed by a predetermined distance or more in the real space. When it is determined whether or not the person has moved to the position and it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more, it is determined that the person to be watched has left the bed. By referring to the depth of each pixel included in the extracted foreground region for the detection step and the captured image acquired after detecting that the watching target person has left the bed, in the real space. , It is determined whether or not the foreground area appearing outside the bed has moved onto the bed, and when it is determined that the foreground area appearing outside the bed has moved onto the bed, the watching target person is the bed. If the elapsed time from the step of detecting that the person to be watched has gone to bed and the time elapsed since the person to be watched to leave the bed exceeds a certain period of time before detecting the person to be watched to go to bed, the watched object is to be watched. It is a program for executing a step of giving a notification to notify that a person has been out of bed for a certain period of time or longer.

According to the present invention, it is possible to accurately execute an alarm regarding getting out of bed of a person to be watched over.

FIG. 1 schematically illustrates an example of a situation in which the present invention is applied. FIG. 2 illustrates the hardware configuration of the image analysis device according to the embodiment. FIG. 3 illustrates the relationship between the depth acquired by the camera according to the embodiment and the subject. FIG. 4 illustrates the functional configuration of the image analysis apparatus according to the embodiment. FIG. 5 illustrates a processing procedure for watching over a person to be watched in the image analysis device according to the embodiment. FIG. 6 illustrates a captured image acquired by the camera according to the embodiment. FIG. 7 illustrates the coordinate relationship in the captured image according to the embodiment. FIG. 8 illustrates the positional relationship between an arbitrary point (pixel) of the captured image according to the embodiment and the camera in the real space. FIG. 9 illustrates a three-dimensional distribution of a subject within a shooting range specified based on depth data included in a shot image. FIG. 10 illustrates a three-dimensional distribution of the foreground region extracted from the captured image. FIG. 11A schematically illustrates an operation when the watching subject gets out of bed. FIG. 11B schematically illustrates an operation when the watch subject gets out of bed. FIG. 12A schematically illustrates a foreground region extracted when the watch subject gets out of bed. FIG. 12B schematically illustrates a foreground region extracted when the watch subject gets out of bed. FIG. 13A schematically illustrates an operation when the person to be watched goes to bed. FIG. 13B schematically illustrates an operation (end sitting position) when the person to be watched goes to bed. FIG. 13C schematically illustrates an operation (sleeping) when the person to be watched goes to bed. FIG. 14A schematically illustrates a foreground region extracted when the person to be watched goes to bed. FIG. 14B schematically illustrates a foreground region extracted when the person to be watched goes to bed. FIG. 14C schematically illustrates a foreground region extracted when the person to be watched goes to bed. FIG. 15A schematically illustrates a scene in which a person other than the person to be watched is working toward the bed. FIG. 15B schematically illustrates a scene in which the watching subject is sitting on the bed.

Hereinafter, embodiments according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. Although the data appearing in the present embodiment is described in natural language, more specifically, it is specified in a pseudo language, a command, a parameter, a machine language, etc. that can be recognized by a computer.

§1 Application scene First, a scene in which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates a scene in which the image analysis device 1 according to the present embodiment is used. The image analysis device 1 according to the present embodiment is an information processing device that detects the behavior of the watching target person by photographing the watching target person with the camera 2 and analyzing the captured image 3 obtained thereby. Therefore, the image analysis device 1 according to the present embodiment can be widely used in a scene of watching over a person to be watched over.

Specifically, as illustrated in FIG. 1, a camera 2 is installed to watch the behavior of the watching target person in the bed, and the image analysis device 1 photographs the watching target person by the camera 2. The camera 2 corresponds to the "photographing device" of the present invention. In addition, the person to be watched over is a person to be watched over, for example, an inpatient, a resident of a facility, and a person requiring long-term care.

The camera 2 is configured to be able to acquire the depth corresponding to each pixel in the captured image 3. In the present embodiment, the camera 2 includes a depth sensor (depth sensor 21 described later) that measures the depth of the subject so that the depth of each pixel can be acquired. Therefore, the captured image 3 captured by the camera 2 includes depth data indicating the depth obtained for each pixel, as illustrated in FIG. 6 described later. The depth of each pixel indicated by the depth data expresses the depth of the subject reflected in each pixel. Therefore, according to this depth data, it is possible to specify the position of the subject in the photographed image 3 in the real space without depending on the installation position and the photographing direction of the camera 2.

The captured image 3 may include data indicating the depth of the subject within the shooting range (angle of view). For example, the depth of the subject within the shooting range is distributed in a two-dimensional manner (for example, depth). It may be a map). Further, the captured image 3 may include an RGB image together with the depth data. Further, the captured image 3 may be a moving image, or may be one or a plurality of still images.

In the present embodiment, the image analysis device 1 acquires such a captured image 3 from the camera 2. Then, the image analysis device 1 calculates the difference between the acquired photographed image 3 and the background image (background image 4 described later) set as the background of the photographed image 3, thereby causing the foreground region in the photographed image 3. Is extracted. That is, the image analysis device 1 extracts the foreground region of the captured image 3 based on the so-called background subtraction method.

This foreground area is extracted from the background image at the place where the change occurs. Therefore, when the watching target person performs some action, the foreground area corresponding to the watching target person is extracted at the place where the watching target person moves due to the action. For example, as illustrated in FIG. 1, when the watch subject gets up on the bed, the foreground region appears in a predetermined height range on the bed surface. That is, the foreground area appears in a place corresponding to the behavior of the person to be watched over. Therefore, it is possible to analyze the behavior of the watching target person based on the positional relationship between the extracted foreground area and the bed in the real space.

Specifically, when the person to be watched gets out of bed, it is assumed that the foreground area appears on the bed when the person to be watched gets up, and the foreground area appearing in such a place moves away from the bed. Will be done. On the other hand, it is assumed that a person other than the person to be watched (hereinafter, also referred to as a “third party”) does not perform such an operation even if it exists in the shooting range of the camera 2. That is, in general, it is assumed that the foreground region corresponding to a third party only occurs around the bed and does not occur on the bed.

Therefore, the image analysis device 1 according to the present embodiment refers to the depth of each pixel included in the foreground region extracted from the captured image 3, so that the foreground region appearing on the bed in the real space can be seen from the bed. It is determined whether or not the user has moved to a position separated by a predetermined distance or more. Then, the image analysis device 1 according to the present embodiment detects that the watching target person has left the bed when it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more. .. Therefore, according to the present embodiment, even if a third party exists in the shooting range of the camera 2, the foreground area corresponding to the third party is watched over and erroneously recognized as the foreground area corresponding to the target person. Therefore, it is possible to appropriately detect the person being watched over getting out of bed. Therefore, according to the present embodiment, it is possible to accurately start the measurement of the elapsed time from the detection of the person being watched over getting out of bed.

In addition, when the person to be watched goes to bed, it is assumed that the foreground area is extracted at a place outside the bed when the person to be watched who has left the bed returns to the vicinity of the bed. For example, when the watching target person leaves the shooting range of the camera 2 due to getting out of bed, the foreground area is extracted near the frame of the shot image 3 when the watching target person enters the shooting range of the camera 2 again. .. Then, when the person to be watched goes to bed, it is assumed that the foreground area appearing in such a place moves on the bed. On the other hand, it is assumed that the third party does not go to bed of the person to be watched even if he / she enters the shooting range of the camera 2. That is, it is assumed that the foreground area corresponding to the third party only moves around the bed and does not move on the bed.

Therefore, the image analysis device 1 according to the present embodiment refers to the depth of each pixel included in the extracted foreground region for the photographed image 3 acquired after detecting the leaving of the bed of the person to be watched over in the real space. Above, it is determined whether or not the foreground area that appears outside the bed has moved onto the bed. Then, the image analysis device 1 according to the present embodiment detects that the person to be watched has gone to bed when it is determined that the foreground region appearing outside the bed has moved onto the bed. Therefore, according to the present embodiment, even if a third party enters the shooting range of the camera 2 after the watch target person gets out of bed, the foreground area corresponding to the third party is watched and the foreground corresponds to the target person. It is possible to appropriately detect the bedtime of the person being watched over without erroneously recognizing it as an area. Therefore, according to the present embodiment, it is possible to accurately stop the measurement of the elapsed time from the detection of the person being watched over getting out of bed.

Then, the image analysis device 1 according to the present embodiment performs the monitoring target when the elapsed time from detecting the leaving of the monitoring target person exceeds a certain time before detecting the bedtime of the monitoring target person. Notify that the person has been out of bed for a certain period of time or longer. That is, according to the present embodiment, in a situation where it is possible to appropriately detect getting out of bed and going to bed of the person being watched over, the person being watched over is based on the elapsed time from detecting the person being watched over getting out of bed. It is possible to execute an alarm regarding getting out of bed. Therefore, according to the present embodiment, it is possible to prevent erroneous detection of the person being watched over getting out of bed due to the presence of a third party in the shooting range of the camera 2, thereby giving an alarm about getting out of bed of the person being watched over. Can be executed accurately.

That is, in the present embodiment, the image analysis device 1 continuously acquires the captured image 3, and extracts the foreground region for each of the continuously acquired captured images 3. Based on the continuously acquired captured image 3, the image analysis device 1 first executes a process of detecting and determining the person being watched over to get out of bed. This process of detecting and determining that the person to be watched has left the bed is repeated until the person to be watched has left the bed.

Then, after detecting the bed leaving of the watching target person, the image analysis device 1 stops the processing of the detection determination of the bed leaving, and starts the processing of the detection determination of the bed of the watching target person. That is, the image analysis device 1 executes the process of detecting and determining the bed of the watching target person based on the captured image 3 acquired after detecting the leaving of the watching target person. This process of detecting and determining the bed of the person to be watched is repeated until the bed of the person to be watched is detected.

In addition, while repeating the process of detecting and determining the bed of the person to be watched, the process of determining whether or not the elapsed time since the person to be watched leaves the bed exceeds a certain time is executed. Then, in the image analysis device 1 according to the present embodiment, when the elapsed time after the monitoring target person leaves the bed exceeds a certain time before detecting the bedtime of the monitoring target person, the monitoring target person Executes a notification process to notify that the person has been out of bed for a certain period of time or longer.

As a result, according to the present embodiment, when the person to be watched is absent for a certain period of time or more in the space (bed) to be watched, the person to be watched is absent for a certain period of time or more. It is possible to inform the watchers who perform the above. Therefore, according to the image analysis device 1 according to the present embodiment, it is possible to prevent the person to be watched from leaving the bed for a long time, and to appropriately watch the person to be watched. Can be done.

In addition, in FIG. 1, the camera 2 is arranged so as to photograph the person to be watched from the footboard side of the bed. However, the installation position of the camera 2 is not limited to such an example, and may be arranged at any place as long as it is possible to photograph the behavior of the person to be watched over in the bed.

Further, the arrangement location of the image analysis device 1 can be appropriately determined according to the embodiment as long as the captured image 3 can be acquired from the camera 2. For example, the image analysis device 1 may be arranged so as to be close to the camera 2 as illustrated in FIG. Further, the image analysis device 1 may be connected to the camera 2 via a network, or may be arranged at a place completely different from the camera 2.

§2 Configuration example <Hardware configuration>
Next, the hardware configuration of the image analysis device 1 will be described with reference to FIG. FIG. 2 illustrates the hardware configuration of the image analysis device 1 according to the present embodiment. As illustrated in FIG. 2, the image analysis device 1 stores a control unit 11 including a CPU, a RAM (Random Access Memory), a ROM (Read Only Memory), and a program 5 executed by the control unit 11. A unit 12, a touch panel display 13 for displaying and inputting an image, a speaker 14 for outputting audio, an external interface 15 for connecting to an external device, a communication interface 16 for communicating via a network, and A computer to which a drive 17 for reading a program stored in the storage medium 6 is electrically connected. In FIG. 2, the communication interface and the external interface are described as "communication I / F" and "external I / F", respectively.

Regarding the specific hardware configuration of the image analysis device 1, it is possible to omit, replace, and add components as appropriate according to the embodiment. For example, the control unit 11 may include a plurality of processors. Further, for example, the touch panel display 13 may be replaced with an input device and a display device that are individually and independently connected to each other. Further, for example, the speaker 14 may be omitted. Further, for example, the speaker 14 may be connected to the image analysis device 1 as an external device instead of as an internal device of the image analysis device 1. Further, the image analysis device 1 may have a built-in camera 2. Further, the image analysis device 1 may include a plurality of external interfaces 15 and may be connected to a plurality of external devices.

In the present embodiment, the camera 2 is connected to the image analysis device 1 via the external interface 15 and is installed to photograph the person to be watched over. The camera 2 according to the present embodiment includes a depth sensor 21 for measuring the depth of the subject. The type and measurement method of the depth sensor 21 may be appropriately selected according to the embodiment. For example, as the depth sensor 21, a sensor such as a TOF (Time Of Flight) method can be mentioned.

However, the configuration of the camera 2 is not limited to such an example as long as the depth of the subject can be acquired, and can be appropriately selected according to the embodiment. For example, the camera 2 may be a stereo camera. Since the stereo camera shoots a subject within the shooting range from a plurality of different directions, the depth of the subject can be recorded. Further, the camera 2 may be replaced with the depth sensor 21 alone.

In addition, the place where the person to be watched is photographed may be dark. Therefore, the depth sensor 21 may be an infrared depth sensor that measures the depth based on the irradiation of infrared rays so that the depth can be acquired without being affected by the brightness of the photographing place. Examples of relatively inexpensive imaging devices including such an infrared depth sensor include Kinect from Microsoft, Xtion from ASUS, and Structure Sensor from Occipital.

Here, the depth measured by the depth sensor 21 according to the present embodiment will be described in detail with reference to FIG. FIG. 3 shows an example of a distance that can be treated as the depth according to the present embodiment. The depth expresses the depth of the subject. As illustrated in FIG. 3, the depth of the subject may be expressed by, for example, the distance A of the straight line between the camera 2 and the object, or the distance B of the perpendicular line drawn from the horizontal axis with respect to the subject of the camera 2. It may be expressed by.

That is, the depth according to the present embodiment may be a distance A or a distance B. In the present embodiment, the distance B is treated as the depth. However, the distance A and the distance B can be converted to each other by using, for example, the three-square theorem. Therefore, the following description using the distance B can be applied to the distance A as it is. By using such a depth, the image analysis device 1 according to the present embodiment can specify the position of the subject in the real space.

The camera 2 is installed so that the area around the bed can be photographed in order to watch the behavior of the person to be watched over in the bed. For example, as described above, in FIG. 1, the camera 2 is arranged so as to photograph the person to be watched from the footboard side of the bed. However, in the present embodiment, the camera 2 is configured to be able to measure the depth of the subject in this way. Therefore, in the present embodiment, the position of the subject in the real space can be specified without depending on the installation location and the shooting direction of the camera 2. Therefore, the installation location and shooting direction of the camera 2 can be set relatively freely.

Further, in the present embodiment, the storage unit 12 stores the program 5. This program 5 is a program for causing the image analysis device 1 to execute a processing procedure related to watching over a person to be watched, which will be described later, and corresponds to the "image analysis program" of the present invention. This program 5 may be recorded on the storage medium 6.

The storage medium 6 stores the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the computer or other device, the machine or the like can read the information of the recorded program or the like. It is a medium to do. The storage medium 6 corresponds to the "storage medium" of the present invention. Note that FIG. 2 illustrates a disc-type storage medium such as a CD (Compact Disk) or a DVD (Digital Versatile Disk) as an example of the storage medium 6. However, the type of the storage medium 6 is not limited to the disc type, and may be other than the disc type. Examples of storage media other than the disk type include semiconductor memories such as flash memories.

The image analysis device 1 may be, for example, a device designed exclusively for the provided service, or a general-purpose device such as a PC (Personal Computer) or a tablet terminal. Further, the image analysis device 1 may be implemented by one or more computers.

<Function configuration example>
Next, the functional configuration of the image analysis device 1 will be described with reference to FIG. FIG. 4 illustrates the functional configuration of the image analysis device 1 according to the present embodiment. In the present embodiment, the control unit 11 of the image analysis device 1 expands the program 5 stored in the storage unit 12 into the RAM. Then, the control unit 11 interprets and executes the program 5 expanded in the RAM by the CPU to control each component. As a result, the image analysis device 1 functions as a computer including an image acquisition unit 111, a foreground extraction unit 112, a bed leaving detection unit 113, a bedtime detection unit 114, and a notification unit 115.

The image acquisition unit 111 acquires the captured image 3 captured by the camera 2. Since the camera 2 includes the depth sensor 21, the acquired captured image 3 includes depth data indicating the depth of each pixel in the captured image 3. The foreground extraction unit 112 extracts the foreground region of the acquired photographed image 3 by calculating the difference between the acquired photographed image 3 and the background image 4 set as the background of the photographed image 3. That is, the foreground extraction unit 112 extracts the foreground region, which is a region changed from the background, in the captured image 3 based on the so-called background subtraction method.

As described above, the foreground region is extracted from the background image 4 at the place where the change occurs. Therefore, when the watching target person performs some action, the foreground area corresponding to the watching target person is extracted at the place where the watching target person moves due to the action. Therefore, the bed leaving detection unit 113 moves the foreground region appearing on the bed to a position separated from the bed by a predetermined distance or more in the real space by referring to the depth of each pixel included in the extracted foreground region. Determine if it has been done. Then, the bed leaving detection unit 113 detects that the watching target person has left the bed when it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more.

In addition, the bedtime detection unit 114 refers to the depth of each pixel included in the extracted foreground region for the photographed image 3 acquired after detecting that the person to be watched has left the bed, so that the real space Above, it is determined whether or not the foreground area that appears outside the bed has moved onto the bed. Then, when it is determined that the foreground area appearing outside the bed has moved onto the bed, the bed detection unit 114 detects that the person to be watched has gone to bed.

Then, the notification unit 115 leaves the watch target person for a certain period of time or more when the elapsed time from detecting the bed leaving of the watch target person exceeds a certain time before detecting the bed of the watch target person. Make a notification to let you know that you are. That is, when it is determined that the person being watched over has been out of bed for a certain period of time or longer, the notification unit 115 executes an alarm notifying that the person being watched over has been out of bed for a certain period of time or longer.

In this embodiment, an example in which all of these functions are realized by a general-purpose CPU is described. However, some or all of these functions may be realized by one or more dedicated processors. Further, regarding the functional configuration of the image analysis device 1, the functions may be omitted, replaced, or added as appropriate according to the embodiment. Each function will be described in detail in an operation example described later.

§3 Operation example Next, an operation example of the image analysis apparatus 1 will be described with reference to FIG. FIG. 5 illustrates a processing procedure for watching over a person to be watched by the image analysis device 1. The processing procedure for watching over the person to be watched described below corresponds to the "image analysis method" of the present invention. However, the processing procedure for watching over the person to be watched described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced and added as appropriate according to the embodiment.

(Overview)
In the present embodiment, the control unit 11 first executes a determination process of detecting the person being watched over to get out of bed by the processes of steps S101 to S104. The processes of steps S101 to S104 are repeated until it is detected that the person to be watched has left the bed. When the monitoring target person has detected the bed leaving, the control unit 11 then executes the determination process of detecting the monitoring target person's bedtime by the processing of steps S105 to S109, and after the watching target person leaves the bed. Determine the elapsed time. Then, if the monitoring target person detects the bedtime before the elapsed time from getting out of bed exceeds a certain time, the control unit 11 ends the process related to this operation example. On the other hand, if the elapsed time from the person to be watched getting out of bed exceeds a certain time before detecting the bedtime of the person to be watched over, in step S110, the person to be watched over has left the bed for a certain period of time or more. Make a notification to let you know. Hereinafter, each process will be described in detail.

(Step S101)
In step S101, the control unit 11 functions as an image acquisition unit 111 to acquire the captured image 3 captured by the camera 2. Then, when the captured image 3 is acquired, the control unit 11 proceeds to the next step S102.

As described above, in the present embodiment, the camera 2 includes the depth sensor 21. Therefore, the captured image 3 acquired in this step S101 includes depth data indicating the depth of each pixel. The control unit 11 acquires, for example, the photographed image 3 illustrated in FIG. 6 as the photographed image 3 including the depth data.

FIG. 6 shows an example of the captured image 3 including the depth data. The photographed image 3 illustrated in FIG. 6 is an image in which the shading value of each pixel is determined according to the depth of each pixel. The blacker the pixel, the closer it is to the camera 2. On the other hand, the whiter the pixel, the farther it is from the camera 2. The captured image 3 may be referred to as a depth image. Based on this depth data, the control unit 11 can specify the position of each pixel in the real space to be captured. That is, the control unit 11 can specify the position of the subject in each pixel in the three-dimensional space (real space) from the coordinates (two-dimensional information) and the depth of each pixel in the captured image 3. .. Hereinafter, with reference to FIGS. 7 and 8, a calculation example in which the control unit 11 specifies the position of each pixel in the real space is shown.

FIG. 7 illustrates the coordinate relationship in the captured image 3. Further, FIG. 8 illustrates the positional relationship between an arbitrary pixel (point s) of the captured image 3 and the camera 2 in the real space. The left-right direction of FIG. 7 corresponds to the direction perpendicular to the paper surface of FIG. That is, the length of the captured image 3 shown in FIG. 8 corresponds to the length (H pixel) in the vertical direction exemplified in FIG. 7. Further, the horizontal length (W pixel) exemplified in FIG. 7 corresponds to the length in the vertical direction of the paper surface of the captured image 3 not shown in FIG.

As illustrated in FIG. 7, assuming that the coordinates of an arbitrary pixel (point s) of the captured image 3 are (x _s , y _s ), the horizontal angle of view of the camera 2 is V _x , and the vertical image. Suppose the angle is V _y . Further, it is assumed that the number of pixels in the horizontal direction of the captured image 3 is W, the number of pixels in the vertical direction is H, and the coordinates of the center point (pixel) of the captured image 3 are (0,0).

The control unit 11 can acquire information indicating the angle of view (V _x , V _y ) of the camera 2 from the camera 2. However, the method of acquiring the information indicating the angle of view (V _x , V _y ) of the camera 2 is not limited to such an example, and the control unit 11 indicates the information indicating the angle of view (V _x , V _y ) of the camera 2. May be acquired based on user input, or may be acquired as a preset setting value. Further, the control unit 11 can acquire the coordinates (x _s , y _s ) of the point s and the number of pixels (W × H) of the photographed image 3 from the photographed image 3. Further, the control unit 11 can acquire the depth Ds of the point s by referring to the depth data included in the captured image 3.

By using this information, the control unit 11 can specify the position of each pixel (point s) in the real space. For example, the control unit 11 has a vector S (S _x , S _y , S _z) from the camera 2 to the point s in the camera coordinate system exemplified in FIG. 8 based on the relational expression shown by the following equations 1 to 3. , 1) can be calculated. As a result, the position of the point s in the two-dimensional coordinate system in the captured image 3 and the position of the point s in the camera coordinate system can be mutually converted.

However, the vector S is a vector of a three-dimensional coordinate system centered on the camera 2. The camera 2 may be tilted with respect to the horizontal direction, as illustrated in FIG. That is, the camera coordinate system may be tilted from the world coordinate system of the three-dimensional space (real space). Therefore, the control unit 11 applies the projective transformation using the roll angle, pitch angle (α in FIG. 8) and yaw angle of the camera 2 to the vector S, thereby changing the vector S of the camera coordinate system to the world coordinate system. The position of the point s in the world coordinate system may be calculated by converting to the vector of.

The captured image 3 may be a moving image, or may be one or a plurality of still images. Further, the control unit 11 may acquire such a captured image 3 in synchronization with the video signal of the camera 2. Then, the control unit 11 may immediately execute the processes of steps S102 to S104 described later for the captured image 3 acquired in synchronization with the camera 2. By continuously and continuously executing such an operation, the image analysis device 1 can detect the getting out of bed of the watching target person existing in the shooting range of the camera 2 in real time. In the following, an example of detecting the getting out of bed of the watching target person by the captured image 3 continuously acquired in this way will be described.

(Step S102)
Returning to FIG. 5, in the next step S102, the control unit 11 functions as the foreground extraction unit 112, and the difference between the captured image 3 acquired in step S101 and the background image 4 set as the background of the captured image 3. Is calculated to extract the foreground region of the captured image 3. The background image 4 is data used for extracting a foreground region in the captured image 3 based on the background subtraction method. The control unit 11 can create the background image 4 from the captured image 3 acquired at an arbitrary timing such as when the monitoring target person is started to be watched.

The method of creating the background image 4 from the captured image 3 may be appropriately selected depending on the embodiment. For example, the control unit 11 may set the captured image 3 for one frame obtained by the camera 2 as the background image 4 as it is. Further, for example, the control unit 11 may create the background image 4 by calculating the average of the captured images 3 for several frames obtained by the camera 2. As a result, the background image 4 including the depth data indicating the depth of each pixel can be generated.

In this step S102, the control unit 11 extracts the foreground region in the captured image 3 acquired in step S101 by calculating the difference between the background image 4 created in this way and the captured image 3 acquired in step S101. To do. For example, the control unit 11 can extract a foreground region in the captured image 3 as illustrated in FIGS. 9 and 10.

FIG. 9 illustrates a three-dimensional distribution of each pixel (subject) within a shooting range specified based on the depth data included in the shot image 3. FIG. 10 illustrates the three-dimensional distribution of the foreground region extracted from the captured image 3 among the subjects illustrated in FIG. As described above, the position of each pixel of the captured image 3 in the three-dimensional space (real space) can be specified based on the depth data. Therefore, the control unit 11 plots each pixel in the three-dimensional space according to the position and depth in the captured image 3, so that the captured image 3 illustrated in FIG. 6 to the three-dimensional distribution exemplified in FIG. 9 are plotted. Can be created. Further, the background image 4 also includes depth data like the captured image 3. Therefore, the control unit 11 can also specify the three-dimensional distribution of each pixel as illustrated in FIG. 9 for the background image 4.

Therefore, the control unit 11 compares the depths of the corresponding pixels of the captured image 3 and the background image 4. Then, the control unit 11 extracts pixels having different depths between the captured image 3 and the background image 4 as the foreground region. As a result, the control unit 11 can extract the foreground region as illustrated in FIG. When the foreground region is extracted from the captured image 3 acquired in step S101, the control unit 11 proceeds to the next step S103.

(Step S103)
Returning to FIG. 5, in the next step S103, the control unit 11 functions as the bed leaving detection unit 113, and executes a determination process for detecting the bed leaving of the watching target person. That is, the control unit 11 refers to the depth of each pixel included in the foreground region extracted in step S102 so that the foreground region appearing on the bed is located at a position separated from the bed by a predetermined distance or more in the real space. Determine if it has moved.

Then, in the present step S103, when it can be determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more, the control unit 11 detects that the person to be watched has left the bed and then leaves the bed. The process proceeds to the next step S104. On the other hand, if such a determination cannot be made, the control unit 11 proceeds to the next step S104 without detecting the person being watched over from bed.

Here, with reference to FIGS. 11A, 11B, 12A and 12B, the determination process for detecting the leaving of the watched person in the step S103 will be specifically described. FIG. 11A schematically illustrates a scene in which the watching subject gets up from the bed when getting out of bed. FIG. 11B schematically illustrates a scene in which the person to be watched moves to a position separated from the bed by a predetermined distance HA or more. FIG. 12A schematically illustrates the position of the foreground region extracted during the scene of FIG. 11A. Further, FIG. 12B schematically illustrates the position of the foreground region extracted in the scene of FIG. 11B. In the following, for convenience of explanation, the direction connecting the headboard and footboard of the bed is referred to as the "front-back direction", and the direction connecting the side portions (side frames) of the bed is referred to as the "left-right direction". The vertical direction of is referred to as "vertical direction".

When getting out of bed, the person being watched first moves his / her body on the bed to get up from the bed. For example, as illustrated in FIG. 11A, the watching subject performs an action of raising the upper body on the bed. Then, the person to be watched stands up on the side of the bed and moves in the direction away from the bed as illustrated in FIG. 11B.

Therefore, in the series of processes of steps S101 to S104 that are repeated until the monitoring target person gets out of bed, in the step S101 that is repeatedly executed, as a result, a series of movements of the monitoring target person are photographed. The image group of the image 3 is acquired. Then, in step S102, in the first captured image 3 included in this image group, as illustrated in FIG. 12A, the foreground region corresponding to the person to be watched is extracted above the bed surface SU. On the other hand, in the last captured image 3 included in this image group, as illustrated in FIG. 12B, as the watching target person moves away from the bed, the foreground area corresponding to the watching target person also moves away from the bed. Will be extracted with.

Therefore, in the present embodiment, the control unit 11 detects the person to be watched to get out of bed as follows. That is, the control unit 11 first captures the foreground region that appears on the bed. Specifically, in step S102, as illustrated in FIG. 12A, when the foreground region is extracted above the bed surface SU, the control unit 11 detects that the person to be watched leaves the bed in step S103. No, but this foreground area is the target of tracking. The range of the bed surface SU may be preset on the system or may be appropriately set by the user. As a result, the control unit 11 can track the foreground region in which the same target (watching target person) appears in the image group of the captured image 3 repeatedly acquired in the subsequent steps S101.

Then, the control unit 11 calculates the distance between the tracking foreground region and the bed, and determines whether or not the calculated distance is equal to or greater than the predetermined distance HA. The distance between the foreground area and the bed can be appropriately calculated by a known method. If the distance between the foreground area being tracked and the bed is not greater than or equal to the predetermined distance HA, the control unit 11 does not detect the person being watched over and continues tracking the foreground area. On the other hand, when the distance between the tracking foreground area and the bed is a predetermined distance HA or more, the control unit 11 moves the foreground area appearing on the bed to a position separated from the bed by a predetermined distance HA or more. It is determined that the person has left the bed, and the person to be watched over is detected.

When the person to be watched gets up on the bed, the foreground region can be extracted from the bed surface SU at a position equal to or higher than a predetermined height. Therefore, the control unit 11 may target the tracking only in the foreground region that appears above the predetermined height from the bed surface SU. This predetermined height may be preset on the system or may be appropriately set by the user. The value of this predetermined height may be appropriately set based on the height of the foreground region that may occur on the bed when the person to be watched leaves the bed.

In this case, the control unit 11 may treat the position of an arbitrary point included in the foreground region as the position of the foreground region. For example, the control unit 11 may treat the position of the center of gravity of the foreground region calculated by the following equation 4 as the position of the foreground region. Hereinafter, the same applies to the position of the foreground region in other scenes.

Note that w indicates the length of an arbitrary pixel (point s) in the real space in the horizontal direction. Further, h indicates the length of an arbitrary pixel (point s) in the real space in the vertical direction. w and h can be calculated by the numbers 5 and 6, which will be described later, respectively. Here, M is the area average of the vector S of the pixels (points) included in the foreground region F. That is, M indicates the average position of the foreground region in the real space. Therefore, the position indicated by M can be treated as the position of the center of gravity of the foreground region.

Further, the control unit 11 may provide a capture area in a predetermined range above the bed surface SU, and may target only the foreground area appearing in this capture area as a tracking target. In this case, the lengths of the capture area in the front-rear direction, the left-right direction, and the up-down direction may be preset on the system or may be appropriately set by the user. Similar to the above, this capture area is appropriately set so that the foreground area that may occur on the bed when the person to be watched leaves the bed can be appropriately captured.

Further, the value of the predetermined distance HA, which is a reference for detecting the person to be watched to get out of bed, may be set in advance on the system or may be appropriately set by the user. Further, FIG. 12B illustrates a scene in which the subject can get out of bed while watching over both sides of the bed in the left-right direction. However, the direction for detecting the leaving of the bed of the person to be watched over is not limited to such an example, and may be appropriately selected according to the embodiment. For example, the direction in which the person to be watched can get out of bed may be limited to one direction instead of the two directions as illustrated in FIG. 12B. Further, for example, the control unit 11 may detect that the person to be watched leaves the bed based on the distance between the foreground region and the bed in the front-rear direction.

Further, the foreground region extracted in step S102 may correspond to an object other than a person (for example, a curtain or the like). Therefore, in order to exclude the foreground region corresponding to such an object other than the person from the processing target of step S103, the control unit 11 sets the foreground area extracted in step S102 to be an area in which a person such as a person to be watched is captured. You may perform the process of determining whether or not there is. For example, such a determination process can be performed as follows.

That is, when the foreground area extracted in step S102 is an area in which a person such as a person to be watched is captured, the shape of this foreground area corresponds to the shape of the person. Therefore, the control unit 11 may determine whether or not the shape of the foreground region extracted in step S102 corresponds to a person such as a person to be watched by image processing such as pattern matching. Then, as a result of pattern matching, the control unit 11 may exclude the foreground area corresponding to the shape of the person from the processing target of step S103, and may set the foreground area corresponding to the shape of the person as the processing target of step S103.

Further, when the foreground area extracted in step S102 is an area in which a person such as a person to be watched is captured, the area of this foreground area corresponds to the size of the person. Therefore, the control unit 11 may determine whether or not the area of the foreground region is included in the predetermined area range. This predetermined area range is set to cover a value that can be taken as the area of the foreground area in which the person to be watched is captured. Then, the control unit 11 may exclude the foreground area having an area not included in the predetermined area range from the processing target in step S103, and may set the foreground area having an area included in the predetermined area range as the processing target in step S103.

At this time, the area of the foreground region may be given by the number of pixels included in the foreground region. However, the farther the subject is from the camera 2, the smaller the image of the subject in the captured image 3, and the closer the subject is to the camera 2, the larger the image of the subject in the captured image 3. The depth of the subject captured in the captured image 3 is acquired with respect to the surface of the subject, but the area of the surface portion of the subject corresponding to each pixel of the captured image 3 does not always match between the pixels.

Therefore, in order to exclude the influence of the perspective of the subject, the control unit 11 may calculate the area of the extracted foreground region in the real space by using the depth of each pixel in this step S103. The area of the foreground region in the real space can be calculated, for example, as follows. That is, first, the control unit 11 first has the lateral length of any point s (1 pixel) exemplified in FIGS. 7 and 8 in the real space based on the following relational expressions of the equations 5 and 6. w and / or the length h in the vertical direction are calculated respectively.

Subsequently, the control unit 11 calculates the area of one pixel in the real space at the depth Ds by the square of w, the square of h, or the product of w and h calculated in this way. Then, the control unit 11 calculates the area of the foreground region in the real space by obtaining the total area of each pixel included in the foreground region in the real space. As a result, the area of the foreground region can be obtained after excluding the influence of the perspective of the subject. Hereinafter, the same applies to the area of the foreground area in other scenes.

It should be noted that such an area may change significantly due to noise of depth data, movement of an object other than the person to be watched, and the like. In order to correspond to this, the control unit 11 may use the average of the areas for several frames. Further, when the difference between the area of the corresponding area in the frame to be processed and the average of the area of the corresponding area in several frames in the past than the frame to be processed exceeds a predetermined range, the control unit 11 sets the corresponding area. It may be excluded from the processing target.

In the present embodiment, it is possible to detect the person being watched over from bed by the above processing. However, the method of detecting the person being watched over from bed does not have to be limited to such an example, and may be appropriately changed according to the embodiment.

(Step S104)
Returning to FIG. 5, in the next step S104, the control unit 11 determines whether or not the monitoring target person has detected that he / she has left the bed in step S103. If the monitoring target person does not get out of bed in step S103, the control unit 11 returns the process to step S101. That is, the control unit 11 repeats the determination process (steps S101 to S104) for detecting the leaving of the watching target person until it detects that the watching target person has left the bed.

On the other hand, when it is detected in step S103 that the person to be watched leaves the bed, the control unit 11 proceeds to the next step S105. That is, when the monitoring target person has detected that he / she has left the bed, the control unit 11 describes the determination process for detecting the monitoring target person's bedtime and the elapsed time since the watching target person has left the bed (hereinafter, also referred to as “the elapsed bed leaving time”). ) Starts execution of the process of determining whether or not it exceeds a certain period of time. At this time, the control unit 11 may start a timer and measure the elapsed bed leaving time by this timer in order to specify the elapsed bed leaving time in the process of step S109 described later. Further, the control unit 11 may hold the time when the person to be watched over has detected that he / she has left the bed.

(Step S105 and Step S106)
In the next step S105, the control unit 11 functions as the image acquisition unit 111, and acquires the captured image 3 from the camera 2 in the same manner as in the step S101. Then, after acquiring the captured image 3, in step S106, the control unit 11 functions as the foreground extraction unit 112, and similarly to step S102, the difference between the captured image 3 and the background image 4 is calculated. The foreground region of the captured image 3 is extracted.

As described above, the processes of steps S105 to S109 correspond to the determination process of detecting the bedtime of the person to be watched over and the process of determining whether or not the elapsed time of getting out of bed exceeds a certain time. The processes of steps S105 to S109 are executed until the bedtime of the person being watched over is detected or the elapsed time of getting out of bed exceeds a certain time. Therefore, the processes of steps S105 and S106 are repeatedly executed in the same manner as in steps S101 and S102 until the bedtime of the person to be watched is detected or the elapsed time of getting out of bed exceeds a certain time. That is, the captured image 3 is continuously acquired in step S105 until the bedtime of the person to be watched is detected or the elapsed time of getting out of bed exceeds a certain time, and the foreground of the captured image 3 is continuously acquired in step S106. The area is extracted.

(Step S107)
In the next step S107, the control unit 11 functions as the bedtime detection unit 114, and executes a determination process for detecting the bedtime of the monitoring target person with respect to the captured image 3 acquired after detecting the bed leaving of the monitoring target person. To do. That is, the control unit 11 refers to the depth of each pixel included in the foreground region extracted in step S106 to determine whether or not the foreground region appearing outside the bed has moved onto the bed in the real space. judge.

Then, in this step S107, when it can be determined that the foreground area appearing outside the bed has moved onto the bed, the control unit 11 detects the bed of the person to be watched and then moves to the next step S108. Proceed with processing. On the other hand, if such a determination cannot be made, the control unit 11 proceeds to the next step S108 without detecting the bed of the person to be watched over.

Here, with reference to FIGS. 13A to 13C and FIGS. 14A to 14C, the determination process for detecting the bedtime of the person to be watched over will be specifically described in this step S107. FIG. 13A schematically illustrates a scene in which a watching subject who has left the bed returns to the vicinity of the bed. FIG. 13B schematically illustrates a scene in which the person to be watched is sitting on the floor when going to bed. FIG. 13C schematically illustrates a scene in which the watching subject goes to bed on the bed. 14A to 14C schematically illustrate the position of the foreground region extracted in each of the scenes of FIGS. 13A to 13C, respectively.

When going to bed, first, the person to be watched moves from a position away from the bed to approach the bed, as illustrated in FIG. 13A. Then, the person to be watched sits on the side of the bed as illustrated in FIG. 13B, and goes to bed on the bed as illustrated in FIG. 13C. Alternatively, the watching subject sleeps on the bed as illustrated in FIG. 13C, omitting the end sitting position exemplified in FIG. 13B.

Therefore, in step S105, which is repeatedly executed until the monitoring target person is detected to go to bed, as a result, the watching target person goes to bed after sitting on the edge or goes to bed without sitting on the edge. An image group of the captured image 3 obtained by photographing a series of operations can be acquired. Then, in step S106, in the first captured image 3 included in this image group, the foreground region corresponding to the person to be watched is extracted outside the bed, as illustrated in FIG. 14A. This foreground area moves around the bed and eventually approaches the vicinity of the bed.

After that, when the watching target person sits on the edge, the foreground area corresponding to the watching target person temporarily stops at the side of the bed. That is, in step S106, in the photographed image 3 acquired while the watching target person is sitting on the edge, the foreground region corresponding to the watching target person is extracted on the bed side as illustrated in FIG. 14B. Will be done.

On the other hand, when the person to be watched goes to bed on the bed without sitting on the edge, the foreground area approaching the vicinity of the bed moves to the bed as it is. That is, in step S106, in the captured image 3 acquired when the watch target person goes to bed, the foreground area corresponding to the watch target person is relatively low directly above the bed surface SU, as illustrated in FIG. 14C. Extracted by position.

Therefore, in the present embodiment, the bedtime of the person to be watched over is detected based on the two states of the sitting position and the sleeping posture. That is, when the foreground area appearing outside the bed moves to a predetermined area DB provided on the side of the bed, or when the control unit 11 moves to a position lower than the predetermined height HC directly above the bed surface SU. , Detects the bed of the person being watched over.

(1) Bedhold detection based on the end sitting position First, a method of detecting bedtime based on the end sitting position will be described. As described above, the foreground region appears outside the bed as illustrated in FIG. 14A by the time the bed is detected based on the sitting position. Then, the foreground region that appears outside the bed moves to the side of the bed as illustrated in FIG. 14B.

Therefore, the control unit 11 can detect the bedtime of the person to be watched over as follows. That is, the control unit 11 first captures the foreground region that appears outside the bed. Specifically, in step S106, as illustrated in FIG. 14A, when the foreground region is extracted outside the bed, the control unit 11 does not detect the bed of the person to be watched over in step S107. , This foreground area is the target of tracking. As a result, the control unit 11 can track the foreground region in which the same target (watching target person) appears in the image group of the captured image 3 repeatedly acquired in the subsequent steps S105.

Then, the control unit 11 determines whether or not the tracked foreground area has moved into a predetermined area DB provided on the bed side. This predetermined area DB is appropriately set at a place where the foreground area is extracted when the person to be watched sits on the side of the bed.

As a result of this determination, if the tracked foreground area has not moved into the predetermined area DB, the control unit 11 does not detect the bed of the person to be watched and continues tracking the foreground area. .. On the other hand, when the tracked foreground area moves into the predetermined area DB, the control unit 11 determines that the foreground area appearing outside the bed has moved onto the bed, and causes the person to be watched to go to bed. Detect.

As in step S103, the control unit 11 may limit the foreground region to be processed based on the shape and / or area of the foreground region. Further, the area for capturing the foreground area appearing outside the bed may be an area outside the bed, and may be appropriately set according to the embodiment.

Further, it may be determined whether or not the tracked area has moved into the predetermined area DB based on the center of gravity of the foreground area as illustrated in FIG. 14B. The center of gravity of this foreground region can be calculated by the above equation 4. In this case, the predetermined area DB is appropriately set so as to include a range in which the center of gravity of the foreground area extracted when the person to be watched sits on the side of the bed can exist. It should be noted that the determination as to whether or not the tracked area has moved into the predetermined area DB may be performed based on the entire foreground area or a part of the foreground area. .. The predetermined area DB is appropriately set according to the reference of the foreground area to be determined.

Further, in FIG. 14B, the predetermined area DB is set on each bed side portion in the left-right direction. However, the setting of the predetermined area DB may be omitted on either bed side. That is, the control unit 11 may set the predetermined area DB only on one of the bed side portions. Further, in FIG. 14B, two predetermined region DBs are provided so as to be separated from each other in the left-right direction. However, the method of setting the predetermined area DB does not have to be limited to such an example, and by connecting the two predetermined area DBs in the left-right direction, one predetermined area DB extending in the left-right direction can be set. May be done. In addition, such a predetermined area DB may be preset on the system or may be appropriately set by the user.

Further, while the person to be watched is sitting on the side of the bed, the foreground area corresponding to the person to be watched is continuously extracted on the side of the bed. Therefore, the control unit 11 determines that the foreground area appearing outside the bed has moved onto the bed when the foreground area appearing outside the bed remains in the predetermined area DB for a certain period of time or longer. The bed of the person to be watched over may be detected. As a result, when some object passes through the side of the bed, it is possible to prevent the watching target person from erroneously detecting the bed. It should be noted that the threshold value (constant time) that serves as a reference for detecting the bedtime of the person to be watched over may be set in advance on the system or may be appropriately set by the user.

(2) Bedtime detection based on sleeping posture Second, a method of detecting bedtime based on sleeping posture will be described. As described above, the foreground region appears outside the bed as illustrated in FIG. 14A by the time the bed is detected based on the sleeping posture. Then, the foreground region appearing outside the bed moves to a relatively low position directly above the bed surface SU, as illustrated in FIG. 14C.

Therefore, the control unit 11 can detect the bedtime of the person to be watched over as follows. That is, first, in the same manner as described above, when the foreground region is extracted outside the bed as illustrated in FIG. 14A in step S106, the control unit 11 controls this in step S107. Target the foreground area for tracking. As a result, the control unit 11 can track the foreground region in which the same target (watching target person) appears in the image group of the captured image 3 repeatedly acquired in the subsequent steps S105.

Then, the control unit 11 determines whether or not the tracked foreground region has moved to a position lower than a predetermined height HC directly above the bed surface SU. The value of this predetermined height HC is appropriately set so as to be higher than the height of the foreground region extracted when the person to be watched goes to bed on the bed surface SU.

As a result of this determination, if the tracked foreground area has not moved to a position lower than the predetermined height HC directly above the bed surface SU, the control unit 11 does not detect the bed of the person to be watched over. , Continue tracking its foreground area. On the other hand, when the tracked foreground region moves to a position lower than the predetermined height HC directly above the bed surface SU, the control unit 11 determines that the foreground region appearing outside the bed has moved onto the bed. Then, it detects the bed of the person to be watched over.

In the same manner as described above, the control unit 11 may limit the foreground region to be processed based on the shape and / or area of the foreground region. Further, the area for capturing the foreground area appearing outside the bed may be an area outside the bed, and may be appropriately set according to the embodiment.

Further, the determination as to whether or not the tracked area has moved to a position lower than the predetermined height HC directly above the bed surface SU may be performed based on the center of gravity of the foreground area as described above. , It may be done based on the whole foreground area, or it may be done based on a part of the foreground area. The predetermined height HC is appropriately set according to the reference of the foreground region to be determined. This predetermined height HC may be preset on the system or may be appropriately set by the user.

(3) Others In the present embodiment, the control unit 11 can detect the bed of the person to be watched over by the above two determination processes. However, the method of detecting the bedtime of the person to be watched over may not be limited to such an example, and may be appropriately changed depending on the embodiment. For example, the control unit 11 may omit either of the above two determination processes. Further, for example, the control unit 11 may detect the bed of the person to be watched by a process other than the above two determination processes.

(Step S108)
Returning to FIG. 5, in the next step S108, the control unit 11 determines whether or not it is detected in step S107 that the watching target person has gone to bed. If the monitoring target person does not detect the bed in step S107, the control unit 11 proceeds to the next step S109.

On the other hand, when it is detected in step S107 that the person to be watched goes to bed, the control unit 11 ends the process related to this operation example. At this time, the control unit 11 may specify the time to get out of bed from the detection of the person to be watched to get out of bed to the time to detect the person to be watched to go to bed, and record the time to get out of bed in the storage unit 12 or the like. The method for specifying the time to get out of bed may be appropriately selected depending on the embodiment. For example, when the elapsed time of getting out of bed is measured by a timer, the control unit 11 can specify the time of getting out of bed by stopping the counting of the timer at this point and referring to the timer. Further, for example, when the time when the person being watched over is detected to get out of bed is held, the control unit 11 calculates the difference between the time when the person being watched over is detected to get out of bed and the time at the time of this processing. With, the time to get out of bed can be specified.

In this case, the person to be watched over is in bed. Therefore, the control unit 11 may start the process again from step S101 in preparation for the person to be watched to get out of bed again. That is, the control unit 11 may continue watching the watching target person by starting from the determination process of detecting the leaving of the watching target person.

(Step S109)
In the next step S109, the control unit 11 determines whether or not the elapsed time from getting out of bed after detecting the getting out of bed of the watching target person exceeds a certain time in step S103. As a result of this determination, if the elapsed time of getting out of bed does not exceed a certain time, the control unit 11 returns the process to step S105. That is, until the monitoring target person's bedtime is detected or the elapsed bed leaving time exceeds a certain time, the determination process for detecting the bedtime of the watching target person and the determination process for determining whether or not the bed leaving elapsed time exceeds a certain time are performed. repeat.

On the other hand, when the elapsed time of getting out of bed exceeds a certain time, the control unit 11 proceeds to the next step S110. Here, in step S110, which will be described later, an alarm is issued to notify that the person to be watched has left the bed for a certain period of time or longer. That is, the threshold value (constant time) that serves as a reference for this branch defines the elapsed time for getting out of bed to issue the alarm. This threshold value (constant time) may be preset on the system or may be appropriately set by the user.

The elapsed time for getting out of bed at this point can be specified as appropriate. For example, when the elapsed time for getting out of bed is measured by a timer, the control unit 11 can specify the time for detecting getting out of bed at this time by referring to the timer at this time. Further, for example, when the time when the person being watched over is detected to get out of bed is held, the control unit 11 calculates the difference between the time when the person being watched over is detected to get out of bed and the time at the time of this processing. Therefore, the time for detecting getting out of bed at this time can be specified.

Further, the process of this step S109 may be executed at an arbitrary timing. For example, the process of this step S109 may be executed separately from the process of steps S105 to S108. When the process of this step S109 is executed before the process of step S108, when it is determined in step S108 that the bed of the person to be watched is not detected, the control unit 11 processes in step S105. Return.

(Step S110)
In the next step S110, the control unit 11 functions as the notification unit 115, and gives a notification to notify that the person to be watched has left the bed for a certain period of time or longer. That is, when the control unit 11 determines in step S109 that the elapsed time of getting out of bed exceeds a certain time before detecting the bedtime of the person to be watched in step S107, the person to be watched is absent for a certain period of time or more. Execute an alarm to inform you that there is.

The notification destination and the notification method can be appropriately selected according to the embodiment. The control unit 11 may, for example, give the notification to a third party other than the watching target person, particularly to the watching person who watches the behavior of the watching target person. When the person to be watched is an inpatient, a resident of a facility, a person requiring long-term care, etc., the person to be watched is, for example, a nurse, a facility staff, a caregiver, or the like. In addition, the control unit 11 may give the notification to the person to be watched over.

For example, when the image analysis device 1 according to the present embodiment is used in a facility such as a hospital, the image analysis device 1 can be connected to equipment such as a nurse call system via an external interface 15. In this case, the control unit 11 may give a notification to notify that the person to be watched has left the bed for a certain period of time or more in cooperation with the equipment such as the nurse call system. That is, the control unit 11 may control the nurse call system via the external interface 15. Then, the control unit 11 may make a call by the nurse call system as a notification for notifying that the person to be watched has left the bed for a certain period of time or more. As a result, it is possible to appropriately notify the nurse or the like who is watching over the person to be watched that the person to be watched is away from the bed for a certain period of time or more.

Further, for example, the control unit 11 may give a notification to notify that the person to be watched has left the bed for a certain period of time or more by displaying the screen on the touch panel display 13. Further, for example, the control unit 11 may output a predetermined sound from the speaker 14 connected to the image analysis device 1 to notify that the person to be watched is out of bed for a certain period of time or longer. Good. By installing the touch panel display 13 and the speaker 14 in the room of the watcher, it is possible to appropriately notify the watcher that the watcher is away from the bed for a certain period of time or longer.

Further, for example, the control unit 11 may use e-mail, short message service, push notification, or the like to give a notification to notify that the person to be watched is out of bed for a certain period of time or longer. When making such a notification, the e-mail address, telephone number, etc. of the user terminal to be notified may be registered in advance in the storage unit 12. Then, the control unit 11 may use the pre-registered e-mail address, telephone number, or the like to give a notification to notify that the person to be watched is out of bed for a certain period of time or longer.

In the present embodiment, after giving such a notification, the control unit 11 ends the process related to this operation example. However, when the process is completed in this way, unlike the case where the bed of the monitoring target person is detected and the process is completed in step S108, the bed of the watching target person is not detected. That is, the person being watched over has not returned to bed while leaving the bed. Therefore, in this case, the control unit 11 may repeat the processes of steps S105 to S110 until the monitoring target person is detected to go to bed. At this time, since the control unit 11 starts to newly measure the elapsed bed leaving time, the bed leaving elapsed time measured in the previous processes may be reset.

(Action / effect)
As described above, the image analysis device 1 according to the present embodiment first executes the determination process of detecting the person being watched over from bed by the process of steps S101 to S104. Next, after detecting the person being watched over from bed, the image analysis device 1 executes a determination process for detecting the bedtime of the person being watched over and a process for determining whether or not the elapsed time for getting out of bed exceeds a certain time. .. Then, if the elapsed time of getting out of bed exceeds a certain time before detecting the bedtime of the person to be watched over, the image analysis device 1 has left the person to be watched over for a certain period of time or more by the process of step S110. Make a notification to let you know.

Here, the image analysis device 1 according to the present embodiment is the monitoring target based on the fluctuation of the foreground region extracted by the series of actions of the monitoring target person exemplified in FIGS. 11A and 11B in step S103. Detects a person getting out of bed. Therefore, according to the present embodiment, even if a third party exists in the shooting range of the camera 2, the foreground area corresponding to the third party is watched over and erroneously recognized as the foreground area corresponding to the target person. Therefore, it is possible to appropriately detect the person being watched over getting out of bed.

Similarly, the image analysis device 1 according to the present embodiment is the monitoring target based on the fluctuation of the foreground region extracted by the series of actions of the watching target person exemplified in FIGS. 13A to 13C in step S107. The person's bed is detected. Therefore, according to the present embodiment, even if a third party enters the shooting range of the camera 2 after the watch target person gets out of bed, the foreground area corresponding to the third party is watched and the foreground corresponds to the target person. It is possible to appropriately detect the bedtime of the person being watched over without erroneously recognizing it as an area.

Therefore, according to the present embodiment, it is possible to prevent erroneous detection of getting out of bed and going to bed of the person to be watched due to the presence of a third party in the shooting range of the camera 2, and to get out of bed and going to bed of the person to be watched over. Each can be detected appropriately. Therefore, according to the present embodiment, it is possible to accurately execute an alarm regarding getting out of bed of the person to be watched over.

In particular, in the present embodiment, in step S107, the control unit 11 detects the bed of the watching target person based on the change in the foreground area when the watching target person sits on the bed or goes to bed. It is assumed that a third party other than the person to be watched may enter the shooting range of the camera 2 but will not sit on the bed or go to bed. Therefore, according to the present embodiment, the bedtime of the watching target person is detected based on the operation that is assumed that the person other than the watching target person does not perform. Therefore, according to the present embodiment, it is possible to improve the accuracy of detecting the bed of the person being watched over, and thereby it is possible to accurately execute the alarm regarding the person being watched over getting out of bed.

§4 Modifications Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention.

For example, in the above embodiment, in step S107, the control unit 11 sets a predetermined area DB on the bed side. Then, the control unit 11 is based on the fact that the foreground region, which fluctuates with a series of movements when the watch target person performs the end sitting position on the bed side, enters the predetermined area DB, and the watch target person Detecting bedtime. This detection method may cause the following problems.

That is, a third party such as a caregiver may perform work such as bed-making in the vicinity of the bed side. In such a case, since the body of the third party may be located above the bed surface SU, the foreground area corresponding to the third party may be placed in the predetermined area DB depending on the setting range of the predetermined area DB. May move from outside. That is, there is a possibility that the above-mentioned movement of a third party other than the watching target person may be erroneously detected as the sleeping movement of the watching target person.

Here, with reference to FIGS. 15A and 15B, a scene in which a third party works in the vicinity of the bed side and a scene in which the watching target person sits on the edge are compared. FIG. 15A schematically illustrates a scene in which a third party is working toward the bed. Further, FIG. 15B schematically illustrates a scene in which the watching subject is sitting on the bed.

As illustrated in FIGS. 15A and 15B, when the predetermined area DB is arranged further outside the bed, in addition to the foreground area extracted when the watching target person performs the end sitting position, a third party The foreground area extracted when the work is performed near the bed side can enter the predetermined area DB. Therefore, when the range of the predetermined area DB is set in this way, there is a possibility that the movement of a third party other than the watching target person may be erroneously detected as the sleeping movement of the watching target person.

However, as illustrated in FIG. 15A, it is generally assumed that a third party performing such work such as bed-making is facing the inside of the bed. Therefore, the center of gravity of the head of this third party is located inside the bed rather than the center of gravity of the shoulder. On the other hand, as illustrated in FIG. 15B, it is generally assumed that the watching subject who sits on the side of the bed faces the outside of the bed. Therefore, the center of gravity of the head of the person being watched over is located inside the bed rather than the center of gravity of the shoulder.

Therefore, in step S107, when the tracking foreground area moves into the predetermined area DB, the control unit 11 further sets the tracking head area as the head area in the tracking foreground area. You may infer that it is the shoulder area where the shoulder is reflected. Such region estimation can be performed by known image processing such as pattern matching.

Then, the control unit 11 may determine the positional relationship between the estimated center of gravity of the head region and the center of gravity of the shoulder region. The center of gravity of the head region and the center of gravity of the shoulder region can be calculated by the calculation of the above equation 4. As a result of the determination, when the center of gravity of the head region is located outside the bed of the shoulder region, the control unit 11 detects that the person to be watched has gone to bed. On the other hand, when the center of gravity of the head region is located inside the bed with respect to the center of gravity of the shoulder region, the control unit 11 continues the process without detecting the bed of the person to be watched over. If the center of gravity of this head area is located inside the bed from the center of gravity of the shoulder area for a certain period of time or longer, the foreground area being tracked can correspond to a third party other than the person being watched over. Therefore, the control unit 11 may cancel the tracking of the foreground region.

According to the modification, it is possible to prevent a person other than the watching target person who works toward the inside of the bed in the vicinity of the bed from erroneously detecting the bed of the watching target person. it can. Therefore, according to the modification, the accuracy of detecting the bed of the person to be watched can be improved, and thereby the alarm regarding the person to be watched to get out of bed can be accurately executed.

In addition, the image analysis device 1 according to the above embodiment may detect actions other than getting out of bed and going to bed of the person to be watched, such as getting up on the bed. The image analysis device 1 can detect the behavior of the watching target person based on the positional relationship between the foreground area corresponding to the watching target person and the bed (for example, the bed surface SU). For example, the image analysis device 1 may detect that the person to be watched rises on the bed when the foreground region appears at a position higher than a predetermined height directly above the bed surface SU. As a result, various behaviors of the person to be watched in the bed can be detected, and the image analysis device 1 can enhance the watching of the person to be watched. Further, in this case, when the image analysis device 1 does not detect any behavior of the monitoring target person for several hours or more after starting the detection processing of the behavior of the monitoring target person, the monitoring target person has an abnormality. It may have occurred. Therefore, if the image analysis device 1 does not detect an action such as getting up of the person to be watched over for a predetermined time or longer, the image analysis device 1 may give a notification to notify the fact.

1 ... Image analyzer,
2 ... camera, 21 ... depth sensor,
3 ... photographed image, 4 ... background image,
5 ... program, 6 ... storage medium,
11 ... Control unit, 12 ... Storage unit, 13 ... Touch panel display,
14 ... speaker, 15 ... external interface, 16 ... communication interface,
17 ... Drive,
111 ... image acquisition unit, 112 ... foreground extraction unit, 113 ... bed leaving detection unit,
114 ... Bed detection unit, 115 ... Notification unit

Claims (4)

An image acquisition unit that acquires a photographed image including depth data indicating the depth of each pixel in the photographed image, which is a photographed image taken by a photographing device installed to monitor the behavior of the person to be watched in the bed. ,
A foreground extraction unit that extracts the foreground region of the acquired photographed image by calculating the difference between the acquired photographed image and the background image set as the background of the photographed image.
By referring to the depth of each pixel included in the extracted foreground region, it is determined whether or not the foreground region appearing on the bed has moved to a position separated from the bed by a predetermined distance or more in the real space. When it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more, the bed leaving detection unit for detecting that the watching target person has left the bed.
With respect to the photographed image acquired after detecting that the person to be watched has left the bed, the image appears outside the bed in the real space by referring to the depth of each pixel included in the extracted foreground area. When it is determined whether or not the foreground area has moved onto the bed and it is determined that the foreground area appearing outside the bed has moved onto the bed, the person to be watched has laid down on the bed. Bed detection unit that detects
If the elapsed time from detecting the bed leaving of the watching target person exceeds a certain time before detecting the bedtime of the watching target person, it is determined that the watching target person has left the bed for a certain period of time or more. A notification unit that gives notifications to notify
Bei to give a,
When the foreground area appearing outside the bed moves into a predetermined area provided on the side of the bed, the bed detection unit further captures the head of the person to be watched in the foreground area. The foreground that appears outside the bed when the head region and the shoulder region where the shoulder is reflected are estimated and the estimated center of gravity of the head region is located outside the bed from the center of gravity of the shoulder region. It is determined that the area has moved onto the bed, and it is detected that the watching target person has fallen asleep on the bed.
Image analyzer. According to the bed detection unit, when the foreground area appearing outside the bed moves to a position lower than a predetermined height directly above the bed surface, the foreground area appearing outside the bed moves onto the bed. Judging and detecting that the person to be watched has gone to bed.
The image analysis apparatus according to claim 1 . The computer
A step of acquiring a captured image including depth data indicating the depth of each pixel in the captured image, which is a captured image taken by a photographing device installed to monitor the behavior of the person to be watched in the bed.
A step of extracting the foreground region of the acquired photographed image by calculating the difference between the acquired photographed image and the background image set as the background of the photographed image.
By referring to the depth of each pixel included in the extracted foreground region, it is determined whether or not the foreground region appearing on the bed has moved to a position separated from the bed by a predetermined distance or more in the real space. When it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more, the step of detecting that the person to be watched has left the bed.
With respect to the photographed image acquired after detecting that the person to be watched has left the bed, the image appears outside the bed in the real space by referring to the depth of each pixel included in the extracted foreground area. When it is determined whether or not the foreground area has moved onto the bed and it is determined that the foreground area appearing outside the bed has moved onto the bed, the person to be watched has laid down on the bed. Steps to detect and
If the elapsed time from detecting the bed leaving of the watching target person exceeds a certain time before detecting the bedtime of the watching target person, it is determined that the watching target person has left the bed for a certain period of time or more. The steps to make a notification to let you know,
The execution,
In the step of detecting that the person has gone to bed, when the foreground area appearing outside the bed moves into a predetermined area provided on the side of the bed, the computer further in the foreground area. When the head area where the head of the person to be watched is reflected and the shoulder area where the shoulder is reflected are estimated, and the estimated center of gravity of the head area is located outside the bed from the center of gravity of the shoulder area. , It is determined that the foreground area appearing outside the bed has moved onto the bed, and it is detected that the person to be watched has gone to bed.
Image analysis method. On the computer
A step of acquiring a captured image including depth data indicating the depth of each pixel in the captured image, which is a captured image taken by a photographing device installed to monitor the behavior of the person to be watched in the bed.
A step of extracting the foreground region of the acquired photographed image by calculating the difference between the acquired photographed image and the background image set as the background of the photographed image.
By referring to the depth of each pixel included in the extracted foreground region, it is determined whether or not the foreground region appearing on the bed has moved to a position separated from the bed by a predetermined distance or more in the real space. When it is determined that the foreground area appearing on the bed has moved to a position separated from the bed by a predetermined distance or more, the step of detecting that the person to be watched has left the bed.
With respect to the photographed image acquired after detecting that the person to be watched has left the bed, the image appears outside the bed in the real space by referring to the depth of each pixel included in the extracted foreground area. When it is determined whether or not the foreground area has moved onto the bed and it is determined that the foreground area appearing outside the bed has moved onto the bed, the person to be watched has laid down on the bed. Steps to detect and
If the elapsed time from detecting the bed leaving of the watching target person exceeds a certain time before detecting the bedtime of the watching target person, it is determined that the watching target person has left the bed for a certain period of time or more. The steps to make a notification to let you know,
To execute ,
In the step of detecting that the person has gone to bed, when the foreground area appearing outside the bed moves into a predetermined area provided on the side of the bed, the computer further in the foreground area. When the head area where the head of the person to be watched is reflected and the shoulder area where the shoulder is reflected are estimated, and the estimated center of gravity of the head area is located outside the bed from the center of gravity of the shoulder area. , It is determined that the foreground area appearing outside the bed has moved onto the bed, and it is detected that the person to be watched has gone to bed.
Image analysis program for.