JP6119938B2 - Image processing system, image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present disclosure relates to an image processing technique, and more particularly, to an image processing system, an image processing apparatus, an image processing method, and an image processing program for determining a person's action.

  There is an image processing technique for discriminating a person's action from an image. This image processing technology is applied to, for example, an image processing apparatus that watches the behavior of a cared person who needs care for an elderly person or the like. The image processing device detects that the cared person has taken an action involving a fall or the like, and notifies the caregiver of this. Thereby, the caregiver can prevent the care receiver from falling over.

  With regard to such an image processing apparatus, Japanese Patent Application Laid-Open No. 2014-90913 (Patent Document 1) discloses a nurse that can detect a state in which the patient wakes up or gets out of bed based on an image captured by a camera that images the patient. A call system is disclosed. Japanese Patent Laying-Open No. 2014-174627 (Patent Document 2) discloses an information processing apparatus that estimates the behavior of a person being watched over by a simple method.

JP 2014-90913 A JP 2014-174627 A

  The nurse call system disclosed in Patent Document 1 notifies a nurse of a state in which the patient wakes up and a patient getting out of bed. Patients with poor back may not be able to wake up sufficiently. Therefore, the patient tries to get out of bed before getting up. Since the nurse call system disclosed in Patent Document 1 reports to the nurse when a state in which the patient has raised his / her upper body is detected, the timing of the report is late and the fall or fall from the bed cannot be sufficiently prevented. In addition, if the notification is made at the timing when the state of being separated from the bed is detected, the notification timing is too late, and the fall or fall from the bed cannot be sufficiently prevented.

  The information processing apparatus disclosed in Patent Document 2 detects the size of a patient protruding from the bed in an image using an optical flow, and when the size exceeds a predetermined size, the patient Notify that you have passed the fence. As shown in FIG. 20, when the camera is installed so as to photograph the patient 9 from an oblique direction, the patient 9 protrudes from the bed 20 on the image only by raising his / her upper body. Therefore, in the information processing apparatus disclosed in Patent Document 2, misinformation to the patient 9 increases, and the burden on the patient 9 increases. In addition, when the camera is installed directly above the patient 9 and when the patient 9 gets off the bed with his feet first, the optical flow indicating the movement of the feet vertically down on the floor is almost displayed on the image. Does not appear. For this reason, the information processing apparatus disclosed in Patent Document 2 cannot accurately detect the behavior of the patient 9 and cannot sufficiently prevent the occurrence of a fall or fall.

  Therefore, there is a demand for an image processing system that can discriminate the behavior of a person such as a patient or a cared person by a method different from the conventional one and can reduce the occurrence of the person's fall or fall.

  The present disclosure has been made in order to solve the above-described problems, and an object in one aspect is to provide an image processing system capable of discriminating a person's action by a method different from the conventional one. An object in another aspect is to provide an image processing apparatus capable of discriminating a person's behavior by a method different from the conventional one. Still another object of the present invention is to provide an image processing method capable of discriminating a person's action by a method different from the conventional method. Still another object of the present invention is to provide an image processing program capable of discriminating a person's behavior by a method different from the conventional method.

According to a certain aspect, an image processing system capable of discriminating a person's behavior is to set a boundary between an object region including an object and another region in a reference image obtained by photographing the object. A boundary receiving unit for receiving, a person detecting unit for detecting a person area in an input image obtained by photographing an object from the same viewpoint as the reference image, and a method for overlapping the boundary and the person area in the input image And a discriminating unit for discriminating the behavior of the person. The determination unit determines the action according to the width of the portion where the boundary and the person region overlap.
According to another aspect, an image processing system capable of discriminating a person's behavior is a region in a reference image obtained by photographing an object, and setting a boundary between the object region including the object and another region A boundary receiving unit for receiving an image, a person detecting unit for detecting a person area in an input image obtained by photographing an object from the same viewpoint as the reference image, and a method for overlapping the boundary and the person area in the input image And a discriminator for discriminating a person's action based on the above. The person detection unit detects a human foot region in the input image. The discriminating unit discriminates an action based on how the boundary and the foot region overlap in the input image.
According to another aspect, an image processing system capable of discriminating a person's behavior is a region in a reference image obtained by photographing an object, and setting a boundary between the object region including the object and another region A boundary receiving unit for receiving an image, a person detecting unit for detecting a person area in an input image obtained by photographing an object from the same viewpoint as the reference image, and a method for overlapping the boundary and the person area in the input image And a discriminator for discriminating a person's action based on the above. The object is bedding. The behavior discriminated by the discriminating unit includes at least one of getting up and entering the floor. The determination unit determines that the action is to wake up or enter the floor when the width of the portion where the boundary and the person region overlap exceeds a predetermined value.

  Preferably, the image processing system further includes a direction detection unit for detecting the movement direction of the person in the input image. The discriminating unit discriminates that the person is getting up from the bedding when the width exceeds the predetermined value and the movement direction is directed from the inside to the outside of the object area in the input image, and the width is set to the predetermined value. If the movement direction exceeds the object area in the input image, the person is determined to be in the bedding.

  Preferably, the person detection unit detects a specific part of the person by using at least one of the shape of the person area and the image information in the person area.

  Preferably, the image processing system further includes a boundary part receiving unit for receiving the setting of the boundary part used for the action determining process of the determining unit from the set boundaries.

According to another aspect, an image processing apparatus capable of discriminating a person's behavior is to set a boundary between an object region including the object and another region in the reference image obtained by photographing the object A boundary receiving unit for receiving an image, a person detecting unit for detecting a person area in an input image obtained by photographing an object from the same viewpoint as the reference image, and a method for overlapping the boundary and the person area in the input image And a discriminator for discriminating a person's action based on the above. The determination unit determines the action according to the width of the portion where the boundary and the person region overlap.
According to another aspect, an image processing apparatus capable of discriminating a person's behavior is to set a boundary between an object region including the object and another region in the reference image obtained by photographing the object A boundary receiving unit for receiving an image, a person detecting unit for detecting a person area in an input image obtained by photographing an object from the same viewpoint as the reference image, and a method for overlapping the boundary and the person area in the input image And a discriminator for discriminating a person's action based on the above. The person detection unit detects a human foot region in the input image, and the determination unit determines a behavior based on how the boundary and the foot region overlap in the input image.
According to another aspect, an image processing apparatus capable of discriminating a person's behavior is to set a boundary between an object region including the object and another region in the reference image obtained by photographing the object A boundary receiving unit for receiving an image, a person detecting unit for detecting a person area in an input image obtained by photographing an object from the same viewpoint as the reference image, and a method for overlapping the boundary and the person area in the input image And a discriminator for discriminating a person's action based on the above. The object is bedding. The behavior discriminated by the discriminating unit includes at least one of getting up and entering the floor. The determination unit determines that the action is to wake up or enter the floor when the width of the portion where the boundary and the person region overlap exceeds a predetermined value.

According to another aspect, an image processing method capable of discriminating a person's action is to set a boundary between an object region including an object and another region in a reference image obtained by photographing the object. The step of detecting the person area in the input image obtained by photographing the object from the same viewpoint as the reference image, and the action of the person based on how the boundary and the person area in the input image overlap. Determining. In the determining step, the action is determined according to the width of the portion where the boundary and the person region overlap.
According to another aspect, an image processing method capable of discriminating a person's action is to set a boundary between an object region including an object and another region in a reference image obtained by photographing the object. The step of detecting the person area in the input image obtained by photographing the object from the same viewpoint as the reference image, and the action of the person based on how the boundary and the person area in the input image overlap. Determining. The detecting step detects a human foot region in the input image. In the determining step, the action is determined based on how the boundary and the foot region overlap in the input image.
According to another aspect, an image processing method capable of discriminating a person's action is to set a boundary between an object region including an object and another region in a reference image obtained by photographing the object. The step of detecting the person area in the input image obtained by photographing the object from the same viewpoint as the reference image, and the action of the person based on how the boundary and the person area in the input image overlap. Determining. The object is bedding. The action determined in the determining step includes at least one of getting up and entering. In the determining step, when the width of the portion where the boundary and the person region overlap exceeds a predetermined value, it is determined that the action is to wake up or enter.

According to another aspect, an image processing program capable of discriminating a person's behavior is an area in a reference image obtained by photographing an object on a computer and includes the object area including the object and another area. Based on the step of accepting the setting of the boundary, the step of detecting the person area in the input image obtained by photographing the object from the same viewpoint as the reference image, and the way in which the boundary and the person area in the input image overlap And a step of discriminating the behavior of. In the determining step, the action is determined according to the width of the portion where the boundary and the person region overlap.

In a certain situation, it is possible to determine a person's behavior by a method different from the conventional method.
The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the present invention taken in conjunction with the accompanying drawings.

It is a figure which shows an example of a structure of the image processing system according to 1st Embodiment. It is a figure which shows the time-sequential image obtained by image | photographing a person from right above. It is a figure which shows the time-sequential image obtained by image | photographing a person from diagonally so that the longitudinal direction of a bed may be included. It is a figure which shows the image obtained by image | photographing a person from diagonally so that the transversal direction of a bed may be included. It is a figure which shows the screen transition diagram of the image processing system according to 1st Embodiment. It is a figure which shows an example of a main screen. It is a figure which shows an example of a setting mode top screen. It is a figure which shows an example of an area | region setting screen. It is a figure which shows an example of a normal time screen. It is a figure which shows an example of the screen at the time of report. It is a block diagram which shows an example of a function structure of the image processing system according to 1st Embodiment. It is a flowchart which shows the image processing which the image processing system according to 1st Embodiment performs. It is a flowchart which shows an action discrimination | determination process. It is a conceptual diagram which shows a person detection process roughly. It is a block diagram which shows the main hardware constitutions of the image processing system according to 1st Embodiment. It is a figure which shows an example of a selection screen. It is a block diagram which shows an example of a function structure of the image processing system according to 2nd Embodiment. It is a figure which shows the example which has the center of an input image before the bed area | region. It is a figure which shows the example in which the center of an input image is contained in the bed area | region. It is a figure which shows the image obtained by image | photographing a person from diagonally.

  Embodiments according to the present invention will be described below with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. Each embodiment and each modified example described below may be selectively combined as appropriate.

<First Embodiment>
[Configuration of Image Processing System 300]
The configuration of the image processing system 300 will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of the configuration of an image processing system 300.

  The image processing system 300 is used for watching the behavior of the care recipient 10, for example. As shown in FIG. 1, the image processing system 300 includes an indoor terminal 100 as an image processing apparatus and a management server 200. The indoor terminal 100 and the management server 200 are connected to each other via the network 400.

  The indoor terminal 100 is installed in, for example, a medical facility, a nursing facility, a home, or the like. The indoor terminal 100 includes a camera 105 (see FIG. 15) described later. FIG. 1 shows a state where the camera 105 is photographing the care receiver 10 and the bed 20 from the ceiling. The indoor terminal 100 determines the behavior of the care receiver 10 based on a time-series image (video) obtained from the camera 105. As an example, the behavior that can be identified by the indoor terminal 100 includes at least one of getting up and entering the care receiver 10. Here, “wake-up” refers to an action from when the cared person 10 wakes up on the bed 20 until it stands up. “Entering the floor” means an action from when the care recipient 10 approaches the bed 20 until entering the bed 20.

  When the indoor terminal 100 detects a behavior to be notified (for example, wake-up or the like), the indoor terminal 100 transmits information indicating the type of the behavior to the management server 200. When wake-up is detected as the action to be notified, the management server 200 notifies the caregiver that the care receiver 10 has woken up. Thereby, the caregiver can assist the care receiver 10 to get up from the bed 20, and can prevent the fall or fall that occurs when the care receiver 10 wakes up.

  1 illustrates an example in which the image processing system 300 includes one indoor terminal 100, the image processing system 300 may include a plurality of indoor terminals 100. 1 illustrates an example in which the image processing system 300 includes one management server 200, the image processing system 300 may include a plurality of management servers 200. Furthermore, although the indoor terminal 100 and the management server 200 are configured as separate devices in FIG. 1, the indoor terminal 100 and the management server 200 may be configured integrally.

[Processing Overview of Image Processing System 300]
With reference to FIGS. 2 to 4, an outline of processing of image processing system 300 according to the first embodiment will be described. FIG. 2 is a diagram showing time-series images obtained by photographing the care receiver 10 from directly above. FIG. 3 is a diagram showing time-series images obtained by photographing the care recipient 10 from an oblique direction so as to include the longitudinal direction of the bed 20. FIG. 4 is a diagram illustrating an image obtained by photographing the care receiver 10 from an oblique direction so as to include the short direction of the bed 20.

  The image processing system 300 extracts the person region 12 from the input image 32A. The image processing system 300 determines the behavior of the care recipient 10 based on how the person region 12 and the bed boundary 40 overlap. The bed boundary 40 is a part or all of the boundary between the bed area and the other area in the input image 32A. The bed boundary 40 is preset as will be described later.

  In one aspect, as shown in the input image 32B, the image processing system 300 determines whether the care receiver 10 has the caregiver 10 according to the width of the portion where the bed boundary 40 and the person region 12 overlap (that is, the overlap width W1). Determine the behavior. When the overlap width W1 exceeds a predetermined value (hereinafter also referred to as “threshold Thw”), the image processing system 300 determines that the action of the care recipient 10 is to wake up or enter the floor.

  As shown in the input image 32C, when there are a plurality of overlapping portions between the bed boundary 40 and the person region 12, the image processing system 300 determines the behavior of the care recipient 10 according to the sum of the overlapping widths W2 and W3. Is determined. In the image processing system 300, when the sum of the overlap widths W2 and W3 exceeds the threshold Thw, the cared person 10 gets up from the bed 20 or the cared person 10 enters the bed 20 Determine. Note that the image processing system 300 may compare the width between the left end of the overlap width W2 and the right end of the overlap width W3 with the threshold Thw.

  If the cared person 10 cannot raise the upper body sufficiently, he / she tries to get out of the bed before getting up. Since the image processing system 300 determines the behavior of the cared person 10 according to how the bed boundary 40 and the person area 12 overlap, the image processing system 300 determines the care receiver 10's getting up before the cared person 10 wakes up. it can. The image processing system 300 informs the caregiver of the caregiver 10 getting up, so that the caregiver can quickly rush to the caregiver 10. As a result, the caregiver can prevent the cared person 10 from falling or the cared person 10 from falling from the bed 20 in advance.

  Preferably, the image processing system 300 detects a specific part of the care recipient 10 using at least one of the shape of the person area 12 of the input images 32A to 32C and the image information in the person area 12. To do. The specific part includes the foot of the care recipient 10. The image processing system 300 determines the behavior of the care receiver 10 based on how the bed boundary 40 overlaps the foot regions in the input images 32A to 32C. Thereby, the image processing system 300 can detect the state where the foot of the cared person 10 protrudes from the bed 20, and the image processing system 300 can further determine the action of the cared person 10 about to go out of the bed 20. It can be detected accurately.

  More preferably, the image processing system 300 detects the movement direction of the care recipient 10 in the input images 33A to 33C. The image processing system 300 determines that the care receiver 10 is getting up from the bed 20 when the overlap width W1 exceeds the threshold Thw and the operation direction is from the inside to the outside of the bed area. In addition, the image processing system 300 determines that the care receiver 10 has entered the bed 20 when the overlap width W1 exceeds the threshold Thw and the operation direction is from the outside to the inside of the bed area. To do. The image processing system 300 can more accurately discriminate the behavior of the cared person 10 by using the movement direction of the cared person 10 for the action discriminating process.

  In addition, in FIG. 2, although the example which performs an action discrimination | determination process using the input images 32A-32C obtained by image | photographing the cared person 10 from right above was demonstrated, the cared person 10 is not necessarily right above. It does not have to be taken from. For example, as shown in the input images 33 </ b> A to 33 </ b> C of FIG. 3, the care recipient 10 may be taken from an oblique direction so as to include the longitudinal direction of the bed 20. In this case, the bed boundary 40 is set by the administrator as a boundary between the upper surface of the bed 20 and the side surface of the bed 20.

  Further, as shown in the input image 34 of FIG. 4, the care receiver 10 may be photographed from an oblique direction so as to include the short direction of the bed 20. In this case, the image processing system 300 corrects the threshold value Thw used for action determination according to at least one information of the camera angle, the bed 20 angle, and the bed 20 position. The image processing system 300 compares the corrected threshold value Thw with the overlap width W4. Alternatively, the image processing system 300 may correct the overlap width W4 and compare the corrected overlap width W4 with the threshold Thw.

  In the above description, the example in which the overlapping widths W1 to W4 are detected based on the overlapping portion between the person region 12 and the bed boundary 40 has been described. However, the overlapping widths W1 to W4 are detected by other methods. Also good. For example, the image processing system 300 stores a foot-shaped template in advance and searches for a portion similar to the template in the vicinity of the bed boundary 40. The image processing system 300 determines the overlapping widths W1 to W4 according to the sizes of similar templates.

  In addition, the image processing system 300 may search for a skin color area having a foot width in the visible light image, and detect overlap widths W1 to W4 between the skin color area and the bed boundary 40. In addition, the image processing system 300 may search for a high temperature region from the thermographic image and detect overlap widths W1 to W4 between the high temperature region and the bed boundary 40. In addition, as the feature amount used for action determination, another feature amount such as the area of the region that protrudes from the bed boundary 40 or the amount of protrusion (for example, the distance from the bed edge) is used instead of the overlap widths W1 to W4. May be.

[Screen Transition of Image Processing System 300]
A screen example displayed on the image processing system 300 will be described with reference to FIGS. FIG. 5 is a diagram showing a screen transition diagram of the image processing system 300.

  The image processing system 300 displays the main screen 310 as an initial screen when the image processing program according to the present embodiment is executed. The administrator can make a transition from the main screen 310 to the setting mode top screen 320 or the normal screen 340. The administrator can make a transition from the setting mode top screen 320 to the main screen 310 or the area setting screen 330. The administrator can make a transition from the area setting screen 330 to the setting mode top screen 320. The administrator can make a transition from the normal screen 340 to the main screen 310 or the notification time screen 350. The administrator can make a transition from the notification time screen 350 to the normal screen 340.

  Hereinafter, screen examples of the main screen 310, the setting mode top screen 320, the region setting screen 330, the normal time screen 340, and the notification time screen 350 will be described in order.

(Main screen 310)
FIG. 6 is a diagram illustrating an example of the main screen 310. The image processing system 300 displays the main screen 310 as an initial screen when the image processing program according to the present embodiment is executed.

  The main screen 310 includes a button 312 for accepting the start of the action determination process and a button 314 for opening a setting screen related to the action determination process. When detecting that the button 312 is pressed, the image processing system 300 displays a normal screen 340. In addition, when the image processing system 300 detects that the button 314 is pressed, the image processing system 300 displays a setting mode top screen 320.

(Setting mode top screen 320)
FIG. 7 is a diagram illustrating an example of the setting mode top screen 320. The setting mode top screen 320 is displayed at the time of initial setting or maintenance of the image processing system 300.

  The setting mode top screen 320 accepts parameter settings related to behavior determination processing. For example, the setting mode top screen 320 receives parameters related to the frame rate of the camera 105 described later. In addition, the setting mode top screen 320 receives parameters relating to the brightness of the image output from the camera 105. Furthermore, the setting mode top screen 320 accepts parameters relating to the detection sensitivity of the care receiver's behavior. Furthermore, the setting mode top screen 320 receives parameters regarding the height of the ceiling on which the camera 105 is installed. When the “update” button on the setting mode top screen 320 is pressed, each parameter is reflected in the image processing system 300.

  When the image processing system 300 detects that the button 322 is pressed, the image processing system 300 displays an area setting screen 330. The image processing system 300 displays the main screen 310 when it detects that the button 324 is pressed.

  The setting mode top screen 320 may accept input of other parameters. For example, the setting mode top screen 320 may accept a parameter relating to the contrast of the input image, a parameter relating to the zoom adjustment of the camera, a parameter relating to the pan / tilt adjustment of the camera, and the like as the parameters relating to the camera 105. In addition, the setting mode top screen 320 may accept a compression rate of an image transmitted from the indoor terminal 100 to the image processing system 300. In addition, the setting mode top screen 320 may accept a setting such as a time zone for determining an action such as getting up or entering.

(Area setting screen 330)
FIG. 8 is a diagram illustrating an example of the area setting screen 330. The area setting screen 330 receives the setting of the bed boundary 40 with respect to the setting image 30 (reference image). The set bed boundary 40 is used in the action determination process as described above.

  The area setting screen 330 receives the setting of the bed boundary 40 by receiving the settings of the points 41A to 41D, for example. As an example, the points 41A to 41D are input by a pointer 332 that is interlocked with a mouse operation. The image processing system 300 stores information (for example, coordinates) for specifying the bed boundary 40 in the setting image 30 based on accepting an operation for saving the bed boundary 40 set by the administrator. .

  In FIG. 8, the example of setting the points 41 </ b> A to 41 </ b> D has been described as the method for setting the bed boundary 40, but the bed boundary 40 may be set by other methods. For example, the area setting screen 330 may accept the setting of the bed boundary 40 by accepting the setting of the line. As another method, the area setting screen 330 receives the setting of the bed boundary 40 by receiving the setting of the surface. In this case, the administrator designates a range in which the bed 20 is shown by a drag operation on the area setting screen 330. As described above, as a method for setting the bed boundary 40, any method capable of designating a part or all of the boundary between the bed region and the other region may be employed.

  In FIG. 8, an example in which a rectangular boundary is set as a method for setting the bed boundary 40 has been described. However, the bed boundary 40 may be set in another shape. For example, the bed boundary 40 may be set in other shapes such as a circle, an ellipse, and a polygon (for example, a hexagon). Alternatively, the shape of the bed boundary 40 may be a line, an arc, or the like. The line or arc may have a predetermined thickness.

  In addition, although an example in which the bed boundary 40 is set with the pointer 332 has been described in FIG. 8, the bed boundary 40 may be set by other operations such as a touch operation.

  Moreover, although the example which sets the bed boundary 40 with respect to the bed 20 was demonstrated in FIG. 8, the target object in which a boundary is set is not limited to a bed. For example, the object for which the boundary is set includes bedding such as a futon, a chair, and other objects used by the care recipient.

  In FIG. 8, an example in which the bed boundary 40 is manually set by the administrator has been described. However, the image processing system 300 automatically detects the bed boundary 40 by image processing such as edge extraction or template matching. May be. Alternatively, the image processing system 300 may detect the bed boundary 40 by other sensors such as a 3D sensor, a position sensor attached to the leg of the bed 20, and a carpet having a pressure sensor.

(Normal screen 340)
FIG. 9 is a diagram illustrating an example of the normal screen 340. The normal screen 340 is a screen that is displayed when the monitored care recipient 10 is taking a non-hazardous action (for example, going to sleep) during the execution of the action determination process by the image processing system 300. . As an example, the image processing system 300 displays an image (video) obtained by photographing the care recipient 10 as it is as a normal screen 340.

(Notification screen 350)
FIG. 10 is a diagram showing an example of the notification time screen 350. The notification time screen 350 is a screen that is displayed when the care receiver 10 to be monitored takes a risky action during execution of the action determination process by the image processing system 300. The image processing system 300 may inquire of the administrator whether or not the notification time screen 350 can be displayed before displaying the notification time screen 350.

  As shown in FIG. 10, the image processing system 300 notifies the caregiver of the care receiver 10 to wake up based on the detection that the care receiver 10 has woken up. In one aspect, the image processing system 300 notifies the caregiver of the care receiver 10 to wake up with a message 352. In another aspect, the image processing system 300 notifies the caregiver of the care receiver 10 to wake up by sound such as voice. In yet another aspect, the image processing system 300 displays an image or video when the care receiver 10 is detected to wake up. As a result, even if the image processing system 300 makes a false alarm, the caregiver can confirm the action of the care receiver 10 at the time of action detection with an image or video. Therefore, it is not necessary to rush to the care recipient 10.

  The action to be notified is not limited to wake up. For example, the action to be notified includes entering the floor, leaving the floor, and other actions that cause danger to the care recipient 10.

[Functional Configuration of Image Processing System 300]
The functions of the image processing system 300 will be described with reference to FIG. FIG. 11 is a block diagram illustrating an example of a functional configuration of the image processing system 300. As shown in FIG. 11, the image processing system 300 includes an indoor terminal 100 and a management server 200. Below, the function of the indoor terminal 100 and the management server 200 is demonstrated in order.

(Functional configuration of indoor terminal 100)
As illustrated in FIG. 11, the indoor terminal 100 includes a reception unit 110, a person detection unit 120, a direction detection unit 130, a determination unit 140, and a transmission unit 160 as functional configurations.

  The receiving unit 110 receives the bed boundary 40 set for the setting image 30 (see FIG. 8) from the management server 200. The receiving unit 110 stores the received bed boundary 40 in the storage device 106.

  The person detection unit 120 detects a person region in an input image obtained by photographing an object (for example, a bed) from the same viewpoint as the setting image 30. Details of the person detection process will be described later (see FIG. 14).

  The direction detection unit 130 detects the movement direction of the care recipient with respect to an object (for example, a bed) in the input image. As an example, the direction detection unit 130 detects the movement direction of the cared person in the input image based on the change in the area of the person region in the bed boundary 40. For example, when the area decreases by a predetermined value or more compared to the previous area, the direction detection unit 130 determines that the care receiver is moving from the inside of the bed area to the outside. When the area increases by a predetermined value or more compared to the previous area, the direction detection unit 130 determines that the care recipient is moving from the outside of the bed area to the inside. Note that the direction detection unit 130 may detect the operation direction based on a change in the position of the person area detected by the person detection unit 120.

  The determination unit 140 determines the behavior of the care recipient based on how the person region in the input image overlaps the bed boundary 40 in the input image. Details of the action determination process will be described later (see FIG. 13).

  The transmission unit 160 transmits the setting image 30 obtained from the camera 105 (see FIG. 15) and the type of action determined by the determination unit 140 to the management server 200.

(Functional configuration of management server 200)
The functional configuration of the management server 200 will be described with reference to FIG. As illustrated in FIG. 11, the management server 200 includes a reception unit 210, a notification unit 230, a boundary reception unit 240, and a transmission unit 250 as functional configurations.

  The receiving unit 210 receives the setting image 30 obtained from the camera 105 (see FIG. 15) and the type of action determined by the determining unit 140 from the indoor terminal 100.

  When the behavior to be notified is detected, the notification unit 230 notifies that the behavior has been detected. The behavior to be notified includes, for example, getting up, getting in, getting out of bed, and other behaviors that are dangerous for the care recipient to be monitored. As an example of the notification unit, the notification unit 230 displays information indicating the type of action as a message or outputs it by voice. Or the alerting | reporting part 230 displays the information which shows the kind of action as a message on the portable terminal (not shown) which a caregiver has, outputs a sound to the said portable terminal, or vibrates the said portable terminal.

  The boundary receiving unit 240 receives the setting of the bed boundary 40 between the area including the bed and the other area in the setting image 30 obtained by photographing the bed. The bed boundary 40 corresponds to a part or the whole between the bed area and another area. Preferably, the bed boundary 40 corresponds to part or all of the edge portion of the bed. More preferably, the bed boundary 40 corresponds to part or all of the boundary of the upper surface of the bed.

  The transmission unit 250 transmits the setting of the bed boundary 40 for the setting image 30 received by the boundary reception unit 240 to the indoor terminal 100.

[Control Structure of Image Processing System 300]
A control structure of the image processing system 300 will be described with reference to FIGS. FIG. 12 is a flowchart showing image processing executed by the image processing system 300. The processing of FIG. 12 is executed by the CPU 102 (see FIG. 15) of the indoor terminal 100 and the CPU 202 (see FIG. 15) of the management server 200. In other aspects, some or all of the processing may be performed by circuit elements or other hardware.

  In step S50, image processing system 300 inputs an image obtained by photographing the care recipient to be monitored into the image processing program according to the present embodiment.

  In step S60, the image processing system 300 performs a behavior determination process as the above-described determination unit 140 (see FIG. 11). Details of the action determination process will be described later (see FIG. 13).

  In step S70, image processing system 300 determines whether or not to end image processing according to the present embodiment. For example, image processing system 300 determines that the image processing according to the present embodiment is to be terminated when an operation for interrupting the processing is received from the administrator (YES in step S70). Otherwise (NO in step S70), image processing system 300 switches control to step S80.

  In step S <b> 80, the image processing system 300 acquires the next input image from the camera 105. As a result, the image processing system 300 sequentially executes image processing according to the present embodiment on the time-series images.

(Behavior discrimination processing)
With reference to FIGS. 13 and 14, the action determination process executed in step S60 of FIG. 12 will be described in detail. FIG. 13 is a flowchart showing the action determination process. FIG. 14 is a conceptual diagram schematically showing the person detection process executed in step S100 of FIG.

  In step S100, the image processing system 300 detects a person area from the input image as the above-described person detection unit 120 (see FIG. 11). The person area is detected by, for example, a background difference that makes a difference between background images from an input image or a time difference that makes an difference between images whose shooting times are before and after.

  FIG. 14 schematically shows a process of extracting the person region 12 from the input image 32 based on the background difference. More specifically, the image processing system 300 acquires a background image 35 in which no person is shown in advance. The background image 35 may be the same image as the setting image 30 described above (see FIG. 8), or may be an image obtained separately from the setting image 30.

  When the image processing system 300 acquires the input image 32 from the camera 105 (see FIG. 15), the image processing system 300 subtracts the background image 35 from the input image 32. Thereby, the image processing system 300 can obtain the background difference image 36 excluding the background from the input image 32. The image processing system 300 sets a region having a pixel value equal to or greater than a predetermined value as the person region 12.

  The person area 12 may be extracted by a method different from the method shown in FIG. For example, the image processing system 300 prepares a characteristic portion (that is, a feature amount) of the care recipient 10 as a template in advance, and searches the region similar to the template by scanning the input image 32. If the image processing system 300 can search for an area similar to the template in the input image 32, the image processing system 300 sets the area as the person area 12. In addition, the person region 12 may be extracted by other image processing techniques such as optical flow and tracking.

  Preferably, the image processing system 300 extracts a foot area from the person area 12. The method for extracting the foot region is arbitrary. As an example, the image processing system 300 determines whether pixel value information such as skin color degree and luminance, shape information such as the width of the person area 12, and motion information such as optical flow are within a predetermined range. Then, a region that satisfies the condition of being within a predetermined range is extracted as a foot region. Alternatively, the image processing system 300 holds a foot-shaped template in advance and searches for a portion similar to the template in the vicinity of the bed boundary 40. When the image processing system 300 can search for a region similar to the template, the image processing system 300 sets the region as a foot region.

  In step S102, the image processing system 300 detects the direction of movement of the care recipient relative to an object (for example, a bed) in the input image as the above-described direction detection unit 130 (see FIG. 11). As an example, the image processing system 300 detects the direction of movement of the cared person in the image based on a change in the area S1 of the person area in the bed area. For example, when the current area S1 has decreased by a predetermined value or more compared to the previous area S1, the image processing system 300 determines that the cared person is moving from the inside to the outside of the bed. When the current area S1 increases by a predetermined value or more compared to the previous area S1, the direction detection unit 130 determines that the care recipient is moving from the outside of the bed to the inside. Note that the image processing system 300 may detect the direction of movement of the care recipient in the image based on the change in the area S2 of the person area outside the bed area.

  In step S110, the image processing system 300 determines whether the moving direction of the care recipient in the input image is from the inside of the bed to the outside. If the image processing system 300 determines that the moving direction of the person in the input image is from the inside of the bed to the outside (YES in step S110), the control is switched to step S120. If not (NO in step S110), the image processing system 300 ends the action determination process.

  In step S120, the image processing system 300 determines whether or not the overlap width W between the bed boundary 40 and the person region 12 is larger than the threshold value Thw. When image processing system 300 determines that overlap width W is larger than threshold value Thw (YES in step S120), control is switched to step S122. If not (NO in step S120), the image processing system 300 ends the action determination process.

  In step S122, the image processing system 300 notifies the care receiver that he / she has woken up as the above-described notification unit 230 (see FIG. 11). As an example of notification means, the image processing system 300 displays information indicating the type of action as a message or outputs it by voice. Alternatively, the image processing system 300 displays information indicating the type of action as a message on a portable terminal (not shown) possessed by the caregiver, outputs sound to the portable terminal, or vibrates the portable terminal. .

[Hardware Configuration of Image Processing System 300]
An example of the hardware configuration of the image processing system 300 will be described with reference to FIG. FIG. 15 is a block diagram illustrating a main hardware configuration of the image processing system 300. As shown in FIG. 15, the image processing system 300 includes an indoor terminal 100, a management server 200, and a network 400. The indoor terminal 100 and the management server 200 are connected via a network 400. Hereinafter, the hardware configuration of the indoor terminal 100 and the hardware configuration of the management server 200 will be described in order.

(Hardware configuration of indoor terminal 100)
As shown in FIG. 15, the indoor terminal 100 includes a ROM (Read Only Memory) 101, a CPU 102, a RAM (Random Access Memory) 103, a network I / F (interface) 104, a camera 105, and a storage device. 106.

  The ROM 101 stores an operating system (OS), a control program executed by the indoor terminal 100, and the like. The CPU 102 controls the operation of the indoor terminal 100 by executing various programs such as an operating system and a control program for the indoor terminal 100. The RAM 103 functions as a working memory and temporarily stores various data necessary for program execution.

  A communication device such as an antenna or a NIC (Network Interface Card) is connected to the network I / F 104. The indoor terminal 100 transmits / receives data to / from other communication terminals via the communication device. Other communication terminals include, for example, the management server 200 and other terminals. Indoor terminal 100 may be configured to download image processing program 108 for realizing various processes according to the present embodiment via network 400.

  The camera 105 is, for example, a surveillance camera or other imaging device capable of photographing a subject. The camera 105 may be configured separately from the indoor terminal 100, or may be configured integrally with the indoor terminal 100 as shown in FIG.

  The storage device 106 is a storage medium such as a hard disk or an external storage device. As an example, the storage device 106 stores a bed boundary 40 set for a set image and an image processing program 108 for realizing various processes according to the present embodiment. The bed boundary 40 is information for specifying a region where a bed is reflected in a setting image or an input image. For example, the bed boundary 40 is indicated by a coordinate value indicating the end point of the bed.

  Note that the image processing program 108 may be provided by being incorporated in a part of an arbitrary program, not as a single program. In this case, processing according to the present embodiment is realized in cooperation with an arbitrary program. Even such a program that does not include some modules does not depart from the spirit of the image processing system 300 according to the present embodiment. Furthermore, part or all of the functions provided by the image processing program 108 according to the present embodiment may be realized by dedicated hardware. Furthermore, the management server 200 may be configured in the form of a so-called cloud service in which at least one server realizes processing according to the present embodiment.

(Hardware configuration of management server 200)
Next, the hardware configuration of the management server 200 will be described. As illustrated in FIG. 15, the management server 200 includes a ROM 201, a CPU 202, a RAM 203, a network I / F 204, a monitor 205, and a storage device 206.

  The ROM 201 stores an operating system, a control program executed by the management server 200, and the like. The CPU 202 controls the operation of the management server 200 by executing various programs such as an operating system and a control program of the management server 200. The RAM 203 functions as a working memory and temporarily stores various data necessary for program execution.

  A communication device such as an antenna or a NIC is connected to the network I / F 204. The management server 200 transmits / receives data to / from other communication terminals via the communication device. Other communication terminals include, for example, the indoor terminal 100 and other terminals. Management server 200 may be configured to download a program for realizing various processes according to the present embodiment via network 400.

  The monitor 205 displays various screens displayed by executing the image processing program 208 according to the present embodiment. For example, the monitor 205 includes a main screen 310 (see FIG. 6), a setting mode top screen 320 (see FIG. 7), an area setting screen 330 (see FIG. 8), a normal screen 340 (see FIG. 9), and a notification time. A screen such as a screen 350 (see FIG. 10) is displayed. The monitor 205 may be realized as a touch panel in combination with a touch sensor (not shown). The touch panel accepts a setting operation of the bed boundary 40, a screen switching operation, and the like by a touch operation.

  The storage device 206 is a storage medium such as a hard disk or an external storage device. As an example, the storage device 206 stores an image processing program 208 for realizing various processes according to the present embodiment.

[Brief Summary]
As described above, image processing system 300 according to the present embodiment determines the behavior of the care recipient based on how the bed boundary and the person area overlap. Thereby, the image processing system 300 can detect passing through the bed boundary, and can detect the rising or entering of the care recipient. The image processing system 300 informs the caregiver of the caregiver's wake-up and the like so that the caregiver can quickly rush to the caregiver. As a result, the caregiver can prevent the care receiver from falling or the care receiver falling from the bed.

<Second Embodiment>
[Overview]
The image processing system 300 according to the first embodiment discriminates the behavior of the cared person using all preset bed boundaries. On the other hand, the image processing system 300A according to the second embodiment determines the care receiver's behavior by partially using a preset bed boundary.

  Below, with reference to FIG. 16, the outline | summary of 300 A of image processing systems according to 2nd Embodiment is demonstrated. FIG. 16 is a diagram illustrating an example of the selection screen 360. The selection screen 360 accepts selection of a boundary (hereinafter also referred to as “boundary portion”) to be used for determination processing from among the set bed boundaries 40.

  In the example of FIG. 16, the boundary portion 43 is selected by a pointer 332 that is linked to a mouse operation. The boundary portion 43 corresponds to one side of the bed boundary 40. The image processing system 300A determines the behavior of the care recipient in accordance with how the selected boundary portion 43 and the person area overlap. That is, the image processing system 300A does not use the boundary portion that has not been selected for the action determination process. As a result, the image processing system 300A can exclude a boundary that does not pass when the cared person wakes up, so that erroneous detection of the cared person's wakeup can be reduced.

  In addition, although FIG. 16 demonstrated the example in which the boundary part used for an action discrimination | determination process was selected, the boundary part which is not used for an action discrimination | determination process may be selected.

  The boundary portion may be selected by other methods. For example, the image processing system 300A may automatically select the long side of the bed area as the boundary portion. Alternatively, the image processing system 300A may automatically select the boundary portion based on information such as the distance from the image center and the presence / absence of a bed fence. Alternatively, paying attention to the fact that the cared person passes through a specific boundary portion when waking up or entering the floor, the image processing system 300A automatically uses the edge of the bed that frequently overlaps the person area as the boundary portion. You may choose.

  Furthermore, the selection screen 360 may accept selection of a plurality of boundary portions. When a plurality of boundary portions are selected, the image processing system 300A may change the action determination processing according to the position of each boundary portion, as will be described later.

[Functional Configuration of Image Processing System 300A]
With reference to FIG. 17, functions of image processing system 300A according to the second embodiment will be described. FIG. 17 is a block diagram illustrating an example of a functional configuration of the image processing system 300A. As illustrated in FIG. 17, the image processing system 300 </ b> A includes an indoor terminal 100 and a management server 200. The image processing system 300A is different from the image processing system 300 according to the first embodiment in that the management server 200 further includes a boundary portion receiving unit 245. Since other functional configurations are as described above, description thereof will not be repeated.

  The boundary part reception unit 245 receives the setting of the boundary part used for the action determination process from the set bed boundary 40. As an example, the boundary portion receiving unit 245 receives an operation of selecting a specific side from the end sides of the bed boundary 40, and sets the selected side as the boundary portion. The boundary part reception unit 245 outputs the selected boundary part to the transmission unit 250.

[Behavior determination processing of image processing system 300A]
With reference to FIG. 18 and FIG. 19, the action determination process of image processing system 300A according to the second embodiment will be described. FIG. 18 is a diagram illustrating an example in which the image center 45 of the input image 37 is in front of (below) the bed boundary 40. FIG. 19 is a diagram illustrating an example in which the image center 45 of the input image 38 is included in the bed boundary 40.

  The way the cared person appears in the input image and the way the bed appears vary depending on the position in the input image. For this reason, the way in which the person area overlaps the bed boundary also changes depending on the position of the care recipient and the position of the bed. Focusing on this point, the image processing system 300A changes the action determination process for the same action according to the position of the bed boundary 40 in the input image. In other words, the image processing system 300A changes the action determination process according to the positional relationship between the bed boundary 40 and the camera 105 (see FIG. 15).

  As an example, the image processing system 300A specifies the position of the image center 45 with respect to the bed boundary 40 (or the boundary portion 43), and changes the action determination process according to the specified position. For example, as shown in FIG. 18, it is assumed that the image center 45 is positioned in front of (below) the bed boundary 40. In this case, the image processing system 300 </ b> A selects “front” in the table 50 and executes an action determination process associated with the “front” in advance. Further, it is assumed that the image center 45 is located within the bed boundary 40 as shown in FIG. In this case, the image processing system 300 </ b> A selects “directly above” in the table 50 and executes a behavior determination process associated with the “directly above” in advance.

  Note that the method for specifying the positional relationship between the bed boundary 40 (or the boundary portion 43) and the camera 105 (see FIG. 15) is not limited to the examples shown in FIGS. For example, it is assumed that the camera 105 is installed so as to have a tilted downward angle. In this case, the image processing system 300A determines a position (not the center of the image) immediately below the camera 105 in the input image based on a preset tilt angle or a tilt angle automatically calculated from the input image. calculate. The image processing system 300A changes the action determination process according to the positional relationship between the calculated position and the bed boundary 40.

[Brief Summary]
As described above, the image processing system 300 </ b> A according to the second embodiment accepts selection of a boundary portion used for the action determination process from the bed boundary 40. The image processing system 300 </ b> A determines the behavior of the care recipient in accordance with how the selected boundary portion and the person area overlap. As a result, the image processing system 300A can exclude a boundary that does not pass when the cared person wakes up, so that erroneous detection of the cared person's wakeup can be reduced.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  9 patients, 10 care recipients, 12 person areas, 20 beds, 30 setting images, 32, 32A to 32C, 33A to 33C, 34, 37, 38 input images, 35 background images, 36 background difference images, 40 bed boundaries, 41A to 41D point, 43 boundary portion, 45 image center, 100 indoor terminal, 101,201 ROM, 102,202 CPU, 103,203 RAM, 104,204 network I / F, 105 camera, 106,206 storage device, 108 , 208 Image processing program, 110, 210 receiving unit, 120 person detecting unit, 130 direction detecting unit, 140 discriminating unit, 160, 250 transmitting unit, 200 management server, 205 monitor, 230 notifying unit, 240 boundary receiving unit, 245 boundary Partial acceptance unit, 300, 300A image processing system, 10 main screen, 312,314,322,324 button, 320 Setting mode top screen, 330 area setting screen, 332 pointers, 340 normal time display, 350 alarm time screen, 352 message, 360 selection screen 400 network.

Claims (13)

An image processing system capable of discriminating human actions,
A boundary receiving unit for receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object;
A person detection unit for detecting a person region in an input image obtained by photographing the object from the same viewpoint as the reference image;
A determination unit for determining the action of the person based on how the boundary and the person area overlap in the input image;
The image processing system , wherein the determination unit determines the action according to a width of a portion where the boundary and the person region overlap . An image processing system capable of discriminating human actions,
A boundary receiving unit for receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object;
A person detection unit for detecting a person region in an input image obtained by photographing the object from the same viewpoint as the reference image;
A determination unit for determining the action of the person based on how the boundary and the person area overlap in the input image;
The person detecting unit detects a foot region of the person in the input image;
The said determination part is an image processing system which discriminate | determines the said action based on how the said boundary and the said foot area | region in the said input image overlap . An image processing system capable of discriminating human actions,
A boundary receiving unit for receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object;
A person detection unit for detecting a person region in an input image obtained by photographing the object from the same viewpoint as the reference image;
A determination unit for determining the action of the person based on how the boundary and the person area overlap in the input image;
The object is bedding;
The action discriminated by the discriminating unit includes at least one of getting up and entering the floor,
The said determination part is an image processing system which discriminate | determines that the said action is the said getting-up or the said entrance when the width | variety of the part with which the said boundary and the said person area | region has exceeded predetermined value . The image processing system further includes a direction detection unit for detecting a movement direction of the person in the input image,
The discrimination unit
When the width exceeds the predetermined value and the movement direction is directed from the inside to the outside of the object area in the input image, it is determined that the person is getting up from the bedding,
When the width exceeds the predetermined value and the movement direction is directed from the outside to the inside of the object area in the input image, it is determined that the person is entering the bedding. Item 4. The image processing system according to Item 3. The person detecting unit, and the shape of the human region, using at least one of information of the image information of the person within the area, detecting a particular portion of the person, any one of claims 1-4 The image processing system described in 1. The image processing system further comprises a boundary portion receiving unit for accepting the setting of boundary used for the action determination processing of the determination unit from among the boundaries set, in any one of claims 1 to 5 The image processing system described. An image processing apparatus capable of discriminating human behavior,
A boundary receiving unit for receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object;
A person detection unit for detecting a person region in an input image obtained by photographing the object from the same viewpoint as the reference image;
A determination unit for determining the action of the person based on how the boundary and the person area overlap in the input image;
The determination unit is an image processing apparatus that determines the behavior according to a width of a portion where the boundary and the person region overlap . An image processing apparatus capable of discriminating human behavior,
A boundary receiving unit for receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object;
A person detection unit for detecting a person region in an input image obtained by photographing the object from the same viewpoint as the reference image;
A determination unit for determining the action of the person based on how the boundary and the person area overlap in the input image;
The person detecting unit detects a foot region of the person in the input image;
The said discrimination | determination part is an image processing apparatus which discriminate | determines the said action based on how the said boundary and the said foot area | region in the said input image overlap . An image processing apparatus capable of discriminating human behavior,
A boundary receiving unit for receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object;
A person detection unit for detecting a person region in an input image obtained by photographing the object from the same viewpoint as the reference image;
A determination unit for determining the action of the person based on how the boundary and the person area overlap in the input image;
The object is bedding;
The action discriminated by the discriminating unit includes at least one of getting up and entering the floor,
The determination unit is an image processing device that determines that the action is the wake-up or the entrance when the width of a portion where the boundary and the person region overlap exceeds a predetermined value . An image processing method capable of determining a person's behavior,
Receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object; and
Detecting a human region in an input image obtained by photographing the object from the same viewpoint as the reference image;
Determining the action of the person based on how the boundary and the person area overlap in the input image,
The determining step is an image processing method in which the action is determined according to a width of a portion where the boundary and the person region overlap . An image processing method capable of determining a person's behavior,
Receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object; and
Detecting a human region in an input image obtained by photographing the object from the same viewpoint as the reference image;
Determining the action of the person based on how the boundary and the person area overlap in the input image,
The detecting step detects a foot region of the person in the input image,
The determining step is an image processing method in which the action is determined based on how the boundary and the foot region overlap in the input image . An image processing method capable of determining a person's behavior,
Receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object; and
Detecting a human region in an input image obtained by photographing the object from the same viewpoint as the reference image;
Determining the action of the person based on how the boundary and the person area overlap in the input image,
The object is bedding;
The action determined in the determining step includes at least one of getting up and entering the floor,
The determining step is an image processing method in which, when the width of a portion where the boundary and the person region overlap exceeds a predetermined value, the action is determined to be the wake-up or the entrance . An image processing program capable of discriminating human actions,
The image processing program is stored in a computer.
Receiving a setting of a boundary between an object region including the object and a region in a reference image obtained by photographing the object; and
Detecting a human region in an input image obtained by photographing the object from the same viewpoint as the reference image;
Determining the action of the person based on how the boundary and the person area overlap in the input image,
The determining step is an image processing program for determining the action according to a width of a portion where the boundary and the person region overlap .