JP2015203881A - Observation device, display method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce load applied to an assistant, in an observation system.SOLUTION: An observation device comprises: a calculation part for calculating a feature amount indicating motion of an object person, based on an image obtained by imaging the object person; a registration part for associating the feature amount calculated by the calculation part with a parameter according to similarity with a feature amount calculated based on other image of the object person which has already been registered, and registering it in a database for every object person; and a display control part for determining whether or not the image is displayed based on the parameter, and when it is determined that the image is displayed, displaying the image.

Description

  The present invention relates to a monitoring device, a display method, and a program.

  2. Description of the Related Art Conventionally, in a hospital or a nursing facility, a monitoring system is known that monitors the operation of a patient lying on a bedding, a care recipient, etc. (referred to as a “watched person”) using an imaging device. By using the watching system, a nurse, a caregiver, or the like (referred to as “assistant”) can grasp from a remote position that assistance is required by the operation of the watching target person. However, in the case of such a monitoring system, the assistant needs to constantly monitor the captured image in front of the monitor, and the burden on the assistant is great.

  On the other hand, an image processing technique for detecting and displaying a specific captured image from a plurality of captured images is known. If such an image processing technology is applied to the above-mentioned monitoring system, only the photographed image when a specific state (a state where assistance by the assistant is required) is brought about by the operation of the person being watched over is notified to the assistant. It will be possible to do that.

JP 2009-140175 A

  However, the daily activities of the person being watched over vary, and even the same person being watched over changes depending on the progress of rehabilitation, the state of medication, and the like. For this reason, the state where assistance by an assistant is required changes every day for each person to be watched, and only a specific photographed image obtained by photographing the state requiring assistance by the assistant is accurately detected from a plurality of photographed images. Is not easy. This is because, in order to accurately detect a specific captured image while the state changes every day, it is necessary to accumulate captured images corresponding to the daily activities of each watching target person.

  On the other hand, if erroneous detection frequently occurs in the watching system, the assistant needs to frequently check the notified captured image, and the burden on the assistant is not reduced.

  One aspect aims to reduce the burden on the assistant in the watch system.

  According to an aspect, a calculation unit that calculates a feature amount indicating an operation of the target person from a captured image obtained by shooting the target person, and the feature amount calculated by the calculation unit is the target person. A registration unit for registering in the database for each target person in association with a parameter corresponding to the degree of similarity with a feature amount calculated from another captured image that has already been registered, and the captured image based on the parameter And a display control unit that displays the captured image when it is determined to display.

  The burden on the assistant can be reduced in the watching system.

It is a figure which shows an example of a watching system. It is a figure which shows the hardware constitutions of an analyzer. It is the figure which showed an example of the picked-up image information stored in a picked-up image database. It is a figure for demonstrating the 1st and 2nd feature-value extracted from a picked-up image. It is a figure for demonstrating the 3rd feature-value extracted from a picked-up image. It is a figure which shows the function structure of an analyzer. It is a figure for demonstrating the outline | summary of the picked-up image analysis process performed by an analyzer. It is a flowchart of the picked-up image analysis process performed by the analyzer. It is a flowchart of the registration process performed by an analyzer. It is a flowchart of the display process performed by an analyzer. It is a flowchart of the action result acquisition process performed by an analyzer. It is a flowchart of the weight change process performed by an analyzer. It is another flowchart of the display process performed by an analyzer. It is a figure for demonstrating the extraction method and display method of a digest image. It is another flowchart of the display process performed by an analyzer. It is another flowchart of the action result acquisition process performed by an analyzer.

  Hereinafter, embodiments will be described with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, the duplicate description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[First Embodiment]
First, the watching system in the first embodiment will be described. FIG. 1 is a diagram illustrating an example of a watching system.

  The monitoring system 100 includes an imaging unit 110, an analysis device 120 that is an example of a monitoring device, and a user interface device 130. The imaging unit 110 and the analysis device 120 are connected via a network 140. . The imaging unit 110 is installed beside the bedding (bed), for example, in each hospital room or each nursing room. The user interface device 130 is installed in, for example, a nurse station in a hospital or a work room for a caregiver in a care facility.

  The imaging unit 110 is a near-infrared camera, and captures an action of a patient lying on a bed in each room of a hospital or a nursing facility, a cared person, or the like (referred to as a “watched person”). A captured image captured by the imaging unit 110 is transmitted to the analysis device 120 via the network 140.

  The analysis device 120 processes the captured image captured by the imaging unit 110 and registers the captured image in the captured image information stored in the captured image database 121 (hereinafter, the database is abbreviated as DB). Further, the analysis device 120 performs control for displaying the captured image on the user interface device 130. Furthermore, the analysis device 120 determines whether or not there is assistance for the person to be watched by a nurse or a caregiver (referred to as “assistant”) when the captured image is displayed on the user interface device 130, and according to the determination result. The photographed image information stored in the photographed image DB 121 is updated. Note that an analysis program 122 is installed in the analysis device 120, and these processes in the analysis device 120 are realized by the analysis device 120 executing the analysis program 122.

  The user interface device 130 displays a photographed image and accepts an input indicating whether or not there is assistance for the person being watched over by the assistant. The user interface device 130 transmits the presence / absence of assistance input by the assistant to the analysis device 120 as information on the behavior result of the assistant.

  FIG. 2 is a diagram illustrating a hardware configuration of the analysis apparatus. As shown in FIG. 2, the analysis device 120 includes a CPU 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203. The analysis device 120 includes a storage unit 204, a connection unit 205, and a communication unit 206. Each unit of the analysis device 120 is connected to each other via a bus 207.

  The CPU 201 is a computer that executes various programs stored in the storage unit 204.

  The ROM 202 is a nonvolatile memory. The ROM 202 stores various programs, data, and the like necessary for the CPU 201 to execute various programs stored in the storage unit 204. Specifically, a boot program such as BIOS (Basic Input / Output System) or EFI (Extensible Firmware Interface) is stored.

  The RAM 203 is a main storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). The RAM 203 functions as a work area that is expanded when various programs are executed by the CPU 201 in the storage unit 204.

  The storage unit 204 stores various programs installed in the analysis device 120, data generated by executing the programs, and the like. The connection unit 205 is connected to the user interface device 130 and transmits / receives data to / from the user interface device 130. The communication unit 206 communicates with the imaging unit 110 via the network 140.

  Next, captured image information stored in the captured image DB 121 of the analysis apparatus 120 will be described. FIG. 3 is a diagram illustrating an example of photographed image information stored in the photographed image DB. The photographed image DB 121 stores photographed image information for each watching target person, and FIG. 3 is a diagram illustrating an example of the photographed image information of the watching target person of “patient ID” = “AAAA”. is there.

  As shown in FIG. 3, the captured image information 300 includes “image ID”, “image file”, “feature”, “weight parameter”, and “counter value” as information items. In “Image ID”, an identifier for identifying a captured image registered in the captured image information 300 is stored.

  The “image file” stores a captured image file. The photographed image file referred to here is a file including a plurality of frames of photographed image groups photographed during body movement time described later. For example, if the frame period is 30 frames / second and the body movement time is 5 seconds, the captured image file includes 150 frames of captured images.

  In the “feature amount”, a feature amount extracted from each captured image file stored in the “image file” is stored. Note that the captured image information 300 includes the size of the body motion, the body motion region, and the body motion time as the feature amount. A detailed description of each feature amount will be described later.

  Stored in the “weight parameter” is a value indicating the degree to which the action of the person to be watched represented by each feature amount is an action that requires assistance from the assistant. Note that an operation requiring assistance from an assistant refers to an operation that hardly occurs or an operation that is different from usual. In addition, movements that are not likely to occur or actions that are not normal occur, for example, during a nighttime period, in a general life rhythm such as going to bed → sleeping → getting up, the person being watched over is violent or waking up Refers to the operation. On the other hand, the actions that can occur normally and the usual actions refer to actions in which the person being watched over turns over. These motions can be expressed by a combination of the size of the body motion, the body motion region, and the body motion time.

  The “counter value” stores a value indicating the frequency with which each captured image file stored in the captured image information 300 is used.

  Next, the body motion magnitude and body motion region, which are the first and second feature amounts extracted from the captured image, will be described. FIG. 4 is a diagram for explaining the first and second feature amounts extracted from the photographed image, and is a diagram specifically showing the size of the body motion and the body motion region.

  FIG. 4A shows a captured image of the watching target photographed at each time. In the example of FIG. 4 (a), the person to be watched who has opened both arms in a figure of eight at time = t−1 moves his right arm to the front of the body at time = t. Further, at time = t + 1, the right arm is moved to near the left arm, and at time = t + 2, the right arm is returned to the original position.

  FIG. 4 (b) calculates the luminance difference of each pixel in the captured image of the person being watched taken at two consecutive times. The pixel having a luminance difference equal to or greater than a predetermined threshold is 1, and the luminance difference is predetermined. A binary image obtained by binarizing the pixels less than the threshold as 0 is shown. In addition, in order to make it easy to understand the binarized part, in the example of FIG. 4B, the watching target person is represented by a dotted line.

  In FIG. 4B, a black area A (t−1) has a luminance difference between a captured image captured at time = t−1 and a captured image captured at time = t equal to or greater than a predetermined threshold. It is an area. In addition, the area A (t) is an area where the luminance difference between the captured image captured at time = t and the captured image captured at time = t + 1 is greater than or equal to a predetermined threshold. Further, the area A (t + 1) is an area where the luminance difference between the captured image captured at time = t + 1 and the captured image captured at time = t + 2 is equal to or greater than a predetermined threshold.

  If the luminance value of each pixel (x, y) in the captured image is f (x, y) and the predetermined threshold is TH, a binary image can be obtained based on the following equation.

Among these, the region A (t−1) is a region where f ′ ₁ (x, y) = 1, and the region A (t) is a region where f ′ ₂ (x, y) = 1, and the region A (t + 1) is a region where f ′ ₃ (x, y) = 1.

  FIG. 4C shows an area of a pixel having a luminance value of 1 in any of two consecutive binary images. In FIG. 4C, a black area B (t) indicates an overlapping area between the area A (t−1) and the area A (t). Also, a black area B (t + 1) indicates an overlapping area between the area A (t) and the area A (t + 1).

  That is, the region B (t) and the region B (t + 1) can be expressed by the following formula.

  In the present embodiment, the region B (t) is referred to as a “body motion region” at time = t, and the region B (t + 1) is referred to as a “body motion region” at time = t + 1.

  In this embodiment, the distance L (t) between the center of gravity of the region B (t) and the center of gravity of the region B (t + 1) is expressed as “body motion between time = t and time = t + 1”. Called "the size of".

  Next, the body movement time, which is the third feature amount extracted from the captured image, will be described. FIG. 5 is a diagram for explaining the third feature amount extracted from the photographed image, and specifically shows the body movement time.

  In FIG. 5, the horizontal axis indicates time, and the vertical axis indicates the magnitude of body movement at each time. In FIG. 5, as an example, the magnitude of body movement (L (t)) from time = t to time = t + 1 is clearly indicated by a dotted line.

  As shown in FIG. 5, the body movement time starts when the magnitude of the body movement becomes greater than a predetermined threshold, and after the start point, the magnitude of the body movement first reaches the predetermined threshold. This is the time between the start point and the end point when the time point is the end point.

  In the present embodiment, among the feature amounts extracted based on the captured image of each frame included in the captured image file and stored in the captured image information 300, “the size of body motion” is the body motion time. The average body movement. The “body motion region” is an average body motion region in the body motion time. That is, the feature amount V is obtained in units of one body movement.

  Next, a functional configuration of the analysis device 120 will be described. FIG. 6 is a diagram illustrating a functional configuration of the analysis device 120. The analysis apparatus 120 realizes functions of each unit described later by the CPU 201 executing the analysis program 122. In the present embodiment, the captured image DB 121 of the analysis apparatus 120 is provided in the storage unit 204, for example.

  As illustrated in FIG. 6, the analysis apparatus 120 includes a captured image acquisition unit 601, a feature amount calculation unit 602, a registration unit 603, an initial weight management unit 604, a display control unit 605, an action result acquisition unit 606, and a weight management unit 607. Have.

  The captured image acquisition unit 601 acquires, from the imaging unit 110, a captured image obtained by capturing an image of the watching target person in the imaging unit 110.

  The feature amount calculation unit 602 calculates a feature amount indicating the motion of the watching target person in units of one body movement from the captured image acquired by the captured image acquisition unit 601.

  The registration unit 603 determines whether or not a captured image can be registered in the captured image information 300 stored in the captured image DB 121 based on the feature amount extracted by the feature amount calculation unit 602. The captured image is registered in the captured image information 300.

  The initial weight management unit 604 calculates the similarity between the feature amount of the photographed image newly registered by the registration unit 603 and the feature amount of the photographed image already registered. In addition, a weight parameter corresponding to the calculated similarity is derived, and the derived weight parameter is stored in the captured image information 300 as a weight parameter of a newly registered captured image.

  The display control unit 605 performs control for displaying a captured image on the user interface device 130. Specifically, based on the weight parameter derived in the initial weight management unit 604, it is determined whether or not the captured image is displayed on the user interface device 130. If it is determined that the captured image is displayed, the captured image is displayed on the user interface device. 130.

  The action result acquisition unit 606 acquires information related to the action result of the assistant input in response to the captured image being displayed on the user interface device 130. Specifically, by looking at the captured image displayed on the user interface device 130, the assistant determines that it is necessary to assist the person being watched over. Then, when the determination result is input to the user interface device 130, the determination result is acquired as information on the action result of the assistant and is notified to the weight management unit 607.

  The weight management unit 607 changes the weight parameter of the captured image stored in the captured image information 300 of the captured image DB 121 when the behavior result acquisition unit 606 is notified of information related to the behavior result of the assistant. Further, the weight management unit 607 changes the count value of the captured image stored in the captured image information 300 in accordance with the change of the weight parameter of the captured image.

  Next, the flow of the captured image analysis process executed by the analysis device 120 will be described. First, an overview of the captured image analysis process executed by the analysis device 120 will be described. FIG. 7 is a diagram for explaining the outline of the captured image analysis process executed by the analysis device 120.

As illustrated in FIG. 7, when the captured image acquisition unit 601 acquires a captured image 700 (“captured image file” = “Imag_n”) for one body motion, the feature amount calculation unit 602 uses the feature amount from the captured image 700. V _n = (f _n1 , f _n2 , f _n3 ) is calculated.

Based on the calculated feature value V _n = (f _n1 , f _n2 , f _n3 ), the registration unit 603 determines whether or not the captured image 700 can be registered in the captured image information 300. When it is determined that all of f _n1 , f _n2 , and f _n3 included in the calculated feature amount V _n are _equal to or greater than a predetermined threshold, it is possible to register the captured image 700 in the captured image information 300. judge.

  Here, the predetermined threshold value is a value that excludes minute movements (movements that temporarily move limbs and movements associated with breathing) that occur regardless of the sleep or awakening of the person being watched over. Is set.

If it is determined that the captured image 700 can be registered in the captured image information 300, the registration unit 603 sets the captured image 700 of “captured image file” = “Imag_n” as “captured image” with “image ID” = “000n”. Register in the information 300. Also, the registration unit 603 stores the calculated feature amount V _n in association with the captured image file.

Further, the initial weight management unit 604 calculates the similarity between the feature amount V _n of the captured image 700 and the feature amount of each captured image already registered in the captured image information 300. For example, the similarity S _(V n of the feature quantity _{V n,} wherein the amount _{V 1} of the captured image "captured image file" = "Imag_1" of the captured image 700 ( "captured image file" = "Imag_n"), V ₁ ) is calculated based on the following equation.

In initial weight management unit 604, similarly, the similarity _S (V _{n, V} 2) between the feature quantity _{V n} and the feature quantity _{V 2,} the similarity S _(V n between the feature quantity _{V n} and the feature quantity _{V 3,} V ₃ ),... Are calculated. The initial weight management unit 604 further extracts a similarity degree equal to or higher than a predetermined threshold from the calculated similarity degrees, and stores the weight parameter stored in association with the captured image when the extracted similarity degree is calculated. Of these, the largest weight parameter is selected.

Further, the initial weight management unit 604 stores the selected weight parameter in the captured image information 300 as the weight parameter W _n of the captured image 700. Note that the initial weight management unit 604 stores 1 in the captured image information 300 as a weight parameter of the captured image 700 when the calculated similarity does not have a similarity greater than or equal to a predetermined threshold.

  When the initial weight management unit 604 stores the weight parameter of the photographed image 700 in the photographed image information 300, the display control unit 605 displays the photographed image 700 on the user interface device 130 based on the stored weight parameter. It is determined whether or not.

  Specifically, when the weight parameter stored by the initial weight management unit 604 is 1, or when the weight parameter stored by the initial weight management unit 604 is equal to or greater than a predetermined threshold, the captured image 700 is displayed. It is determined to be displayed on the user interface device 130.

  In other words, when the weight parameter is greater than 1 and less than the predetermined threshold, the display control unit 605 determines that the captured image 700 is not displayed on the user interface device 130. Note that when the weight parameter is 1, the captured image 700 is displayed as a new action that has not occurred in the past, so that the assistant can determine whether or not the action requires assistance. It is.

  A display screen 710 shows the captured image 700 displayed on the user interface device 130. On the display screen 710, in addition to the photographed image 700 determined to be displayable by the display control unit 605, a button for inputting the action result of the assistant is arranged.

  The “corresponding” button 711 is an object to be watched when it is determined that the person to be watched performs a specific action (an action that requires assistance from the assistant) by looking at the captured image 700 displayed on the display screen 710. It is a button pressed by a person.

  On the other hand, the “watch” button 712 indicates that the person to be watched does not perform a specific action by looking at the captured image 700 displayed on the display screen 710 (an action that does not require assistance from the assistant). When it is determined, the button is pressed by the person being watched over.

  The action result acquisition unit 606 determines which of the button 711 and the button 712 the assistant has pressed in response to the captured image 700 being displayed on the display screen 710. If the action result acquisition unit 606 determines that the button 712 is pressed, the captured image information 300 is not changed.

On the other hand, when the action result acquisition unit 606 determines that the button 711 has been pressed, the weight management unit 607 changes the weight parameter stored in the captured image information 300. Specifically, it increments the weight parameter W _n associated with the captured image 700. Thus, the weight parameter associated with the captured image 700 is W _n +1.

The weight management unit 607 calculates the similarity between the feature amount V _n of the captured image 700 and the feature amount of each captured image already registered in the captured image information 300 (when the captured image 700 is registered). If the calculated degree of similarity is retained, recalculation here is unnecessary). Then, among the calculated similarities, the weight parameter associated with the captured image for which the similarity equal to or greater than a predetermined threshold is calculated is incremented.

  In the example of FIG. 7, the weight parameter “captured image file” = “Imag — 2” and the weight parameter “captured image file” = “Imag — 5” are incremented.

  When the weight management unit 607 updates the weight parameter, the corresponding counter value is returned to the initial value. Also, the weight management unit 607 decrements the counter value corresponding to the weight parameter that has not been updated by one. In the weight management unit 607, when the counter value becomes zero, the captured image information 300 deletes the corresponding captured image.

  Next, the detailed flow of the captured image analysis process executed by the analysis device 120 will be described. FIG. 8 is a flowchart of the captured image analysis process executed by the analysis device 120.

  In step S <b> 801, the captured image acquisition unit 601 acquires a captured image 700. In step S <b> 802, the feature amount calculation unit 602 calculates a feature amount from the acquired captured image 700.

  In step S <b> 803, the registration unit 603 performs a registration process for registering the acquired captured image 700 in the captured image information 300. The initial weight management unit 604 derives a weight parameter to be stored in the captured image information 300 in association with the captured image 700 when registering the captured image 700 based on the extracted feature amount. The initial weight management unit 604 stores the derived weight parameter in association with the captured image 700.

  In step S804, the display control unit 605 performs display processing for displaying the captured image 700 on the user interface device 130 according to the weight parameter of the captured image registered in step S803.

  In step S <b> 805, the behavior result acquisition unit 606 performs a behavior result acquisition process for acquiring information related to the behavior result of the assistant input in response to the captured image being displayed on the user interface device 130.

  In step S806, the weight management unit 607 performs change processing for changing the weight parameter of the photographed image 700 registered in the photographed image information 300 according to the information regarding the action result of the assistant acquired in step S805.

  In step S807, the captured image acquisition unit 601 determines whether or not a shooting end instruction has been input. If it is determined that a shooting end instruction has not been input, the process returns to step S801. On the other hand, if it is determined that a shooting end instruction has been input, the shot image analysis process ends.

  Next, details of the registration process (step S803) will be described. FIG. 9 is a flowchart of registration processing executed by the analysis device 120.

In step S901, the registration unit 603 determines whether or not all of the feature amounts f _n1 , f _n2 , and f _n3 extracted from the captured image 700 that is the processing target are equal to or greater than a predetermined threshold value. If it is determined in step S901 that any of the feature quantities f _n1 , f _n2 , and f _n3 is less than the predetermined threshold value, the process proceeds to step S908. In step S <b> 908, the registration unit 603 determines not to register the captured image 700 in the captured image information 300.

On the other hand, if it is determined in step S901 that all of the feature quantities f _n1 , f _n2 , and f _n3 are greater than or equal to a predetermined threshold value, the process proceeds to step S902.

  In step S <b> 902, the initial weight management unit 604 substitutes 1 as a default value for the weight parameter of the captured image 700.

In step S903, the initial weight management unit 604 obtains the feature amount V _n of the captured image 700 and the feature amounts V ₁ , V ₂ , V ₃ ,... Of each captured image already registered in the captured image information 300. The similarity is calculated.

In step S904, the initial weight management unit 604, the similarity _S calculated in step _{S903 (V n, V 1)} , S (V n, V 2), S (V n, V 3), ··· of Among these, it is determined whether or not the maximum similarity is less than a predetermined threshold.

  If it is determined in step S904 that the maximum similarity is less than the predetermined threshold value, the process proceeds to step S907. On the other hand, if it is determined in step S904 that the maximum similarity is greater than or equal to a predetermined threshold, the process proceeds to step S905.

  In step S <b> 905, the initial weight management unit 604 acquires, from the captured image information 300, the maximum weight parameter among captured images whose similarity is equal to or greater than a predetermined threshold. In step S906, the initial weight management unit 604 substitutes the weight parameter acquired in step S905 for the weight parameter of the captured image 700.

  In step S907, the registration unit 603 stores the captured image 700 in the captured image information 300 together with the weight parameter substituted in step S902 or in step S906 and the feature amount calculated in step S802. At this time, the registration unit 603 stores the initial value in the counter value.

  Next, details of the display process (step S804) will be described. FIG. 10 is a flowchart of display processing executed by the analysis device 120.

  In step S <b> 1001, the display control unit 605 determines whether the weight parameter of the captured image 700 is 1. If it is determined in step S1001 that the weight parameter of the captured image 700 is 1, the process proceeds to step S1003.

  On the other hand, if it is determined in step S1001 that the weight parameter of the captured image 700 is not 1, the process proceeds to step S1002. In step S1002, the display control unit 605 determines whether the weight parameter of the captured image 700 is equal to or greater than a predetermined threshold.

  If it is determined in step S1002 that the weight parameter of the captured image 700 is greater than or equal to a predetermined threshold, the process proceeds to step S1003.

  In step S <b> 1003, the display control unit 605 controls to display the captured image 700 on the user interface device 130.

  On the other hand, if it is determined in step S1002 that the weight parameter of the captured image 700 is less than the predetermined threshold, the process proceeds to step S1004. In step S <b> 1004, the display control unit 605 determines not to display the captured image 700 on the user interface device 130.

  As described above, the display control unit 605 displays the captured image 700 on the user interface device 130 when the weight parameter is the default value (= 1), or when the weight parameter is equal to or greater than the predetermined threshold. On the other hand, it is not displayed on the user interface device 130 in other cases.

  Next, details of the action result acquisition process (step S805) will be described. FIG. 11 is a flowchart of the action result acquisition process executed by the analysis device 120.

  In step S <b> 1101, the action result acquisition unit 606 determines whether a predetermined time has elapsed since the captured image 700 was displayed on the user interface device 130. If it is determined in step S1101 that the predetermined time has not elapsed, the process proceeds to step S1102.

  In step S1102, the action result acquisition unit 606 determines whether the button 711 or the button 712 on the display screen 710 has been pressed. If it is determined in step S1102 that no button is pressed, the process returns to step S1101.

  If it is determined in step S1101 that no button has been pressed within a predetermined time, the process proceeds to step S1106. In step S <b> 1106, the action result acquisition unit 606 determines that the operation of the watching target person indicated by the captured image 700 is not an operation that requires assistance from the assistant.

  On the other hand, if it is determined in step S1102 that either button 711 or button 712 on display screen 710 has been pressed, the process proceeds to step S1103.

  In step S1103, the action result acquisition unit 606 identifies which button on the display screen 710 has been pressed. As a result of the identification, if the pressed button is the button 711, the process proceeds from step S1104 to step S1105. In step S <b> 1105, the behavior result acquisition unit 606 determines that the operation of the watching target person indicated by the captured image 700 is an operation that requires assistance from the assistant.

  On the other hand, as a result of the identification, if the pressed button is the button 712, the process proceeds from step S1104 to step S1106. In step S <b> 1106, the action result acquisition unit 606 determines that the operation of the watching target person indicated by the captured image 700 is not an operation that requires assistance from the assistant.

  Next, details of the weight change process (step S806) will be described. FIG. 12 is a flowchart of the weight change process by the analysis device 120.

  In step S <b> 1201, the weight management unit 607 determines that the action result acquisition unit 606 determines that the action of the watching target person indicated by the captured image 700 is an action that requires assistance from the assistant, or is an unnecessary action. Is determined.

  If it is determined in step S1201 that the operation is unnecessary, the process returns to step S807 in FIG. On the other hand, if it is determined in step S1201 that the operation is necessary, the process proceeds to step S1202.

  In step S <b> 1202, the weight management unit 607 increments the weight parameter of the captured image 700 stored in the captured image information 300.

In step S1203, the weight management unit 607 calculates the feature amount V _n of the captured image 700 and the feature amounts V ₁ , V ₂ , V ₃ ,... Of each captured image already registered in the captured image information 300. Calculate similarity.

  In step S1204, the weight management unit 607 returns the counter value associated with the captured image for which the similarity equal to or greater than the predetermined threshold among the similarities calculated in step S1203 is returned to the initial value.

  In step S1205, the weight management unit 607 increments the weight parameter associated with the captured image for which the similarity equal to or greater than a predetermined threshold is calculated among the similarities calculated in step S1203.

  Further, in step S1206, the weight management unit 607 decrements the counter value associated with the captured image for which the similarity less than a predetermined threshold is calculated among the similarities calculated in step S1203.

  In step S <b> 1207, the weight management unit 607 determines whether there is a captured image with a counter value less than 1 in the captured image information 300. If it is determined in step S1207 that there is a captured image with a counter value less than 1, the captured image with a counter value less than 1 is deleted from the captured image information 300, and the process returns to step S807 in FIG.

  On the other hand, if it is determined in step S1207 that there is no captured image with a counter value less than 1, the process returns to step S807 in FIG.

  As is clear from the above description, the analysis device 120 extracts a feature amount representing the operation of the watching target person from a photographed image obtained by shooting the watching target person, and derives it according to the extracted feature amount. The weight parameter thus determined is managed for each person to be watched in association with the captured image. Further, the analysis device 120 determines whether to display the captured image to the assistant based on the derived weight parameter, and displays it to the assistant when it is determined to display the captured image.

  Thus, in order to determine whether or not to display the newly acquired captured image to the assistant based on the weight parameter reflecting the determination result determined by the assistant in the past as an operation requiring assistance, It is possible to accurately detect and display an operation that requires assistance from an assistant. That is, it is possible to perform display according to the difference in daily life operation for each watching target person.

  Further, the analysis device 120 feeds back a result of determination as to whether or not the assistant needs to assist the person to be watched according to the displayed captured image, and reflects the result in the weight parameter of the captured image.

  Thereby, even if the daily life operation of the person being watched over changes, it is possible to accurately detect an operation that requires assistance from the assistant. That is, it is possible to dynamically reflect the judgment criteria that the assistant has perceived sensuously.

  As a result, it is possible to suppress erroneous detection in the watching system, and the burden on the assistant can be reduced.

[Second Embodiment]
In the second embodiment, the analysis device 120 displays, on the user interface device 130, a captured image that satisfies a predetermined condition from a plurality of frames of captured images included in the acquired captured image as a digest image.

  FIG. 13 is a flowchart of the display process in the second embodiment. Of the steps in the flowchart shown in FIG. 13, steps that are the same as those in the flowchart shown in FIG. 10 are given the same reference numerals, and descriptions thereof are omitted here.

  In step S1301, the display control unit 605 acquires all the frames included in the captured image 700.

  In step S1302, the display control unit 605 obtains the difference value (luminance difference) between the luminance values of the pixels in the captured images of the two frames captured at two consecutive times for all the frames acquired in step S1301. And the total value of the luminance differences is calculated.

  In step S1303, the display control unit 605 extracts a predetermined number of frames having a large total value of luminance differences.

  In step S1304, the display control unit 605 displays the frame at the start of body movement and the predetermined number of frames extracted in step S1303 on the user interface device 130 as a digest image.

  FIG. 14 is a diagram for explaining a digest image extraction method and a display method. In FIG. 14A, the horizontal axis indicates time, and the vertical axis indicates the luminance difference between the pixels in the captured images of two frames captured at two consecutive times for all frames included in the captured image 700. The result of calculating the total value of is shown. A large luminance difference indicates that the person being watched over has moved greatly between two consecutive times.

  In FIG. 14A, a circle 1401 indicates the time at the start of body movement. Circles 1402 to 1404 are frames shot within the same body movement time as the shot image 700, and indicate times when the watching target person moved greatly.

  In addition to the frame at the start of body movement, the display control unit 605 displays a frame captured at the time specified by the circles 1402 to 1404 as a digest image on the display screen. FIG. 14B shows an example of a display screen displaying a digest image. In FIG. 14B, a frame at the start of body movement is displayed in an area 1411. In addition, in the areas 1412 to 1414, frames taken at the time specified by the circles 1402 to 1404 are displayed.

  In the area 1420, the captured image acquired by the captured image acquisition unit 601 is displayed in real time.

  In addition, when displaying a digest image, the frame at the time of a body motion start is included because it can grasp | ascertain the state of a monitoring subject when a body motion starts. For example, if the watched person performs a large motion while his / her foot is out of the bedding and the frame at the start of body movement is not included, the assistant will check whether the foot has come out of the bedding due to the motion. I can't judge. For this reason, it cannot be determined whether the operation was caused by the surrounding environment such as a high room temperature or the operation was caused by another reason.

  On the other hand, by including the frame at the start of body movement in the digest image, the assistant can intuitively grasp the history of the operation and the like.

  As is clear from the above description, when the analysis device 120 determines to display based on the derived weight parameter, it does not include all the claims included in the determined captured image, but the frame extracted from the determined captured image. indicate.

  As a result, it is possible to display a captured image effective for the assistant to determine whether or not the operation requires assistance. As a result, the assistant can grasp the entire image of the watching target person in a short time.

  In the above description, a frame satisfying a predetermined condition is extracted from a plurality of frames included in the same body movement time and displayed as a digest image. For example, a digest of a past captured image is displayed. Images may be displayed together.

  FIG. 15 is another flowchart of the display process executed by the analysis device 120. Of the steps in the flowchart shown in FIG. 15, the same steps as those in the flowchart shown in FIG. 13 are given the same reference numerals, and the description thereof is omitted here.

  As shown in FIG. 15, in step S1501, the display control unit 605 weights each of a plurality of captured images that the assistant has determined in the past that the operation needs assistance in accordance with the time difference from the current time. Perform. Further, the display control unit 605 calculates a similarity with the captured image at the current time, and extracts a past captured image with the highest similarity.

  In steps S1502 to S1504, the display control unit 605 extracts a frame of the digest image from the extracted past captured images by the same processing as in steps S1302 to S1304. And it displays in parallel with the frame of the digest image of the captured image at the current time. Thereby, the assistant can more accurately determine the operation of the person being watched over.

[Third Embodiment]
In the third embodiment, the analysis device 120 determines whether or not assistance by an assistant is performed on a person to be watched based on a captured image.

  FIG. 16 is a flowchart of action result acquisition processing according to the third embodiment. Of the steps in the flowchart shown in FIG. 16, the same steps as those in the flowchart shown in FIG. 11 are given the same reference numerals, and the description thereof is omitted here.

  In step S1602, the behavior result acquisition unit 606 acquires the captured image acquired by the captured image acquisition unit 601 for a predetermined time after the captured image 700 is displayed on the display screen 710.

  In step S1603, the action result acquisition unit 606 calculates a body movement region based on the captured image acquired in step S1602.

  In step S1604, the action result acquisition unit 606 determines whether or not there is a change in the body movement region. If it is determined that there is no change, the action result acquisition unit 606 returns to step S1101. On the other hand, if it is determined that there is a change in the body movement region, the process proceeds to step S1105, where the action result acquisition unit 606 determines that the assistant has performed assistance and determines that the operation requires assistance.

  Thus, by determining whether or not the assistant has provided assistance based on the captured image, it is possible to save the assistant from pressing the button 711 or 712 each time the captured image is displayed.

[Fourth Embodiment]
In the first embodiment, the captured image is deleted from the captured image information 300 according to the counter value. However, the captured image may be deleted from the captured image information 300 according to the weight parameter. Specifically, when the weight parameter is not updated for a certain time, the weight parameter is decremented, and when the weight parameter becomes less than 1, the captured image is deleted from the captured image information 300. Good.

  In the first embodiment, the assistant has to determine whether or not the operation needs assistance for the displayed captured image, and presses either the button 711 or 712. You may install the button which shows the hold of judgment. Thereby, the assistant can judge after watching the state of the person being watched for a while.

  In the first embodiment, the information items of the captured image information 300 include “image ID”, “image file”, “feature”, “weight parameter”, and “counter value”. Furthermore, coordinates or a part indicating a body movement region may be included.

In addition, in the disclosed technology, forms such as the following additional notes are possible.
(Appendix 1)
A calculation unit that calculates a feature amount indicating an operation of the subject from a captured image obtained by photographing the subject;
The feature amount calculated by the calculation unit is associated with a parameter according to the similarity with the feature amount calculated from another captured image already registered for the target person, and is stored in the database for each target person. A registration section to register;
A monitoring apparatus, comprising: a display control unit configured to determine whether to display the captured image based on the parameter, and to display the captured image when it is determined to display the captured image.
(Appendix 2)
The feature amount is
A body movement area indicating the size of the body movement area of each area of the subject,
The size of the subject's body movement;
The watch apparatus according to appendix 1, including a body motion time indicating a start to a finish of the subject's body motion.
(Appendix 3)
Change in which the presence or absence of assistance by an assistant assisting the target person is determined and the parameter is changed according to the determination result when the captured image is displayed according to whether or not to display based on the parameter The watching device according to appendix 1 or 2, further comprising a portion.
(Appendix 4)
When the similarity is equal to or greater than a predetermined threshold, the registration unit corresponds to the feature amount calculated by the calculation unit by using a parameter associated with the other photographed image for which the similarity is calculated. The watching device according to any one of the supplementary notes 1 to 3, wherein the registration device is attached and registered.
(Appendix 5)
If the similarity is less than a predetermined threshold, the registration unit registers a predetermined value as a parameter in association with the feature amount calculated by the calculation unit. 4. The monitoring device according to any one of the supplementary notes.
(Appendix 6)
The display control unit
A frame extracted from a plurality of captured images included in the extracted captured image by extracting a captured image similar to the captured image determined to be displayed based on the parameter from other registered captured images Is further displayed. The monitoring device according to attachment 1, wherein
(Appendix 7)
The display control unit
The watching device according to any one of appendices 1 to 6, wherein a frame extracted from a plurality of frames of captured images included in the captured image determined to be displayed based on the parameter is displayed.
(Appendix 8)
The watching device according to appendix 3, wherein the changing unit changes a parameter of the photographed image when the assistant assists the subject by displaying the photographed image.
(Appendix 9)
When the similarity is equal to or higher than a predetermined threshold, the changing unit changes a parameter associated with the other captured image for which the similarity equal to or higher than the predetermined threshold is calculated. The monitoring device according to attachment 3.
(Appendix 10)
The watching device according to appendix 3, wherein the registration unit varies a counter value associated with the parameter in accordance with the change of the parameter by the changing unit.
(Appendix 11)
The monitoring apparatus according to appendix 10, wherein the registration unit deletes a captured image, a feature amount, and a parameter associated with the counter value according to a change in the counter value.
(Appendix 12)
The registration unit varies the parameter when the parameter is not changed by the changing unit for a predetermined time, and the captured image, the feature amount, and the parameter associated with the parameter are changed according to the parameter change. 4. The watching device according to appendix 3, wherein the monitoring device is deleted.
(Appendix 13)
From the captured image obtained by photographing the subject, a feature amount indicating the behavior of the subject is calculated,
The calculated feature amount is associated with a parameter according to the similarity with the feature amount calculated from another captured image already registered for the subject, and is registered in the database for each subject.
It is determined whether to display the captured image based on the parameter, and when it is determined to display, the captured image is displayed.
A program that causes a computer to execute processing.
(Appendix 14)
A display method by a computer, wherein the computer
From the captured image obtained by photographing the subject, a feature amount indicating the behavior of the subject is calculated,
The calculated feature amount is associated with a parameter according to the similarity with the feature amount calculated from another captured image already registered for the subject, and is registered in the database for each subject.
It is determined whether to display the captured image based on the parameter, and when it is determined to display, the captured image is displayed.
A display method characterized by that.

  It should be noted that the present invention is not limited to the configuration shown here, such as a combination with other elements in the configuration described in the above embodiment. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form.

100: Monitoring system 110: Imaging unit 120: Analysis device 121: Captured image DB
122: Analysis program 130: User interface device 140: Network 300: Captured image information 601: Captured image acquisition unit 602: Feature amount calculation unit 603: Registration unit 604: Initial weight management unit 605: Display control unit 606: Action result acquisition unit 607: Weight management unit

Claims (7)

A calculation unit that calculates a feature amount indicating an operation of the subject from a captured image obtained by photographing the subject;
The feature amount calculated by the calculation unit is associated with a parameter according to the similarity with the feature amount calculated from another captured image already registered for the target person, and is stored in the database for each target person. A registration section to register;
A monitoring apparatus, comprising: a display control unit configured to determine whether to display the captured image based on the parameter, and to display the captured image when it is determined to display the captured image. The feature amount is
A body movement area indicating the size of the body movement area of each area of the subject,
The size of the subject's body movement;
The monitoring apparatus according to claim 1, further comprising: a body movement time indicating a start to an end of body movement of the subject.   Change in which the presence or absence of assistance by an assistant assisting the target person is determined and the parameter is changed according to the determination result when the captured image is displayed according to whether or not to display based on the parameter The watching device according to claim 1, further comprising a portion. The display control unit
A frame extracted from a plurality of captured images included in the extracted captured image by extracting a captured image similar to the captured image determined to be displayed based on the parameter from other registered captured images The monitoring device according to claim 1, further comprising: The display control unit
The monitoring apparatus according to claim 1, wherein a frame extracted from a plurality of frames of captured images included in the captured image determined to be displayed based on the parameter is displayed. From the captured image obtained by photographing the subject, a feature amount indicating the behavior of the subject is calculated,
The calculated feature amount is associated with a parameter according to the similarity with the feature amount calculated from another captured image already registered for the subject, and is registered in the database for each subject.
It is determined whether to display the captured image based on the parameter, and when it is determined to display, the captured image is displayed.
A program that causes a computer to execute processing. A display method by a computer, wherein the computer
From the captured image obtained by photographing the subject, a feature amount indicating the behavior of the subject is calculated,
The calculated feature amount is associated with a parameter according to the similarity with the feature amount calculated from another captured image already registered for the subject, and is registered in the database for each subject.
It is determined whether to display the captured image based on the parameter, and when it is determined to display, the captured image is displayed.
A display method characterized by that.

Images