JP7391536B2 - Observation system and observation method - Google Patents

Description

The present invention relates to an observation system and an observation method for observing an object.

Conventionally, techniques for observing the condition of a target person are known. For example, the system disclosed in Patent Document 1 is a system consisting of a terminal of an observed person such as an elderly person, a terminal of a doctor or family member, and a monitoring server, and emergency notification conditions corresponding to the level of care required are set. When the detection information of the observation equipment (human sensor, Doppler sensor, etc.) connected to the terminal reaches the specified emergency alarm information, an emergency alarm is sent to the terminal of the specified person (physician, family member, etc.) paired with the observed person. I do.

Japanese Patent Application Publication No. 2017-151755

However, with conventional technology, it is not possible to issue a warning until an emergency situation occurs, so the problem is that only after-the-fact responses are required, and it is not possible to prevent emergency situations in advance or take early responses. was there.

The present invention has been made in order to solve the problems of the prior art described above, and aims to realize early response to the person being observed.

In order to solve the above-mentioned problems and achieve the objects, the present invention includes: an imaging means for imaging an observed person; an extraction means for extracting motion characteristics from an image of the observed person captured by the imaging means; An observation system comprising: an observer terminal that receives an alert regarding the abnormal behavior when the extracted motion feature corresponds to an abnormal motion, the motion feature extracted by the extraction means; However, if the above-mentioned abnormality precursor falls under the abnormality precursor, which is a behavior that has the possibility of transitioning to abnormal behavior, excluding those that were determined not to transition to abnormal behavior due to notifications related to past abnormal behavior. The present invention is characterized in that an alarm regarding a sign of abnormality is issued to the observer terminal.

Further, in the above invention, the present invention provides that the observer terminal has a display unit, receives information based on an image of the observed person when receiving an alarm related to the abnormality sign, and displays the It is characterized by being displayed in the section.

Further, the present invention is characterized in that, in the above invention, the observer terminal further includes an input unit, and after receiving the alert, receives an input as to whether or not the alert is appropriate.

Further, in the above invention, the observer terminal transmits information based on the input to the extraction means after receiving the input as to whether or not the alarm was appropriate, and the observation system includes: The storage means further comprises a storage means, after receiving the information based on the input, stores the information based on the input in association with the motion characteristic corresponding to the input, and the extraction means Before issuing another alarm to the observer terminal, search the operating characteristics stored in the storage means for operating characteristics similar to the abnormality sign that caused the alarm, and detect the corresponding operating characteristics. The present invention is characterized in that the alarm is stopped when the information stored corresponding to the operation feature is not information corresponding to an appropriate alarm.

Further, in the present invention, in the above invention, the extraction means controls the alarm using a learned model that has learned whether or not the operation feature corresponds to the abnormal operation and/or the abnormal sign. , the information based on the input is used to update the learned model.

Further, in the above invention, the present invention provides that the imaging means is a three-dimensional camera capable of acquiring depth information, and the imaging means or the extraction means extracts information from the observed person based on the facial image of the observed person. It is characterized by authenticating whether a person is suitable as an observation subject.

Further, the present invention includes an imaging means for imaging a person to be observed, and a motion feature that is extracted from an image of the observed person captured by the imaging means, and when there is a sign of abnormality in the extracted motion feature. , an extraction means for issuing a notification regarding the abnormality sign to an observer terminal; and, when the extracted operation feature corresponds to an abnormal operation, receiving a notification regarding the abnormal operation from the extraction means. An observation system including an observer terminal that sends a warning, and a storage means, wherein the observer terminal receives an input as to whether or not the warning was appropriate, and then sends information based on the input to the extraction means. After receiving the information based on the input, the storage means associates and stores the information based on the input and the operation characteristic corresponding to the input, and the extraction means stores another information based on the input. Before issuing an alarm to the observer terminal, search the operating characteristics stored in the storage means for an operating feature similar to the abnormality sign that caused the alarm, and apply the corresponding operating feature to the operating feature stored in the storage means. If the information stored in correspondence is not information corresponding to appropriate notification, the notification is stopped.
The present invention also provides an imaging means for taking an image of the observed person, an extraction means for extracting a motion feature from an image of the observed person captured by the imaging means, and the extracted motion feature corresponds to an abnormal motion. an observation method in an observation system that includes an observer terminal that receives an alert regarding the abnormal behavior, the extracted behavior feature being a behavior that may lead to the abnormal behavior; Among the abnormality signs that correspond to the abnormality signs excluding those that were determined not to progress to abnormal behavior due to notifications related to past abnormal operations, a notification related to the abnormality sign is sent to the observer terminal. The method is characterized by having an alarming step performed by.

According to the present invention, early response to the observed person can be realized.

FIG. 1 is an explanatory diagram for explaining the concept of an observation system. FIG. 2 is a system configuration diagram showing the configuration of the observation system. FIG. 3 is a functional block diagram showing the functional configuration of the imaging device shown in FIG. 1. FIG. 4 is a functional block diagram showing the functional configuration of the server device shown in FIG. 1. FIG. 5 is a functional block diagram showing the functional configuration of the terminal device shown in FIG. 1. FIG. 6 is a flowchart showing the processing procedure of the server device. FIG. 7 is an explanatory diagram of a display example when no abnormal operation or sign of abnormality occurs. FIG. 8 is an explanatory diagram of a display example when an abnormal operation occurs. FIG. 9 is an explanatory diagram of updating the abnormality sign model data using corrected data. FIG. 10 is an explanatory diagram of a case where correction data is referred to at the time of determination. FIG. 11 is an explanatory diagram of the detention facility observation system.

Hereinafter, an observation system and an observation method according to an embodiment will be described with reference to the accompanying drawings.

FIG. 1 is an explanatory diagram for explaining the concept of an observation system. The observation system shown in FIG. 1 has an imaging device 10 installed in a living room of a welfare facility for the elderly, and can image and observe residents. The resident who is the subject of this observation is called the observed person.

The imaging device 10 performs imaging in chronological order and transmits a chronological group of images (videos) to the server device 20 . The server device 20 extracts motion characteristics from the image of the observed person captured by the imaging device 10, and determines abnormal motion and signs of abnormality.

Abnormal movements include accidents such as falls, movements indicating poor physical condition, and the like. If the extracted motion feature corresponds to an abnormal motion, the server device 20 notifies the observer's terminal device 30 of the abnormal motion. The observer is a staff member of a welfare facility for the elderly, and when the terminal device 30 receives a notification of an abnormal operation, the observer takes action to deal with the abnormal operation.

An abnormality sign is an operation that does not correspond to an abnormal operation but has the possibility of turning into an abnormal operation. For example, crouching or unsteady walking. Additionally, unusual behavior can also be treated as a sign of abnormality. If the extracted operational feature corresponds to an abnormality sign, the server device 20 notifies the observer's terminal device 30 of the abnormality sign. The observer is a staff member of a welfare facility for the elderly, and when the terminal device 30 receives a notification of a sign of abnormality, the observer takes action on the sign of abnormality, thereby realizing prevention and early response to the abnormality.

For convenience, the server device 20 notifying an abnormal operation or a sign of an abnormality is called an alarm. In addition, notification of abnormal operation is called abnormality notification, and notification of abnormality sign is called predictive notification.

In this way, the observation system according to the present embodiment images the observed person, extracts motion characteristics from the image of the observed person, and when the motion characteristics correspond to abnormal motion, the observation system uses the terminal device 30 of the observer to An abnormality warning is issued to the terminal device 30, and when there is an abnormality sign in the operating characteristics, an abnormality warning is issued to the terminal device 30. Therefore, advance prevention and early response to emergencies can be realized.

Next, the configuration of the observation system will be explained. FIG. 2 is a system configuration diagram showing the configuration of the observation system. As shown in FIG. 2, an imaging device 10 is provided in each of a plurality of living rooms, and the imaging device 10 is connected to a server device 20 via a predetermined network.

Furthermore, a terminal device 30 for an observer is also connected to the server device 20 via a predetermined network. The terminal device 30 for the observer may be installed in the observer's waiting room, or may be a portable device that the observer can carry.

The server device 20 acquires the imaging results from the plurality of imaging devices 10, extracts the motion characteristics of the person to be observed in each living room, and, if it corresponds to abnormal motion or a sign of abnormality, sends the result to the corresponding terminal device 30. Make a report.

FIG. 3 is a functional block diagram showing the functional configuration of the imaging device 10 shown in FIG. 1. As shown in FIG. 3, the imaging device 10 includes a visible light camera 11, a depth sensor 12, a communication section 13, a storage section 14, and a control section 15.

The visible light camera 11 is an imaging device that captures visible light images in time series. The depth sensor 12 is a device that acquires depth data by emitting infrared light and receiving reflected waves. The communication unit 13 is a communication interface that communicates with the server device 20. The storage unit 14 is a storage device such as a hard disk drive, and stores visible light image data 14a, depth image data 14b, and the like.

The control unit 15 is a control unit that controls the entire imaging device 10, and includes a visible light image processing unit 15a, a depth image processing unit 15b, and a transmission processing unit 15c. In reality, programs corresponding to these functional units are stored in a ROM or non-volatile memory (not shown), and by loading these programs into the CPU and executing them, the visible light image processing unit 15a and the depth image processing The unit 15b and the transmission processing unit 15c are caused to execute corresponding processes, respectively.

The visible light image processing unit 15a generates visible light image data 14a from the imaging results of the visible light camera 11, and performs a process of storing the data in the storage unit 14. The depth image processing unit 15b generates depth image data 14b using the depth data acquired by the depth sensor 12, and performs a process of storing the data in the storage unit 14. The transmission processing unit 15c performs a process of associating the visible light image data 14a and the depth image data 14b and transmitting them to the server device 20 as appropriate.

FIG. 4 is a functional block diagram showing the functional configuration of the server device 20 shown in FIG. 1. As shown in FIG. As shown in FIG. 4, the server device 20 includes a display section 21, an input section 22, a communication section 23, a storage section 24, and a control section 25.

The display unit 21 is a display device such as a liquid crystal display. The input unit 22 is an input device such as a keyboard or a mouse. The communication unit 23 is a communication interface that communicates with the imaging device 10 and the terminal device 30.

The storage unit 24 is a storage device such as a hard disk drive, and stores abnormal operation model data 24a, abnormal sign model data 24b, observation management data 24c, image history data 24d, notification history data 24e, and the like.

The abnormal behavior model data 24a is data used to determine abnormal behavior. This abnormal behavior model data 24a may be generated by performing deep learning using, for example, motion feature data for learning and correct answer data (data indicating whether or not the motion is abnormal).

The abnormality sign model data 24b is data used to determine abnormality signs. This abnormality sign model data 24b may be generated by performing deep learning using, for example, motion feature data for learning and correct answer data (data indicating whether or not it is an abnormality sign).

The observation management data 24c is data for managing information about the room, the person to be observed, the imaging device 10, the observer, and the terminal device 30. That is, it is used to manage which person to be observed is in which room, which imaging device 10 is installed, which observer is in charge, which terminal device 30 is located where, etc.

The image history data 24d is data in which images received from the imaging device 10 are accumulated. Specifically, identification information of the imaging device 10, date and time, visible light image data, depth image data, etc. are stored in association with each other.

The notification history data 24e is data that accumulates the history of notifications made to the terminal device 30. Specifically, the identification information of the terminal device 30, the date and time, the content of the notification, the result of the response, the suitability of the notification, etc. are stored in association with each other. Note that the response results and the suitability of notification are received from the terminal device 30.

The control unit 25 is a control unit that controls the entire server device 20, and includes a judgment model management unit 25a, an observation management unit 25b, an authentication processing unit 25c, an image composition unit 25d, a motion feature extraction unit 25e, an abnormal motion notification unit 25f, and It has an abnormality sign notification section 25g. In reality, programs corresponding to these functional units are stored in a ROM or non-volatile memory (not shown), and by loading these programs into the CPU and executing them, the judgment model management unit 25a and the observation management unit 25b , the authentication processing section 25c, the image composition section 25d, the motion feature extraction section 25e, the abnormal operation notification section 25f, and the abnormality sign notification section 25g, respectively, to execute corresponding processes.

The determination model management unit 25a is a processing unit that manages the abnormal behavior model data 24a and the abnormality sign model data 24b. The abnormal operation model data 24a and the abnormal sign model data 24b may be generated by the determination model management section 25a, or may be acquired from outside and stored in the storage section 14.

Furthermore, if the observer who has confirmed the abnormality report or warning sign determines that the abnormal operation or warning sign does not apply, correction data is transmitted from the terminal device 30. The determination model management unit 25a modifies the abnormal behavior model data 24a and the abnormality sign model data 24b based on the modified data to improve the accuracy of the determination.

The observation management unit 25b is a processing unit that manages the operating state of the observation system by registering information on the room, the person to be observed, the imaging device 10, the observer, and the terminal device 30 in the observation management data 24c and updating it as appropriate. be.

The authentication processing unit 25c is a processing unit that authenticates the person to be observed. Specifically, characteristic data indicating the external appearance characteristics of the person to be observed is registered in the observation management data 24c, and the authentication processing unit 25c uses the image of the person included in the visible light image data received from the imaging device 10. The observed person is authenticated by detecting and comparing it with the characteristic data of the observation management data 24c.

The image synthesis section 25d is a processing section that synthesizes visible light image data and depth image data and generates an image to be output to the viewer. Specifically, an image is generated by replacing the area of the image of the observed person included in the visible light image data with depth image data. In this way, by replacing the image of the person to be observed with a depth image, it is possible to protect the privacy of the person to be observed while accurately indicating the condition of the person to be observed.

The motion feature extraction unit 25e is a processing unit that uses visible light image data and depth image data received from the imaging device 10 to extract motion features that indicate the motion characteristics of the observed person.

The abnormal behavior notification unit 25f uses the behavior feature extracted by the behavior feature extraction unit 25e and the abnormal behavior model data 24a to determine whether the behavior feature corresponds to an abnormal behavior. Then, when the operation characteristic corresponds to an abnormal operation, the terminal device 30 is notified of the abnormal operation (abnormality notification).

The abnormality sign notification unit 25g uses the movement feature extracted by the movement feature extraction unit 25e and the abnormality sign model data 24b to determine whether the movement feature corresponds to an abnormality sign. Then, when the operational feature corresponds to an abnormality sign, the terminal device 30 is notified of the abnormality sign (predictive notification).

In this way, the motion feature extracting section 25e, the abnormal motion notifying section 25f, and the abnormality sign notifying section 25g cooperate and operate as the extraction means described in the claims.

FIG. 5 is a functional block diagram showing the functional configuration of the terminal device 30 shown in FIG. 1. As shown in FIG. 5, the terminal device 30 includes a display section 31, an input section 32, a communication section 33, a storage section 34, and a control section 35.

The display unit 31 is a display device such as a liquid crystal display. The input unit 32 is an input device such as a button. The display section 31 and the input section 32 may be integrally formed using a touch panel display or the like. The communication unit 33 is a communication interface that communicates with the server device 20. The storage unit 34 can be used to store various data.

The control unit 35 is a control unit that controls the entire terminal device 30, and includes an observation output processing unit 35a, a notification output processing unit 35b, and a notification correction processing unit 35c. In reality, programs corresponding to these functional units are stored in a ROM or non-volatile memory (not shown), and by loading these programs into the CPU and executing them, the observation output processing unit 35a and the notification output processing unit 35b and the notification correction processing unit 35c are caused to execute corresponding processes, respectively.

The observation output processing section 35a is a processing section that outputs the state of the living room. Specifically, it is possible to display a list of the status of each room and display images of the rooms. The status of the room can include absence, no abnormality, occurrence of abnormality, presence of signs, etc.

The notification output processing unit 35b is a processing unit that, when receiving a notification of an abnormal operation or a sign of an abnormality from the server, makes the observer recognize the notification by performing display output or audio output. The observer confirms the contents of the notification and takes measures to deal with abnormal operations and signs of abnormality.

When the observer who has confirmed the abnormality report or predictive warning determines that the abnormal behavior or predictive warning does not apply, the notification correction processing unit 35c accepts a correction input, and generates correction data indicating the notification to be corrected and the content of the correction. The process of transmitting the information to the server device 20 is performed.

FIG. 6 is a flowchart showing the processing procedure of the server device 20. When visible light image data and depth image data are acquired from the imaging device 10 (step S101), the authentication processing unit 25c detects the image of the person included in the visible light image data and compares it with the characteristic data of the observation management data 24c. In this way, the person to be observed is authenticated (step S102).

If the authentication fails (step S103; No), the process ends because the person to be observed is absent. If the authentication is successful (step S103; Yes), the motion feature extraction unit 25e uses the visible light image data and the depth image data to extract motion features indicating the motion characteristics of the observed person (step S104).

The abnormal operation notification unit 25f uses the extracted operation feature and the abnormal operation model data 24a to determine whether the operation feature corresponds to an abnormal operation (step S105). If the operation feature corresponds to an abnormal operation (step S106; Yes), the abnormal operation notification unit 25f notifies the terminal device 30 of the abnormal operation (abnormality notification) (step S107).

If the motion feature does not correspond to an abnormal motion (step S106; No), the abnormality sign reporting unit 25g uses the extracted motion feature and the abnormality sign model data 24b to determine whether the motion characteristic corresponds to an abnormality sign. (Step S108). When the operation characteristic corresponds to an abnormality sign, the abnormality sign notifying unit 25g notifies the terminal device 30 of the abnormality sign (predictive notification) (step S110). If the motion feature does not correspond to an abnormal motion (step S109; No), the process ends.

After making the notification in step S107 or step S110, the determination model management unit 25a determines whether correction data has been received from the terminal device 30 (step S111). If the correction data is received (step S111; Yes), the determination model management unit 25a reflects the correction data in the abnormal behavior model data 24a and the abnormality sign model data 24b (step S112), and ends the process. Furthermore, if the correction data has not been received (step S111; No), the process is immediately terminated.

Next, a display example of the terminal device 30 will be explained. FIG. 7 is an explanatory diagram of a display example when no abnormal operation or sign of abnormality occurs. FIG. 7A is a list display screen, and the terminal device 30 displays a message "No abnormality" and a room selection button. "No abnormality" on the list display screen indicates that no abnormal operation or sign of abnormality has occurred in any room. Furthermore, the display mode of the room selection button, such as color, can be made different depending on whether the user is absent or when the user is present and there is no abnormality.

When the room selection button is operated on the list display screen, the screen shifts to the room image screen shown in FIG. 7(b). In 7(b), a message "No abnormality", room number "202", and an image of the living room are displayed. "No abnormality" on the room image screen indicates that no abnormal operation or signs of abnormality are occurring in the room. Furthermore, the image of the living room is an image generated by replacing the image of the observed person in the visible light image data with depth image data.

FIG. 8 is an explanatory diagram of a display example when an abnormal operation occurs. FIG. 8A is a list display screen, and the terminal device 30 displays a message "Fall has occurred" and a room selection button. "Fall Occurrence" on the list display screen indicates that a fall, which is one of the abnormal movements, has occurred in any room. Furthermore, the display correspondence of the room selection button can be changed depending on the presence or absence of an abnormal operation, and in FIG. 8(b), the button for "Room 202" has a display mode indicating an abnormal operation. It should be noted that notification can be made in the same way when an abnormality sign occurs, and the display mode of the room selection button can be made different depending on the abnormal operation and abnormality sign.

When the room selection button is operated on the list display screen, the screen shifts to the room image screen shown in FIG. 8(b). In 8(b), a message saying "Fall has occurred", a room number "202", an image of the living room, and a "Modify" button are displayed. "Fall Occurrence" on the living room image screen indicates that a fall, which is one of the abnormal movements, has occurred in the living room. Further, when an operation of the "correct" button is accepted, correction data indicating that the notification was inappropriate is generated and transmitted to the server device 20.

Next, reflection of modified data will be explained. FIG. 9 is an explanatory diagram of updating the abnormality sign model data using corrected data. In FIG. 9, as a result of the server device 20 extracting the operational characteristics and determining the abnormality sign, the server device 20 determines that the operational characteristics correspond to the abnormality sign, and notifies the terminal device 30 of the abnormality sign.

If the observer who has confirmed the abnormality sign notification determines that the abnormality sign does not apply, the observer inputs a correction into the terminal device 30. The terminal device 30 receives the modification input, generates modification data, and transmits it to the server device 20. The server device 20 updates the abnormality sign model data 24b using the corrected data. If the abnormality sign model data 24b is a learned model, the abnormality sign model data 24b can be updated by performing learning again using the corrected data as correct data. In addition, in FIG. 9, although the determination of an abnormal sign was explained as an example, the abnormal operation model data 24a can be updated in the same way.

FIG. 10 is an explanatory diagram of a case where correction data is referred to at the time of determination. In FIG. 10, as a result of the server device 20 extracting the operational characteristics and determining the abnormality sign, the server device 20 determines that the operational characteristics correspond to the abnormality sign, and notifies the terminal device 30 of the abnormality sign.

If the observer who has confirmed the abnormality sign notification determines that the abnormality sign does not apply, the observer inputs a correction into the terminal device 30. The terminal device 30 receives the modification input, generates modification data, and transmits it to the server device 20. After receiving the modified data, the server device 20 stores the modified data and the corresponding operational characteristics in association with each other. In subsequent abnormality sign determination, when it is determined that a motion feature corresponds to an abnormality sign based on the abnormality sign model data 24b, correction data associated with a motion feature similar to the motion feature is searched, and the corresponding motion feature is searched for. If there is corrected data indicating that the warning was not appropriate, the notification of the abnormality sign is stopped. Note that although FIG. 10 has been described using an example of determining an abnormal sign, the present invention can be similarly applied to determining an abnormal operation.

In the explanation so far, we have used an observation system in a welfare facility for the elderly and treated residents as the subjects to be observed. It is also possible to do so.

FIG. 11 is an explanatory diagram of the detention facility observation system. In FIG. 11, an imaging device 10 and a Doppler sensor 40 are installed in a detention facility. The imaging device 10 transmits a three-dimensional image in which visible light image data and depth image data are associated with each other to the server device 120. The Doppler sensor 40 is a device that performs sensing using microwaves, detects the heartbeat and breathing of the person being observed, and transmits the detected information to the server device 120.

The server device 120 identifies abnormal behavior using the observed person's movement obtained from the three-dimensional image and the observed person's biological information such as heartbeat and breathing, determines the degree of risk, and performs the following as necessary. A notification is sent to the terminal device 130 for the observer.

As described above, the observation system of this embodiment images the observed person, extracts motion characteristics from the image of the observed person, and when the motion characteristics correspond to abnormal motion, sends an image to the observer's terminal device. An abnormality report is issued, and if there is an abnormality sign in the operating characteristics, an alarm related to the abnormality sign is issued to the terminal device. Therefore, advance prevention and early response to emergencies can be realized.

Furthermore, the observation system of the present embodiment uses a motion that does not correspond to an abnormal motion and has a possibility of transitioning to an abnormal motion as an abnormality sign.

In addition, in the observation system of this embodiment, the observer's terminal device has a display unit, and when receiving an alarm regarding an abnormal sign, it receives information based on the image of the observed person and displays it on the display unit. can do.

Furthermore, in the observation system of this embodiment, the observer's terminal device has an input unit, and after receiving the alert, can accept an input as to whether the alert was appropriate.

In addition, in the observation system of this embodiment, the observer's terminal device, after receiving an input as to whether or not the alarm was issued, transmits information based on the input to the server device, and the server device transmits information based on the input. After receiving the information, the information based on the input and the operation characteristics corresponding to the input are stored in association with each other, and before another alarm is sent to the observer's terminal device, the information that was the source of the alarm is stored. An operation feature similar to an abnormality sign is searched from the stored operation features, and when information stored corresponding to the corresponding operation feature is not information corresponding to an appropriate alarm, the alarm is stopped.

In addition, the observation system of this embodiment controls the alarm using a trained model that has learned whether or not the motion characteristics correspond to abnormal motion and/or signs of abnormality, and uses information based on input to the learned model. It can be used for updating.

In addition, the observation system of this embodiment captures images using a three-dimensional camera that can acquire depth information, and authenticates whether or not the person being observed is suitable as an observation target based on the person's face image. be able to.

Note that each configuration illustrated in this embodiment is a functional schematic, and does not necessarily need to be physically configured as illustrated. In other words, the form of dispersion/integration of each device is not limited to the one shown in the diagram, but all or part of it can be functionally or physically distributed/integrated in arbitrary units depending on various loads and usage conditions. Can be configured.

For example, in this embodiment, a configuration in which the server device performs authentication has been described as an example, but a configuration in which the authentication of the observed person is performed by an imaging device may also be adopted. Further, in the present embodiment, the configuration has been described using as an example the configuration in which the server device extracts the motion characteristics, but a configuration may also be adopted in which the imaging device extracts the motion characteristics.

Furthermore, in this embodiment, the case where the motion characteristics are evaluated in consideration of the time series and the motion features that have the possibility of transitioning to abnormal motion are determined as an abnormality sign has been described as an example, but other indicators may also be used. It may also be used to determine signs of abnormality. For example, it is also possible to use a combination of the environment of the room and the condition of the person being observed to determine signs of abnormality. Specifically, if the living room is cluttered and the person being observed is using a cane, an early warning is issued as there is a risk of falling.

Alternatively, the behavior of the observed person may be accumulated for each individual, and a predictive warning may be issued when the observed person behaves for the first time or behaves differently than usual. There is a high possibility that these actions will be judged by the observer as being okay, but in that case, they can be reflected in subsequent judgments by corrective input.

Furthermore, although the present embodiment has been described using an example of a welfare facility for the elderly and a detention center, the present invention is not limited thereto, and can be widely applied to medical facilities, public facilities, and the like.

The observation system and observation method of the present invention are useful for realizing early response to the observed person.

10 Imaging device 11 Visible light camera 12 Depth sensor 13, 23, 33 Communication unit 14, 24, 34 Storage unit 14a Visible light image data 14b Depth image data 15, 25, 35 Control unit 15a Visible light image processing unit 15b Depth image processing Section 15c Transmission processing section 20 Server device 21, 31 Display section 22, 32 Input section 24a Abnormal operation model data 24b Abnormal sign model data 24c Observation management data 24d Image history data 24e Notification history data 25a Judgment model management section 25b Observation management section 25c Authentication processing unit 25d Image synthesis unit 25e Motion feature extraction unit 25f Abnormal operation notification unit 25g Abnormality sign notification unit 30 Terminal device 35a Observation output processing unit 35b Notification output processing unit 35c Notification correction processing unit

Claims (8)

an imaging means for capturing an image of the observed person; an extraction means for extracting a motion feature from the image of the observed person captured by the imaging means; and, when the extracted motion feature corresponds to an abnormal motion , An observation system comprising: an observer terminal that receives a notification regarding abnormal operation;
Among the abnormal signs in which the motion characteristics extracted by the extraction means are motions that have a possibility of transitioning to the abnormal motion, excluding those that have been determined not to transition to abnormal motion due to past notifications regarding abnormal motions. An observation system characterized in that, when the abnormality sign corresponds to an abnormality sign, a notification regarding the abnormality sign is issued to the observer terminal. The observer terminal has a display section,
2. The observation system according to claim 1 , wherein information based on an image of the observed person is received and displayed on the display unit when receiving the alarm regarding the abnormal sign. The observer terminal further includes an input section,
3. The observation system according to claim 1 , further comprising receiving an input as to whether or not the warning was appropriate after receiving the warning. The observer terminal receives the input as to whether or not the alarm was appropriate, and then transmits information based on the input to the extraction means,
The observation system further includes a storage means,
After receiving the information based on the input, the storage means stores the information based on the input in association with the operation characteristic corresponding to the input,
The extraction means searches the operation characteristics stored in the storage means for a motion characteristic similar to the abnormal sign that caused the alarm, before issuing another alarm to the observer terminal. 4. The observation system according to claim 3 , wherein the observation system stops issuing the alarm when the information stored corresponding to the corresponding operation feature is not information corresponding to appropriate issuing of the alarm. The extracting means controls the alarm using a trained model that has learned whether or not the operation feature corresponds to the abnormal operation and/or the abnormality sign, and uses the information based on the input to the learned model. 4. The observation system according to claim 3 , wherein the observation system is used for updating an already completed model. The imaging means is a three-dimensional camera capable of acquiring depth information, and the imaging means or the extraction means authenticates whether the observed person is suitable as an observation target based on the facial image of the observed person. The observation system according to any one of claims 1 to 5, characterized in that the observation system performs the following. an imaging means for taking an image of the observed person; a motion feature is extracted from the image of the observed person captured by the imaging means; and if the extracted motion feature has an abnormality sign, the abnormality sign is related to the abnormality sign; an extractor for issuing an alarm to an observer terminal; and an observer terminal for receiving an alarm regarding the abnormal operation from the extracting means when the extracted operation feature corresponds to an abnormal operation; An observation system comprising: a storage means;
The observer terminal receives an input as to whether or not the warning was appropriate, and then transmits information based on the input to the extraction means,
After receiving the information based on the input, the storage means stores the information based on the input in association with the operation characteristic corresponding to the input,
The extraction means searches the operation characteristics stored in the storage means for a motion characteristic similar to the abnormal sign that caused the alarm, before issuing another alarm to the observer terminal. An observation system characterized in that, if information stored corresponding to the corresponding operation feature is not information corresponding to an appropriate alarm, the observation system stops issuing the alarm. an imaging means for capturing an image of the observed person; an extraction means for extracting a motion feature from the image of the observed person captured by the imaging means; and, when the extracted motion feature corresponds to an abnormal motion , An observation method in an observation system including: an observer terminal that receives a notification regarding abnormal operation;
The extracted operation feature is an abnormality sign that is a movement that has a possibility of transitioning to the abnormal operation, excluding the abnormality sign that is determined not to transition to an abnormal operation due to a notification related to a past abnormal operation. An observation method comprising the step of issuing an alarm regarding the abnormality sign to the observer terminal , if applicable .

Images