JP2022048830A - Detection system, detection device, terminal device, display control device, detection method, and program - Google Patents

Abstract

To provide a technique capable of flexibly meeting various requests of a use.SOLUTION: A detection system comprises: a terminal device which receives a setting operation of a detection object event concerning an observation object person; and a detection device which detects the detection object event on the basis of a setting content by the setting operation. For example, by a setting operation using a setting screen 200, the detection object event is set as an event satisfying conditions concerning a three-dimensional position of a specific portion in the observation object person, variation in the three-dimensional position, and/or the variation speed of the three-dimensional position and the like.SELECTED DRAWING: Figure 20

Description

The present invention relates to a detection device for detecting a detection target event related to an observation target person and a technique related thereto.

There is a technique for detecting a detection target event (fall, etc.) related to an observation target person (watch target person, etc.).

For example, Patent Document 1 describes a technique for monitoring the state of a resident of a long-term care facility (also referred to as an observation target person), which is a monitoring target (watching target). The monitored state (also referred to as a detection target event) includes a fall of the care recipient (person requiring long-term care) in the living room, leaving the room (abnormal leaving), and the like.

In the technique of Patent Document 1, a program for detecting the specific operation (detection target event) to be monitored is prepared (see paragraph 0038 of Patent Document 1). Then, each specific action is detected by operating each program.

Japanese Unexamined Patent Publication No. 2017-228042

In the above-mentioned technique of Patent Document 1, it is necessary to prepare a program in advance for each operation to be detected. Therefore, the event to be detected is so-called fixed.

However, there are various demands in actual nursing care sites and the like. With the above-mentioned fixed method, it is difficult to flexibly respond to such various demands.

Therefore, the first object of the present invention is to provide a technique capable of flexibly responding to various demands of a user.

In addition, in the above-mentioned technology, the situation in the living room of the care recipient or the like may be monitored.

However, at the time of such monitoring, the fact that the photographed image of the person in the living room (also referred to as the person to be observed) is always displayed as it is on the screen for the administrator or the like is appropriately considered for the privacy of the person. It cannot be said that it is.

Therefore, the second object of the present invention is to provide a technique that appropriately considers the privacy of the person to be observed.

In order to solve the first problem, the detection system according to the present invention has a terminal device that accepts a setting operation of a detection target event related to an observation target person, and a detection that detects the detection target event based on the setting contents of the setting operation. It is characterized by being provided with an apparatus.

The detection target event may be set as an event that satisfies the condition regarding the three-dimensional position of the specific portion in the observation target person.

The detection target event may be set as an event that satisfies the condition regarding the change in the three-dimensional position of the specific portion in the observation target person.

The detection target event may be set as an event that satisfies the condition regarding the change speed of the three-dimensional position of the specific portion in the observation target person.

The detection target event may be set as an event that satisfies the conditions relating to the three-dimensional position of the specific portion and the change rate of the three-dimensional position of the specific portion in the observation target person.

The detection target event may be set as an event that satisfies the conditions relating to the change in the three-dimensional position of the specific portion and the change rate of the three-dimensional position of the specific portion in the observation target person.

The detection target event may be set as an event that satisfies at least a height condition among the three-dimensional positions of the specific portion.

The detection target event is set as an event that satisfies at least a condition regarding a plane position among the three-dimensional positions of the specific portion, and the plane position in the detectable space is divided into a plurality of areas and is related to the plane position. The condition may include a condition relating to a designated area from the plurality of areas.

The detection target event may be set as an event that satisfies the condition regarding the positional relationship between the specific portion of the observation target person and a predetermined object other than the observation target person.

The terminal device may display an image showing a plurality of specific parts relating to a person, and may accept an input for designating a desired specific part among the plurality of specific parts as the specific part.

The terminal device may accept a setting operation for setting each detection target event related to a plurality of rooms for each room.

The terminal device accepts a setting operation for setting the height information of the observation target person for each room, and the detection device excludes a person who does not match the height information from the detection target person and detects the detection target event. You may.

The terminal device may accept an operation of setting an alarm level according to the event to be detected among a plurality of alarm levels.

The detection device generates a skeleton image of the observation target person from a photographed image of the observation target person, and also generates an observation image of the observation target person based on the skeleton image, and the terminal device is the observation target. The observation image relating to the person may be displayed, and the observation image may be an image having no captured image of the observation target person itself and having the skeleton image of the observation target person.

The observation image may be an image generated by synthesizing the skeleton image with a background image in which a person does not exist.

The background image may be generated based on a captured image in which no person is present.

When a predetermined condition is satisfied, the terminal device may display a photographed image of the person to be observed in place of the observation image or together with the observation image.

The predetermined condition may include that the event to be detected has been detected.

The predetermined condition may include that the notification level set for the detection target event among the plurality of notification levels has a degree of importance equal to or higher than the predetermined level.

In order to solve the first problem, the terminal device according to the present invention performs the setting operation for the reception unit that accepts the setting operation of the detection target event related to the observation target person and the detection device that detects the detection target event. It is characterized by including a control unit for transmitting setting information based on the above.

In order to solve the first problem, the program according to the present invention has a) a step of accepting a setting operation of a detection target event related to an observation target person, and b) the setting of the detection device for detecting the detection target event. It is characterized in that it is a program for causing a computer to execute a step of transmitting setting information based on an operation.

In order to solve the first problem, the detection method according to the present invention is a detection method for detecting a detection target event related to an observation target person, and a) a step of accepting a setting operation of the detection target event related to the observation target person. , B) It is characterized by comprising a step of detecting the detection target event based on the setting content by the setting operation.

In order to solve the first problem, the detection device according to the present invention has a reception unit that accepts a setting operation of a detection target event related to an observation target person, and a control that detects the detection target event based on the setting content of the setting operation. It is characterized by having a part and.

In order to solve the second problem, the detection system according to the present invention includes a detection device for detecting a detection target event related to the observation target person and a display device for displaying an observation image of the observation target person. The detection device generates a skeleton image of the observation target person from the captured image of the observation target person, and also generates the observation image of the observation target person based on the skeleton image, and the observation image is the observation target. The image is characterized in that it does not have a photographed image of the person itself and has the skeleton image of the person to be observed.

In order to solve the second problem, the display control device according to the present invention is a display control device that controls the display on the display unit in the detection system that detects the detection target event related to the observation target person, and is the observation target. Control to generate a skeleton image of the observation target person from a photographed image of the person, generate an observation image of the observation target person based on the skeleton image, and display the observation image of the observation target person on the display unit. The observation image is characterized by having a captured image of the observation target person itself and having the skeleton image of the observation target person.

According to the inventions of claims 1 to 23, it is possible to flexibly respond to various requests of the user.

According to the inventions of claims 24 to 25, it is possible to realize a display that gives due consideration to the privacy of the person to be observed.

It is a figure which shows the schematic structure of the detection system. It is a functional block diagram which shows the schematic structure of the detection device (image processing device). It is a functional block diagram which shows the schematic structure of a management device. It is a functional block diagram which shows the schematic structure of the mobile terminal apparatus. It is a flowchart which shows the flow of processing in a detection device. It is a conceptual diagram which shows the generation of 3D position information about each specific part of a person. It is a figure which shows "getting up from a bed". It is a figure which shows "the end sitting position". It is a figure which shows "getting out of bed". It is a figure which shows "fall on the floor". It is a figure which shows "lying on the floor". It is a figure which shows "the whole body slides off from a bed". It is a figure which shows "the lower body slides off from a bed". It is a figure which shows "one-handed extension from a bed". It is a figure which shows the selection screen which selects the room to be displayed. It is a figure which shows the observation screen of the observation target person in a certain room. It is a figure which shows the observation screen of the observation target person in a certain room. It is a figure explaining the generation process of the observation image. It is a figure which shows the setting screen (registration screen) of the event to be detected. It is a figure which shows the setting screen of the event to be detected "get up (on the bed)". It is a figure which shows the setting screen of the detection target event "end sitting position (bed)". It is a figure which shows the setting screen of the detection target event "end sitting position (bed)". It is a figure which shows the setting screen of the detection target event "getting out of bed". It is a figure which shows the setting screen of the detection target event "fall (on the floor)". It is a figure which shows the setting screen of the event to be detected "lying (on the floor)". It is a figure which shows the setting screen of the detection target event "whole body slip from a bed". It is a figure which shows the setting screen of the detection target event "lower body slipping from a bed". It is a figure which shows the setting screen of the event to be detected "one hand extension from a bed". It is a figure which shows the setting screen of the event to be detected "one hand extension from a bed". It is a flowchart which shows the process in a management apparatus. It is a figure which shows "the fall on the bed". It is a figure which shows the setting screen of the detection target event "fall".

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. System overview>
FIG. 1 is a diagram showing a detection system 1 according to the present invention. As shown in FIG. 1, the detection system 1 includes a plurality of detection devices 10 and a plurality of terminal devices 70 and 80. The terminal device 70 is also referred to as a management device 70, and the terminal device 80 is also referred to as a mobile terminal device 80. The detection system 1 is a system that detects a detection target event (“fall” or the like) related to an observation target person (care recipient or the like). Since the detection device and the detection system watch over the behavior of a person, they are also referred to as a watching device and a watching system.

Each detection device 10 and each terminal device 70, 80 are connected to each other via a network 108. The network 108 is composed of a LAN (Local Area Network), the Internet, and the like. Further, the connection mode to the network 108 may be a wired connection or a wireless connection. For example, the management device 70 is wiredly connected to the network 108, and each detection device 10 and each mobile terminal device 80 are wirelessly connected to the network 108. Alternatively, all devices 10, 70, 80 may be wirelessly connected to the network 108.

In the detection system 1, various input / output processes (operation input process, display process, etc.) and the like related to each detection device 10 are performed by using the management device 70 and the mobile terminal device 80. That is, the management device 70 and the mobile terminal device 80 each function as terminal devices related to the plurality of detection devices 10.

In the following, a mode in which the detection system 1 is used in a long-term care facility will be mainly exemplified. The detection system 1 is not limited to this, and may be used in a nursing facility (hospital or the like) or a factory.

Each detection device 10 is arranged in each living room 90 (for example, a private room of each care recipient) of each observation target person (here, the care recipient). Each detection device 10 is a device that detects a detection target event (“fall” or the like) related to the observation target person based on a photographed image or the like related to the observation target person. As will be described later, the "detection target event" is set (defined) by the setting operation in the terminal device 70. Then, the detection device 10 detects the event to be detected based on the setting content of the setting operation.

Each detection device 10 includes a camera unit 20 and a processing unit 30. The camera unit 20 is installed on the ceiling or the like of the living room 90, and can photograph the inside of the living room 90. The processing unit 30 is a three-dimensional position of a plurality of specific parts (chest, waist, shoulders, elbows, wrists, knees, ankles, etc.) of the person to be observed based on an image (particularly a moving image) taken by the camera unit 20. (Especially the time series information) can be obtained. Here, the camera unit 20 and the processing unit 30 are provided separately, but the present invention is not limited to this, and the camera unit 20 and the processing unit 30 may be provided integrally.

The camera unit 20 is a three-dimensional camera. The camera unit 20 acquires a captured image with depth information. Specifically, the camera unit 20 captures a captured image (infrared image, etc.) 110 (see FIG. 6) of a subject object (person, wall, floor 91, bed 92, etc.) and each of the captured images 110. Acquires the depth information (depth distance information) 120 of the pixel. The depth information 120 of each pixel in the captured image is information on the distance (distance from the camera unit 20) to the subject object (person, wall, floor, bed, etc.) corresponding to each pixel in the captured image 110. This is information on the distance in the direction perpendicular to the captured image 110. In other words, the depth information is distance information (depth distance information) in the normal direction of the captured image plane.

For example, the camera unit 20 includes a sensor unit 23 (FIG. 2), and the sensor unit 23 includes an infrared camera (infrared image sensor or the like) and an infrared projector. The infrared camera acquires a photographed image 110 (for example, an infrared image) of the subject object. Further, the infrared projector irradiates a predetermined dot pattern by infrared rays toward the target object, and the infrared camera also acquires an infrared image obtained by capturing the dot pattern irradiated on the target object. The distance (depth distance information) 120 to the target object is calculated (acquired) by utilizing the fact that the geometric shape of the captured dot pattern changes depending on the distance to the target object. The calculation process of the depth distance information is executed by the controller 21 or the like incorporated in the camera unit 20. The controller 21 has the same hardware configuration as the controller 31 (described later) and the like. The sensor unit 23 may be provided with an RGB image sensor or the like for capturing a visible light image (color image or the like), and a color image may be captured.

In this way, the camera unit 20 has a photographed image (photographed image information) 110 (see FIG. 6) of the subject person and depth information of each pixel in the photographed image (information on the distance to an object corresponding to each pixel (see FIG. 6). Depth distance information)) 120 and is acquired.

FIG. 6 is a conceptual diagram showing that information on the three-dimensional position of each specific portion of the subject person is acquired (calculated) based on the captured image 110 and the depth information 120. In FIG. 6, the captured image 110, the depth information 120, and the like are schematically shown. For example, in the actual captured image 110, the person is captured as it is captured, whereas in the captured image 110 of FIG. 6, the person is represented (simplified) by a figure in which ellipses are connected. There is. The same applies to the other depth information 120. Further, the actual depth information 120 has information on the distance to the object corresponding to each pixel, whereas in the depth information 120 of FIG. 6, the distance to the object corresponding to each pixel is each pixel. It is expressed in terms of the concentration of. In detail, the magnitude of the distance is expressed by the magnitude (shading) of the density.

First, the processing unit 30 (particularly the controller 31) analyzes the captured image 110 and acquires the skeleton information (skeleton model information) 140 (see the lower left portion of FIG. 6) of the subject person. The skeleton information 140 includes a plurality of specific parts (specifically, chest, nose, shoulders, elbows, wrists, hips, knees, ankles, eyes, ears, etc.) (mainly joints) and the plurality of specific parts of the subject person. It is information that expresses (simplified) the skeleton of the person concerned by using the skeleton line (link) that connects the above. In the skeleton information 140 of FIG. 6, each specific part is indicated by a "point", and each skeleton line (connecting line) is indicated by a "line segment". The skeleton information 140 based on the captured image has information on a plurality of specific parts (here, B0 to B17) (planar position information in the image of each part, etc.).

Further, the processing unit 30 is based on the skeleton information 140 of the subject person, and the plane position information (2 in the photographed image in the camera coordinate system) in the photographed image 110 of each specific part (for example, each representative position) of the subject person. Dimensional position information) is acquired.

Further, the processing unit 30 has distance information (camera) from the camera unit 20 to each specific portion based on the depth information 120 of one or a plurality of pixels in the plane position (in the captured image 110) corresponding to each specific portion. Depth position information in the coordinate system) is also acquired. The distance information is also expressed as depth information in the normal direction of the captured image plane.

In this way, the processing unit 30 has information on the plane position in the captured image of each specific portion of the subject person (two-dimensional position information in the captured image) and distance information (depth information) to each specific portion in the subject person. ) And get.

Then, the processing unit 30 is based on the information on the plane position in the captured image of each specific part of the subject person and the distance information (depth information) to the specific part, and the three-dimensional position information 150 of each specific part. (3D position information in the living room space) is acquired by coordinate conversion or the like. Specifically, the processing unit 30 uses the coordinate system Σ2 fixed to the living room to provide position information (planar position in the captured image and depth position in the normal direction of the captured image) in the camera coordinate system Σ1. Is converted into the three-dimensional position information 150 of. The camera coordinate system Σ1 is, for example, a three-dimensional Cartesian coordinate system based on three orthogonal axes of two orthogonal axes parallel to the captured image plane and one axis extending in a direction perpendicular to the captured image plane. Further, the coordinate system Σ2 after conversion is, for example, a three-dimensional orthogonal coordinate system based on three orthogonal axes of two orthogonal axes parallel to the horizontal plane and one axis extending in the vertical direction (height direction).

In this way, the processing unit 30 acquires (calculates) information on the three-dimensional position of each specific portion of the subject person. Here, the subject person in the captured image is, in principle, regarded as the person to be observed. When there are a plurality of subject persons in the field of view of the camera, all the subject persons may be specified as the observation target persons. However, the present invention is not limited to this, and a desired observation target object may be specified from a plurality of subject persons using information such as height (or using face recognition technology or the like).

Here, the controller 31 of the processing unit 30 generates the skeleton information 140, the three-dimensional position information 150 of each specific part, and the like, but the present invention is not limited to this.

For example, the controller 21 of the camera unit 20 may generate the skeleton information 140. Further, the controller 21 of the camera unit 20 may generate the three-dimensional position information 150 of each specific portion. Specifically, the controller 21 acquires information on the position of each specific part of the subject person in the captured image and distance information (depth information) to each specific part, and obtains three-dimensional position information 150 of each specific part. It may be acquired by coordinate conversion or the like. Then, the controller 31 of the processing unit 30 may acquire the three-dimensional position information 150 of each specific portion from the camera unit 20. Alternatively, both the controllers 21 and 31 of the camera unit 20 and the processing unit 30 may cooperate to execute these various processes. In other words, the skeleton information 140 and the three-dimensional position information 150 can be generated by both or one of the controllers 21 and 31.

Further, here, a three-dimensional camera based on a pattern irradiation method that acquires depth information of each pixel based on the irradiation of a dot pattern is exemplified, but the present invention is not limited to this. The three-dimensional camera may be a stereo viewing type camera, a TOF (Time of Flight) type camera, or the like. It suffices to acquire a photographed image of the person to be observed and depth information (depth information of each pixel, etc.) which is information on the distance to the subject object in the photographed image by a three-dimensional camera of an arbitrary method.

Further, since the detection device 10 is a device that executes image processing related to a captured image or the like, it is also expressed as an image processing device. The detection system 1 is also referred to as an image processing system or the like.

<2. Detection device 10>
FIG. 2 is a functional block diagram showing a schematic configuration of the detection device 10.

The detection device 10 includes a camera unit 20 and a processing unit 30 as shown in the functional block diagram of FIG.

As described above, the camera unit 20 includes an infrared camera, an infrared projector, and the like.

Further, the processing unit 30 includes a controller 31, a storage unit 32, a communication unit 34, and an operation unit 35.

The controller 31 is a control device built in the processing unit 30 and controls the detection device 10.

The controller 31 is configured as a computer system including a CPU (Central Processing Unit) (also referred to as a microprocessor or a hardware processor) and the like. The controller 31 executes various software programs (hereinafter, also simply referred to as programs) stored in the storage unit (non-volatile storage unit such as ROM and / or hard disk) 32 in the CPU. Realize the processing. The program (specifically, a program module group) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the detection device 10. Alternatively, the program may be downloaded via a communication network or the like and installed in the detection device 10.

The controller 31 acquires photographed image information about the person to be observed from the camera unit 20, and obtains distance information (depth information) from the camera unit 20 to each specific part of the person to be observed (subject person) from the camera unit 20. get. Then, the controller 31 acquires the three-dimensional position information of each specific portion based on the captured image information (particularly, the plane position information of each specific portion of the subject in the captured image) and the distance information (depth information of the subject). ..

Further, the controller 31 detects the "detection target event" (described later) based on the three-dimensional position information of each specific portion. Specifically, the controller 31 sets the "detection target event" (described later) according to the setting operation accepted by the terminal devices 70 and 80. Then, the controller 31 detects the event to be detected based on the setting content by the setting operation.

Further, the controller 31 controls a process of displaying a monitoring screen or the like regarding the person to be observed on the terminal devices 70, 80. Specifically, the controller 31 generates image data or the like of the observation image 330 (FIG. 18) as described later, and transmits the image data or the like to the terminal devices 70 and 80. Accordingly, the terminal devices 70 and 80 display the observation image 330 based on the image data and the like.

The storage unit 32 is composed of a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 32 stores (stores) time-series data and the like of three-dimensional positions of a plurality of specific parts related to the person to be observed.

The communication unit 34 can perform network communication via the network 108. In this network communication, for example, various protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol) are used. By using the network communication, the detection device 10 can exchange various data with and from a desired destination (for example, terminal devices 70 and 80).

The operation unit 35 includes an operation input unit 35a for receiving an operation input to the detection device 10 and a display unit 35b for displaying and outputting various information. In the system 1, the terminal devices 70 and 80 mainly exhibit the input / output functions (operation input reception function and output function (display function, etc.)) related to the detection device 10. Therefore, the operation unit 35 may be of a type that is attached to the processing unit 30 only at the time of maintenance or the like.

<3. Terminal devices 70, 80>
Next, the configurations of the terminal devices 70 and 80 will be described.

Of the terminal devices 70 and 80, the management device 70 is a device that manages the entire detection system 1, and is mainly operated by an administrator. On the other hand, the mobile terminal device 80 functions as a display device or the like that displays various information in the detection system 1. The mobile terminal device 80 is carried by a person (caregiver) who cares for the care recipient, and displays various information about the care recipient and the like. The management device 70 also functions as a display device for displaying various information in the detection system 1. Further, the management device 70 also functions as a setting reception device that accepts a setting operation by the user regarding a detection target event (described later) or the like.

The terminal devices 70 and 80 are information input / output terminal devices (information processing devices) capable of network communication with other devices (10 and the like). The terminal devices 70 and 80 are configured as a smartphone, a tablet type terminal, a personal computer (either a fixed type (stationary type) or a portable type) or the like. For example, the terminal device 70 is a fixed personal computer, and the terminal device 80 is a portable device (portable terminal device), more particularly a smartphone.

First, the management device 70 will be described.

FIG. 3 is a functional block diagram showing a schematic configuration of the management device 70.

As shown in the functional block diagram of FIG. 3, the management device 70 includes a controller 71, a storage unit 72, a communication unit 74, an operation unit 75, and the like, and various types can be obtained by operating each of these units in a complex manner. Realize the function.

The controller (control unit) 71 is a control device that controls the management device 70.

The controller 71 has the same hardware configuration as the controller 31. Further, the controller 71 realizes various processes by executing a predetermined software program stored in the storage unit (nonvolatile storage unit such as ROM and / or hard disk) 72 in the CPU. The program (specifically, a program module group) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the management device 70. Alternatively, the program may be downloaded via a communication network or the like and installed in the management device 70.

The controller 71 executes the program and executes various processes such as the following.

The controller 71 cooperates with the operation unit 75 to receive various setting operations and the like. For example, the controller 71 accepts a setting operation of a detection target event related to an observation target person. More specifically, the controller 71 generates a setting screen 200 for a detection target event related to the observation target person, displays the setting screen 200 on the display unit 75b, and accepts a user operation using the setting screen 200. Further, the controller 71 transmits the setting contents by the setting operation to the detection device 10, and causes the detection device 10 to execute the registration process (setting process) for registering the detection target event.

Further, the controller 71 is based on the display image information (observation image 330 (see FIGS. 16 and 17) (described later) and the like) transmitted from the detection device 10, and information about the observation target person (skeleton image 325 (described later)). ) Etc.) are displayed on the display unit 75b.

The storage unit 72 has the same hardware configuration as the storage unit 32. The storage unit 72 (particularly the non-volatile storage unit) stores the setting contents (setting contents and the like related to the event to be detected) by the management device 70. The storage unit 72 stores (stores) three-dimensional position information and the like (time-series data and the like of three-dimensional positions of a plurality of specific parts relating to the observation target person) acquired by the detection device 10 and transferred from the detection device 10. You may.

The communication unit 74 has the same hardware configuration as the communication unit 34. By using the network communication by the communication unit 74, the management device 70 can exchange various data with and from a desired destination (for example, the devices 10 and 80).

The operation unit 75 includes an operation input unit 75a for receiving an operation input to the management device 70, and a display unit 75b for displaying and outputting various information. A mouse, keyboard or the like is used as the operation input unit 75a, and a display (liquid crystal display or the like) is used as the display unit 75b.

Next, the mobile terminal device 80 will be described.

FIG. 4 is a functional block diagram showing a schematic configuration of the mobile terminal device 80.

As shown in the functional block diagram of FIG. 4, the mobile terminal device 80 includes a controller 81, a storage unit 82, a communication unit 84, an operation unit 85, and the like, and various types can be obtained by operating each of these units in a complex manner. To realize the function of.

The controller 81, the storage unit 82, the communication unit 84, and the operation unit 85 each have the same hardware configuration as the controller 71, the storage unit 72, the communication unit 74, and the operation unit 75.

The controller (control unit) 81 is a control device (also referred to as a controller) that controls the mobile terminal device 80.

The controller 81 realizes various processes by executing a predetermined software program stored in the storage unit (nonvolatile storage unit such as ROM and / or hard disk) 82 in the CPU. The program (specifically, a program module group) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the portable terminal device 80. Alternatively, the program may be downloaded via a communication network or the like and installed in the mobile terminal device 80.

The controller 81 executes the program and executes various processes such as the following.

For example, the controller 81 is based on display image information (observation image 330 (see FIGS. 16 and 17) (described later) and the like) transmitted from the detection device 10, and information about an observation target person (skeleton image 325 (described later)). ) Etc.) are displayed on the display unit 85b.

The mobile terminal device 80 can exchange various data with and from a desired destination (for example, each device 10, 70) by using the network communication by the communication unit 84.

The mobile terminal device 80 is provided with a touch panel 87 (see FIG. 1). The touch panel 87 is configured by embedding various sensors and the like in a liquid crystal display panel, and is capable of displaying various information and receiving various operation inputs from an operator (user). The touch panel 87 functions as a part of the operation input unit 85a and also as a part of the display unit 85b.

<4. Event to be detected>
Next, the event to be detected will be described.

In the detection system 1, it is possible to detect various detection target events related to the observation target person. The "detection target event" is an event (event to be detected) that should be detected with respect to the observation target person. Specifically, the "detection target event" includes at least one of the state of the person and the movement of the person.

"Events to be detected" include, for example, "getting up from the bed", "sitting on the edge", "getting out of bed", "falling on the floor", "lying on the floor", "sliding the whole body from the bed", "" Examples include "sliding the lower half of the body from the bed" and "stretching one hand from the bed". 7 to 14 are diagrams showing these detection target events (events E1 to E8), respectively.

Event E1 "Getting up from bed" (see FIG. 7) is an action in which the upper body of the observation target person (who had a recumbent position) gets up on the bed. The event E1 is, for example, a condition that the chest height (chest height from the upper surface of the bed) of the observation target person is increased by 30 cm or more in the bed area R1 (the plane area where the bed is arranged). Detected when satisfied.

Event E2 "edge sitting position" (see FIG. 8) is a state in which the observation target person is sitting (having a sitting position) at the bed edge. The "end sitting position" is one of the events "boundary position" that occurs near the boundary between the bed area R1 and the adjacent area. In the "end sitting position", for example, the terminal part (ankle, etc.) of the lower body of the observation target person moves from the bed area R1 to the bedside area R2, and some other parts (waist, etc.) of the observation target person. Is detected when the condition that is left in the bed area R1 is satisfied. The "end sitting position" is a compound event of the movement of both ankles of the observation target person from the bed area R1 to the bedside area R2 and the state in which the waist of the observation target person exists in the bed area R1. , Also expressed.

Event E3 "getting out of bed" (see FIG. 9) is an action in which the person to be observed moves away from the bed to a place other than the bed. "Getting out of bed" is detected when, for example, the condition that the whole body (all specific parts) of the observation target person has moved from the bed area R1 to the bedside area R2 is satisfied.

Event E4 "falling on the floor" (see FIG. 10) is an action in which the person to be observed falls on the floor. "Falling on the floor" means, for example, that a specific part (chest, etc.) of the person to be observed is at a predetermined speed (1 m / s, etc.) or higher in a flat area other than the bed area R1 and a predetermined height (50 cm, etc.) on the floor surface. It is detected when the condition of entering (space) below is satisfied.

The events E1, E2, E3, and E4 may occur in this order. In order to prevent the occurrence of the most dangerous event E4 among these events E1 to E4, it is preferable that the events E1, E2, and E3 that occur before the event E4 are detected (sequentially). By detecting each event E1, E2, E3 and notifying the caregiver or the like, it is possible to prevent the occurrence of the dangerous event E4.

Event E5 "Lying on the floor" (see FIG. 11) is a state in which the person to be observed is lying on the floor for a predetermined period or longer (having a lying position). "Lying on the floor" means, for example, a space where a specific part of the upper body (nose, chest, etc.) is at a predetermined height (for example, 50 cm) or less from the floor surface outside the range of the bed area R1 for a predetermined time (for example, 5 seconds) or more. It is detected when the condition to stay is satisfied.

Event E6 "whole body sliding off the bed" (see FIG. 12) is an action in which the whole body of the observation target person slides down from the bed to the bedside. "Whole body slip from bed" is detected when the following conditions are met. The condition is that, for example, the positions of a plurality of major parts (such as the chest of the upper body and the waist of the lower body) at a predetermined speed (for example, 1 m / s) or more from the bed area R1 to the bedside area R2 are at a predetermined height (for example). It is to move to a space of 50 cm) or less.

Event E7 "sliding the lower body from the bed" (see FIG. 13) is an action in which the lower body of the person to be observed slides down from the bed to the bedside. "Lower body slippage from bed" is detected when the following conditions are met. The condition is that, for example, the position of the terminal portion (waist, etc.) of the lower body of the observation target person at a predetermined speed (for example, 1 m / s) or more from the bed area R1 to the bedside area R2 is a predetermined height (for example, 50 cm). It is to move to the following space.

Event E8 "stretching one hand from the bed" (see FIG. 14) is a state in which one hand of the person to be observed is stretched to the bedside. "One-handed extension from the bed" is one of the events "boundary position" that occurs near the boundary between the bed area and the other area.

"Extending one hand from the bed" means that, for example, the part (wrist) of one hand of the person to be observed has moved from the bed area R1 to the bedside area R2, and the waist of the person to be observed remains in the bed area R1. Is detected when the condition of is satisfied. In "extending one hand from the bed", one wrist of the observation target person moves from the bed area R1 to the bedside area R2, and the waist of the observation target person exists in the bed area R1. It is also expressed as a compound event.

Event E8 is an event that may lead to the occurrence of dangerous events E5 to E7, and is preferably detected before the occurrence of these events E5 to E7. By detecting the event E8 and notifying the caregiver or the like, it is possible to prevent the occurrence of dangerous events E5 to E7.

<5. Processing flow>
FIG. 5 is a flowchart showing a processing flow in the detection device 10 (particularly the processing unit 30). The processing in the detection system 1 will be described with reference to FIG.

First, the management device 70 accepts a setting operation of a detection target event related to the observation target person. Specifically, the management device 70 displays the setting screen 200 on the display unit 75b, and receives an operation input from an operation user (administrator or the like). The management device 70 transmits the setting information regarding the operation input from the operation user to the detection device 10. The detection device 10 sets (registers) a "detection target event" based on the received setting information (step S1). The setting operation and the like using the management device 70 will be described later.

After that, when the user gives an instruction to start the observation process (detection process, etc.), the detection device 10 performs the process (detection process, etc.) in steps S11 to S15 for a very short time (for example, as shown in FIG. 5). Repeat every few tens of milliseconds to hundreds of milliseconds). Specifically, a detection process or the like for detecting a detection target event related to the observation target person is executed. In other words, the event to be detected is detected based on the information of the three-dimensional position in the time series regarding the plurality of specific parts of the person to be observed.

In step S11, the detection device 10 obtains the skeleton information 140 of the observation target person and the three-dimensional position information 150 of a plurality of specific parts of the observation target person based on the captured images (110, 120) with depth information regarding the observation target person. To get.

For example, as described above, the detection device 10 has skeleton information of the observation target person based on the captured image with depth information (particularly, the captured image relating to the observation target person) (110, 120) acquired by the camera unit 20. Get 140. Further, the detection device 10 acquires the plane position information (information about the two-dimensional position) in the captured image of each specific part of the observation target person and the distance information (depth information) to each specific part of the observation target person. do. Then, the detection device 10 acquires the three-dimensional position information 150 (three-dimensional position information in the living room space) of each specific portion by coordinate conversion or the like based on these information.

In step S12, the detection device 10 generates an image for monitoring display (also referred to as an observation image). Specifically, the detection device 10 generates a skeleton image 325 (see FIG. 18) (described later) of the observation target person from the captured image 110 of the observation target person, and observes the observation target person based on the skeleton image 325. Generate image 330. The observation image 330 is an image that does not have a photographed image of the observation target person itself and has a skeleton image of the observation target person.

Then, the detection device 10 transmits image data related to the observation image 330 to the terminal devices 70 and 80 (specifically, a device that sends a monitoring display request (described later) to the detection device 10), and the observation image 330 is transmitted to the terminal. It is displayed on the display unit of the device 70 or the like (step S12). According to this, the observed image 330 of the observation target person is not monitored and displayed as it is, but the observation image 330 of the observation target person is monitored and displayed. Therefore, it is possible to realize a display that gives due consideration to the privacy of the person to be observed.

Further, in step S12, the time-series observation image 330 is stored in the storage unit 32 as a log image (recording image).

In step S13, the detection device 10 executes detection processing of a plurality of detection target events (for example, events E1 to E8) based on three-dimensional position information of a plurality of specific parts of the observation target person. The detection device 10 detects each detection target event based on the setting content (content set in step S1) by the setting operation received by the management device 70 or the like.

In step S14, it is determined whether or not any of the detection target events has been detected.

If none of the detection target events is detected, the process returns to step S11.

On the other hand, if any of the detection target events is detected, the process proceeds to step S15. In step S15, processing (reporting processing, etc.) according to the detection result is performed. For example, when the event E4 "fall on the floor" is detected, a "warning" is issued to the terminal devices 70, 80 and the like. For example, warning sounds are output from the terminal devices 70 and 80, and warning screens including characters such as "Tumble in room 101!" Are displayed on the display units 75b and 85b. This allows the caregiver or the like to be aware of the occurrence of event E4 "fall on the floor".

<6. Setting operation, etc.>
FIG. 30 is a flowchart showing a process of the management device 70 (a process of accepting a setting operation related to a detection target event, etc.). The process of step S1 (FIG. 5) in the detection device 10 is performed in accordance with the process of FIG. 30 in the management device 70.

In step S31, the management device 70 displays the setting screen 200 (see FIG. 19) on the display unit 75b in response to a predetermined operation of the operating user (administrator or the like). The setting screen (also referred to as a registration screen) 200 is a screen for setting (registering) a detection target event related to the observation target person. Then, the management device 70 accepts the setting operation of the detection target event regarding the observation target person by using the setting screen 200.

Next, in step S32, the management device 70 transmits the setting information based on the setting operation received in step S31 to the detection device 10 (processing unit 30). The detection device 10 sets (registers) a detection target event based on the setting information received from the management device 70 (see step S1 in FIG. 5). In other words, the event to be detected is customized and registered according to the user's request. The setting information transmitted from the management device 70 to the detection device 10 may be the information itself received by the management device 70 or the information processed by the management device 70.

In such processing, a certain detection target event satisfies the condition regarding the three-dimensional position (particularly the plane position (particularly the plane areas R1, R2 (described later) and / or the height)) of the specific part in the observation target person. Set as an event. In particular, a certain detection target event is set as an event that satisfies the condition regarding the positional relationship between the specific part of the observation target person and a predetermined object (bed or the like) other than the observation target person.

In addition, a certain detection target event is an event that satisfies the conditions related to the change in the three-dimensional position of a specific part in the observation target person (movement of the plane position (particularly movement between plane areas) and / or height change, etc.). Is set as. Further, a certain detection target event is set as an event that satisfies the condition regarding the change speed (movement speed, etc.) of the three-dimensional position of the specific part in the observation target person.

Further, each detection target event may be set as an event that satisfies the conditions relating to any combination of the "three-dimensional position", the "change in the three-dimensional position", and the "change speed of the three-dimensional position".

Hereinafter, the setting operation (step S31) using the setting screen 200 will be described.

FIG. 19 is a diagram showing a setting screen 200.

The setting screen 200 has a room designation area 201, a person information registration button 202, an area setting area 210, a skeleton information display area 220, a rule designation area 230, a rule editing area 240, an alarm level designation area 280, a registration button 291 and a cancel button. It is equipped with 292.

The room designation area 201 is an area for designating a room (living room) to be set. The operation user specifies a desired room (for example, "Room 101") from a plurality of rooms ("Room 101", "Room 102", "Room 103" ...) displayed in the drop-down list. It is possible. Then, the operating user can set one or a plurality of detection target events related to the designated room.

The person information registration button 202 is a button for registering information on a person (person to be observed) who lives in the room. When the person information registration button 202 is pressed, an information registration screen (not shown) of the person to be observed for each room is displayed in a pop-up. By using the information registration screen or the like, a setting operation for setting information (for example, height information) of the person to be observed is accepted.

The area setting area 210 includes an area addition button 211, an area deletion button 212, an area selection area 213, and a camera image display area 215.

Here, the plane position (specifically, a part or all of the plane position) in the room (specifically, in the detectable space thereof) is a plurality of areas (planar area) by the setting operation using the area setting area 210. It is divided into R1, R2, .... As will be described later, the condition regarding the plane position among the three-dimensional positions of each specific part of the observation target person is set by designating a desired area among a plurality of areas.

The camera image display area 215 is an area for displaying an image captured by the camera unit 20 regarding the detection target space as an area designation image. For example, a captured image is displayed in a situation where the care recipient does not exist. In addition, the present invention is not limited to this, and a schematic graphic image or the like obtained by processing the captured image may be displayed as an area designation image.

The area addition button 211 is a button for adding an area, and the area deletion button 212 is a button for deleting an area. Further, the area selection area 213 is an area in which the area names (“R1”, “R2”, etc.) of the generated one or a plurality of areas are displayed. The operation user can select the area to be edited from the areas displayed in the area selection area 213.

For example, when the area addition button 211 is first pressed, a new name "R1" is added to the area selection area 213, and a hatch area indicating the area R1 is displayed in the camera image display area 215. The hatched area is superimposed and displayed on the camera image (captured image). The operating user can set the position and size of the area R1 by appropriately shaping the position and size of the hatched area by operating the mouse or the like.

The height from the floor of each area (a constant value within the area) is set for each area. In detail, the height of each area is automatically set to the height of the three-dimensional position of the object corresponding to the position of the center of gravity (center) of the hatching region. For example, the height of the area R1 arranged on the bed is automatically set to the height (measured value) of the upper surface of the bed (50 cm from the floor surface, etc.). However, the height of the area R1 is not limited to this, and may be specified by the operating user.

Further, the boundary position (planar position) of each area is automatically converted and registered to the coordinate position in the coordinate system Σ2 in the three-dimensional space fixed in the room. For example, the area R1 is registered as a rectangular area (a rectangular area having two points (0,100) and (80,280) as diagonal vertices (counter-vertices)) in a horizontal plane parallel to the floor surface.

In addition, the present invention is not limited to this, and the coordinate values of the diagonal vertices (two points) of each area having a rectangular area may be input by the operation user, and the position and size of each area may be specified. For example, the coordinate values (0,100,50) and (80,280,50) of the two points in the three-dimensional coordinate system Σ2 fixed to the room may be input. The Z coordinate value (“50”) may or may not be specified.

The skeleton information display area 220 is an area for displaying an image showing skeleton information of a person (human). The skeletal information of a person includes information on a plurality of specific parts of the person (chest, waist, shoulders, elbows, wrists, knees, ankles, etc.). In the skeleton information display area 220, each specific part of a human being is shown together with a designated number thereof. Specifically, each part of "nose", "chest", "right shoulder", "right elbow", "right wrist", "left shoulder", "left elbow", and "left wrist" is the respective part number. It is shown together with "B0" to "B7". In addition, each part of "right hip", "right knee", "right ankle", "left hip", "left knee", and "left ankle" is shown together with the respective part numbers "B8" to "B13". ing. Further, each part of the "right eye", "left eye", "right ear", and "left ear" is shown together with the respective part numbers "B14" to "B17".

Such skeleton information (part information) is an operation of designating a desired part with a designated number (including numbers and / or characters) when creating a rule (creating a detection rule for a detection target event) as described later. Used for. The operation user can specify a desired specific part among a plurality of specific parts while referring to an image (an image showing the skeleton information) in the skeleton information display area 220. The image in the skeleton information display area 220 is an image for designating a plurality of specific parts related to a person (an image displaying a list of a plurality of specific parts), and is an "image for specifying a part" (or a "part list display"). Also called "image").

For example, a desired specific part is designated by a designated number corresponding to each specific part (designated number associated with the corresponding part in the part designation image) (“B1” or the like). Not limited to this, by clicking (or touching on the screen) the display part (point part corresponding to B1) of each specific part of the image for specifying the part with the mouse, the specific part corresponding to the display part can be specified. A site (desired specific site) may be designated.

The rule designation area 230 includes a rule addition button 231, a rule deletion button 232, and a rule selection area 233.

The rule addition button 231 is a button for adding a detection rule for a detection target event (also referred to as a detection condition for the detection target event), and the rule deletion button 232 is a button for deleting the detection rule. Further, the rule selection area 233 is an area for displaying the names of one or a plurality of generated rules as options. The rule selection area 233 is an area for selecting a desired rule (from a plurality of rules) as a rule to be edited. The content of the rule selected in the rule selection area 233 is defined (set) in detail in the rule editing area 240. When the rule addition button 231 is pressed, the "new rule" is additionally displayed as an option in the rule selection area 233 as shown in FIG.

The rule editing area 240 is an area for editing the detailed contents of the rule to be edited. The rule (detection rule (detection condition) to be edited) is designated (defined) by inputting appropriate contents in various input fields in the rule editing area 240.

The rule editing area 240 includes a rule name input field 241 and a detailed condition designation field 250.

An arbitrary rule name is input in the rule name input field 241. The "rule name" is also expressed as the name of the event to be detected. The name entered in the rule name input field 241 (for example, "getting up (on the bed)") is determined as the rule name, and the name is displayed as an option in the rule selection area 233 (see FIG. 20). ..

The detailed condition designation column 250 includes a part designation column 251 and an operation or the like designation column 252.

The site designation field 251 is an input field for designating a specific site to be noted. The operation user can specify a desired option (for example, "B1") from a plurality of options (B0 to B17, etc.) in the first drop-down list in the site designation field 251. That is, the operating user can specify a desired specific part among a plurality of specific parts. For example, when the operating user wants to specify the "chest" as a specific part, the designated number B1 corresponding to the "chest" may be specified by referring to the display content in the skeleton information display area 220.

Further, the operating user can specify two or more specific parts by pressing the "Add part" button in the part designation field 251. For example, the drop-down list for specifying the second part and the drop-down list for specifying the AND / OR condition are additionally displayed in response to the pressing of the part addition button (see FIG. 21). Then, the operation user specifies the second specific part in the drop-down list for specifying the second part, and the operation user mutually of the two or more specific parts in the drop-down list for specifying the AND / OR condition. It is possible to specify the relationship between (AND condition / OR condition). Further, the third and subsequent specific parts can be specified in the same manner.

The operation or the like designation field 252 is an input field for designating the operation or state of the specific part designated by the part designation field 251. For example, in the drop-down list in the operation designation field 252, a plurality of options such as "move" and "stay" are displayed. The operating user can specify a desired option from the plurality of options.

Further, the detailed condition designation column 250 further includes an area designation column 253, a height designation column 254, a speed designation column 255, and a period designation column 256.

The operating user can put a check mark in the check box at the head of the desired designated field among these designated fields 253 to 256 and specify the detailed contents of the desired designated field. The contents of the designated fields of the check boxes that are not marked with a check mark are invalidated.

The area designation field 253 is an input field for designating a detection area (planar position) related to the operation or state of a specific part. The operation user specifies an area (for example, "R1") in the first drop-down list for specifying an area, and ancillary conditions (for example, "inside") in the drop-down list for specifying ancillary conditions related to the area. Can be specified (see FIG. 20). As the incidental condition, there are a plurality of options such as "inside", "outside", "inside", "outside", and "from (FROM)".

If "kara (FROM)" is specified in this drop-down list for specifying ancillary conditions, a second drop-down list for specifying an area and a drop-down list for specifying ancillary conditions related to the area are added. It is displayed (see FIG. 21). In this drop-down list for specifying incidental conditions, options such as "heto (TO)" are provided. Thereby, it is also possible to specify that the specific portion moves “from the area R1 to the area R2” and the like.

The height designation field 254 is an input field for designating a height related to the movement or state of a specific part. In the first drop-down list in the height designation field 25, there are options (“current location”, “floor surface”, etc.) indicating the height standard. Further, it is possible to specify "30 cm or more" (more specifically, "30" cm "or more") or the like in another input field in the height designation field 25. In addition, it is possible to specify an arbitrary numerical value as the numerical value, and it is also possible to specify the numerical value "or less" instead of the numerical value "greater than or equal to". The same applies to the other columns.

Further, here, in the height designation column 254, the height in a certain state or the height after the movement is specified, but the height is not limited to this, and for example, the height before the movement (before the state transition) and the movement. The height after (after the state transition) may be specified.

The speed designation field 255 is an input field for designating the speed related to the operation of the specific portion. Specifically, it is possible to specify that the moving speed of the specific part is "1" m / s "or more" (see FIG. 24).

The period designation field 256 is an input field for designating a period related to the state (or operation) of a specific part. Specifically, it is possible to specify that the duration of the state (or the duration of the operation) is equal to or longer than a predetermined length (see FIG. 25).

The operating user can set the detection target event by using such a detailed condition designation field 250.

For example, an event as shown in FIG. 20 can be set as a detection target event having the name of “getting up (on the bed)”. In FIG. 20, an event that satisfies the condition that "the part B1 (chest) moves within the area R1 so that the height from the current location (current height) is 30 cm or more" is set (both specified and defined). Is expressed).

Further, by pressing the condition addition button 242 in the rule editing area 240, it is possible to specify a complicated condition consisting of two or more conditions (see FIGS. 21 and 22). For example, when the condition addition button 242 is pressed after the first condition is specified in the detailed condition specification field 250 of FIG. 21, the detailed condition specification field 260 (see FIG. 22) for designating the second condition is displayed. To. The detailed condition designation column 260 has the same configuration as the detailed condition designation column 250. Further, in FIG. 22, a button 243 and a drop-down list 244 are provided (instead of the rule name input field 241). The relationship between the first condition and the second condition (AND / OR, etc.) is input to the drop-down list 244. Further, in response to the pressing of the button 243, the detailed condition designation field 250 relating to the first condition is displayed again.

The alert level designation area 280 is an area for designating a plurality of alert levels (“warning”, “caution”, “daily operation”). The operating user can specify a desired level from a plurality of alert levels by selecting a desired option (eg, "warning") from the drop-down list in the alert level designation area 280. For example, "daily movement" is specified for the detection target event of "getting up (on the bed)", and "caution" is specified for the detection target event of "end sitting position (bed)". In addition, a "warning" is specified for the event to be detected of "falling (on the floor)". According to this, it is possible to specify (change) the alarm level for each event to be detected. In particular, it is possible to change the alarm level for each detection target event for each facility (or for each room) according to the needs of the site.

The registration button 291 is a button for registering (confirming) the content edited by using the rule editing area 240, the detailed condition designation field 250, the alarm level designation area 280, and the like. The edited content is confirmed by pressing the registration button 291.

The cancel button 292 is a button for discarding the edited content. When the cancel button 292 is pressed, the edited content is discarded and the setting content before the start of editing is restored.

By using such a setting screen 200, the above-mentioned events E1 to E8 and the like can be set.

FIG. 20 is a diagram showing a setting screen of the event to be detected E1 “getting up (on the bed)”. In the setting screen 200 of FIG. 20, the height (height from the current location) of the chest (site B1) of the person to be observed is 30 cm or more in the bed area R1 (the plane area where the bed is arranged). The conditions are stipulated. When the condition is satisfied, event E1 (movement of the upper body of the observation target person rising on the bed) (see FIG. 7) is detected.

The detection target event E1 in FIG. 20 is defined as an event that satisfies the condition regarding the three-dimensional position (plane area R1 and / or height) of the specific portion B1 in the observation target person. In particular, the detection target event E1 is a condition relating to the positional relationship between the specific portion B1 of the observation target person and a predetermined object (here, the bed) other than the observation target person (“the specific portion B1 is in the bed area R1”). ) Is specified as an event that satisfies. Further, the detection target event E1 is defined as an event that also satisfies the condition regarding the change in the three-dimensional position (change in height) of the specific portion in the observation target person.

21 and 22 are diagrams showing a setting screen of the event to be detected E2 “end sitting position (bed)”. In FIG. 21, the condition (first condition) that "the ankles (sites B10 and B13) of the observation target person move from the bed area R1 to the bedside area R2" is defined. Further, in FIG. 22, a condition (second condition) to the effect that "the waist (site B8 and site B11) of the observation target person remains in the bed area R1" is defined. Further, "AND" (with the condition on the previous page (first condition in FIG. 21)) is specified in the drop-down list 244. That is, it is also stipulated that the condition of the logical product (AND) of the first condition and the second condition is satisfied.

In this way, the ankles (sites B10 and B13) of the observer move from the bed area R1 to the bedside area R2, and the waists (sites B8 and B11) of the observer remain in the bed area R1. The conditions to the effect are stipulated. When the condition is satisfied, event E2 (a state in which the person to be observed is sitting at the edge of the bed) (see FIG. 8) is detected.

The detection target event E2 in FIGS. 21 and 22 is a condition relating to a change in the three-dimensional position (moving from the plane area R1 to the plane area R2) of the specific parts B10 and B13 (right ankle and left ankle) in the observation target person. Is defined as an event that satisfies. In particular, the detection target event E2 is defined as an event that satisfies the condition regarding the change in the positional relationship between the specific parts B10 and B13 of the observation target person and the bed. Specifically, the condition is "the specific portions B10 and B13 move from the inside of the bed area R1 to the bedside area R2 outside the bed area R1".

Further, the detection target event E2 is defined as an event that satisfies the condition regarding the three-dimensional position (plane area R1) of the specific parts B8 and B11 in the observation target person. In particular, the detection target event E2 is defined as an event that satisfies the condition regarding the positional relationship between the specific parts B8 and B11 of the observation target person and a predetermined object (bed) other than the observation target person. Specifically, the condition is "the specific sites B8 and B11 are present in the bed area R1".

FIG. 23 is a diagram showing a setting screen of the event to be detected E3 “getting out of bed”. In the setting screen 200 of FIG. 23, there is a condition that the whole body (all specific parts B0 to B17) of the observation target person moves from the bed area R1 to another area R3 (area away from the bed area R1). It is stipulated. When the condition is satisfied, event E3 (movement of the person to be observed to move away from the bed to a place other than the bed) (see FIG. 9) is detected.

The detection target event E3 in FIG. 23 is defined as an event that satisfies the condition regarding the change in the three-dimensional position (moving from the plane area R1 to the plane area R3) of the specific parts B0 to B17 in the observation target person. In particular, the detection target event E3 is a condition relating to a change in the positional relationship between the specific parts B0 to B17 of the observation target person and the bed (“The specific parts B0 to B17 move from the inside of the bed area R1 to the area R3 outside the bed area R1). What to do ") is defined as an event that satisfies.

FIG. 24 is a diagram showing a setting screen of the event to be detected E4 “falling (on the floor)”. In the setting screen 200 of FIG. 24, the condition that the chest (specific portion B1) of the person to be observed is 1 m / s or more and enters a space of a predetermined height (50 cm) or less on the floor surface is defined outside the bed area R1. Has been done. When the condition is satisfied, event E4 (movement of the person to be observed to fall on the floor) (see FIG. 10) is detected.

The detection target event E4 in FIG. 24 is defined as an event that satisfies the condition regarding the three-dimensional position of the specific portion B1 in the observation target person (“outside the bed area R1”). Further, the detection target event E4 is defined as an event that also satisfies the condition regarding the change (change in height) of the three-dimensional position of the chest (specific portion B1) in the observation target person. Further, the detection target event E4 is defined as an event that also satisfies the condition (“1 m / s or more”) regarding the change speed (movement speed) of the three-dimensional position of the chest (specific portion B1) in the observation target person.

In particular, "falling" is an action in which the posture of the person to be observed (such as a care recipient) changes against his / her own will. Therefore, the moving speed in "falling" is usually above a certain level. In FIG. 24, the event to be detected is set more appropriately by utilizing such a characteristic. According to this, the movement of the person to be observed (the care recipient, etc.) to fall against his / her will (see FIG. 10) and the movement of the person to be observed to lie down slowly (at a speed less than a certain speed) by his / her own will (see FIG. 10). It is possible to properly distinguish from (not shown).

FIG. 25 is a diagram showing a setting screen of the event to be detected E5 “lying (on the floor)”. In the setting screen 200 of FIG. 25, there is a condition that the nose (specific part B0) and chest (specific part B1) of the person to be observed stay in a space 50 cm or less from the floor surface for 5 seconds or more outside the range of the bed area R1. It is stipulated. When the condition is satisfied, event E5 (a state in which the person to be observed is lying on the floor for a predetermined period or longer (having a recumbent position)) (see FIG. 11) is detected.

The detection target event E5 in FIG. 25 is defined as an event that satisfies the conditions regarding the three-dimensional positions of the specific parts B0 and B1 in the observation target person (“height is 50 cm or less from the floor surface” and “outside the bed area R1”). There is. Further, the detection target event E5 is defined as an event that also satisfies the condition (“5 seconds or more”) regarding the continuity (non-change) of the three-dimensional positions of the specific parts B0 and B1 in the observation target person.

FIG. 26 is a diagram showing a setting screen of the event to be detected E6 “whole body slipping off the bed”. In the setting screen 200 of FIG. 26, the portion B0 (nose) and the portion B8 (right waist) of the person to be observed from the bed area R1 to the bedside area R2 are 1 m / s or more and a predetermined height (50 cm) on the floor surface. The conditions for moving to the following space are stipulated. When the condition is satisfied, event E6 (movement of the whole body of the person to be observed sliding down from the bed to the bedside) (see FIG. 12) is detected.

The detection target event E6 in FIG. 26 satisfies the conditions regarding the change in the three-dimensional positions of the specific parts B0 and B8 in the observation target person (“movement from bed area R1 to bedside area R2” and “change in height”). It is defined as an event to be done. Further, the detection target event E6 is defined as an event that also satisfies the condition (“1 m / s or more”) regarding the change speed (movement speed) of the three-dimensional positions of the specific parts B0 and B8 in the observation target person.

FIG. 27 is a diagram showing a setting screen of the event to be detected E7 “sliding the lower half of the body from the bed”. In the setting screen 200 of FIG. 27, the right hip (site B8) and the left hip (part B11) of the person to be observed from the bed area R1 to the bedside area R2 are 1 m / s or more and a predetermined height (50 cm) on the floor surface. ) The condition of moving to the following space is stipulated. When the condition is satisfied, event E7 (movement of the lower body of the person to be observed sliding down from the bed to the bedside) (see FIG. 13) is detected.

Here, "sliding of the whole body" is defined by the movement of the specific part B0 (nose) of the upper body and the specific part B8 (right waist) of the lower body, while "sliding of the lower body" is the identification of the lower body. It is defined by the movement of sites B8 and B11 (right and left hips). That is, "sliding of the lower body" is different from "sliding of the whole body" in that the movement of a specific part of the upper body is not a condition. In addition, according to this regulation content (setting content), when the "whole body" of a person actually slips off the bed, both "sliding of the lower body" and "sliding of the whole body" are detected. However, the present invention is not limited to this, and the two may be detected separately from each other by, for example, specifying another condition. For example, "sliding of the lower body" may be detected on condition that the face (nose, etc.) of the person is present at a predetermined height (for example, 50 cm) or more from the floor.

28 and 29 are diagrams showing a setting screen of the event to be detected E8 “one-handed extension from the bed”. In FIG. 28, a condition (first condition) is set to the effect that "one of both wrists (part B4 or part B7) of the person to be observed moves from the bed area R1 to the bedside area R2". Further, in FIG. 29, a condition (second condition) to the effect that "the waist (part B8 and part B11) of the person to be observed remains in the bed area R1" is defined. Further, "AND" (with the condition on the previous page (first condition in FIG. 28)) is specified in the drop-down list 244. That is, it is also stipulated that the condition of the logical product (AND) of the first condition and the second condition is satisfied.

In this way, one wrist (site B4 or site B7) of the observation target person moves from the bed area R1 to the bedside area R2, and the waist (site B8 and site B11) of the observation target person is in the bed area R1. The conditions for staying are stipulated. When the condition is satisfied, event E8 (a state in which one hand of the observation target person extends to the bedside) (see FIG. 14) is detected.

The detection target event E8 in FIGS. 28 and 29 satisfies the condition regarding the change in the three-dimensional position (moving from the area R1 to the area R2) of the specific part B4 or B7 (right wrist or left wrist) in the observation target person. It is defined as an event to be done. In particular, the detection target event E8 is defined as an event that satisfies the condition regarding the change in the positional relationship between the specific parts B4 and B7 of the observation target person and the bed. Specifically, the condition is "the specific portion B4 or B7 moves from the inside of the bed area R1 to the bedside area R2 outside the bed area R1".

Further, the detection target event E8 is defined as an event that satisfies the condition regarding the three-dimensional position (plane area R1) of the specific parts B8 and B11 in the observation target person. In particular, the detection target event E8 is a condition relating to the positional relationship between the specific parts B8 and B11 of the observation target person and a predetermined object (bed) other than the observation target person (“specific parts B8 and B11 exist in the bed area R1”. It is defined as an event that satisfies "that").

As described above, each detection target event can be flexibly set (defined) by the setting operation using the setting screen 200 or the like. Therefore, it is possible to flexibly respond to various requests of the user.

In detail, the user can easily set the detection target event itself (in other words, the detection rule (detection condition) of the detection target event). For example, the user (facility staff, etc.) can easily customize and register the setting contents (part, operation / state, three-dimensional position, speed, etc.) of each of the above events E1 to E8 according to the circumstances of each facility. It is possible. In addition, the user can determine the event to be detected according to the circumstances of each facility and appropriately register (set) the event to be detected as the event to be detected. Further, when the number of events to be newly detected increases, it can be easily realized that the event is newly additionally registered as the event to be detected.

In particular, the event to be detected can be specified by the state and / or operation of a specific part of the person, and the specific part can be arbitrarily specified (appropriately specified according to needs and the like). The combination of each specific part and the operation content can be arbitrarily specified. In short, the customization performance is high.

In particular, as an event that satisfies one or more of the three elements of "three-dimensional position", "change of the three-dimensional position", and "change speed of the three-dimensional position" of a specific part of a person. , The event to be detected is specified. Therefore, it is possible to specify the event in the real space (three-dimensional space) in detail and appropriately. Furthermore, since the event to be detected can be defined as an event that satisfies the conditions relating to two or more of these three elements, it is possible to specify the event in the real space (three-dimensional space) more finely and more appropriately. .. In addition, since the detection target event can be set (specified) including the change speed of the three-dimensional position, it is possible to set the detection target event (event having a characteristic change speed) such as a fall very appropriately. be.

Further, by utilizing the "height" of the three-dimensional position of the person to be observed, it is easy to appropriately detect a large change in the posture of the person (a large change in a fall or the like).

Further, the "planar position" of the three-dimensional positions of the person to be observed can be designated by the area (R1, R2, R3, etc.) preset by the user. In detail, the plane area in the real space is divided into a plurality of areas (R1, R2, R3, etc.) in advance based on the camera image (photographed image of the room) or the like based on the user's operation or the like. Then, an appropriate plane area can be designated by the user from the plurality of areas (R1, R2, R3, etc.) and the detection target event can be set. According to this, the user can create each plane area with a high degree of freedom, and can flexibly specify the plane area as the existence area, the movement source area and / or the movement destination area of each specific part. It is possible.

In addition, the name of the event to be detected can be appropriately set as a common name for each facility. Therefore, when introducing the above system, it is possible to prevent the user (the person who cares (the caregiver), etc.) from feeling uncomfortable with the name of the event to be detected.

Further, the alert level (warning, attention, daily operation) can be set for each event to be detected. In other words, the terminal device 70 accepts an operation of setting the alarm level according to the event to be detected among the plurality of alarm levels. As a result, the event to be detected is notified by level. In addition, it is possible to flexibly set the level classification according to the circumstances of each facility.

It should be noted that the occurrence of each detection target event is notified in different notification modes according to the notification level. For example, when an event of "attention" level occurs, the name of the detected event ("end sitting position") on the observation screen 310 (specifically, the character display area 312) (see FIG. 16) of the terminal devices 70 and 80. "Etc.) is displayed in yellow and medium-sized characters. When an event of "warning" level (higher than "caution" level) occurs, the name of the detected event ("fall", etc.) on the observation screen 310 (see FIG. 17) of the terminal devices 70 and 80. ) Is displayed in red and large size characters. In this case, a warning sound is further output from the terminal devices 70 and 80. Further, when an event of the "daily operation" level (a level lower than the "attention" level) occurs, the name of the detected event (such as "getting up") is black on the observation screen 310 of the terminal devices 70 and 80. Displayed in small size letters.

Further, one or more detection target events (detection conditions) are set for each room. Specifically, first, the terminal device 70 accepts a setting operation for setting each detection target event related to a plurality of rooms for each room. Then, each detection device 10 provided for each room sets each detection target event for each room according to the setting operation, and each detection target event is set based on the setting content (setting content for each room) by the setting operation. Is detected.

Therefore, it is possible to make detailed settings according to the occupants of each room. For example, when the care level of the resident is different for each room, it is possible to appropriately customize and set the detection target event (detection condition) for each room (in other words, for each resident of each room). It is possible.

Further, when the information of the person residing in each room (height of the person to be observed for each room, etc.) is registered in response to the pressing of the person information registration button 202 described above, it is detected as follows. It is possible to improve the accuracy. Specifically, the detection device 10 may exclude a person who does not match the height information of the observation target person in the room from the detection target person and detect each detection target event in the room.

More specifically, the detection device 10 measures the height of one or a plurality of subject persons in the captured image based on the information (photographed image, depth information, etc.) acquired by the three-dimensional camera. When the difference between the height of the subject person (measured height) and the height of the observation target person (pre-registered height) is equal to or greater than a predetermined value, the detection device 10 determines that the subject person is the observation target person. It may be determined that there is no such subject, and the subject person may be excluded from the detection target.

According to this, when a person different from the person to be observed (for example, a child whose height is shorter than the person to be observed by a certain degree or more) is staying in the room for a sympathy, etc. (playing). Even if the child falls, it is not necessary to mistakenly determine that the person to be observed has fallen.

Alternatively, if there are more people in a room than the number of people to be observed (predetermined number (for example, one person)), the detection target event should not be issued even when it is detected. May be good. In a situation where a person (another person) other than the observation target person exists in the room, it is often possible for the person (another person) to recognize the occurrence of a detection target event related to the observation target person. In such a situation, it is possible to suppress an originally unnecessary alarm by considering that the necessity of the alarm is lower than a predetermined level and not issuing the alarm even when the event to be detected is detected. It is also possible to suppress an increase in the load of the detection device 10 due to the detection operation of a plurality of subject persons.

Here, the management device 70 accepts the setting operation, but the setting operation is not limited to this. For example, the mobile terminal device 80 may accept the setting operation.

<7. Monitoring display by skeleton display>
Next, the monitoring display of the person to be observed will be described.

FIG. 15 is a diagram showing a selection screen 300 for selecting a room to be displayed. On the selection screen 300, a plurality of buttons 301 to 306 corresponding to each of the plurality of (here, 6) rooms are displayed. Then, when a desired button is selected from the plurality of buttons (301 to 306), the observation screen 310 relating to the room corresponding to the desired button is displayed. For example, when the button 301 is pressed, the observation screen 310 (see FIG. 16 and the like) relating to "Room 101" associated with the button 301 is displayed. The selection screen 300 and the observation screen 310 are displayed, for example, on the display unit 75b of the management device 70. However, the present invention is not limited to this, and the selection screen 300 and the observation screen 310 may be displayed on the display unit 85b (touch panel 87 or the like) of each mobile terminal device 80.

The display processing of these screens 300 and 310 is controlled by the cooperation between each terminal device 70 (or 80) and each detection device 10. More specifically, first, the terminal device 70 (or 80) sends a monitoring display request to the detection device 10 corresponding to the target room. The detection device 10 generates display image data and the like (display data of the observation screen 310 (including the observation image 330)) in response to the monitoring display request, and uses the display image data and the like as the terminal device. It is transmitted to 70 and the like (step S12 in FIG. 5). The terminal device 70 or the like displays the observation screen 310 based on the received display data.

16 and 17 are views showing an observation screen 310 of an observation target person in a certain room (here, “Room 101”).

As shown in FIG. 16 and the like, the observation screen 310 includes an image display area 311 for displaying the observation image 330 (observation image) and a character display area 312.

An observation image (video (moving image)) 330 of the person to be observed is displayed in real time in the video display area 311. As shown in FIG. 16 and the like, the observation image 330 is an image having no captured image of the observation target person itself and having a skeleton image 325 of the observation target person.

The skeleton image 325 and the observation image 330 are generated by the processing unit 30 based on the captured image 110 captured by the camera unit 20.

FIG. 18 is a diagram illustrating a process of generating the observation image 330 and the like.

Specifically, the processing unit 30 of the detection device 10 first generates a skeleton image 325 of the observation target person from the captured image 110 of the observation target person (specifically, the skeleton information 140 based on the captured image 110). As shown in FIG. 18 (and FIG. 6) and the like, the skeleton image 325 is an image in which, for example, each specific part of the person to be observed is indicated by a “point” and each skeleton line (connecting line) is indicated by a “line segment”. Is.

Then, the processing unit 30 generates an observation image 330 regarding the observation target person based on the skeleton image 325. The observation image 330 is generated, for example, by synthesizing a background image 327 (also referred to as 327a) in which no person exists and a skeleton image 325. The skeleton image 325 is arranged at the same position as the existing position in the captured image 110 in the background image 327.

The background image 327a in which no person exists is generated based on, for example, a photographed image 110 (also referred to as 326) which has been photographed in advance and has no person. Specifically, when the detection device 10 determines that a person does not exist in the photographed image (specifically, the photographed image in which the target room is photographed in advance) 110 at a certain point in time, the photographed image (moving image) Image) Extract a still image from 110. Then, the still image is generated as the background image 327a.

In this way, the observation image 330 at a certain point in time is generated.

After that, by repeating such an operation, the observation image 330 at each time point is sequentially generated. In short, the observation image 330 (including the skeleton image 325) is generated in real time based on the captured image 110 and the like.

The background image 327a may always be the same image. However, the present invention is not limited to this, and for example, different background images 327a may be generated and used depending on the time zone (daytime / nighttime) and the like. Specifically, in the daytime time zone, the captured image 110 captured in the daytime and in the absence of a person may be used as the background image 327a. Further, in the night time zone, the captured image 110 captured at night and in the absence of a person may be used as the background image 327a.

Further, the background image 327 may be a fixed image 327b (a plain image, a three-dimensional schematic diagram of a room, etc.) prepared in advance.

By the way, on a general monitoring screen (also referred to as a screen according to a comparative example), the image taken by the camera unit 20 itself is displayed instead of the skeleton image 325. In detail, the photographed image of a person (also referred to as an observation target person) in the living room is always displayed as it is on the administrator screen or the like.

However, it cannot be said that such a monitoring display gives appropriate consideration to the privacy of the person concerned.

On the other hand, in the monitoring display according to the above embodiment, the skeleton image 325 of the person to be observed is displayed as the observation image 330 instead of the photographed image of the person to be observed. Therefore, it is possible to give appropriate consideration to the privacy of the person to be observed. In particular, according to the skeleton information (skeleton image 325 (observation image 330)), information on the face of the observation target person is lost, and the body shape (fat / thin) and gestures of the observation target person are recorded. The amount of information is reduced to the extent that it is almost unrecognizable. Therefore, it is possible to give appropriate consideration to the privacy of the person concerned. Therefore, it is also possible to reduce the psychological resistance of the person being watched over (such as a care recipient).

On the other hand, the caregiver or the like can appropriately grasp the posture and the like of the observation target person by visually recognizing the skeleton image of the observation target person.

<8. Modification example regarding display>
In the monitoring display as described above, when a predetermined condition is satisfied, the captured image 110 relating to the observation target person may be displayed in place of the observation image 330 or together with the observation image 330. In short, switching from the observation image 330 (skeleton image 325) to the captured image 110 may be performed as needed.

<Predetermined condition 1>
Examples of the predetermined conditions for which such switching or the like should be performed include the detection of a predetermined detection target event.

For example, on a predetermined setting screen (200, etc.), an option (option regarding the display target image) regarding which of the "skeleton image" and the "captured image" should be displayed when each detection target event occurs (at the time of detection) is selected. I will set it up. It is preferable that the option is provided for each event to be detected. Then, the type of image (“skeleton image” / “photographed image”) set by the option may be displayed when each detection target event occurs. For example, when a "captured image" is set as a display target image when a certain detection target event ("fall" or the like) occurs, the captured image 110 is monitored and displayed when the detection target event is detected. On the other hand, when the detection target event is not detected, the skeleton image 325 (observation image 330) is displayed.

According to this, when an event having a high risk such as "falling over" occurs, the user can visually recognize the actual captured image 110 and confirm a more detailed situation. As a result, it is possible to grasp the situation more appropriately and take appropriate measures.

In this way, the skeleton image 325 (observation image 330) is normally displayed, and the captured image 110 is displayed when a predetermined detection target event occurs. According to this, it is possible for the user (caregiver or the like) to confirm the actual photographed image in a predetermined situation while considering privacy. In other words, the observer (caregiver, etc.) can confirm the photographed image of the person to be observed, if necessary. As a result, the observer (caregiver, etc.) can perform appropriate treatment.

Further, by setting the pros and cons of displaying the captured image for each event to be detected, it is possible to realize a fine-tuned response.

<Predetermined condition 2>
Further, as a predetermined condition for which switching or the like should be performed as described above, it is also exemplified that the alarm level set for the event to be detected has a higher importance than the predetermined level among a plurality of alarm levels. Will be done.

For example, the relationship between the display target image (observation image 330) and the plurality of alarm levels may be set on the setting screen for alerting (see FIG. 20 and the like). Examples of the plurality of alarm levels are "warning", "caution", and "daily operation". "Warning" is a notification level with a higher importance than "Caution", and "Caution" is a notification level with a higher importance than "Daily movement". It is assumed that one of a plurality of alarm levels is assigned to each detected event.

For example, on the setting screen for alerting, a predetermined level (for example, "warning" or higher) at which switching from the skeleton image 325 (observation image 330) to the captured image 110 should be performed is set among a plurality of alerting levels. Will be done.

Based on such settings, when the detection target event is not detected, the skeleton image 325 (observation image 330) is displayed, while when a certain detection target event is detected, the detection target event is displayed. Images are displayed according to the assigned alarm level.

Specifically, when a detection target event occurs, it is determined whether or not the alarm level of the detection target event has a predetermined level (“warning”) or higher. If the alert level of the event to be detected does not have a predetermined level (“warning”) or higher, the skeleton image 325 (observation image 330) is continuously displayed. On the other hand, when the alarm level of the detection target event has a predetermined level (“warning”) or higher, the display target image is changed from the skeleton image 325 (observation image 330) to the captured image 110.

As described above, the skeleton image 325 (observation image 330) is normally displayed, and the captured image 110 is displayed when a detection target event having a predetermined level of importance or higher occurs. According to this, it is possible to confirm the actual captured image and perform appropriate processing as necessary while considering privacy. In addition, since it is possible to set whether or not to display the captured image for each event to be detected, it is possible to realize a detailed response.

In particular, it is possible to change whether to display the skeleton image 325 (observation image 330) or the captured image 110 according to the importance of the event to be detected that has occurred. In particular, when an event with a high risk (event having a predetermined level of importance or higher) such as "falling over" occurs, the user may visually check the actual captured image 110 and confirm a more detailed situation. can. As a result, it is possible to grasp the situation more appropriately and take appropriate measures.

Further, according to such an aspect, the user does not have to set the display target image for each detection target event. Therefore, it is possible to reduce the time and effort for setting as compared with the case where the user sets both the alert level and the display target image for each detection target event.

<Other modifications>
Further, in each of the above modified examples, when a predetermined condition is satisfied, a photographed image relating to the person to be observed is displayed instead of the observation image 330 (skeleton image 325), but the present invention is not limited to this. When the predetermined condition is satisfied, the photographed image 110 relating to the person to be observed may be displayed together with the observation image 330 (skeleton image 325).

Further, in the above-described embodiment and the like, the moving image is displayed as the observation image 330 and the skeleton image 325 in the terminal devices 70 and 80, but the present invention is not limited to this, and the still image is the observation image 330 in the terminal devices 70 and 80. And may be displayed as a skeleton image 325.

Further, in the above embodiment, the transfer of image data is started to the terminal device 70 or the like in response to the monitoring display request sent from the terminal device 70 (or 80) to the detection device 10, and the image (skeleton image or the like) is started. Has started to be displayed, but is not limited to this. For example, in response to the detection of each detection target event by the detection device 10, the terminal devices 70 and 80 are notified of the detection of the detection target event, and the management device 70 or the like automatically notifies the image (skeleton image). Etc.) may be started to be displayed. The displayed image (skeleton image 325, observation image 330, etc.) at this time may also be a moving image or a still image.

Further, in the above-described embodiment and the like, an embodiment in which the skeleton image 325 (observation image 330) is displayed at the time of monitoring display is exemplified, but the present invention is not limited to this. For example, the skeleton image 325 may be displayed when the observation image 330 (log image) stored in the storage unit 32 is subsequently reproduced. Alternatively, when the recorded photographed image 110 (recorded image) is subsequently reproduced, the skeleton image 325 based on the photographed image 110 may be generated and displayed.

Further, here, the detection device 10 executes a generation process (in other words, a display control process) of the skeleton image 325 and the observation image 330, but the present invention is not limited to this. For example, the terminal device 70 (, 80) may execute a process of generating a skeleton image 325 and / or an observation image 330 and the like. Specifically, the captured image 110, the depth information 120, and the like may be transmitted from the detection device 10 to the terminal device 70 and the like, and the skeleton image 325 and the like may be generated by the transmission destination terminal device 70 and the like. However, from the viewpoint of improving security or reducing the amount of data transmitted, it is preferable that the detecting device 10 on the transmitting side (rather than the management device 70 on the receiving side) executes the generation process or the like.

In this way, the detection device 10 may perform the display control process, or the terminal devices 70 and 80 may perform the display control process. That is, the detection device 10 may operate as a display control device, or the terminal devices 70 and 80 may operate as a display control device. In other words, the display control device may be provided as a device separate from the display device (70, 80), or may be provided integrally with the display device (70, 80).

<9. Standard height for each area>
In the above, event E4 “falling on the floor” (see FIG. 10) was exemplified as one of the events to be detected. In the event to be detected E4, as a condition regarding the "height" of the "chest" (site B1) of the person to be observed, it is adopted that the height from the floor surface changes from a value of 50 cm or more to a value of 50 cm or less. ing.

However, "falling" can occur not only on the floor but also on the bed and the like. If the same criteria as the criteria for falls on the floor are used when determining a fall on the bed, an appropriate determination cannot be made. Specifically, "the height of the chest is 50 cm or more from the floor" is not always normal, and it is required to use another standard when falling on the bed.

Therefore, as another event to be detected similar to the event E4, the event E12 "falling on the bed (the person to be observed falls on the bed)" (see FIG. 31) is defined, and the following conditions are specified. The detection target event E12 may be detected according to the establishment of. The condition is, for example, that the height of the "chest" (site B1) of the person to be observed from the "top surface of the bed" in the "bed area R1" changes from a value of 50 cm or more to a value of 50 cm or less, and at that time. The moving speed of the specific part B1 is "1 m / s or more".

However, the present invention is not limited to this, and event E13 "fall" (an event in which two events E4 and E12 are integrated) may be detected without distinguishing between a fall on the floor and a fall on the bed.

The event to be detected E13 is detected when the following conditions are satisfied. The condition is that, for example, the "height from the reference plane" of the "chest" (part B1) of the person to be observed changes from a value of 50 cm or more to a value of 50 cm or less, and the moving speed of the specific part B1 at that time. Is "1 m / s or more". In other words, the conditions in the two events E4 and E12 may both be defined by the height with respect to the "reference plane".

Specifically, first, the detection device 10 predetermines the height of the "reference plane" for each area (planar area) (prior to the processing of FIG. 5).

Specifically, the detection device 10 has a plurality of areas R1, R2, .. Divide into. In other words, a plurality of flat areas R1, R2, ... Are provided in the room. Then, the detection device 10 sets the "height of the reference plane" for each of the plurality of areas R1, R2, .... For example, "45 cm" is set for the bed area R1, "0 cm" is set for the bedside area R2, and "0 cm" is set for the other areas R3 as the heights of the reference planes. Specifically, the detection device 10 may set the height of the reference plane for each area according to the user's operation input or the like.

The height of the reference plane of each plane area (R1, R2, R3, etc.) may be automatically set. Specifically, the height of the reference plane is the height in the three-dimensional space of the object (floor, bed, etc.) corresponding to the representative position of each plane area based on the information from the camera unit 20 (three-dimensional camera). In addition, it may be set automatically. For example, the height of the reference surface of the flat area R1 on which the bed is placed is automatically set to the height of the upper surface of the bed (value measured by a three-dimensional camera or the like) (45 cm from the floor surface or the like). Further, the height of the reference surface of the flat area R2 on which the bed is not placed on the floor is automatically set to the height of the floor surface (0 cm from the floor surface, etc.).

Then, the "reference height" for each area is determined based on the height of the "reference plane" for each area. For example, the reference height of the flat area (bed area) R1 is set to the height "45 cm" of the reference plane (upper surface of the bed) set for the flat area R1 and "50 cm" (height from the reference plane). The added value is "95 cm". On the other hand, the reference height of the flat area R2, R3, etc. (non-bed area (floor surface area)) is "50 cm" at the height "0 cm" of the reference plane (floor surface) set for the flat area R2. It is a value "50 cm" to which (height from the reference plane) is added.

When the area where the person to be observed exists is determined as described below, the reference height of the area is obtained based on the "reference height" for each area. In other words, a "reference height" is set for each existing area. For example, when the bed area (area on the bed) R1 is the area where the person to be observed exists, the reference height of the area is "95 cm". On the other hand, when the area on the floor (areas R2, R3, etc. other than the bed area R1) is the area where the person to be observed exists, the reference height of the area is "50 cm". The reference height for each existing area differs depending on whether the existing area is on the bed or on the floor.

The detection target event E13 may be set according to the setting operation using the setting screen 200 of FIG. 32, for example. FIG. 32 is a diagram showing a setting screen 200 for the event to be detected E13 “fall”. In the setting screen 200 (particularly the detailed condition designation field 250) of FIG. 32, it is stated that the chest (specific portion B1) of the person to be observed enters a space of 1 m / s or more and a predetermined height (50 cm) or less from the "reference plane". Conditions are stipulated. Compared with the setting screen 200 (FIG. 24) relating to the event to be detected E4, there is a particular difference between the point where the plane area of the detection target is not defined and the point where the height from the "reference plane" is defined.

After that, when the process of FIG. 5 is started, in steps S11 to S15 (particularly S11 and S13), the detection device 10 determines the area where the person to be observed exists, and the area of the person to be observed is determined according to the area where the person to be observed exists. Find (set) the reference height of the existing area. Then, the detection device 10 determines the condition regarding the height of each specific portion based on the reference height for each existing area. Further, the detection device 10 detects the event to be detected based on these determination results. The area where the observation target person exists may be determined based on the existence position (planar position) of a predetermined portion (chest, waist, etc.) of the observation target person.

Specifically, the detection device 10 detects the event to be detected based on the comparison result between the reference height for each existing area and the height of the specific portion.

For example, when the person to be observed is on the bed (inside the bed area R1), the reference height of the bed area R1 (“95 cm”) and the height of the specific part B1 (“chest”) of the person to be observed are set. Based on the result of the comparison, the detection target event E13 is detected. Specifically, the height of the "chest" (part B1) of the person to be observed from the floor changes from a value of 95 cm (height of the reference surface "45 cm" + "50 cm") or more to a value of 95 cm or less. The detection target event E13 is detected on condition that the detection target event E13 is performed.

On the other hand, when the person to be observed is on the floor (for example, area R3), the reference height (“50 cm”) of the area R3 is compared with the height of the specific part B1 (“chest”) of the person to be observed. Based on the result of, the detection target event E13 is detected. Specifically, the height of the "chest" (site B1) of the person to be observed from the floor changes from a value of 50 cm (height of the reference surface "0 cm" + "50 cm") or more to a value of 50 cm or less. The detection target event E13 is detected on condition that the detection target event E13 is performed.

In this way, the detection target event E13 is detected based on the reference height for each existing area (in other words, the reference height set for each existing area). It is preferable that the event E13 to be detected is detected by adding the above-mentioned conditions related to the moving speed (moving speed is “1 m / s or more”, etc.) in addition to the conditions related to the “height”.

As described above, the detection device 10 (controller 31) may determine the existing area of the person to be observed and determine the condition regarding the height of the specific portion based on the reference height for each existing area. In detail, a different reference height may be set for each area where the observation target person exists, and the detection target event may be detected based on the comparison result between the reference height and the height of the specific portion. According to this, even if the normal range of the height of the specific portion differs for each existing area, an appropriate detection process can be executed based on an appropriate reference height for each existing area of the person to be observed. Therefore, it is possible to appropriately detect the event to be detected as compared with the case where the detection process is executed based on a single reference height (reference height that does not depend on the existing area). In addition, it is not necessary to separately specify the event of "falling" as a plurality of detection target events (E4, E12, etc.) for each area, and the plurality of detection target events (E4, E12, etc.) are detected as one. It can be treated as a target event E13 (“fall”) in a unified manner.

<10. Others>
Although the embodiments of the present invention have been described above, the present invention is not limited to the contents described above.

For example, the specific part of the person to be observed is not limited to each of the above-mentioned specific parts, and may be the head, neck, clavicle, or the like of the person.

Further, in the above-described embodiment and the like, the detection device 10 (particularly, the processing unit 30) executes the detection processing of each detection target event, but the present invention is not limited to this. For example, the management device 70 may execute the detection process of each detection target event.

Further, in the above embodiment and the like, the processing unit 30 (controller 31) generates the skeleton information 140 and the three-dimensional position information 150 of each specific part, but the present invention is not limited to this. For example, the management device 70 (controller 71) may generate the three-dimensional position information 150 of each specific part. Further, the management device 70 (controller 71) may also generate the skeleton information 140.

Further, in each of the above embodiments, the detection device 10 and the management device 70 are configured as separate devices, but the present invention is not limited to this. For example, the detection device 10 may be constructed as a device that also executes a process of the management device 70 (such as a process of accepting a setting operation).

Further, the display unit for displaying the observation image 330 may be provided in the terminal device 70 (, 80) as in each of the above embodiments, but is not limited to this, and may be provided in, for example, the detection device 10. good.

1 Detection system 10 Detection device 20 Camera unit 30 Processing unit 70 Management device 80 Mobile terminal device 110 Photographed image 120 Depth information 140 Skeletal information 150 3D position information 200 Setting screen 202 Person information registration button 210 Area setting area 220 Skeletal information display area 230 Rule specification area 240 Rule edit area 250 Detailed condition specification field 280 Notification level specification area 300 Selection screen 310 Observation screen 325 Skeletal image 327 Background image 330 Observation image

Claims (25)

A terminal device that accepts setting operations for detection target events related to the observation target person,
A detection device that detects the event to be detected based on the setting contents of the setting operation, and
A detection system characterized by being equipped with. The detection system according to claim 1, wherein the detection target event is set as an event that satisfies the condition regarding the three-dimensional position of the specific portion in the observation target person. The detection system according to claim 1 or 2, wherein the detection target event is set as an event that satisfies a condition relating to a change in a three-dimensional position of a specific portion in the observation target person. The detection according to any one of claims 1 to 3, wherein the detection target event is set as an event that satisfies the condition regarding the change speed of the three-dimensional position of the specific portion in the observation target person. system. The first aspect of claim 1, wherein the detection target event is set as an event that satisfies the conditions relating to the three-dimensional position of the specific portion and the change rate of the three-dimensional position of the specific portion in the observation target person. Detection system. The detection target event is set as an event that satisfies the conditions relating to the change in the three-dimensional position of the specific portion and the change rate of the three-dimensional position of the specific portion in the observation target person. The detection system described in. The detection system according to any one of claims 2 to 6, wherein the detection target event is set as an event that satisfies at least a height condition among the three-dimensional positions of the specific portion. The detection target event is set as an event that satisfies at least a plane position among the three-dimensional positions of the specific part.
The plane position in the detectable space is divided into multiple areas.
The detection system according to any one of claims 2 to 7, wherein the condition relating to the plane position includes a condition relating to an area designated from the plurality of areas. Claim 2 to claim 2, wherein the detection target event is set as an event that satisfies the condition regarding the positional relationship between the specific portion of the observation target person and a predetermined object other than the observation target person. The detection system according to any one of 8. 2. The terminal device is characterized in that it displays an image showing a plurality of specific parts relating to a person and accepts an input for designating a desired specific part among the plurality of specific parts as the specific part. The detection system according to any one of claims 9. The detection system according to any one of claims 1 to 10, wherein the terminal device accepts a setting operation for setting each detection target event related to a plurality of rooms for each room. The terminal device accepts a setting operation for setting the height information of the person to be observed for each room.
The detection system according to claim 11, wherein the detection device excludes a person who does not conform to the height information from the person to be detected and detects the event to be detected. The detection system according to any one of claims 1 to 12, wherein the terminal device accepts an operation for setting an alarm level according to the event to be detected among a plurality of alarm levels. The detection device generates a skeleton image of the observation target person from the captured image of the observation target person, and also generates an observation image of the observation target person based on the skeleton image.
The terminal device displays the observation image of the observation target person, and displays the observation image.
The aspect according to any one of claims 1 to 13, wherein the observation image is an image that does not have a photographed image of the observation target person itself and has the skeleton image of the observation target person. Detection system. The detection system according to claim 14, wherein the observation image is an image generated by synthesizing the skeleton image with a background image in which a person does not exist. The detection system according to claim 15, wherein the background image is generated based on a captured image in which a person does not exist. One of claims 14 to 16, wherein the terminal device displays a photographed image of the person to be observed in place of the observation image or together with the observation image when a predetermined condition is satisfied. The detection system described in. The detection system according to claim 17, wherein the predetermined condition includes the detection of the event to be detected. The predetermined condition according to claim 18, wherein the notification level set for the detection target event has a importance of a predetermined level or higher among a plurality of notification levels. Detection system. The reception unit that accepts the setting operation of the detection target event related to the observation target person,
A control unit that transmits setting information based on the setting operation to a detection device that detects the event to be detected, and a control unit.
A terminal device characterized by being provided with. a) The step of accepting the setting operation of the detection target event related to the observation target person,
b) A step of transmitting setting information based on the setting operation to the detection device that detects the detection target event, and
A program that lets your computer run. It is a detection method that detects the detection target event related to the observation target person.
a) The step of accepting the setting operation of the detection target event related to the observation target person,
b) A step of detecting the event to be detected based on the setting contents of the setting operation, and
A detection method characterized by being provided with. The reception unit that accepts the setting operation of the detection target event related to the observation target person,
A control unit that detects the event to be detected based on the setting contents of the setting operation, and
A detection device characterized by being equipped with. A detection device that detects the detection target event related to the observation target person,
A display device that displays an observation image of the person to be observed, and
Equipped with
The detection device generates a skeleton image of the observation target person from the captured image of the observation target person, and also generates the observation image of the observation target person based on the skeleton image.
The detection system, characterized in that the observation image is an image having no captured image of the observation target person itself and having the skeleton image of the observation target person. It is a display control device that controls the display on the display unit in the detection system that detects the detection target event related to the observation target person.
A skeletal image of the observation target person is generated from the captured image of the observation target person, an observation image of the observation target person is generated based on the skeleton image, and the observation image of the observation target person is displayed on the display unit. Control unit to display,
Equipped with
The display control device, characterized in that the observation image is an image having no captured image of the observation target person itself and having the skeleton image of the observation target person.

Images