JP2023132846A - Monitoring system, monitoring software, and monitoring method - Google Patents

Abstract

To provide a monitoring system capable of reducing an erroneous detection rate for persons, monitoring software, and a monitoring method.SOLUTION: A monitoring system includes a storage apparatus, an imaging apparatus, and an arithmetic apparatus which acquires a captured image generated by the imaging apparatus. The arithmetic apparatus performs: determination processing for determining whether a candidate of an immobile body is a mobile body or an immobile body when the candidate of the immobile body is detected in the captured image; storage processing for storing the candidate of the immobile body as one of immobile bodies in the storage apparatus when a result of the determination processing indicates the immobile body; collation processing for collating the candidate of the mobile body with one or more immobile bodies stored in the storage apparatus when the candidate of the mobile body is detected in the captured image; and monitoring processing for determining that the candidate of the mobile body is a mobile body when a result of the collation processing indicates absence of a matched immobile body, and monitoring the mobile body. The arithmetic apparatus does not monitor the candidate of the mobile body when the result of the collation processing indicates presence of the matched immobile body, .SELECTED DRAWING: Figure 1

Description

The present invention relates to a monitoring system, monitoring software, and monitoring method.

Patent Document 1 discloses that by analyzing images taken by a camera, the posture of the cared person and the height of the bedding are detected, and based on these, it is determined whether the cared person is in a dangerous position, and Disclosed is a danger detection device that notifies you when it is determined that a situation has occurred.

JP 2017-10383 Publication

Posture estimation technology that detects people along with their posture in captured images may sometimes mistakenly recognize non-human objects as people. Such false detection of people can be reduced to some extent by removing flat people (or objects) based on 3D distance information from a TOF camera, etc., or by removing unmoving objects using inter-frame differences. is possible. However, flat person removal cannot remove cases where a three-dimensional object such as a chair is incorrectly detected, and frame-to-frame difference cannot remove cases where a difference occurs due to something crossing in front of a three-dimensional object such as a chair. . In these cases, objects other than people may be erroneously and continuously monitored.

Therefore, in a monitoring system that detects a person in a photographed image using a posture estimation technique or the like, there is room for improvement in terms of reducing the rate of false detection of a person.

The present invention has been made in view of the above-mentioned current situation, and it is an object of the present invention to provide a monitoring system, monitoring software, and monitoring method that can reduce the rate of false human detection.

In order to solve the above-mentioned problems and achieve the objectives, (1) a monitoring system according to a first aspect of the present disclosure includes a storage device, a photographing device that generates a photographed image, and an image generated by the photographing device. a computing device that acquires the photographed image, the computing device further comprising: when a candidate for an unmoving object is detected in the photographing image, the candidate for the unmoving object is a moving object or an unmoving object; If the result of the determination process is a non-moving object, a storage process to store a candidate for the non-moving object in the storage device as one of the non-moving objects; When a moving object candidate is detected, a matching process is performed to match the moving object candidate with one or more of the unmoving objects stored in the storage device, and as a result of the matching process, there is no matching unmoving object. In this case, a monitoring process is performed in which the moving object candidate is determined to be a moving object and the moving object is monitored, and if the matching unmoving object is found as a result of the matching process, the moving object candidate is monitored. do not have.

(2) In the monitoring system according to (1) above, when the number of the immovable objects stored in the storage device reaches a predetermined value, the arithmetic device further determines that, when the number of the immovable objects stored in the storage device, the most You can delete the old one.

(3) In the monitoring system according to (1) or (2) above, the arithmetic unit further includes the candidate for the immobile object determined to be a moving object as a result of the determination process, and the result of the matching process. , the moving object candidates determined to be moving objects may be stored in the storage device as one of the moving objects, and in the determination processing, the unmoving object candidates may be stored as one of the moving objects. , a second matching process may be performed to match one or more of the moving objects stored in the storage device, and if there is no matching moving object as a result of the second matching process, the unmoving object candidate may be determined to be an unmoving object, and if there is a matching moving object as a result of the second matching process, the unmoving object candidate may be determined to be a moving object.

(4) In the monitoring system according to (3) above, when the number of the moving objects stored in the storage device reaches a predetermined value, the computing device further selects the oldest of the stored moving objects. may be deleted.

(5) In the monitoring system according to any one of (1) to (4) above, when the result of the determination process is that the object is an immovable object, the arithmetic device does not need to monitor the immovable object candidate. Often, when the result of the determination process is a moving object, the candidate for the unmoving object may be determined to be a moving object, and the moving object may be monitored.

(6) In the monitoring system according to any one of (1) to (5) above, the computing device further includes a detection process for detecting an object whose pose has been estimated in the photographed image, and a detection process for detecting the object by the detection process. A primary determination process may be performed to determine whether the detected object is a moving object candidate or an unmoving object candidate.

(7) Further, the monitoring software according to the second aspect of the present disclosure is monitoring software that operates on a computing device connected to a storage device and a photographing device that generates a photographed image, When a candidate for an unmoving object is detected in the photographed image, the arithmetic unit performs a determination process for determining whether the candidate for the unmoving object is a moving object or an unmoving object, and a determination process that determines whether the candidate for the unmoving object is an unmoving object as a result of the determination process. In some cases, a storage process of storing the candidate of the unmoving object in the storage device as one of the unmoving objects, and when a candidate of the moving object is detected in the captured image, storing the candidate of the moving object in the storage device. a matching process to match one or more of the unmoving objects that have been identified, and a monitoring process that determines that the moving object candidate is a moving object and monitors the moving object if there is no matching unmoving object as a result of the matching process; If there is a matching unmoving object as a result of the matching process, the process of monitoring the moving object candidate is not executed.

(8) In the monitoring software according to (7) above, when the number of the immovable objects stored in the arithmetic device and further in the storage device reaches a predetermined value, among the stored immovable objects, You may also perform a process of deleting the oldest one.

(9) In the monitoring software according to (7) or (8) above, the arithmetic unit further includes a candidate for the immobile object determined to be a moving object as a result of the determination process, and a candidate for the immobile object determined to be a moving object as a result of the determination process, and As a result, a second storage process may be executed in which each of the moving object candidates determined to be a moving object is stored in the storage device as one of the moving objects, and in the determination process, A second matching process may be executed to match the candidate with one or more of the moving objects stored in the storage device, and if there is no matching moving object as a result of the second matching process, the immobile object may be matched. A process of determining that the body candidate is a non-moving object may be executed, and if a matching moving object is found as a result of the second matching process, a process of determining that the non-moving object candidate is a moving object may be executed. You may run it.

(10) In the monitoring software according to (9) above, when the number of the moving objects stored in the arithmetic device and further in the storage device reaches a predetermined value, the oldest of the stored moving objects A process for deleting things may also be executed.

(11) In the monitoring software according to any one of (7) to (10) above, when the result of the determination process is that the object is an immovable object, the arithmetic unit is configured to perform a process of monitoring the immovable object candidate. If the result of the determination process is a moving object, a process of determining the unmoving object candidate to be a moving object and monitoring the moving object may be executed.

(12) In the monitoring software according to any one of (7) to (11) above, the computing device further includes a detection process for detecting an object whose pose has been estimated in the captured image, and a detection process for detecting an object whose pose has been estimated in the captured image. A primary determination process for determining whether the detected object is a moving object candidate or an unmoving object candidate may be performed.

(13) Further, the monitoring method according to the third aspect of the present disclosure includes an image acquisition step of acquiring a photographed image generated by a photographing device, and when a candidate for an unmoving object is detected in the photographed image, a determination step of determining whether an immovable object candidate is a moving object or an immovable object; and if the result of the determination step is an immovable object, storing the immovable object candidate in a storage device as one of the immovable objects; a storage step; when a moving object candidate is detected in the photographed image, a matching step of matching the moving object candidate with one or more of the unmoving objects stored in the storage device; and a result of the matching step. and a monitoring step of determining that the moving object candidate is a moving object if there is no matching unmoving object, and monitoring the moving object.

(14) In the monitoring method described in (13) above, further, when the number of the immovable objects stored in the storage device reaches a predetermined value, the oldest one of the memorized immovable objects is deleted. It may have a step.

(15) In the monitoring method according to (13) or (14) above, the candidate for the immobile object that is determined to be a moving object as a result of the determination step and the candidate that is determined to be a moving object as a result of the collation step are further provided. The method may include a step of performing a second storage process of storing each of the determined moving object candidates in the storage device as one of the moving objects, and in the determination step, the unmoving object candidate is A second matching process may be performed to match one or more of the moving objects stored in the storage device, and if there is no matching moving object as a result of the second matching process, the candidate for the unmoving object is selected. It may be determined that the object is an unmoving object, and if there is a matching moving object as a result of the second matching process, the candidate for the unmoving object may be determined to be a moving object.

(16) The monitoring method according to (15) above further includes the step of deleting the oldest of the stored moving objects when the number of the moving objects stored in the storage device reaches a predetermined value. May have.

(17) In the monitoring method according to any one of (13) to (16) above, further, if the result of the determination step is a moving object, the unmoving object candidate is determined to be a moving object, and the It may also include a step of monitoring.

(18) The monitoring method according to any one of (13) to (17) above further includes a detection step of detecting an object whose pose has been estimated in the captured image; It may also include a primary determination step of determining whether the object is a moving object candidate or an unmoving object candidate.

According to the present disclosure, it is possible to provide a monitoring system, monitoring software, and monitoring method that can reduce the person false detection rate.

1 is a schematic diagram showing the configuration of a monitoring system according to Embodiment 1. FIG. 5 is a flowchart illustrating a process of storing an immovable object by the arithmetic unit of the monitoring system according to the first embodiment. 7 is a flowchart illustrating a specific example of storage processing by the arithmetic unit of the monitoring system according to the first embodiment. FIG. 2 is a schematic diagram illustrating an example of unmoving object candidates detected in a photographed image. FIG. 3 is a schematic diagram showing an example of immovable objects already stored in the storage device of the monitoring system according to the first embodiment. FIG. 3 is a schematic diagram for explaining a method of calculating an area coincidence rate using IoU. 5 is a flowchart illustrating a moving object candidate matching process performed by the arithmetic unit of the monitoring system according to the first embodiment. 7 is a flowchart illustrating a specific example of verification processing performed by the arithmetic unit of the monitoring system according to the first embodiment. FIG. 2 is a schematic diagram illustrating a case where an unmoving object that is not a person is mistakenly detected as a moving object candidate, and shows a scene where a jacket hung on a chair is mistakenly detected. FIG. 2 is a schematic diagram illustrating a case where an unmoving object that is not a person is mistakenly detected as a moving object candidate, and shows a scene in which a person moves while overlapping an erroneously detected jacket. FIG. 3 is a schematic diagram illustrating a case where an unmoving object that is not a person is mistakenly detected as a moving object candidate, and shows a scene where a person passes by a jacket that has been incorrectly detected. 12 is a flowchart illustrating processing by the arithmetic unit of the monitoring system according to the second embodiment. 7 is a flowchart illustrating a specific example of monitoring processing performed by the arithmetic unit of the monitoring system according to the second embodiment.

Hereinafter, embodiments of a monitoring system, monitoring software, and monitoring method according to the present disclosure will be described in detail with reference to the drawings. The monitoring target of the monitoring system, monitoring software, and monitoring method according to the present disclosure is not particularly limited as long as it is a moving object, but an example in which a human target is used will be described below.

Note that, in the following description, repeated descriptions of configurations having the same or similar functions will be omitted as appropriate.

(Embodiment 1)
First, an overview of the monitoring system according to the first embodiment will be explained. In the monitoring system according to the present embodiment, image patterns of non-human non-moving objects that were erroneously detected in the past are registered, image patterns of moving objects detected after that are compared with the registered patterns, and matching registered patterns are checked. If there is, that moving object will not be monitored. Thereby, objects that are erroneously detected as moving objects (for example, objects that are not people) can be removed from the monitoring target.

FIG. 1 is a schematic diagram showing the configuration of a monitoring system according to the first embodiment.

The monitoring system 100 according to this embodiment is used in facilities such as elderly care facilities and hospitals. The monitoring system 100 is a system that detects a person to be monitored, such as a resident or a hospitalized patient, as a moving object from a captured image and monitors the person. As shown in FIG. 1, a monitoring system 100 according to the present embodiment includes a storage device 10, a photographing device 20 that generates photographed images, and a calculation device 30 that acquires photographed images generated by the photographing device 20. We are prepared.

The photographing device 20 is a camera that is installed in the room of the monitored person and photographs the monitored person within the living room area. The photographing device 20 generates a photographed image, for example, an infrared image, and transmits the generated photographic image to the arithmetic device 30 via a communication network. The photographed image is a moving image. Photographing by the photographing device 20 and generation of photographed images may be performed continuously at all times, or may be performed in response to the person to be monitored entering the photographing range.

Note that the living room area is an area where the monitored person lives, is hospitalized, stays for a long period of time, etc. Both a room where a person to be monitored stays alone (a private room) and a room where a plurality of people to be monitored stay together are included in the living room area. When a facility has a plurality of living areas, the imaging device 20 may be provided in each living area, or the imaging device 20 may be provided in some living areas.

Further, the photographing device 20 may be provided in a common area of the facility, and may photograph a person to be monitored within the common area. The common area is an area in which the monitored person is allowed to enter, pass through, and stay, excluding the living room, and includes, for example, a passageway, a dining room, a rehabilitation room, a recreation room, a visiting area, and the like. A plurality of imaging devices 20 may be provided in one shared area.

The computing device 30 is a computer, and is placed in an office of the facility or the like. The computing device 30 analyzes the photographed image generated by the photographing device 20, and performs monitoring when the person to be monitored in the photographed image is a moving object.

The computing device 30 may include a communication section, a display section (display device), an input section, and a control section.

The communication unit is a mechanism that transmits and receives data to and from the photographing device 20 and the storage device 10 via a communication network or the like.

The display unit is a mechanism that displays various screens to present information to facility staff, and specifically includes, for example, a liquid crystal display device.

The input unit is a mechanism that receives operation input, and specifically includes a keyboard, a mouse, and the like.

The control unit is a mechanism that controls the operation of the arithmetic device 30, and executes various processes described below. In reality, each process is executed by storing programs corresponding to each process, such as a monitoring program, in a ROM or non-volatile memory, and loading and executing these programs into a CPU.

Note that the monitoring program may be installed in the monitoring system 100 in advance, or it may be recorded on a computer-readable recording medium as an application program that can run on a general-purpose OS, or provided to the user via a network. may be provided.

The storage device 10 is a device that stores data, and is composed of storage devices such as an HDD, nonvolatile RAM, and ROM. The storage device 10 is used to store programs executed by the arithmetic unit 30 (control unit), temporarily store data during operation of the control unit, and the like.

Next, processing by the arithmetic device 30 will be explained using FIGS. 2 to 11. The main characteristics of the arithmetic device 30 are that it performs processing for storing (registering) immovable objects and processing for comparing moving object candidates with stored immovable objects.

First, the process of storing an immovable object will be explained. FIG. 2 is a flowchart illustrating a process of storing an immovable object by the arithmetic unit of the monitoring system according to the first embodiment.

As shown in FIG. 2, the computing device 30 first acquires a photographed image generated by the photographing device 20 (S10), and when a candidate for an unmoving object is detected in the photographed image, the candidate for the unmoving object is detected. A determination process is performed to determine whether the object is a moving object or an immobile object (S11). This determination process may be performed by comparing the candidate for the unmoving object with a history of past moving objects stored in the storage device 10. This matching process will be described in detail in the second embodiment as a second matching process.

Note that the term "unmoving object candidate" as used herein means an object that has been detected as an unmoving object at least once before the determination process; for example, an unmoving object, etc. detected by inter-frame difference before the determination process. can be mentioned.

If the result of the determination process is an unmoving object (S12: unmoving object), the computing device 30 performs a storage process to store (register) the candidate for the unmoving object in the storage device 10 as one of the unmoving objects (S13). . The information to be registered in the storage device 10 includes, for example, a photographed image of the unmoving object, the posture coordinates of the unmoving object, the angle (orientation) of the photographing device 20 when photographing the unmoving object, and the like.

As a result of the above, an image pattern of an unmoving object (for example, an object that is not a person) to be removed can be registered in the storage device 10.

Note that, as will be described later, when the result of the determination process is an unmoving object, the arithmetic device 30 calculates the matching rate (for example, area A matching rate) may be calculated, and it may be determined whether or not the candidate for the immovable object is stored in the storage device 10 according to the calculated matching rate. For example, if there is an unmoving object with a matching rate equal to or higher than a predetermined threshold, the candidate for the unmoving object is not stored in the storage device 10, and if there is no unmoving object with the matching ratio higher than the threshold, the candidate for the unmoving object is not stored in the storage device 10. may be stored in the storage device 10 as one of the immovable objects.

Further, as shown in FIG. 2, when the result of the determination process is an unmoving object, the arithmetic device 30 does not need to monitor the unmoving object candidate (S14). On the other hand, if the result of the determination process is a moving object (S12: moving object), the arithmetic device 30 may determine that the unmoving object candidate is a moving object, and may monitor the moving object (S15).

The arithmetic device 30 may perform the storage process according to the flow shown in FIG. FIG. 3 is a flowchart illustrating a specific example of storage processing by the arithmetic unit of the monitoring system according to the first embodiment. FIG. 4 is a schematic diagram showing an example of unmoving object candidates detected in a photographed image.

In this case, as shown in FIG. 4, the arithmetic device 30 first calculates the area area 41 of the unmoving object candidate within the photographed image (S20). Specifically, the coordinates of the smallest rotated rectangle that includes all the parts (skeletal points) whose postures have been estimated are determined. FIG. 4 shows an example in which a jacket (three-dimensional object) hung on a chair is erroneously detected.

FIG. 5 is a schematic diagram showing an example of immovable objects already stored in the storage device of the monitoring system according to the first embodiment. FIG. 6 is a schematic diagram for explaining a method of calculating area coincidence rate using IoU.

Next, the calculation device 30 calculates the area matching rate between the calculated area area 41 of the immovable object candidate and the area area 42 of each immovable object already stored in the storage device 10 as shown in FIG. Calculate (S21). FIG. 5 shows an example in which a person (plane) on a poster hung on a wall is stored in the storage device 10 as an immovable object. As the area matching rate, for example, as shown in FIG. 6, IoU (Intersection over Union) can be used.

As a result, if there is an unmoving object with an area matching rate equal to or higher than a predetermined threshold (for example, 50%) (S22: Yes), the arithmetic unit 30 performs storage processing without storing the unmoving object candidate in the storage device 10. (S23), and if it does not exist (S22: No), the arithmetic device 30 stores the immovable object candidate in the storage device 10 as one of the immovable objects (S24). In the examples shown in FIGS. 4 to 6, since the area between the area 41 of the erroneously detected jacket and the area 42 of the registered poster person is 0%, the storage device 10 If there is no other unmoving object, the erroneously detected jacket will be stored in the storage device 10 as an unmoving object.

The arithmetic device 30 may perform storage processing using a FIFO (First In, First Out) method, and as a result of storing the immovable objects in S24, the number of immovable objects stored in the storage device 10 reaches a predetermined value. (For example, 50 items) (S25: Yes), the oldest immovable object stored in the storage device 10 may be deleted (S26). On the other hand, if the number of immovable objects stored in the storage device 10 is less than a predetermined value (for example, 50 objects), the storage process may be ended without being deleted.

Next, a process of comparing moving object candidates with stored unmoving objects will be described. FIG. 7 is a flowchart illustrating a moving object candidate matching process performed by the arithmetic unit of the monitoring system according to the first embodiment.

As shown in FIG. 7, when a moving object candidate is detected in a photographed image, the computing device 30 performs a matching process to match the moving object candidate with one or more unmoving objects stored in the storage device 10. Execute (S30). As will be described later, this matching process may be performed by calculating the difference between the captured image of the moving object candidate and the captured image of the unmoving object to be matched.

Note that the term "moving object candidate" here means an object that has been detected as a moving object at least once before the matching process, and includes, for example, a moving object that has been detected by inter-frame differences before the matching process. .

As a result of the matching process, if there is no matching unmoving object (S31: No), the arithmetic device 30 determines that the moving object candidate is a moving object, and performs a monitoring process to monitor the moving object (S32). On the other hand, if there is a matching unmoving object as a result of the matching process (S31: Yes), the arithmetic device 30 does not monitor moving object candidates (S33). As a result, even if an unmoving object is mistakenly detected as a moving object candidate (for example, when a person, etc. is detected as a moving object candidate by crossing an erroneously detected three-dimensional object that is not a person), the unmoving object If the object is registered in the storage device 10, it can be determined that the object is not a moving object (person) and can be removed from the monitoring target. Therefore, the rate of false human detection by the monitoring system 100 is reduced.

Here, in the case where the moving object candidate matches the unmoving object stored in the storage device 10 in the matching process, the two (both captured images) do not necessarily have to completely match, and the difference between the two captured images does not necessarily have to match completely. It may be less than a predetermined threshold. That is, if the difference between the photographed image of the moving object candidate and the photographed image of the unmoving object to be matched is less than or equal to a predetermined threshold, the calculation device 30 determines that there is a matching unmoving object, and sets the predetermined threshold. If it exceeds the limit, it may be determined that there is no matching immovable object.

In addition, at this time, the immovable objects to be compared may be all the immovable objects stored in the storage device 10, or at least some of the immovable objects stored in the storage device 10, for example, moving object candidates. The object may be an immovable object whose area area matching rate is greater than or equal to a predetermined threshold.

The arithmetic device 30 may perform the matching process according to the flow shown in FIG. FIG. 8 is a flowchart illustrating a specific example of verification processing performed by the arithmetic unit of the monitoring system according to the first embodiment. First, a specific example in which a non-human non-moving object is mistakenly detected as a moving object candidate will be described. FIG. 9 is a schematic diagram illustrating a case where a non-human non-moving object is mistakenly detected as a moving object candidate, and shows a scene where a jacket hung on a chair is mistakenly detected. FIG. 10 is a schematic diagram illustrating a case where an unmoving object that is not a person is mistakenly detected as a moving object candidate, and shows a scene in which a person moves while overlapping an erroneously detected jacket. FIG. 11 is a schematic diagram illustrating a case where an unmoving object that is not a person is mistakenly detected as a moving object candidate, and shows a scene in which a person passes by an erroneously detected jacket.

As shown in Figure 9, if a jacket (three-dimensional object) hanging on a chair is mistakenly detected as a person, the jacket is an immovable object and can be removed from the monitoring target. . However, as shown in FIGS. 10 and 11, when a person moves while overlapping the erroneously detected jacket, that is, when at least a portion of the jacket is hidden, inter-frame differences occur, and as a result, the jacket It will be detected as a moving object candidate. In such a case, although it could not be removed from the monitoring target by plane person removal or inter-frame difference, it can be removed as an unmoving object by the matching process according to the present embodiment.

In the matching process shown in FIG. 8, as shown in FIG. 11, the calculation device 30 first calculates the area area 43 of the moving object candidate within the photographed image (S40). Specifically, the coordinates of the smallest rotated rectangle that includes all the parts (skeletal points) whose postures have been estimated are determined.

Next, the calculation device 30 calculates the area matching rate between the calculated area area 43 of the moving object candidate and the area area of each unmoving object stored in the storage device 10 (S41). The immovable objects stored in the storage device 10 include, for example, the coat hung on the chair shown in FIG. 4, the person in the poster shown in FIG. 5, and the like. For example, IoU can be used as the area matching rate.

Next, the arithmetic device 30 extracts, from among the unmoving objects stored in the storage device 10, unmoving objects whose area matching rate is equal to or higher than a predetermined threshold value (for example, 25%) (S42). In the example shown in Figure 11, the area matching rate with the poster shown in Figure 5 is 0%, so it is not extracted, and the area matching rate with the jacket shown in Figure 4 is 80%. %, so it will be extracted.

At this time, the extracted immovable object is one in which the angle of the photographing device 20 at the time of photographing matches the angle of the photographing device 20 at the time of photographing the moving object candidate (for example, the difference between both angles is within a predetermined threshold). may be limited to cases). That is, an unmoving object whose angle of the photographing device 20 at the time of photographing does not match the moving object candidate may be excluded from the processing targets in the next step S43 and subsequent steps.

Next, the calculation device 30 calculates the difference between the captured image of the moving object candidate and the captured image of each extracted unmoving object (S43). Specifically, the difference in brightness values is calculated between the area area of the moving object candidate in the captured image and the area area of the unmoving object in the captured image.

As a result, if all the differences are below the predetermined threshold (S44: Yes), the arithmetic device 30 determines that there is a matching immovable object and ends the matching process (S45). If even one exists (S44: No), the arithmetic device 30 determines that there is no matching immovable object and ends the matching process (S46).

In the example shown in FIG. 11, since the difference between the area area 43 and the area area 41 of the jacket shown in FIG. If the object is not moving, it will be judged as an immovable object and will not be monitored.

In the present embodiment, when performing monitoring processing for a moving object, the computing device 30 may detect the behavior of the moving object from the captured image, and may determine whether the person to be monitored is in danger based on the detection result. Then, if it is determined that the facility is dangerous, an alarm device may be activated to notify the manager of the facility to that effect. Note that as a method for detecting the behavior of a moving object from a captured image, for example, a behavior detection model based on machine learning can be used.

The alarm device is a device that notifies the staff of the facility that the person to be monitored is in a dangerous state, and is placed in the staff room, nurse station, etc. of the facility. The alarm device may be a liquid crystal display device, an information terminal (smartphone, tablet, etc.) carried by the staff, a speaker that emits sound, a light that emits light, or the like. The alarm device issues an alarm to the staff regarding the person to be monitored who is determined to be in a dangerous state. The alarm may be issued by displaying a screen on the display unit of the alarm device, by sound, by light emission, by vibration, or the like.

As described above, according to the monitoring system 100 according to the present embodiment, it is possible to reduce the person false detection rate.

(Embodiment 2)
In the monitoring system according to the second embodiment, the arithmetic device 30 performs a process of detecting a moving object or a non-moving object candidate from a photographed image before the determination process, and a process of storing (registering) an object determined to be a moving object. The monitoring system according to the first embodiment is different from the monitoring system according to the first embodiment in that the monitoring system further performs processing and also performs a process of comparing unmoving object candidates with stored moving objects in the determination process.

Furthermore, in this embodiment, the imaging device 20 includes a depth sensor such as an FOT camera in addition to an image sensor to obtain three-dimensional coordinates of joints. The depth sensor recognizes the position and depth (depth) of a subject. Note that the depth sensor may be a pattern irradiation type sensor.

Processing by the arithmetic device 30 in this embodiment will be explained using FIGS. 12 and 13. FIG. 12 is a flowchart illustrating processing by the arithmetic unit of the monitoring system according to the second embodiment. Note that in FIG. 12, processes common to those in the first embodiment are given the same reference numerals.

As shown in FIG. 12, in the present embodiment, after acquiring a captured image generated by the imaging device 20 (S10), the computing device 30 first performs a detection process to detect an object whose pose has been estimated in the captured image. (S50). In detail, a human detection model based on machine learning is used to detect an object that is estimated to be a person from a photographed image, extract multiple skeletal points of the object for each time-series frame, and Estimate the coordinate displacement. The coordinates of each skeleton point may be three-dimensional coordinates including depth (distance) information based on a depth sensor. Note that the object detected by this detection process may be an actual person or may be an object that is not a person.

Subsequently, the arithmetic device 30 performs tracking processing of the object detected by the detection processing (S51). Specifically, if the detected object is an already detected object, the position information of the detected object is added to the registered ID. If the detected object is an object that has not been detected yet, a new ID is assigned, and the position information of the detected object is added to the ID.

Next, the calculation device 30 performs a primary determination process to determine whether the object detected by the detection process is a moving object candidate or an immobile object candidate (S52). Specifically, the arithmetic device 30 determines whether the detected object is a moving object or an unmoving object by taking the inter-frame difference of the photographed images, and sets the detected object as a moving object candidate or an unmoving object candidate, respectively.

If the result of the primary determination process is a moving object candidate (S53: moving object), the arithmetic device 30 performs the matching process described in the first embodiment (S30). That is, the moving object candidate is compared with one or more unmoving objects stored in the storage device 10.

If the result of the primary determination process is a candidate for an unmoving object (S53: moving object), the arithmetic device 30 performs the determination process described in Embodiment 1 (S11). That is, it is determined whether the unmoving object candidate is a moving object or an unmoving object. As described above, in the present embodiment, the arithmetic device 30 determines whether the detected object is a moving object or an unmoving object in the primary determination process and the subsequent determination process (secondary determination process). conduct.

Furthermore, the calculation device 30 selects a candidate for an unmoving object that is determined to be a moving object as a result of the determination process (S12: moving object) and a candidate for a moving object that is determined to be a moving object as a result of the matching process (S31: A second storage process is performed to store (register) each moving object as one of the moving objects in the storage device 10 (S54).

Specifically, for example, the amount of activity of the moving object is stored as the amount of human activity in the ID determined in the object tracking process. Here, the human activity amount is a value based on the image difference amount calculated in the primary determination process or the matching process. The amount of human activity is updated at 5 FPS, for example.
It is calculated from the formula: person activity amount (after update) = person activity amount (before update) + image difference amount - activity amount attenuation constant. The activity attenuation constant is a constant determined experimentally. In the primary determination process, the calculation device 30 calculates this amount of human activity, and if the amount of human activity is equal to or greater than a predetermined threshold, it is determined that the object is a moving object candidate, and if it is less than the threshold, it is determined that it is a candidate for an immobile object. It is determined that

The arithmetic device 30 may perform the second storage process using a FIFO method, and if the number of moving objects stored in the storage device 10 reaches a predetermined value as a result of storing the moving objects in S54, The oldest moving object stored in the device 10 may be deleted. On the other hand, if the number of moving objects stored in the storage device 10 is less than the predetermined value, the second storage process may be ended without being deleted.

Then, in the determination process (secondary determination process) in S11, the arithmetic device 30 performs a second collation process in which the unmoving object candidate is collated with one or more moving objects stored in the storage device. If there is no matching moving object as a result of the second matching process, the arithmetic device 30 determines that the unmoving object candidate is an unmoving object (S12: Unmoving object), and as a result of the second matching process, the computing device 30 determines that the unmoving object candidate is an unmoving object. If there is, the arithmetic device 30 determines that the unmoving object candidate is a moving object (S12: moving object). Specifically, the arithmetic device 30 checks whether an object with the same ID as a candidate for a non-moving object is stored in the storage device 10 as a moving object (an object whose amount of human activity is equal to or higher than a predetermined threshold), If there is a history of moving objects, it is determined that there is a matching moving object, and if there is no history of such moving objects, it is determined that there is no matching moving object.

Furthermore, in the present embodiment, the arithmetic device 30 performs a moving object monitoring process after the second storage process (S55).

The arithmetic device 30 may execute the monitoring process (S55) according to the flow shown in FIG. FIG. 13 is a flowchart illustrating a specific example of monitoring processing by the arithmetic unit of the monitoring system according to the second embodiment.

In this case, the calculation device 30 first checks the reliability of the 3D coordinates of the moving object (S60). Specifically, based on the three-dimensional target information of the moving object acquired by the depth sensor, it is determined whether the object is valid as a human. For example, if the moving object exists on a flat surface or if the moving object exhibits an unexpected posture, it is determined that the moving object is unreliable. If the 3D coordinates of the moving object are reliable (S61: Yes), the computing device 30 detects the movement of the moving object from the captured image, and determines whether the person to be monitored is in danger based on the detection result. Good (S62). On the other hand, if the 3D coordinates of the moving object are unreliable (S61: Yes), the arithmetic device 30 may continue to monitor only the moving object without detecting the movement of the moving object (S63).

According to this embodiment as well, similarly to Embodiment 1, it is possible to reduce the person false detection rate.

Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to the above embodiments. Furthermore, the configurations of each embodiment may be combined or modified as appropriate without departing from the gist of the present disclosure.

As described above, the present disclosure is a technique useful for reducing the human false detection rate of a surveillance system using captured images.

100: Monitoring system 10: Storage device 20: Photographing device 30: Arithmetic devices 41, 42, 43: Area area

Claims (8)

a storage device;
a photographing device that generates a photographed image;
A monitoring system comprising: a calculation device that acquires the photographed image generated by the photographing device;
The arithmetic device further includes, when a candidate for a non-moving object is detected in the photographed image, determining whether the candidate for the non-moving object is a moving object or an unmoving object;
If the result of the determination process is a non-moving object, a storage process of storing the candidate of the non-moving object in the storage device as one of the non-moving objects;
When a moving object candidate is detected in the photographed image, a matching process of comparing the moving object candidate with one or more of the unmoving objects stored in the storage device;
As a result of the matching process, if there is no matching unmoving object, performing a monitoring process of determining the moving object candidate as a moving object and monitoring the moving object;
As a result of the matching process, if there is a matching unmoving object, the moving object candidate is not monitored;
A monitoring system characterized by: 2. The arithmetic device further deletes the oldest of the stored immovable objects when the number of the immovable objects stored in the storage device reaches a predetermined value. Surveillance system described in. The arithmetic device further calculates the unmoving object candidate determined to be a moving object as a result of the determination process and the moving object candidate determined to be a moving object as a result of the matching process, respectively. Performing a second storage process to store it as one of the moving objects in the storage device,
In the determination process, a second matching process is performed to match the candidate for the unmoving object with one or more of the moving objects stored in the storage device,
As a result of the second matching process, if there is no matching moving object, determining the unmoving object candidate to be an unmoving object;
3. The monitoring system according to claim 1, wherein if there is a matching moving object as a result of the second matching process, the unmoving object candidate is determined to be a moving object. 4. The arithmetic device further deletes the oldest of the stored moving objects when the number of the moving objects stored in the storage device reaches a predetermined value. monitoring system. If the result of the determination process is that the object is an unmoving object, the arithmetic device does not monitor the unmoving object candidate;
5. The monitoring system according to claim 1, wherein when the result of the determination process is a moving object, the unmoving object candidate is determined to be a moving object, and the moving object is monitored. The arithmetic device further includes a detection process of detecting an object whose posture has been estimated in the photographed image;
A primary determination process for determining whether the object detected by the detection process is a moving object candidate or an immobile object candidate is performed. monitoring system. Monitoring software that operates on a computing device connected to a storage device and a photographing device that generates photographed images,
a determination process in which, when a candidate for a non-moving object is detected in the photographed image, the arithmetic device determines whether the candidate for the non-moving object is a moving object or an unmoving object;
If the result of the determination process is a non-moving object, a storage process of storing the candidate of the non-moving object in the storage device as one of the non-moving objects;
When a moving object candidate is detected in the photographed image, a matching process of comparing the moving object candidate with one or more of the unmoving objects stored in the storage device;
As a result of the matching process, if there is no matching unmoving object, determining the moving object candidate as a moving object and performing a monitoring process of monitoring the moving object;
As a result of the matching process, if there is a matching unmoving object, the process of monitoring the moving object candidate is not executed;
Monitoring software characterized by: an image acquisition step of acquiring a photographed image generated by the photographing device;
a determination step of determining, when a candidate for an unmoving object is detected in the captured image, whether the candidate for the unmoving object is a moving object or an unmoving object;
If the result of the determination step is an immovable object, a storage step of storing the immovable object candidate in a storage device as one of the immovable objects;
When a moving object candidate is detected in the captured image, a matching step of comparing the moving object candidate with one or more of the unmoving objects stored in the storage device;
a monitoring step of determining that the moving object candidate is a moving object and monitoring the moving object if there is no matching unmoving object as a result of the matching step;
A monitoring method characterized by:

Images