JP7089493B2 - Person detectors, methods and programs - Google Patents

Description

The present invention relates to a person detection device, a method and a program, and more particularly to a person detection device, a method and a program for detecting a person who has fallen on a camera image.

Research is being conducted to analyze the movement of each person by extracting the person's area based on the camera image of the person taken outdoors from above from the request for security and crime prevention. In the past, such outdoor cameras were fixedly installed at high places in structures such as buildings, but in recent years, since it has become possible to mount small cameras on so-called drones, any location can be used. It is easy to shoot at the angle and orientation of, and it is expected to be applied to various fields.

Non-Patent Document 1 discloses a technique for identifying an object based on the color characteristics of the image by machine learning.

Non-Patent Document 2 discloses a technique for extracting a human skeleton from only an image.

Patent Document 1 discloses a technique for quickly and accurately tracking an unauthorized intruder into a detection area for detecting an intruder at a monitored site.

Patent Document 2 describes an image capable of efficiently monitoring a suspicious person who may have delivered with a high degree of suspicion in a monitoring area where an unspecified number of people gather and contact each other exclusively for welcoming. The monitoring device is disclosed.

Patent Document 3 discloses a technique of photographing a person from a camera directly above and determining whether or not the person has fallen based on the area of the person area.

Japanese Patent Application No. 2017-216477 Japanese Patent Application No. 2017-238178 Japanese Unexamined Patent Publication No. 2000-207664

SSD: Single Shot Multi Detector (https://arxiv.org/pdf/1512.02325.pdf) Z. Cao, T. Simon, S. Wei and Y. Sheikh, "Realtime Multi-person 2D Pose Optimization Using Part Affinity Fields," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp . 1302-1310.

As shown in FIG. 9, when a person is photographed from above, the upper body of a person in a standing position is located above the lower body regardless of the angle or orientation, whereas the upper body of a person in a fall state is above the lower body. It will not be positioned. Therefore, if the skeletal information of each person can be analyzed and the relative positional relationship between the upper body and the lower body can be identified, it is possible to relatively accurately recognize whether the person is in a standing state or a fallen state. Can be done.

On the other hand, as shown in FIG. 10, when a person falls backward from the camera viewpoint, the camera image appears as if the upper body is located above the lower body, as in the case of shooting a standing person. Therefore, it was difficult to determine whether the person was in a standing state or a fallen state based on the skeletal information.

In Patent Document 3, if a camera image of a person is taken from directly above, it can be determined whether the person has fallen or not. However, it has been difficult to detect a fallen state, for example, on an image such as a bird's-eye view of a person taken at a high place in a building from above and diagonally, especially when the person falls backward from the camera's point of view. ..

An object of the present invention is to solve the above-mentioned technical problems and to provide a person detection device, a method and a program capable of reliably discriminating a person who is in a fall state regardless of the positional relationship between the fall direction and the camera. ..

In order to achieve the above object, the person detection device of the present invention is characterized in that it has the following configurations.

(1) A means for extracting a person area from a camera image, a means for calculating the aspect ratio of the extracted person area, and an aspect ratio for determining whether or not the person is in a fallen state based on the aspect ratio of the person area. It is equipped with a base determination means.

(2) A means for extracting skeleton information from a person area and a skeleton information-based determination means for determining whether or not the person is in a fall state based on the skeleton information are provided, and the aspect ratio-based determination means is a skeleton. The information-based determination means now determines whether or not a person who has not been determined to be in a fall state is in a fall state.

(3) The skeletal information-based determination means determines that the person is in a fall state if the upper body of the person or the main part thereof is not located above the lower body or the main part thereof.

(4) The standard value R _ref of the aspect ratio of the person area on the camera image of the standing person and its range ΔR _ref are held, and the difference between the aspect ratio of the extracted person area and the standard value R _ref is When the person goes out of the range ΔR _ref , the person is judged to be in a fallen state.

(5) The optical axis angle θ _cam of the camera that outputs the camera image is acquired, and the standard value R _ref is calculated based on the optical axis angle.

According to the present invention, the following effects are achieved.

(1) Since it is determined whether or not the person is in a fall state based on the aspect ratio of the person area, even if the person falls backward from the camera viewpoint, the fall state can be reliably detected.

(2) Since the fall state is determined based on the skeletal information base and whether or not the person who is not determined to be in the fall state on the skeleton information base is in the fall state based on the aspect ratio of the person area, the aspect ratio of the person area is determined. Even if the fall state cannot be detected accurately only by the ratio, the person in the fall state can be reliably detected.

(3) If the upper body or the main part of the person is not located above the lower body or the main part of the person, the person is judged to be in a fall state, so that the person in the fall state can be reliably detected.

(4) If the difference between the aspect ratio of the person area and the standard value R _ref deviates from the range ΔR _ref , the person is judged to be in a fallen state. Will be able to detect.

(5) Since the optical axis angle θ _cam of the camera is acquired and the standard value R _ref of the aspect ratio of the person area is calculated based on this optical axis angle, the camera is mounted on the drone or is a handy camera. Even if the standard value R _ref changes according to the change in the optical axis angle θ _cam , it becomes possible to reliably detect a person in a fallen state based on the aspect ratio (H / W) of the person area.

It is a functional block diagram which showed the structure of the main part of the person detection apparatus which concerns on one Embodiment of this invention. It is a figure which showed the Example when the optical axis angle θ _cam of a fixed camera is sufficiently smaller than 45 °. It is a figure which showed the Example when the optical axis angle θ _cam of a fixed camera is sufficiently larger than 45 °. It is a figure which showed the example of the aspect ratio (H / W) of the person area of the person in a standing state. It is a figure which showed the example of the aspect ratio (H / W) of the person area of a person in a fall state. It is a flowchart which showed the operation of 1st Embodiment of this invention. It is a functional block diagram which showed the structure of the main part of the person detection apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which showed the operation of the 2nd Embodiment of this invention. It is a figure which showed the example of the camera image which took the person from above. It is a figure which showed the example of the camera image when a person falls backward from the viewpoint of a camera.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a configuration of a main part of a person detection device 1 according to an embodiment of the present invention, and detects a person who has fallen particularly based on a moving image taken by a fixed camera 2. do.

Such a person detection device 1 can be configured by mounting an application (program) that realizes each function described later on a general-purpose computer or server provided with a CPU, a memory, an interface, a bus connecting them, and the like. Alternatively, it can be configured as a dedicated machine or a single-purpose machine in which a part of the application is made into hardware or programmed.

In the person detection device 1, the frame image acquisition unit 101 acquires a frame image from a camera image showing a person. The person area extraction unit 102 extracts a person area from the frame image. The skeleton information extraction unit 103 extracts skeleton information from the extracted person area.

A technique for extracting a person area as a rectangular Bounding Box from a frame image of a camera image is disclosed in, for example, Non-Patent Document 1. A technique for extracting skeletal information from a person's domain and analyzing and recognizing a person's behavior is disclosed in, for example, Non-Patent Document 2.

The body placement detection unit 104 detects the placement of each body part based on the extracted skeletal information. In the present embodiment, the positions of the upper body and the lower body are detected based on the skeletal information, and the relative positional relationship between the lower body and the upper body is detected. Instead of the upper body, the head and neck, which are the main parts thereof, may be detected. Similarly, instead of the lower body, the thighs and lower legs, which are the main parts thereof, may be detected.

Based on the detection result of the body arrangement, the ratio calculation unit 105 calculates the aspect ratio (H / W) of the person area of a person whose upper body is located above the lower body and can be estimated to be in a standing position. do. However, if the upper body or the main part of the person, or the lower body or the main part cannot be detected, the aspect ratio (H / W) is not calculated. Whether or not the upper body is located above the lower body is determined by the rectangular frame circumscribing the skeleton classified as the upper body when the coordinates of each skeleton or joint are given as the skeletal information of the person. Comparing the relative positions with the rectangular frame circumscribing the skeleton classified in the lower body, the rectangular frame of the upper body is located above the rectangular frame of the lower body, and the horizontal overlap of each rectangular frame is more than a predetermined ratio. If so, it can be determined that the upper body or the like is located above the lower body or the like.

The fall detection unit 106 detects a person who has fallen based on the detection result of the body placement detection unit 104 and the calculation result of the ratio calculation unit 105. In the present embodiment, a person whose upper body is not located above the lower body is determined to be in a fall state based on the detection result by the body placement detection unit 104. On the other hand, a person whose upper body is located above the lower body is determined whether or not he / she is in a fall state based on the aspect ratio (H / W).

In the present embodiment, a person whose aspect ratio (H / W) of the person area satisfies the following equation (1) is determined to be in a standing state, and a person who does not satisfy the following equation (1) is in a standing state or a similar standing state. It is judged to be in a state other than. Here, R _ref is the standard aspect ratio (H / W) when a person in a standing position is photographed at the optical axis angle θ _cam of the fixed camera 2, and ΔR _ref is the aspect ratio (H / W). It is an individual difference range.

｜ H / WR _ref ｜ ≦ ΔR _ref ... (1)

For example, as shown in FIG. 2, when the optical axis angle θ _cam formed by the optical axis O _cam of the fixed camera 2 and the ground surface is sufficiently smaller than 45 °, it is assumed that the fixed camera 2 is located sufficiently far away. Then, as shown in FIGS. 4 and 5, the height H _L in the projected image of the fallen person P _L is lower than the height H _S in the projected image of the standing person Ps, while on the projected image. There is no difference in the width W of. Therefore, the H / _W of the person PL in the fallen state is significantly smaller than the standard aspect ratio R _ref in the standing state corresponding to the optical axis angle θ _cam , and the H / W and R _ref Exceeds the individual difference range ΔR _ref . As a result, since the above equation (1) is not established, it becomes possible to reliably detect the person P _L who has fallen.

Similarly, as shown in FIG. 3, when the optical axis angle θ _cam of the fixed camera 2 is sufficiently larger than 45 °, the height H _L in the projected image of the person P _L in the fallen state is that of the person Ps in the standing state. While it is taller than the height H _S in the projected image, there is no difference in the width W on the projected image. Therefore, the H / _W of the person PL in the fallen state is significantly larger than the standard aspect ratio R _ref in the standing state corresponding to the optical axis angle θ _cam , and the H / W and R _ref Exceeds the individual difference range ΔR _ref . As a result, since the above equation (1) is not established, it becomes possible to reliably detect the person P _L in the fallen state.

FIG. 6 is a flowchart showing the operation of the present embodiment, and in step S1, the frame image acquisition unit 101 acquires a still image from the camera image in frame units. In step S2, the person area extraction unit 102 extracts the person area from the frame image. In step S3, the skeleton information extraction unit 103 extracts skeleton information from the person area.

In step S4, the body placement detecting unit 104 identifies the upper body or its main part and the lower body or its main part based on the skeletal information, and in step S5, whether or not the upper body or the like is located above the lower body or the like. Is judged. If the upper body or the like is not located above the lower body or the like, the process proceeds to step S9, and it is determined that the person has fallen.

On the other hand, if the upper body or the like is located above the lower body or the like, the process proceeds to step S6, and the ratio calculation unit 105 measures the aspect ratio (H / W) of the person area. In step S7, the fall detection unit 106 determines whether or not the above equation (1) holds. If the above equation (1) is not satisfied, the process proceeds to step S9, and it is determined that the person has fallen backward. On the other hand, if the above equation (1) is satisfied, the process proceeds to step S8, and it is determined that the person has not fallen.

FIG. 7 is a functional block diagram showing the configuration of the main part of the person detection device 1 according to the second embodiment of the present invention, and represents the same or equivalent part as described above, and thus the description thereof will be omitted.

In the above first embodiment, it has been described that the camera 2 is fixed and the standard value R _ref regarding the aspect ratio of the person area is registered in advance according to the angle θ _cam between the optical axis and the ground surface. In the present embodiment, the case where the camera 2 is mounted on the drone or is a handy camera and the standard value R _ref cannot be registered in advance because the optical axis angle θ _cam is variable will be described as an example. do.

The camera 2 detects its own posture with a motion sensor and outputs an optical axis angle θ _cam with respect to the ground surface. The fall detection unit 106 includes an R _ref calculation unit 106a, and dynamically calculates a standard value R _ref of the aspect ratio of the person region at the optical axis angle θ _cam based on the optical axis angle θ _cam . The relationship between the optical axis angle θ _cam and the standard value R _ref can be registered in advance in, for example, a map format.

The fall detection unit 106 applies the R _ref calculated by the R _ref calculation unit 106a based on the optical axis angle θ _cam to the above equation (1) together with the calculation result of the aspect ratio (H / W) of the person area. , Determine if the person has fallen.

FIG. 8 is a flowchart showing the operation of the second embodiment of the present invention, and since the same or equivalent processing is executed in the procedure with the same step number as described above, the description thereof will be omitted.

In the present embodiment, when it is determined in step S5 that the upper body is located above the lower body and the standing / falling state cannot be determined only from the skeleton information, in step S6a, the optical axis angle θcam is determined from the camera 2. To be acquired. In step S6b, the R _ref is acquired by referring to the θ cam / R _ref map based on the optical axis angle θ _cam . In step S6c, the aspect ratio (H / W) of the person area is measured.

According to this embodiment, since the standard value R _ref is dynamically changed based on the optical axis angle θ _cam of the camera, it is possible to accurately detect a fallen person even if the optical axis angle θ _cam changes during shooting. It will be like.

1 ... person detection device, 2 ... fixed camera, 101 ... frame image acquisition unit, 102 ... person area extraction unit, 103 ... skeleton information extraction unit, 104 ... body placement detection unit , 105 ... Ratio calculation unit, 106 ... Fall detection unit

Claims (10)

A means of extracting a person area from a camera image,
A means for calculating the aspect ratio of the extracted person area, and
A means for extracting skeletal information from the extracted person area,
A skeleton information-based determination means for determining whether or not the person is in a fall state based on the skeleton information,
It is characterized by being provided with an aspect ratio-based determination means for determining whether or not the person is in a fallen state based on the aspect ratio of a person who is not determined to be in a fallen state based on the skeletal information. Person detection device. A means of extracting a person area from a camera image,
A means for calculating the aspect ratio of the extracted person area, and
It is provided with an aspect ratio-based determination means for determining whether or not the person is in a fallen state based on the aspect ratio.
The aspect ratio-based determination means holds a standard value Rref of the aspect ratio of the person area on the camera image of the person in the standing state and its range ΔRref, and sets the aspect ratio of the extracted person area and the standard value Rref. A person detection device characterized in that when the difference deviates from the range ΔRref, the person is determined to be in a fall state. A means for extracting skeletal information from the extracted person area,
A skeleton information-based determination means for determining whether or not the person is in a fallen state based on the skeleton information is provided.
The person detection device according to claim 2 , wherein the aspect ratio-based determination means determines whether or not a person who has not been determined by the skeletal information-based determination means to be in a fall state is in a fall state. The skeletal information-based determination means according to claim 1 or 3 , wherein the person is determined to be in a fall state if the upper body of the person or the main part thereof is not located above the lower body or the main part thereof. Person detection device. When the standard value Rref of the aspect ratio of the person area on the camera image of the standing person and its range ΔRref are held, and the difference between the aspect ratio of the extracted person area and the standard value Rref deviates from the range ΔRref, The person detection device according to claim 1, wherein the person is determined to be in a fallen state. A means to acquire the optical axis angle θcam of the camera that outputs the camera image,
The person detection device according to claim 2 or 5 , further comprising means for calculating the standard value Rref based on the optical axis angle. The computer
Extract the person area from the camera image and
The aspect ratio of the extracted person area is calculated, and the aspect ratio is calculated.
The skeleton information is extracted from the extracted person area, and the skeleton information is extracted.
Based on the skeletal information, it is determined whether or not the person is in a fallen state.
A person detection method for determining whether or not a person who has not been determined to be in a fallen state based on the skeletal information is in a fallen state based on the aspect ratio. The computer
Extract the person area from the camera image and
The aspect ratio of the extracted person area is calculated, and the aspect ratio is calculated.
Based on the aspect ratio, it is determined whether or not the person is in a fallen state.
When the standard value Rref of the aspect ratio of the person area on the camera image of the standing person and its range ΔRref are held, and the difference between the aspect ratio of the extracted person area and the standard value Rref deviates from the range ΔRref, A person detection method characterized in that the person is determined to be in a fallen state. The procedure for extracting the person area from the camera image and
The procedure for calculating the aspect ratio of the extracted person area and
The procedure for extracting skeletal information from the extracted person area and
A procedure for determining whether or not the person is in a fall state based on the skeletal information, and
For a person who was not determined to be in a fall state based on the skeletal information, a procedure for determining whether or not the person is in a fall state based on the aspect ratio, and
A person detection program that causes a computer to execute. The procedure for extracting the person area from the camera image and
The procedure for calculating the aspect ratio of the extracted person area and
A computer is made to perform a procedure for determining whether or not the person is in a fallen state based on the aspect ratio.
In the above-mentioned procedure, the standard value Rref of the aspect ratio of the person area on the camera image of the standing person and its range ΔRref are held, and the difference between the aspect ratio of the extracted person area and the standard value Rref is obtained. A person detection program characterized in that when the person deviates from the range ΔRref, the person is determined to be in a fall state.

Images