JP7378223B2 - Detection device and its control method - Google Patents

Description

The present invention relates to a technology for detecting objects included in images.

A system that detects when a person has entered a facility by installing surveillance cameras and analyzing the images obtained by the surveillance cameras, with the aim of ensuring the safety of facilities such as apartments, buildings, school buildings, station buildings, and airports. has been introduced. Regarding detection of a human body region in a video, Patent Document 1 discloses a technique of detecting a human body by cascading strong classifiers that detect parts. Further, Patent Document 2 discloses an intruder monitoring device that determines whether a person has entered a prohibited area on an image.

U.S. Patent Application Publication No. 2007/237387 Japanese Patent Application Publication No. 9-50585

By the way, the above-mentioned conventional technology is configured to detect a human body included in an image without distinguishing the state of the human body. Therefore, when an image as shown in FIG. 1 is captured by a camera, a human body (driver 102) riding in a vehicle and a human body (pedestrian 103) walking on the road are detected. It turns out. For example, it is determined that a human body is detected in the broken line rectangle 104 and the broken line rectangle 105, and tracking is started. Then, for example, when the tracked human body passes through the intrusion detection line 110, a warning indicating that an intruder has been detected is output. In other words, it is not possible to support a use case where it is desired to exclude the entry of a vehicle from the detection target (i.e., to exclude the human body inside the vehicle from the detection target).

The present invention has been made in view of such problems, and an object of the present invention is to provide a technology that allows objects to be distinguished and detected according to their states.

In order to solve the above-mentioned problems, a detection device according to the present invention has the following configuration. That is , the detection device includes a detection means that detects a plurality of parts of an object included in a frame image of a moving image, and determines whether the object is a pedestrian depending on whether the parts detected by the detection means include a predetermined part. determination means for determining whether the person is a passenger of the vehicle or not.

According to the present invention, it is possible to provide a technology that allows objects to be distinguished and detected according to their states.

FIG. 3 is a diagram illustrating an example of intrusion detection. FIG. 3 is a diagram illustrating the principle of intrusion detection in the first embodiment. FIG. 3 is a diagram illustrating logic for determining the state of a human body based on detection of a plurality of parts. 1 is a diagram illustrating a hardware configuration of a detection system; FIG. FIG. 2 is a diagram illustrating a functional configuration of a detection device. FIG. 6 is a diagram showing an example of a screen for setting conditions for intrusion detection. It is a flowchart of intrusion detection processing in a 1st embodiment. It is a flowchart of intrusion detection processing in a 2nd embodiment.

An example of an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the following embodiments are merely illustrative, and are not intended to limit the scope of the present invention.

(First embodiment)
As a first embodiment of the detection device according to the present invention, a detection system including a camera that images a predetermined monitoring area and a detection device that detects an object from a moving image acquired by the camera will be exemplified below. Explain. In the following description, a human body is assumed as the object to be detected, but other objects may be used.

<System configuration>
FIG. 4 is a diagram illustrating the hardware configuration of the detection system. As mentioned above, the detection system includes a detection device 400 and a camera 450. Note that although the detection device 400 is shown here as having a configuration in which it receives video from one camera 450 via the network 430, it may be configured to receive video from a plurality of cameras. Alternatively, a configuration may be adopted in which a camera is mounted on the detection device 400 itself and images are directly acquired without going through the network 430.

The detection device 400 includes a CPU 401, a ROM 402, a RAM 403, an external storage I/F 404, an input/output I/F 405, and a network I/F 406. The ROM 402 stores programs executed by the CPU 401 and various setting data. The CPU 401 implements each functional unit described later with reference to FIG. 5 by loading a program stored in the ROM 402 into the RAM 403 and executing it.

The external storage I/F 404 is an interface for connecting an external storage device 407 such as a hard disk drive (HDD). The input/output I/F 405 is an interface for connecting to an input device such as a keyboard 421, a mouse 422, a touch panel (not shown), and an output device such as a display 423. Network I/F 406 is an interface for connecting to network 430 and communicating with external devices such as camera 450. The detection device 400 can be realized by, for example, a general-purpose PC (personal computer), a smartphone, or a tablet, and is not dependent on a specific device type.

The camera 450 includes a CPU 451, a ROM 452, a RAM 453, an imaging section 454, and a network I/F 455. The ROM 452 stores programs executed by the CPU 451 and various setting data. The CPU 451 realizes imaging processing by the imaging unit 454 and communication processing via the network I/F 455 by loading a program stored in the ROM 452 into the RAM 453 and executing it. The imaging unit 454 includes an imaging optical system and an imaging element such as a CCD or CMOS, and generates a moving image (a plurality of frame images) by imaging. A configuration may also be used in which compression encoding and the like are also performed.

FIG. 5 is a diagram illustrating the functional configuration of the detection device. The detection device 400 includes a communication unit 501, an image acquisition unit 502, a preprocessing unit 503, a human body detection unit 510, a human body tracking unit 504, a state determination unit 505, an intrusion determination unit 506, a result generation unit 508, and an area setting unit as functional units. 507. The human body detection unit 510 also includes a plurality of detection units (head detection unit 511, face detection unit 512, upper body detection unit 513, whole body detection unit 514) each detecting a different target region, and a result integration unit 515. include.

The communication unit 501 performs communication with an external device via the network 530. For example, a compression-encoded moving image is received from the camera 450. The image acquisition unit 502 decodes the acquired moving image and stores it in the external storage device 407, for example, as a plurality of frame images. The preprocessing unit 503 corrects the frame image so that feature amounts can be easily extracted.

The human body detection unit 510 detects a human body included in a frame image. Specifically, a head detection section 511, a face detection section 512, an upper body detection section 513, and a whole body detection section 514 each execute a detection process, and a result integration section 515 integrates the detection results to detect a human body. Details of the detection process will be described later with reference to FIGS. 2 and 3.

The human body tracking unit 504 associates the human body detected by the human body detection unit 510 between a plurality of frame images and tracks the human body. The state determining unit 505 determines the state of the human body detected by the human body detecting unit 510. Here, it is determined whether the person is a driver or a pedestrian. The area setting unit 507 sets an image area that is a predetermined area for determining intrusion detection. Here, as shown in Figure 2, it is assumed that the boundary line is set as an intrusion detection line in the shape of a straight line (line segment) for the imaging area shown by the moving image, but it is assumed that the boundary line is set as an intrusion detection line in the shape of a straight line (line segment), but it can also be An intrusion detection line may also be set. Alternatively, it may be set as an intrusion detection area of arbitrary shape. When setting as a region, for example, intrusion detection is determined on the condition that the target human body exists inside the region for a predetermined period of time or more.

The intrusion determination unit 506 determines the intrusion of a human body to be detected, according to the setting by the area setting unit 507. The result generation unit 508 generates information on the detection result by the intrusion determination unit 506 and issues an alarm notification. For example, a warning is displayed on the display unit 522 or a warning notification is sent to an external device (not shown) via the network 530.

<Principle of intrusion detection>
FIG. 2 is a diagram illustrating the principle of intrusion detection in the first embodiment. Similar to FIG. 1, in FIG. 2, an image including a human body riding in a vehicle 201 (driver 202) and a human body walking on the road (pedestrian 203) was captured by a camera. It shows the condition.

An intrusion detection line 210 is set for the video, and the detection device performs an intrusion detection determination when a person passes through the intrusion detection line 210. Further, part detection information 204 and 205 exemplarily shows information indicating detection results of four parts detected by four discriminators included in the detection device.

In the first embodiment, the detection device determines the state of the human bodies 202 and 203 that have passed through the intrusion detection line 210 based on the part detection information 204 and 205. Specifically, it is determined whether the human body is a driver or a pedestrian. Then, it is determined that the human body 202, which is the driver, is not an intruder, and no warning notification is given. On the other hand, a human body 203 that is a pedestrian (a human body that is not the driver) is determined to be an intruder, and a warning notification is given. Note that here, for convenience, the human body riding in the vehicle is referred to as a "driver," but the same applies to other vehicle occupants (human bodies in the passenger seat or rear seat).

FIG. 3 is a diagram illustrating the logic for determining the state of a human body based on detection of a plurality of parts. FIG. 3(a) shows the flow of the human body state determination process, and FIG. 3(b) shows a table showing determination results based on combination patterns of part detection results.

The classifiers 302, 304, 306, and 308 perform detection processing corresponding to different parts (body regions) of the human body, respectively, on the frame images. The classifiers 302, 304, 306, and 308 correspond to the head detection section 511, face detection section 512, upper body detection section 513, and whole body detection section 514 in FIG. That is, in the first embodiment, the "parts" are assumed to be "face 301," "head 303," "upper body 305," and "whole body 307."

Each classifier (strong classifier) is configured, for example, by cascading a plurality of weak classifiers. Weak classifiers detect patterns in image features such as edges and colors. Note that the optimal detection pattern for image features may be obtained by machine learning.

In human body detection processing 309, a human body is detected based on the part detection results of each classifier (strong classifier). The human body detection 309 corresponds to the result integration unit 515 in FIG. Note that, for example, an evaluation function based on a weighted sum of each part can be used to integrate the detection results.

In the state determination process 310, the state of the detected human body is determined based on the region detection results by each classifier (strong classifier). Here, based on the combination pattern of the part detection results, it is determined whether the detected human body is a "driver", a "pedestrian", or "unknown" indicating that it is something else.

For example, as shown in FIG. 3B, if at least one of a face or a head is detected in the frame image and no other parts are detected, the human body is determined to be a driver. Further, if the whole body is detected in the frame image, the human body is determined to be a pedestrian. In other words, when the whole body is detected in a frame image, it is a case where the entire human body is detected, including the lower body, which is unlikely to be detected in the case of a vehicle occupant such as a driver. It is determined that the On the other hand, if the whole body is not detected in the frame image, this means that the entire human body including the lower body has not been detected, so it is determined that the person is a vehicle occupant such as a driver. In addition, in the table of FIG. 3(b), "O" indicates that the detection has occurred, and "x" indicates that the detection has not occurred. Further, "-" indicates that it does not depend on the presence or absence of detection (it is not used for state determination). Note that the table in FIG. 3(b) is merely an example, and other combinations of human body parts may be used, and other conditions may be determined. Furthermore, for example, the plurality of detection units included in the human body detection unit 510 may include two or more detectors among the four described above. For example, it is configured to include two, a head detection section 511 and a whole body detection section 514, and the state is determined based on the detection results by the two detectors. Further, in this embodiment, the configuration includes the four detectors described above, but a configuration may also include a detector for detecting the lower body of the human body. In the case of a vehicle occupant such as a driver, there is a low possibility that the lower body of the human body will be detected in the frame image, so if the lower body is detected by the detector, it is determined that the person is a pedestrian. On the other hand, if the lower body is not detected by the detector, it is determined that the person is a vehicle occupant such as a driver. Examples of the detector that detects the lower body include a detector that detects the entire lower body from the waist down, a detector that detects the area below the ankle, a detector that detects the legs, and the like.

FIG. 6 is a diagram showing an example of a screen for setting conditions for intrusion detection. Here, it is assumed that settings for a detection target, detection area, and detection determination point are received via a dialog 600 displayed on the display 423. Of course, it may be configured to accept other conditions. Here, the dialog 600 includes a radio button 601 for selecting a detection target, a setting button 602 for setting an area, and a display 603 for displaying a detection determination point. Further, an OK button 604 for confirming the settings and a cancel button 605 for discarding the settings are further arranged.

The radio button 601 for selecting a detection target accepts one selection from a plurality of options by clicking with the mouse 422 when limiting the intrusion detection target. For example, "exclude driver" excludes a human body determined to be a "driver" from intrusion detection targets. That is, a human body determined to be a "pedestrian" or "unknown" becomes an intrusion detection target. Furthermore, "detect only pedestrians" excludes human bodies determined to be "unknown" as well as human bodies determined to be "drivers" from intrusion detection targets.

A settings button 602 accepts settings for an area for determining intrusion detection. For example, when the setting button 602 is pressed by clicking with the mouse 422, the scene shown in FIG. 2 is displayed, and a user interface (UI) for accepting changes to the intrusion detection line 210 is displayed. The UI corresponds to the area setting section 507 in FIG. 5. As described above, in addition to being able to set it as an intrusion detection line (line), it is also possible to set it as an intrusion detection region (area). Further, a detection determination point display 603 is a display indicating a detection point in the area for determining intrusion detection. For example, the feet, center, right, left, etc. are displayed as detection points.

<Device operation>
FIG. 7 is a flowchart of intrusion detection processing in the first embodiment. As described above, each functional unit shown in FIG. 5 is realized by the CPU 401 reading a program stored in the ROM 402 into the RAM 403 and executing it.

In S701, the image acquisition unit 502 acquires a moving image from the camera 450 via the network 530. The acquired moving image is then decoded and stored in the external storage device 407, for example, as a plurality of frame images.

In S702, the preprocessing unit 503 corrects the frame image so that feature amounts can be easily extracted, and the human body detection unit 510 detects the human body included in the frame image. Here, as described above, each of the four detection units 511 to 514 performs processing to detect each of the four parts of the human body (head, face, upper body, and whole body). Then, a result integration unit 515 integrates the detection results from the four detection units to detect a human body.

In S703, the human body tracking unit 504 tracks the human body detected by the human body detection unit 510 over a plurality of frame images. For example, a history of human body detection results in preceding (past) frame images is stored in association with each other. Note that in the first embodiment, all human bodies detected by the human body detection unit 510 are tracked.

In S<b>704 , the intrusion determination unit 506 determines intrusion of a human body to be detected according to the setting by the area setting unit 507 . For example, it is determined that the movement vector of the human body intersects the intrusion detection line 210 in a predetermined direction.

In S705, the state determination unit 505 determines the state of the human body that intersects with the intrusion detection line 210. Specifically, it is determined whether the vehicle is a driver, a pedestrian, or an unknown person using the method described with reference to FIG. If the person is the driver, the process advances to S707; if the person is not the driver (that is, if the person is a pedestrian or unknown), the process advances to S706. Further, in S706, if the person is a pedestrian, the process advances to S708, and if the person is not a pedestrian (that is, if it is unknown), the process advances to S709.

In S707, the intrusion determination unit 506 records the target human body as a normal intruder. On the other hand, in S708, the intrusion determination unit 506 records the target human body as an abnormal person (pedestrian). Further, in S709, the intrusion determination unit 506 records the target human body as an abnormal intruder (unknown). Then, the intrusion determination unit 506 records the abnormal intruder in S708 or S709, and then issues an intrusion detection event.

In S711, the intrusion determination unit 506 determines whether the intrusion detection determination in S704 has been performed for all human bodies detected in S702 and tracked in S703. If there is a human body that has not been determined, the process returns to S704 and the remaining human bodies are determined. If the determination for all human bodies is completed, the process advances to S712.

In S712, when an intrusion detection event is issued, the result generation unit 508 issues an alarm notification to make the user aware that an abnormal intruder has been detected. For example, a warning is displayed on the display unit 522 or a warning notification is sent to an external device (not shown) via the network 530.

In S713, the detection device 400 makes an end determination. For example, if the user issues an instruction to end the process, the process ends. If there is no instruction, the process returns to S701 and continues.

As described above, according to the first embodiment, a human body is detected based on the detection results of a plurality of detectors, and the state of the human body (driver, pedestrian, unknown) is determined. The issuance of an intrusion detection event is then controlled based on the determined state. For example, if it is determined that the human body is a driver, the issuance of an intrusion detection event is suppressed. This makes it possible, for example, to suppress warning notifications regarding intrusion of a vehicle (intrusion of a human body aboard a vehicle). In other words, it becomes possible to correspond to a use case where it is desired to exclude the entry of a vehicle from the detection target.

Note that in the above description, a human body is assumed as the object to be detected, but various objects can be designated as the object to be detected. That is, any object may be used as long as it can take on a plurality of states based on detection results from a plurality of discriminators.

(Second embodiment)
In the second embodiment, a mode will be described in which the number of human bodies targeted for tracking and intrusion detection is reduced and the processing load is reduced. Specifically, a mode will be described in which the state of a detected human body is determined, and human bodies (pedestrians and unknown persons) excluding the human body (driver) in a specific state are tracked and intrusion detection is determined. The principle of intrusion detection, state determination logic, and hardware configuration are the same as those in the first embodiment (FIGS. 2 to 4), so explanations thereof will be omitted.

Further, the functional configuration in the second embodiment is almost the same as that in the first embodiment (FIG. 5), but the processing order is different from the first embodiment. Specifically, in the second embodiment, the human body tracking section 504 and the state determining section 505 in FIG. 5 are replaced, and the human body tracking is performed after the state determination is performed. Below, the differences from the first embodiment will mainly be explained.

<Device operation>
FIG. 8 is a flowchart of intrusion detection processing in the second embodiment. Similar to the first embodiment, each functional unit shown in FIG. 5 is realized by the CPU 401 reading a program stored in the ROM 402 into the RAM 403 and executing it.

S801 and S802 are similar to S701 and S702 of the first embodiment. In S803, the state determining unit 505 determines the state of the human body detected in S802. Specifically, it is determined whether the vehicle is a driver, a pedestrian, or an unknown person using the method described with reference to FIG. If the person is the driver, the process advances to S805; if the person is not the driver (that is, if the person is a pedestrian or unknown), the process advances to S804. Further, in S804, if the person is a pedestrian, the process advances to S806, and if the person is not a pedestrian (that is, if it is unknown), the process advances to S807.

In S805, the state determining unit 505 determines that the target human body is a normal campus entry candidate. That is, even if the human body passes through the intrusion detection line 210, it is recorded as a normal person entering the premises. On the other hand, in S806, the intrusion determination unit 506 determines that the target human body is an abnormal entrance candidate (pedestrian). Further, in S807, the intrusion determination unit 506 determines that the target human body is an abnormal trespasser candidate (unknown). Then, in S808, the intrusion determination unit 506 sets the human body recorded as an abnormal trespasser candidate in S806 or S807 as a tracking target.

In S809, the intrusion determining unit 506 determines whether the state determinations in S803 to S808 have been performed for all human bodies detected in S802. If there is a human body that has not been determined, the process returns to S803 and the remaining human bodies are determined. If the determination for all human bodies is completed, the process advances to S810.

In S810, the human body tracking unit 504 tracks the human body set as the tracking target in S808 over a plurality of frame images. Then, in S811, the intrusion determination unit 506 determines intrusion of a human body to be detected according to the setting by the area setting unit 507, and in S812, the intrusion determination unit 506 issues an intrusion detection event. That is, all of the human bodies set as tracking targets in S808 are human bodies (abnormal campus visitor candidates) that will be recorded as abnormal campus visitors if they pass through the intrusion detection line 210.

In S813, the intrusion determination unit 506 determines whether the state determination in S811 has been performed for all human bodies set as tracking targets in S808. If there is a human body that has not been determined, the process returns to S811 and the remaining human bodies are determined. If the determination of all human bodies set as tracking targets is completed in S808, the process advances to S814. Note that S814 and S815 are similar to S712 and S713 in the first embodiment.

As described above, according to the second embodiment, the human body to be tracked and intrusion detected is determined based on the determination result of the state of the detected human body (driver, pedestrian, unknown). That is, in the second embodiment, compared to the first embodiment, it is possible to reduce the number of human bodies for which tracking and intrusion detection determination are performed. Therefore, it is possible to suppress the processing load. Note that, based on the determination result in FIG. 3(b), the result generation unit 508 may count the human bodies determined to be targets for intrusion detection, generate and output the count results.

(Modified example)
In the embodiment described above, the state of the human body (driver, walking, unknown) is determined by integrating the detection results from a plurality of detectors (determiners) each detecting a different part of the human body. On the other hand, one or more detectors that detect each part of the human body and one or more detectors that detect each part of the vehicle are provided, and the detection results from these detectors are integrated to determine the state of the human body. You may.

For example, a horizontally long moving object area included in a video image is determined to be a vehicle. Then, for example, if the human body area is located in the middle of the front (travel direction) of the vehicle area when the human body passes the intrusion detection line 210, it is determined that the human body is the driver. That is, the state of the human body may be determined by integrating the detection results of objects other than the human body.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

400 Detection device; 401 CPU; 402 ROM; 403 RAM; 430 Network; 450 Camera; 501 Communication unit; 502 Image acquisition unit; 510 Human body detection unit; 504 Human body tracking unit; 505 State determination unit; Generation part

Claims (10)

detection means for detecting multiple parts of an object included in a frame image of a video;
determining means for determining whether the object is a pedestrian or a vehicle occupant depending on whether a predetermined region is included in the region detected by the detection means;
A detection device characterized by having: 2. The detection device according to claim 1 , further comprising intrusion determining means for determining that the object has invaded a preset area with respect to the frame image. 3. The detection device according to claim 2 , further comprising notification means for notifying the determination result by the intrusion determination means when the determination means determines that the object is a pedestrian . The notification means provides a notification that makes the user aware that a pedestrian has intruded.
The detection device according to claim 3, characterized in that: further comprising a setting means for accepting settings of the area,
5. The detection device according to claim 2, wherein the setting means accepts the setting of the area as a straight line, a curved line, or a combination thereof indicating a boundary line of the area. 6. The determining means determines that the object is a pedestrian when the parts detected by the detecting means include a predetermined part. Detection device. 7. The detection device according to claim 1, wherein the object is a human body, and the predetermined region is the entire human body or the lower half of the human body. 8. The detection device according to claim 1, wherein the detection means detects a plurality of parts of the object using a plurality of detectors. a detection step of detecting multiple parts of an object included in a frame image of a video;
a determination step of determining whether the object is a pedestrian or a vehicle occupant, depending on whether the region detected in the detection step includes a predetermined region;
A method for controlling a detection device, comprising: A program for causing a computer to function as the detection device according to any one of claims 1 to 8.

Images