JP7314645B2 - OBJECT DETECTION PROGRAM, OBJECT DETECTION METHOD, AND OBJECT DETECTION DEVICE - Google Patents

Description

The present invention relates to an object detection program, an object detection method, and an object detection device.

For example, a large amount of video data acquired from a security camera is analyzed in order to detect an object image, which is an image of a vehicle or a person, which is an object to be searched for in an incident or accident. However, the detection accuracy of the object image by video data analysis is still not 100%, and even if the object image to be searched is included in the video data, it may not be detected appropriately.

Many measures have been taken to bring the detection accuracy close to 100%, but it is difficult to bring the object image detection accuracy close to 100% for all video data shot under various conditions. If the detection accuracy is not 100%, the video data will have to be reexamined, for example, with the naked eye in order to search for object images that may not have been detected even though they are included in the video data, which takes time and effort.

JP 2015-222881 A

In order to facilitate reexamination with the naked eye, there is a technique for generating video data by masking an object image detected by video data analysis. However, it may be difficult to detect an object image that is not detected by image analysis, that is, is difficult to detect, simply by masking the object image.

An object of the present invention, as one aspect, is to improve the detection accuracy of an object image that is difficult to detect from video data.

In one embodiment, a first detection process is executed to detect an object image representing a predetermined object from a detection target image based on set detection conditions, and a masked image is generated by masking the object image detected in the first detection process in the detection target image. After performing processing for enabling over-detection of an object image representing a predetermined object from the generated masked image, second detection processing for detecting an object image from the masked image is performed.

As one aspect of the present invention, it is possible to improve the detection accuracy of an object image that is difficult to detect from video data.

1 is a block diagram showing an example of an object detection device according to an embodiment; FIG. FIG. 4 is a schematic diagram showing an example of a rectangle enclosing an object image; FIG. 4 is a schematic diagram showing an example of a rectangle enclosing an object image; FIG. 4 is a schematic diagram showing an example of an image within a rectangular area; 1 is a schematic diagram showing an example of a still image used to mask an object image; FIG. 1 is a schematic diagram showing an example of an image used to mask an object image; FIG. FIG. 4 is a schematic diagram showing an example of a detection target image in which an object image is masked; FIG. 4 is a schematic diagram showing an example of a masked image in which an object image is masked; FIG. 4 is a schematic diagram showing an example of an image in which a detected object image is emphasized and displayed; FIG. 4 is a schematic diagram showing an example of an image in which a detected object image is emphasized and displayed; FIG. 4 is a schematic diagram showing an example of an image in which a detected object image is emphasized and displayed; FIG. 4 is a schematic diagram showing an example of a rectangle enclosing an object image; FIG. 4 is a schematic diagram showing an example of a rectangle enclosing an object image detected in a masked image; It is a block diagram which shows an example of the hardware of an object detection apparatus. 6 is a flowchart showing an example of object detection processing; FIG. 5 is a schematic diagram illustrating detection results of a first detection unit and detection results of a second detection unit;

An example of an embodiment will be described in detail below with reference to the drawings.

Object detection device 10 shown in FIG. The video data input unit 11 inputs video data captured by a surveillance camera installed on the street, for example, and extracts still images at predetermined time intervals from the video data. The predetermined time may be, for example, 1/30 [second].

The first detection unit 12 performs a first detection process for detecting an object image representing a predetermined object such as a person or vehicle to be detected based on detection conditions set from each of still images that are detection target images. The detection conditions are set in advance by the user, for example, and a known technique is used to detect the object image. The first detection unit 12 acquires the coordinates of the detected object image on the still image. The coordinates of the detected object image may be, for example, the coordinates of the upper left and lower right vertices of a rectangle surrounding the detected object image. Note that the object is not limited to a person or a vehicle. For example, it may be an item worn by a person or an item attached to a vehicle.

FIG. 2A illustrates an upper left vertex A and a lower right vertex B of a rectangle surrounding an object image detected from a detection target image. Here, a "walking person" is detected. As illustrated in FIGS. 2B and 2C, the image P within the rectangular area surrounded by the rectangle in FIG. 2B may be cut out as illustrated in FIG. 2C and used to confirm the detection result.

The masking unit 13 masks the image P within the rectangular area of the detection target image. Specifically, the mask unit 13 determines whether or not the rectangular area RC surrounding the object image detected by the first detection unit 12 this time overlaps with the rectangular area RP on the still image R in which the object image was previously detected by the first detection unit 12. If it is determined that they do not overlap, the image P, which is an image within the rectangular area RC, is replaced with an image at a position corresponding to the rectangular area RC of the still image R, thereby masking the image P. FIG. That is, the masking unit 13 uses the background image of the detection target object corresponding to the position of the object image detected by the first detection unit 12 to mask the detected object image.

Further, if there is no still image R that is determined to be non-overlapping in the above determination, the masking unit 13 masks the rectangular area RC by, for example, filling it with a predetermined color.

FIG. 3A illustrates still images R that are determined not to overlap in the above determination. The still image R may be searched from among still images extracted from the video data input from the video data input unit 11, which have undergone object detection processing. Further, among the still images extracted from the video data captured by the fixed camera that captured the video data input from the video data input unit 11 and stored, the still image that has undergone the object detection processing may be retrieved. FIG. 3B illustrates an image Q at a position corresponding to the rectangular area RC of the still image of FIG. 3A.

FIG. 3C exemplifies a masked image obtained by masking, with an image Q, a rectangular area RC surrounding the object image detected by the first detection unit 12 . If there is no still image R that is determined to be non-overlapping, it is masked by filling the rectangular area RC with a predetermined color, as illustrated in FIG. 3D. The predetermined color may be determined by specifying RGB values in advance by the user, for example. Also, a predetermined pattern may be used instead of the predetermined color.

The second detection unit 14 performs second detection processing for detecting the same object image again from the rectangular area, ie, the masked image, which is a still image obtained by masking the object image detected by the first detection unit 12 . The second detection unit 14 performs detection such that the object image to be detected by the first detection unit 12 is over-detected. That is, the second detection unit 14 either processes the masked image so that the object image is over-detected, or detects by using a detection condition that is obtained by changing the detection conditions used in the first detection unit 12 so that the object image is over-detected.

Based on the detection conditions used by the first detection unit 12, the second detection unit 14 performs processing to enable over-detection of the object image. Therefore, the user does not have to set a separate detection condition for detection by the second detection unit 14 . Further, the second detection unit 14 performs processing for emphasizing and displaying the detected object image.

FIG. 4A illustrates a case where, for example, when the detection condition used by the first detection unit 12 is “a person including a predetermined color,” first, the masked image is subjected to processing for emphasizing the predetermined color, for example, red, i.e., processing to increase only the R value, thereby detecting a person including red. As a result, a person including red, which was not detected by the first detection unit 12, is detected.

Further, the predetermined color, such as red, may be emphasized within the rectangular area surrounding the detected object image. In this case, the predetermined color, for example, only the R value of the portion determined to be red may be changed to a higher value, the predetermined color may be changed to a different color from the predetermined color, for example, yellow, or only the portion may be set to blink when displayed. In addition to the enhancement of the predetermined color, a rectangle F1 surrounding the object image may be superimposed on the still image.

In FIG. 4B, the object image detected by the second detection unit 14, that is, the optical flow, which is a vector indicating the movement of "a person including a predetermined color", is detected, and an arrow OF indicating the detected optical flow is superimposed on the masked image. In addition to optical flow, a rectangle surrounding the object image may be superimposed on the masked image.

FIG. 4C exemplifies an image in which a “moving person” is detected by the second detection unit 14 by setting the detection condition lower than the detection condition used by the first detection unit 12, for example. Setting a low detection condition means, for example, setting the score for determining “a moving person” in detection using a neural network to a value lower than the value used in the first detection unit 12. Here, a rectangle F2 surrounding the person not detected by the first detection unit 12 and a rectangle F3 surrounding the shadow of the masked person detected by the first detection unit 12 are superimposed on the masked image.

For ease of comparison, the upper diagram of FIG. 8 shows the detection result of the first detection unit 12 illustrated in FIG. 2A, and the lower diagram of FIG. 8 shows the detection result of the second detection unit 14 illustrated in FIG. 4C. As illustrated in the upper diagram of FIG. 8, the shadow of the person not detected by the first detection unit 12 and the shadow of the person detected and masked by the first detection unit 12 is detected by the second detection unit 14 as illustrated in the lower diagram of FIG. The shadow of a person is over-detected, which is disadvantageous, but detection by the second detection unit 14 is more beneficial in detecting a person that was not detected by the first detection unit 12 .

The video data output unit 15 generates video data by arranging and combining the masked images including information indicating the detection result of the second detection unit 14 in time series. The information indicating the detection result is information for emphasizing and displaying the detected object image, as described above. The information indicating the detection result is an arrow indicating the optical flow of the detected object image, a rectangle surrounding the detected object image, information emphasizing the object image detection condition, for example, a color emphasizing the color that is the detection condition. The generated video data may be displayed on a display device or may be stored in a storage device.

A large number of object images, for example, three or more, may be detected in one still image that is the detection target image. FIG. 5A illustrates a large number of rectangles F4 surrounding the object image of the person detected by the first detection unit 12. As shown in FIG. In FIG. 5A, since there are many object images of people, there are also many rectangles F4.

FIG. 5B shows an example in which the masking unit 13 masks the object image within the rectangular F4 area, and the second detection unit 14 detects the object image of a person so as to over-detect. In FIG. 5B, the person in the right back, which was not detected by the first detection unit 12, is detected. A rectangle F6 surrounding an object image of a person in the far right is illustrated. Also, FIG. 5B illustrates the rectangle F5 corresponding to the rectangle F4 surrounding the object image in FIG. 5A, but the rectangle F5 may not be displayed on the display device, for example.

The object detection device 10 includes, as an example, a CPU (Central Processing Unit) 51, a primary storage unit 52, a secondary storage unit 53, and an external interface 54, as shown in FIG. The CPU 51 is an example of a processor that is hardware. The CPU 51 , primary storage unit 52 , secondary storage unit 53 and external interface 54 are interconnected via a bus 59 .

The primary storage unit 52 is, for example, a volatile memory such as a RAM (Random Access Memory). The secondary storage unit 53 is, for example, a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The secondary storage unit 53 includes a program storage area 53A and a data storage area 53B. The program storage area 53A stores programs such as an object detection program, for example. The data storage area 53B stores, for example, video data.

The CPU 51 reads out the object detection program from the program storage area 53A and develops it in the primary storage section 52 . By loading and executing an object detection program, the CPU 51 operates as the image data input unit 11, the first detection unit 12, the mask unit 13, the second detection unit 14, and the image data output unit 15 shown in FIG.

A program such as an object detection program may be stored in an external server and expanded to the primary storage unit 52 via a network. Also, a program such as an object detection program may be stored in a non-temporary recording medium such as a DVD (Digital Versatile Disc) and developed in the primary storage unit 52 via a recording medium reading device.

An external device is connected to the external interface 54 , and the external interface 54 controls transmission and reception of various information between the external device and the CPU 51 . The camera 21 connected via the external interface 54 may be a monitoring camera that is installed at a predetermined position and continuously acquires video data.

Although only one camera 21 is illustrated here, there may be a plurality of cameras, and object detection processing may be performed on each of the video data captured by the plurality of cameras. Similarly, the display 22 connected via the external interface 54 displays video data including information indicating the result of the object detection processing of the second detection section 14, for example. The camera 21 and the display 22 may be built in the object detection device 10 instead of being connected to the external interface 54 .

The object detection device 10 may be, for example, a personal computer or a dedicated device.

Next, an outline of the action of the object detection process will be described. FIG. 7 illustrates the flow of object detection processing. The CPU 51 inputs video data in step 101 . Video data acquired by the camera 21 may be input in real time, or video data acquired by the camera 21 and stored in the data storage unit 53B or an external storage device may be input.

At step 102, the CPU 51 extracts still images from the video data at predetermined time intervals. The processing of steps 103 to 108 is performed for each still image. At step 103, the CPU 51 detects an object image using a known detection method based on the set detection conditions. The CPU 51 determines in step 104 whether or not an object image has been detected. If the determination in step 104 is negative, that is, if the object image is not detected, the CPU 51 proceeds to step 109 .

If the determination in step 104 is affirmative, that is, if an object image has been detected, the CPU 51 determines in step 105 whether or not there is a still image that can be used as a mask. For example, it is determined whether or not the rectangular area RC surrounding the object image detected in step 103 overlaps with the rectangular area RP surrounding the object image on the still image R previously detected. If it is determined that they do not overlap, it is determined that the still image R is a mask image that can be used as a mask.

If the determination in step 105 is affirmative, the CPU 51 uses the still image R as a mask image to mask the object image in step 106 . For example, the object image is masked by replacing the rectangular area RC surrounding the detected object image with an image at a position corresponding to the rectangular area RC of the still image R. If the determination at step 105 is negative, the CPU 51 at step 107 masks the rectangular area RC surrounding the detected object image using, for example, a predetermined color.

At step 108 , the CPU 51 detects an object image from the masked image masked at step 106 or 107 . Again, as in step 103, known techniques are used. However, in step 108, detection is performed such that the object image to be detected in step 103 is over-detected. Further, in step 108, a process for emphasizing and displaying the detected object image is performed.

At step 109, the CPU 51 determines whether or not an unprocessed still image exists. If the determination in step 109 is affirmative, that is, if an unprocessed still image exists, the CPU 51 returns to step 103 . If the determination in step 109 is negative, that is, if an unprocessed still image does not exist, the CPU 51 combines in time series the masked images that have undergone processing for emphasizing and displaying the detected object image to generate video data in step 110.

At step 111, the CPU 51 outputs and displays the generated image data on the display 22, for example. The generated video data may be stored, for example, in the data storage area 53B, or may be stored in an external storage device. Note that the shape surrounding the detected object image is not limited to a rectangle. For example, it may be circular or oval. The flowchart in FIG. 7 is an example, and the order of processing may be changed as appropriate.

In this embodiment, a first detection process for detecting an object image representing a predetermined object from the detection target image is executed based on the set detection conditions, and a masked image is generated by masking the object image detected in the detection target image by the first detection process. After performing processing for enabling over-detection of an object image representing a predetermined object from the generated masked image, second detection processing for detecting an object image from the masked image is performed.

That is, in the present embodiment, the object image detected in the first detection process is over-detected in the second detection process from the masked image obtained by masking the object image. As a result, in this embodiment, for example, when reviewing video data with the naked eye, it is possible to improve the detection accuracy of an object image that is difficult to detect from video data.

The following supplementary remarks are further disclosed regarding each of the above embodiments.

(Appendix 1)
executing a first detection process for detecting an object image representing a predetermined object from the detection target image based on the set detection conditions;
generating a masked image obtained by masking the object image detected in the first detection process in the detection target image;
performing a second detection process for detecting the object image from the masked image after performing a process that enables over-detection of the object image representing the predetermined object from the generated masked image;
An object detection program for causing a computer to execute object detection processing.
(Appendix 2)
The process for enabling overdetection of the object image includes:
A process of processing the masked image and/or changing the detection conditions to make it possible to detect the object image that is not detected in the first detection process from the masked image.
The object detection program of Appendix 1.
(Appendix 3)
highlighting the detection result of the second detection process and displaying it on a display device;
The object detection program of appendix 1 or appendix 2.
(Appendix 4)
Emphasizing the detection result of the second detection process includes:
adding optical flow to the detected object image;
emphasizing information corresponding to detection conditions of the detected object image;
including at least one of
The object detection program according to any one of Appendices 1 to 3.
(Appendix 5)
Generating the masked image includes:
masking the detected object image using the background image of the predetermined object corresponding to the position of the object image detected in the first detection process; or
masking the detected object image using a predetermined color or pattern;
including,
The object detection program according to any one of Appendices 1 to 4.
(Appendix 6)
the computer
executing a first detection process for detecting an object image representing a predetermined object from the detection target image based on the set detection conditions;
generating a masked image obtained by masking the object image detected in the first detection process in the detection target image;
performing a second detection process for detecting the object image from the masked image after performing a process that enables over-detection of the object image representing the predetermined object from the generated masked image;
Object detection method.
(Appendix 7)
The process for enabling overdetection of the object image includes:
A process of processing the masked image and/or changing the detection conditions to make it possible to detect the object image that is not detected in the first detection process from the masked image.
The object detection method of appendix 6.
(Appendix 8)
highlighting the detection result of the second detection process and displaying it on a display device;
The object detection method according to appendix 6 or appendix 7.
(Appendix 9)
Emphasizing the detection result of the second detection process includes:
adding optical flow to the detected object image;
emphasizing information corresponding to detection conditions of the detected object image;
including at least one of
The object detection method according to any one of Appendices 6 to 8.
(Appendix 10)
Generating the masked image includes:
masking the detected object image using the background image of the predetermined object corresponding to the position of the object image detected in the first detection process; or
masking the detected object image using a predetermined color or pattern;
including,
The object detection method according to any one of appendices 6 to 9.
(Appendix 11)
a first detection unit that performs a first detection process for detecting an object image representing a predetermined object from the detection target image based on the set detection condition;
a mask unit that generates a masked image by masking the object image detected by the first detection unit;
a second detection unit that performs a second detection process of detecting the object image from the masked image after performing a process that enables over-detection of an object image representing the predetermined object from the masked image generated by the mask unit;
An object detection device, comprising:
(Appendix 12)
The process for enabling overdetection of the object image includes:
A process of processing the masked image and/or changing the detection conditions to make it possible to detect the object image that is not detected in the first detection process from the masked image.
The object detection device of appendix 11.
(Appendix 13)
highlighting the detection result of the second detection process and displaying it on a display device;
The object detection device according to Appendix 11 or Appendix 12.
(Appendix 14)
Emphasizing the detection result of the second detection process includes:
adding optical flow to the detected object image;
emphasizing information corresponding to detection conditions of the detected object image;
including at least one of
The object detection device according to any one of Appendices 11 to 13.
(Appendix 15)
Generating the masked image includes:
masking the detected object image using the background image of the predetermined object corresponding to the position of the object image detected in the first detection process; or
masking the detected object image using a predetermined color or pattern;
including,
The object detection device according to any one of Appendices 11 to 14.

10 object detection device 12 first detection unit 13 mask unit 14 second detection unit 51 CPU
52 primary storage unit 53 secondary storage unit

Claims (7)

executing a first detection process for detecting an object image representing an object exhibiting a predetermined characteristic from the detection target image based on the set detection conditions;
generating a masked image obtained by masking the object image detected in the first detection process in the detection target image;
After performing a process to enable over-detection of an object image representing an object exhibiting the same feature as the object exhibiting the predetermined feature from the generated masked image, performing a second detection process of detecting the object image representing the object exhibiting the same feature as the object exhibiting the predetermined feature from the masked image.
An object detection program for causing a computer to execute object detection processing. The process for enabling overdetection of the object image includes:
At least one of processing the masked image and changing the detection conditions to enable detection of the object image representing the object exhibiting the same characteristics as the object exhibiting the predetermined characteristics not detected in the first detection process, from the masked image.
The object detection program according to claim 1. highlighting the detection result of the second detection process and displaying it on a display device;
3. The object detection program according to claim 1 or 2. Emphasizing the detection result of the second detection process includes:
adding optical flow to the detected object image;
emphasizing information corresponding to detection conditions of the detected object image;
including at least one of
The object detection program according to any one of claims 1 to 3. Generating the masked image includes:
masking the detected object image using a background image of the object showing the predetermined characteristics corresponding to the position of the object image detected in the first detection process; or
masking the detected object image using a predetermined color or pattern;
including,
The object detection program according to any one of claims 1 to 4. the computer
executing a first detection process for detecting an object image representing an object exhibiting a predetermined characteristic from the detection target image based on the set detection conditions;
generating a masked image obtained by masking the object image detected in the first detection process in the detection target image;
After performing a process to enable over-detection of an object image representing an object exhibiting the same feature as the object exhibiting the predetermined feature from the generated masked image, performing a second detection process of detecting the object image representing the object exhibiting the same feature as the object exhibiting the predetermined feature from the masked image.
Object detection method. a first detection unit that performs a first detection process for detecting an object image representing an object exhibiting a predetermined characteristic from the detection target image based on the set detection condition;
a mask unit that generates a masked image by masking the object image detected by the first detection unit;
a second detection unit that performs a second detection process for detecting, from the masked image generated by the mask unit, an object image that represents an object that exhibits the same characteristics as the object that exhibits the predetermined characteristics, and then detects the object image that represents the object that exhibits the same characteristics as the object that exhibits the predetermined characteristics from the masked image;
An object detection device, comprising: