JP2022025878A - Object detection device and program - Google Patents

Abstract

To provide an object detection device with which it is possible to perform good and sufficient machine training even in a situation where there is difficulty in collecting correct-answer images.SOLUTION: A detection unit detects a specific object included in an inputted frame image by using a machine trainable model. A training data supply unit supplies training data which is an aggregate in pairs of frame images for input and correct-answer data representing whether or not a specific object is included in the frame images, the training data being the one used for training models of the detection unit. An amplification unit pastes the image of the specific object to a prescribed position in the frame image for input so as to create a frame image for input of training data, as well as generates correct-answer data representing that the specific object is included in the created frame image and adds a pair of the created frame image and correct-answer data corresponding to said frame image to the training data supplied by the training data supply unit.SELECTED DRAWING: Figure 1

Description

Application for application of Article 30, Paragraph 2 of the Patent Act (1) Date of issue: May, 2nd year of Reiwa (2020), Publication "73rd Broadcasting Technology Report Meeting Kansai Broadcasting and Technology Forum 2020" (Proceedings) , Paper title: "Development and verification of election poster checker by AI", Pages: 5-6 pages, Publisher: NHK Osaka-based Broadcasting Station Engineering Department "Kansai Broadcasting and Technology Forum 2020" Executive Committee (2) Date : May 19, 2020, the name of the debriefing session: 73rd Broadcasting Technology Report Meeting Kansai Broadcasting and Technology Forum 2020, Organizer: NHK Osaka Base Broadcasting Station Engineering Department "Kansai Broadcasting and Technology" Forum 2020 ”Executive Committee, Venue: NHK Osaka Base Broadcasting Station 17th Floor Meeting Room

The present invention relates to an object detection device and a program.

It is useful to be able to automatically detect a specific object in an image (or in a still image). If the detection of a specific object in a video can be automated, tagging of the video can be performed at low cost, and it can be used, for example, to improve the accuracy of searching the video. In addition, for example, in the business of producing video content, it is possible to confirm at low cost whether or not a specific object is shown in the produced video content or the video content used as a material. become. In the prior art, machine learning is also used to detect a specific object in an image.

Non-Patent Documents 1, 2 and 3 describe techniques for detecting an object in an image.

Patent Document 1 describes a technique for automatically cutting out a region including an object from a material image. It is described that the automatically cut out image is used for machine learning to detect the object.

Patent Document 2 describes a technique for detecting an object by a robot that patrols a predetermined path. Further, a technique for excluding noise (false detection) when a robot detects an object is described.

Japanese Unexamined Patent Publication No. 2020-046858 Japanese Unexamined Patent Publication No. 2020-042562

Dong Lao, Ganesh Sundaramoorthi, “Minimum Delay Object Detection From Video”, ICCV 2019, 2019, [online], [Downloaded May 19, 2020], Internet <URL: https://arxiv.org/abs/1908.11092 ＞ Huizi Mao, Xiaodong Yang, William J. Dally, “A Delay Metric for Video Object Detection: What Average Precision Fails to Tell”, ICCV 2019, 2019, [online], [Download May 19, 2020], Internet < URL: https://arxiv.org/abs/1908.06368 ＞ Kai Kang, Hongsheng Li, Tong Xiao, Wanli Ouyang, Junjie Yan, Xihui Liu, Xiaogang Wang, “Object Detection in Videos with Tubelet Proposal Networks”, CVPR 2017, 2017, [online], [Downloaded May 19, 2020] ], Internet <URL: https://arxiv.org/abs/1702.06355>

In order to be able to detect a specific object or the like in an image by using a machine learning method, a good and large amount (predetermined amount or more) of learning data is required. However, it may be difficult to collect the correct image showing the object to be detected. As an example, when producing video content, the photographer who shoots often performs framing so as to avoid reflection on the screen such as a poster or a signboard for a specific purpose. Therefore, among the materials of such video contents, there is no frame image that can be used as correct answer data for detecting the specific object, or even if it exists, there are extremely few. If the correct image cannot be collected sufficiently, there arises a problem that machine learning based on the correct image cannot be sufficiently performed. Then, there is a problem that the accuracy of detecting the object cannot be improved.

The present invention has been made based on the above-mentioned problem recognition, and will provide an object detection device and a program capable of performing good and sufficient machine learning even in a situation where it is difficult to collect correct images. Is to be.

[1] In order to solve the above problems, the object detection device according to one aspect of the present invention includes a detection unit that detects a specific object included in an input frame image using a machine-learnable model, and the above-mentioned object detection device. It is a training data for training the model of the detection unit, and is a set of a pair of a frame image for input and correct answer data indicating whether or not the specific object is included in the frame image. A frame image for input of training data is created and created by pasting an image of the specific object at a predetermined position in a frame image for input and a data supply unit for learning that supplies training data. The learning data supply unit supplies a pair of the created correct answer data indicating that the specific object is included in the frame image and the correct answer data corresponding to the frame image. It is provided with an amplification unit to be added to the data for use.

[2] Further, in one aspect of the present invention, in the object detection device, the correct answer data is in the frame image in which the specific object is captured when the specific object is included in the frame image. It contains position data representing a position.

[3] Further, one aspect of the present invention is at least one of the learning data supplied by the learning data supply unit and the learning data added by the amplification unit in the object detection device. Further, it is provided with a filtering unit for performing a filtering process for blurring at least a part of the frame image of the above.

[4] Further, in one aspect of the present invention, in the above-mentioned object detection device, the detection unit detects a specific object in each of the frame images included in the series of frame images, and the frame image. The series further includes a tracking unit that identifies a specific object detected in each of the frame images and tracks the specific object over a plurality of the frame images.

[5] Further, in one aspect of the present invention, in the object detection device, the detection unit outputs a score indicating the specific object-likeness of the specific object detected in the frame image, and is described above. The score of the specific object identified by the tracking unit in each of the frame images included in the frame image series is specified as the score series corresponding to the frame image series based on the score series. Further, it is provided with a tracking sequence determination unit for determining whether or not an object is truly a specific object.

[6] Further, in one aspect of the present invention, in the above-mentioned object detection device, the detection unit outputs a score indicating the specific object-likeness of the specific object detected in the frame image.

[7] Further, one aspect of the present invention is to learn a detection unit that detects a specific object included in an input frame image and the model of the detection unit using a machine-learnable model. Data for training that supplies training data that is a set of a pair of a frame image for input and correct answer data indicating whether or not the specific object is included in the frame image. The frame image for input of the training data is created by pasting the image of the specific object at a predetermined position in the frame image for input, and the created frame image includes the specific object. An amplification unit that generates correct answer data indicating that the data is generated and adds a pair of the created frame image and the correct answer data corresponding to the frame image to the learning data supplied by the learning data supply unit. It is a program for operating a computer as an object detection device to be equipped.

According to the present invention, machine learning of the model of the detection unit can be performed using abundant learning data. This can be expected to improve the detection accuracy when the detection unit operates.

It is a block diagram which shows the schematic functional structure of the object detection apparatus by 1st Embodiment of this invention. It is a schematic diagram which shows the structural example of the learning data in the same embodiment. It is a schematic diagram which shows the content of the process which the amplification part by the same embodiment pastes the image of a specific object. It is a schematic diagram which shows the example of the image after the amplification part by the same embodiment pasted the image of a specific object. It is a schematic diagram which shows the example of the amplification of the learning data by the amplification part of the same embodiment. It is a block diagram which shows the schematic functional structure of the object detection apparatus by 2nd Embodiment. It is a block diagram which shows the schematic functional structure of the object detection apparatus according to 3rd Embodiment. It is a block diagram which shows an example of the functional structure of the computer which realizes the object detection apparatus by 1st, 2nd, 3rd Embodiment. It is a schematic diagram for demonstrating the identification method of the object in the case where the tracking of an object is not performed.

Next, a plurality of embodiments of the present invention will be described with reference to the drawings. In each of the embodiments described below, the learning data based on the image is amplified (inflated). The object detection device acquires abundant learning data by this method and performs machine learning processing.

[First Embodiment]
FIG. 1 is a block diagram showing a schematic functional configuration of an object detection device according to the first embodiment. As shown in the figure, the object detection device 1 includes a frame image acquisition unit 10, a learning data supply unit 20, an amplification unit 30, a filter processing unit 40, a detection unit 50, and a result output unit 80. It is composed. Each of these functional units can be realized by, for example, a computer and a program. In addition, each functional unit has a storage means, if necessary. The storage means is, for example, a variable on the program or a memory allocated by executing the program. Further, if necessary, a non-volatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used. Further, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit instead of a program.

The frame image acquisition unit 10 acquires a frame image from the outside. The frame image acquisition unit 10 may acquire only one frame image, or may acquire a series of frame images. A type of frame image series is video. The video is composed of frame images having a predetermined frame rate (number of frames per unit time). The frame image acquisition unit 10 may acquire, for example, a series of frame images taken by a video camera or the like via a cable or the like. Further, the frame image acquisition unit 10 may acquire a series of frame images recorded on an information recording medium such as a magnetic hard disk device (HDD). The frame image acquisition unit 10 passes the acquired frame image to the detection unit 50.

The learning data supply unit 20 supplies data for machine learning of the model of the detection unit 50. The training data may include positive and negative examples. A positive example is learning data using an image (for example, a landscape image) showing a specific object. A negative example is learning data using an image (for example, a landscape image) in which a specific object is not shown. The learning data supply unit 20 attaches correct answer data in association with each of the positive and negative images. The learning data of the positive example has the position information of the area in which the specific image is captured (coordinates of the points at the four corners of the area) and the score value 1.0 of the area. The learning data of the negative example does not have the position information of the area where the specific image appears. The training data of the negative example may have a score value of 0.0. That is, the learning data supply unit 20 is learning data for learning the model of the detection unit 50, and represents a frame image for input and whether or not the frame image includes a specific object. The correct answer data and the learning data which is a set of pairs are supplied. Further, the correct answer data includes position data representing a position in the frame image in which the specific object is captured when the frame image includes the specific object.

The amplification unit 30 amplifies the learning data. Here, "amplification" is to increase the amount of data having the same properties (for example, data used for the same purpose). In other words, the amplification unit 30 inflates the amount of learning data by performing the image pasting process. That is, the amplification unit 30 creates additional data to be added to the learning data supplied by the learning data supply unit 20. Therefore, the amplification unit 30 may prepare in advance an image of a specific object to be detected and an image that does not include the specific object. It is desirable that the amplification unit 30 prepare in advance a large number of images of a specific object and a large number of images that do not include the specific object.

That is, the amplification unit 30 creates a frame image for input of learning data by pasting an image of a specific object at a predetermined position in the frame image for input. Further, the amplification unit 30 generates correct answer data indicating that the created frame image includes a specific object, and the pair of the created frame image and the correct answer data corresponding to the frame image is a learning data supply unit. It is added to the learning data supplied by 20. Further, in the learning data added by the amplification unit 30, when the frame image includes a specific object, the correct answer data includes position data representing a position in the frame image in which the specific object appears.

Here, an example of a specific object will be described. As an example, a particular object is a poster or signboard for a particular purpose. Further, the above-mentioned "image not including a specific object" is a landscape image or the like in which the specific object is not shown. However, the specific object is not limited to the poster or signboard as illustrated here, and may be anything. Further, the "image that does not include a specific object" may be another image that is not necessarily a landscape image. Further, the above-mentioned "poster or signboard for a specific purpose" is, for example, a poster or signboard for announcement or promotion of a candidate for a public office election, a political party, or the like. However, the specific object does not have to be related to the election. Further, the specific object to be detected may be anything else. Examples of specific objects include people (which may be specific people), animals, plants, structures such as buildings, vehicles, food and drink, books, machines, electric appliances, and the like. The specific object is not limited to those listed here.

The amplification unit 30 performs a process of pasting the image of the specific object in a predetermined position in the "image not including the specific object". For example, when the specific object is a poster or a signboard, the amplification unit 30 sets the planar shape image in the "image not including the specific object" as the position where the image of the specific object is pasted.

It should be noted that the detection of the plane reflected in the image itself can be performed by using the existing technology. For example, SynthText (https://github.com/ankush-me/SynthText) has the ability to detect planes in an image. In order to detect a plane, the amplification unit 30 performs, for example, a process of estimating the depth of what is reflected in the image, and divides an image region based on the estimated depth distribution. If the degree of change in depth is linear (or almost linear) in all directions in the image in a certain region in the image, the amplification unit 30 recognizes that the region is a plane.

The amplification unit 30 appropriately determines a region in which the image of the specific object is pasted in the detected plane. The amplification unit 30 performs a process of pasting an image of a specific object in a determined area. When the specific object is, for example, a poster or a signboard, the amplification unit 30 may perform affine transformation of the image of the specific object before pasting. By performing the affine transformation, the amplification unit 30 can enlarge / reduce, rotate, or translate, for example, the image of the poster. As an example, the value of each element of the matrix when performing the affine transformation may be set to a value within the range of 0.9 or more and 1.1 or less. As a result, the amplification unit 30 can create an image of a landscape or the like in which the specific object is captured from an image of the specific object and an image of a landscape or the like in which the specific object is not captured.

When the image of the specific object is pasted in the image, the amplification unit 30 also creates correct answer data corresponding to the image (the image to which the image of the specific object is pasted). The correct answer data includes the position information of the pasted specific object in the image. The position information is represented using the coordinate values in the image. For example, when the area including the pasted specific object is a quadrangle, the coordinates of the points at the four corners of the area are a kind of the above-mentioned position information. The above area is not limited to the quadrangle. The above-mentioned region may be a polygonal region other than a quadrangle, or may be a region represented by a curve such as a circle or an ellipse.

The amplification unit 30 may generate learning data by pasting an image of a specific object on a landscape image prepared by itself. Further, the amplification unit 30 attaches the specific object to the image of the learning data supplied by the learning data supply unit 20 (the image including the specific object or the image not including the specific object), so that the learning data can be obtained. You may increase the amount.

The filter processing unit 40 filters the image of the learning data after amplification output by the amplification unit 30. Specifically, the filter processing unit 40 is a region of at least a part of at least one frame image of the learning data supplied by the learning data supply unit 20 and the learning data added by the amplification unit 30. Performs a filter process (Gaussian filter, etc.) to blur the image. The filter processing unit 40 may have both a case where the individual image output by the amplification unit 30 is filtered and a case where the filter processing is not performed. Further, the filter processing unit 40 may output both an image when the filter processing is performed and an image when the filter processing is not performed for one image output by the amplification unit 30. good. The filter processing performed by the filter processing unit 40 is a process of applying a Gaussian filter to at least a part of the image. For example, the filtering unit 40 may perform filtering processing only on a part of the image (for example, a part of a specific object). As a result, the filtering unit 40 can produce the effect of blurring at least a part of the image. The Gaussian-filtered part of the image has Gaussian blur, resulting in an image that is out of focus at the time of shooting or similar. That is, the processing of the filter processing unit 40 can cause at least a part of the image to be out of focus (or a similar state) in the training data.

The detection unit 50 has a function of detecting a specific object in the frame image. The detection unit 50 is configured to include a machine learning model. The parameters of this model can be updated by performing machine learning. This model is realized using, for example, a neural network. That is, the detection unit 50 detects a specific object included in the input frame image by using a machine learning model. When a series of frame images is passed from the frame image acquisition unit 10, the detection unit 50 detects a specific object in each frame image included in the series of frame images. Further, the detection unit 50 outputs a score value representing the specific object-likeness of the specific object detected in the frame image. The score value is a numerical value within a predetermined range. For example, the score value is a real number of 0 or more and 1 or less. The score value represents the likelihood that a candidate for a particular object (or an area in the image) is a particular object. This score is calculated automatically by the model. The model shall be machine-learned to output the appropriate score.

The detection unit 50 detects an object in the input frame image. The detection unit 50 outputs the position information of the area in which the detected object appears and the score at the time of detection. The score is a value indicating the degree (likelihood) that what is reflected in the area is a specific object. As an example, the score takes a value of 0.0 or more and 1.0 or less. However, the range of the score value may be set to be a range other than the above. The position information is represented by, for example, the coordinates of the four vertices of the quadrangle, with the area in which the object appears as a quadrangle. This coordinate value is a numerical value representing the position of the pixel in each of the vertical direction (vertical direction) and the horizontal direction (horizontal direction) of the image. The detection unit 50 operates in either a machine learning mode or a detection execution mode. The operation of the detection unit 50 in each mode is as follows.

(1) Machine learning mode When operating in the machine learning mode, the detection unit 50 performs machine learning using the learning data supplied from the learning data supply unit 20 side. The learning data is given as a set of pairs of an image to be input to the detection unit 50 and correct answer data corresponding to the image. The learning data may include data amplified by the amplification unit 30. Further, the learning data may include data (image data) filtered by the filtering unit 40. The detection unit 50 performs detection processing based on the input image included in these learning data. The detection unit 50 calculates a loss (error) between the result of the detection process and the correct answer data corresponding to the image. The loss calculated here is a value that reflects the difference in the score for each region in the image. Then, the detection unit 50 updates the parameters of the model based on the calculated loss. When the model is configured using a neural network, the detection unit 50 updates the parameters of the model using the backpropagation method. The backpropagation method itself is an existing technology. The detection unit 50 performs the above machine learning process using a large amount of learning data.

(2) Detection Execution Mode When operating in the machine learning mode, the detection unit 50 performs a process for detecting a specific object in the frame image passed from the frame image acquisition unit 10. The detection unit 50 may sequentially process the sequence of frame images passed from the frame image acquisition unit 10. This series of frame images is a video. The frame rate of the video may be, for example, 30 fps (frames per second) or the like. Each frame image is associated with uniquely identifiable identification information. The frame identification information is, for example, a combination of the image identification information for identifying the image (content) and the frame number. The frame number is represented by, for example, the relative time of the frame (hours, minutes, seconds, and frame serial numbers less than seconds). The frame number is, for example, hh: mm: ss. It is expressed in the form of nnn (hh is hour, mm is minute, ss is second, and nnn is a serial number in seconds or less (for example, 000 to 029)). The detection unit 50 passes the result of the detection process to the result output unit 80. The result output by the detection unit 50 is a set of pairs of frame identification information and detection result data. The detection result data is represented as a set of pairs of position information representing the position of a region within a frame and a score of that region. The position information is represented by the coordinate values of the four corners (four points) of the area.

The result output unit 80 outputs the determination result. In the present embodiment, the result output unit 80 outputs the result of detection by the detection unit 50 as it is. Specifically, the result output unit 80 outputs information for identifying a frame image in which a specific object is detected, information for specifying an area in the frame image (coordinate values of four points), and a score for that area. Output.

FIG. 2 is a schematic diagram showing a configuration example of learning data. The learning data may include a positive example (an example of an image in which a specific object to be detected is shown) and a negative example (an example of an image in which a specific object is not shown). In the figure, (A) is a positive example and (B) is a negative example. Both positive and negative data are pairs of images and correct answers.

In the positive example of (A), it means that a specific object is reflected in the hatched area shown in the image. Corresponding to this image, the correct answer data has information about the area. The position information included in the correct answer data is the coordinates of the points at the four corners of the area. The coordinates of each point are represented by the positions of the pixels in the horizontal and vertical directions. In the illustrated example, the coordinates of the points at the four corners of the region are (320,80), (399,80), (320,119), (399,119). The score in this area is 1.000. In the positive example of the training data, the score of the area where the specific object is shown is 1.000. On the contrary, the score of the area where the specific object is not shown is 0.000. The score is 0.000 for the area where the position information data does not exist.

In the negative example of (B), the specific object is not shown in the image. The correct answer data corresponding to this image does not have a region where the score is 1.000.

FIG. 3 is a schematic diagram showing the contents of a process in which the amplification unit 30 pastes an image of a specific object. FIG. (A) shows an example of an image in which a specific object is not shown before amplification. FIG. (B) shows the result of the plane detection process by the amplification unit 30 for the image of (A) above. In the image of (B), the dark black region is a portion detected as a plane as a result of the plane detection process performed by the amplification unit 30. In the image of (A), the wall surface, the floor surface, the upper surface of the table, the side plate surface of the furniture, and the like are detected as flat surfaces. The other area (light gray area) in the image of (B) is a part that was not detected as a plane. In the example of this embodiment, the amplification unit 30 attaches an image of a specific object to a plane region. That is, the area shown in dark black in the image (B) is a candidate for a place to paste the image of the specific object.

FIG. 4 is a schematic view showing an example of an image after the amplification unit 30 has pasted an image of a specific object. The image shown in FIG. 4 corresponds to the image of FIG. 3 (A). The image shown in FIG. 4 is the result of the amplification unit 30 performing a process of pasting an image of a specific object on the image of FIG. 3 (A). An image of a specific object is attached to the lower left portion of the image of FIG. More specifically, the position where the image of the specific object is pasted is (horizontal coordinate, ordinate coordinate) of (30,350), (80,350), (30,400), (80, It is in the area surrounded by the four points of 400). In this example, the image pasted by the amplification unit 30 is an image of a poster which is a specific object.

FIG. 5 is a schematic diagram showing an example of amplification of learning data by the amplification unit 30. In the figure, (A) is an example of learning data before amplification. The image of (A) does not show a specific object. (B) is learning data (normal example) after amplification due to the amplification unit 30 attaching a specific object to a part of the image of (A). The region shown by hatching existing in the lower right portion of the image of (B) is a region including an image of a specific object attached by the amplification unit 30. (C) is learning data (negative example) after amplification, which is used as it is without pasting an image of a specific object on the image of (A). The correct answer data in (A) does not have data regarding the area including the specific object. That is, it is equivalent to setting the score of the existence of a specific object to 0.000 for the entire image of (A). Like the correct answer data in (A), the correct answer data in (C) does not have data regarding the region including the specific object. On the other hand, the correct answer data in (B) has data relating to a region including a specific object. As shown in the figure, the positions of the area are four points (points at the four corners) whose (horizontal coordinates, ordinate coordinates) are (480,280), (559,280), (480,359), and (559,359). ) Is represented by the coordinate value.

For example, the amplification unit 30 can input the learning data of (A) and output the learning data of (B). Further, the amplification unit 30 can input the learning data of (A) and output the learning data of both (B) and (C), for example. Further, the amplification unit 30 may input the learning data of (A) and output the learning data of either (B) or (C) stochastically, for example.

As described above, in the present embodiment, the amplification unit 30 increases the amount of learning data. As a result, the detection unit 50 can perform machine learning using abundant learning data. This makes it possible to improve the accuracy of the detection unit 50.

The amplification unit 30 performs a process of pasting an image of the specific object at a place where the specific object to be detected is likely to appear. Therefore, the amplification unit 30 performs a process of searching for a place in the image where a specific object to be detected is likely to appear. As an example, the place where the specific object to be detected is likely to appear is on a plane. In this case, the amplification unit 30 performs a process of detecting a plane in the image to be pasted.

Further, according to the present embodiment, the filter processing unit 40 includes the blurred image due to the filter processing in the learning data. As a result, the detection unit 50 performs machine learning of the model using the learning data including the blur. As a result, the detection unit 50 can also learn when the input frame image contains blur (blurring due to out of focus). That is, it is expected that the detection unit 50 will be able to detect a specific object even from the input image including the blur.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. The matters already described in the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.

FIG. 6 is a block diagram showing a schematic functional configuration of the object detection device according to the present embodiment. As shown in the figure, the object detection device 2 includes a frame image acquisition unit 10, a learning data supply unit 20, an amplification unit 30, a filter processing unit 40, and the same as the object detection device 1 of the first embodiment. It has a detection unit 50 and a result output unit 80. A feature of this embodiment is that the object detection device 2 includes a tracking unit 60. The tracking unit 60 can also be realized by using, for example, a computer.

The tracking unit 60 tracks a specific object detected by the detection unit 50 in a series of frame images processed by the detection unit 50. That is, the tracking unit 60 identifies the specific object detected in each of the frame images, and tracks the specific object over a plurality of frame images included in the series of frame images. The tracking unit 60 tracks, for example, between frame images of a specific object (candidate) detected by the detection unit 50 and having a score equal to or higher than a predetermined threshold value. The tracking unit 60 uses, for example, a KCF (Kernelized Correlation Filter) algorithm to track a specific object. KCF is also described in the following references.
[References] Joao F. Henriques, Rui Caseiro, Pedro Martins, Jorge Batista, “High-Speed Tracking with Kernelized Correlation Filters”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, https://arxiv.org/abs/1404.7584, 2014.

The tracking unit 60 may perform tracking by using another existing method such as TLD (Tracking Learning Detection) instead of the KCF algorithm.

The tracking unit 60 passes the tracking result to the result output unit 80 together with the detection result data received from the detection unit 50. The result output unit 80 of the present embodiment outputs the detection result (data of the position in the image and the score related to the position) obtained by the detection unit 50, and also outputs the tracking result. The data of the tracking result represents, for example, which of the positions of the specific object detected in one frame corresponds to the position of the specific object detected in another frame (for example, an adjacent frame). It is data. Further, it is easy for the tracking unit 60 to pass a sequence (time series) of scores (scores obtained by the detection unit 50) for each frame to the result output unit 80 for each of the identified specific objects.

According to the present embodiment, the result output unit 80 not only covers the data of the specific object detected for each frame image, but also the information of the specific object identified by tracking between frames and the identified specific object. The value of the score in the above time series can also be output. The user of the object detection device 2 according to the present embodiment can use the information output by the result output unit 80.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. The matters already explained up to the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.

FIG. 7 is a block diagram showing a schematic functional configuration of the object detection device according to the present embodiment. As shown in the figure, the object detection device 3 includes a frame image acquisition unit 10, a learning data supply unit 20, an amplification unit 30, a filter processing unit 40, and the same as the object detection device 1 of the second embodiment. It has a detection unit 50, a tracking unit 60, and a result output unit 80. A feature of this embodiment is that the object detection device 3 includes a tracking sequence determination unit 70. The tracking sequence determination unit 70 can also be realized by using, for example, a computer.

The tracking sequence determination unit 70 determines whether or not the specific object is included in the sequence (video) of the frame image based on the tracking result of the specific object (candidate) in the frame image. Specifically, the tracking sequence determination unit 70 tracks the data (including the score at the time of detection) of the specific object (candidate) detected by the detection unit 50 and the specific object (candidate) identified by the tracking unit 60. Receive information and. The tracking sequence determination unit 70 acquires a score in each of a plurality of frame images tracked by the tracking unit 60 for the identified specific object (candidate). In other words, the tracking sequence determination unit 70 acquires a sequence of scores. That is, the tracking sequence determination unit sets the score in each frame image included in the frame image series of the specific object identified by the tracking unit 60 into the score series as the score series corresponding to the frame image series. Based on this, it is determined whether or not the specific object is truly a specific object. That is, the tracking sequence determination unit 70 does not make a determination based only on the score in one frame image, but determines whether or not it is a specific object based on the sequence of the above scores.

As an example, the tracking sequence determination unit 70 calculates the average value of the sequence of the above scores. Then, the tracking sequence determination unit 70 makes a determination depending on whether or not the average value is a predetermined threshold value. When the average value of the series of the above scores is equal to or greater than the threshold value (only in that case), the tracking sequence determination unit 70 determines that the specific object (candidate) is truly a specific object.

As another example, the tracking sequence determination unit 70 may make a determination based on a statistical value other than the average value of the score series. The tracking sequence determination unit 70 may make a determination based on, for example, the maximum value or the minimum value in the score series. In this case, the tracking sequence determination unit 70 determines whether or not either the maximum value or the minimum value (predetermined side) in the score sequence is equal to or higher than a predetermined threshold value. When the maximum value or the minimum value is equal to or higher than a predetermined threshold value (only in that case), the tracking sequence determination unit 70 determines that the specific object (candidate) is truly a specific object. Further, the tracking sequence determination unit 70 may make a determination based on any one of the average value, the maximum value, and the minimum value of the values of the score series and the variance value of the score in the series. For example, when any one of the average value, the maximum value, and the minimum value (predetermined) of the values in the series of scores is equal to or more than a predetermined threshold value and the variance value is equal to or less than a predetermined threshold value regarding the variance value (its). Only in the case), the tracking sequence determination unit 70 determines that the specific object (candidate) is truly a specific object. Further, the tracking sequence determination unit 70 determines whether or not the specific object (candidate) is truly a specific object based on whether or not other statistical values regarding the values of the score series are within a predetermined range. May be determined.

As yet another example, the tracking sequence determination unit 70 inputs data based on the value of the score series to SVM (Support Vector Machine) or FFNN (Feed Forward Neural Networks). The determination may be made by. In this case, the tracking sequence determination unit 70 inputs statistical values such as an average value, a variance value, a maximum value, and a minimum value of the score series values into the SVM or FFNN. The SVM or FFNN outputs a determination result (information indicating whether or not it is a specific object) based on those input parameters. In this case, for the above SVM and FFNN, learning (adjustment of internal parameters) is performed in advance using the positive example data and the negative example data. The adjustment itself of the internal parameters of SVM and FFNN using the positive example data and the negative example data can be performed by using the existing technology. The positive and negative example data for adjusting the internal parameters are the statistical values related to the values of the score series to be input to SVM and FFNN, and the correct answer (whether or not it is a specific object). It is a pair of data.

As yet another example, the tracking sequence determination unit 70 may make a determination by inputting the score sequence value itself into the SVM or FFNN instead of the statistical value based on the score sequence value. In this case, SVM and FFNN perform a calculation based on the value of a series of input scores, and output a determination result (information indicating whether or not the object is a specific object). Also in this case, for SVM and FFNN, learning (adjustment of internal parameters) is performed in advance using positive example data and negative example data. Each of the positive example data and the negative example data for adjusting the internal parameters is a pair of the value of the score series itself for inputting to SVM or FFNN and the correct answer (whether or not it is a specific object). It is data.

The tracking sequence determination unit 70 passes the result of determination for the score sequence to the result output unit 80 together with the result data received from the detection unit 50 and the tracking unit 60. The result output unit 80 of the present embodiment outputs the detection result (data of the position in the image and the score related to the position) obtained by the detection unit 50, and also outputs the tracking result (including the data of the score series). , Outputs the judgment result based on the score series.

According to the present embodiment, the result output unit 80 is based not only on the data of the specific object detected for each frame image but also on the sequence of scores of the specific object (candidate) identified by tracking between frames. The determination result determined by the tracking sequence determination unit 70 can be output. That is, the user of the object detection device 3 according to the present embodiment can obtain a determination result based on a series of frame images having a predetermined length, not a determination result based on only one frame image.

In the second and third embodiments, the tracking technique was used to identify objects in the frame image sequence, straddling the frame images. Further, in the third embodiment, the score series is obtained corresponding to the series of frame images, and it is determined whether to detect or not to detect based on the score series. These make it possible to remove erroneous detection (overdetection) that is detected only in a moment (single frame image or a small number of frame images) in the video.

FIG. 8 is a block diagram showing an example of the internal configuration of the object detection devices 1, 2, or 3 in each of the above embodiments. The object detection device 1, 2, or 3 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes an instruction included in a program read from RAM 902 or the like. The central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic operations and logical operations according to each instruction. The RAM 902 stores data and programs. Each element contained in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory". The input / output port 903 is a port for the central processing unit 901 to exchange data with an external input / output device or the like. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output ports 903. Bus 906 is a common communication path used inside a computer. For example, the central processing unit 901 reads and writes data in the RAM 902 via the bus 906. Further, for example, the central processing unit 901 accesses the input / output port via the bus 906.

At least a part of the functions of the object detection devices 1, 2, or 3 can be realized by a program running on a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The "computer-readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, or a USB memory, or a storage device such as a hard disk built in a computer system. That means. Furthermore, a "computer-readable recording medium" is a device that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In that case, it may include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or a client. Further, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

[Evaluation experiment]
The results of experiments for evaluating a plurality of embodiments will be described below.

Table 1 shows the results of detecting an object on a still image using a trained model. Here, the collected learning data is 200, and the amplified learning data is about 200,000. In this experiment, the training data and the evaluation data are similar to each other.

The first row of Table 1 is an experiment to be compared, and the training data is not amplified and not filtered. The F1 value in this case was 0.80. The second line is the case where the learning data is amplified and the filtering process is not performed. The F1 value in this case is 0.81, and better results are obtained as compared with the case of the comparison target. The third line is a case where the learning data is amplified and filtered. The F1 value in this case was 0.77, which was a worse result than in the case of the comparison target. Specifically, in the experiment in the case of the third line, as a result of learning using the blurred image, the positive detection increased, but at the same time, the false detection also increased. The deterioration in performance due to the filtering process is due to the extremely small amount of blurred images in both the original training data and the evaluation data. In reality, it is also possible to detect a specific object (here, a poster) that appears in a blurred state. That is, the improvement of the detection performance of the blurred image is obtained while suppressing the deterioration of the detection performance.

In this experiment, it was confirmed that the object detection performance was improved by amplifying the data.

Table 2 shows the results of detecting an object for a moving image using a trained model. In this experiment, evaluation was performed using evaluation data (video) containing images completely different from the learning data.

The first row of Table 2 is an experiment to be compared, and the training data is not amplified, filtered, or tracked. The F1 value in this case is 0.16. The second to fourth lines are cases where the learning data is amplified and filtered. In any of the second to fourth lines, the performance is improved as compared with the case of the comparison target of one business. The second line is the case where tracking is not performed (first embodiment), and the F1 value is 0.19. The third line is the case where simple tracking is performed (second embodiment), and the F1 value is 0.24. The fourth row is a case where tracking is performed and a determination is made based on the average value of a series of scores obtained as a result (third embodiment), and the F1 value is 0.26. That is, when the tracking described in the second embodiment and the third embodiment is performed, the detection performance is significantly improved. Further, even when the second row and the third row are compared, the detection performance of the third row (third embodiment) is further improved.

FIG. 9 is a schematic diagram for explaining a method of identifying an object between frames when the object detection device does not track the object. In each of the figures (A) and (B), the two rectangles are objects or candidates for objects detected in different frame images, respectively. In order to identify an object, an IOU (Intersection over Union) value is calculated between frames (for example, between adjacent frames). The area hatched in the figure (A) is the intersection of the two quadrilaterals. The region hatched in FIG. 3B is a union of two quadrilaterals. The IoU value is calculated by obtaining the area of the intersection and the area of the union as shown in the frame image and dividing the former by the latter. Objects whose calculated IoU value is equal to or higher than a predetermined threshold value (for example, 0.5) are identified as the same object and tracking is performed.

The present invention is available in all industries that require, for example, to detect a particular object in an image or to determine if a particular object in an image is present. As an example, in a business of producing video content, it can be used to confirm that a specific object is shown or not shown in each frame image constituting the video content. In particular, it can be used for a system that realizes a moving object detection task in which it is difficult to collect learning data. However, the scope of use of the present invention is not limited to those exemplified here.

1, 2, 3 Object detection device 10 Frame image acquisition unit 20 Learning data supply unit 30 Amplification unit 40 Filter processing unit 50 Detection unit 60 Tracking unit 70 Tracking sequence determination unit 80 Result output unit 901 Central processing unit 902 RAM
903 I / O ports 904,905 I / O devices 906 buses

Claims (7)

A detector that detects a specific object contained in the input frame image using a machine-learnable model,
It is a learning data for learning the model of the detection unit, and is a set of a pair of a frame image for input and correct answer data indicating whether or not the specific object is included in the frame image. A learning data supply unit that supplies certain learning data,
By pasting the image of the specific object at a predetermined position in the frame image for input, the frame image for input of the training data is created, and the created frame image includes the specific object. An amplification unit that generates correct answer data to be represented and adds a pair of the created frame image and the correct answer data corresponding to the frame image to the learning data supplied by the learning data supply unit.
An object detection device comprising. When the specific object is included in the frame image, the correct answer data includes position data representing a position in the frame image in which the specific object is captured.
The object detection device according to claim 1. Filtering is performed to blur at least a part of the frame image of at least one of the learning data supplied by the learning data supply unit and the learning data added by the amplification unit. Filter processing unit,
The object detection device according to claim 1 or 2, further comprising. The detection unit detects a specific object in each of the frame images included in the series of frame images.
A tracking unit that identifies a specific object detected in each of the frame images in the series of the frame images and tracks the specific object over a plurality of the frame images.
The object detection device according to any one of claims 1 to 3, further comprising. The detection unit outputs a score representing the specific object-likeness of the specific object detected in the frame image.
The score in each of the frame images of the specific object identified by the tracking unit in the frame image series is used as a score series corresponding to the frame image series, based on the score series. Tracking sequence determination unit that determines whether a specific object is truly a specific object,
The object detection device according to claim 4, further comprising. The detection unit outputs a score representing the specific object-likeness of the specific object detected in the frame image.
The object detection device according to any one of claims 1 to 4. A detector that detects a specific object contained in the input frame image using a machine-learnable model,
It is a learning data for learning the model of the detection unit, and is a set of a pair of a frame image for input and correct answer data indicating whether or not the specific object is included in the frame image. A learning data supply unit that supplies certain learning data,
By pasting the image of the specific object at a predetermined position in the frame image for input, the frame image for input of the training data is created, and the created frame image includes the specific object. An amplification unit that generates correct answer data to be represented and adds a pair of the created frame image and the correct answer data corresponding to the frame image to the learning data supplied by the learning data supply unit.
A program for operating a computer as an object detection device, which is equipped with.

Images