JP7074723B2 - Learning equipment and programs - Google Patents

Description

The present invention relates to a learning device and a program capable of learning to improve the detection accuracy of a distant object in object detection by utilizing an image taken while moving by a camera provided in a moving means.

In recent years, connected car technology, which is defined as an automobile having a function as an ICT (Information and Communication Technology) terminal, is constantly connected to a network and enables new services such as autonomous driving, has been developed. In a connected car, various information around the car can be collected, and the information collected by a large number of cars can be shared and used in a system on a network. As a kind of information that can be collected, there is an image taken by an in-vehicle camera. Patent Documents 1 to 3 are mentioned as an example of the prior art relating to the object recognition for the vehicle-mounted camera image.

In Patent Document 1, the recognition conditions of the recognition dictionary as the object recognition dictionary used for object recognition are changed based on the light distribution state of the headlight of the own vehicle, and the image taken in front of the own vehicle is taken based on the changed recognition dictionary. Object recognition is performed from. Patent Document 2 discloses an object recognition device that can detect an object that is difficult for a camera to recognize by another detection means when the traveling environment changes suddenly. Specifically, the disappearance of the object detected in the previous frame is detected, and the distance information of the object is output by irradiating the laser based on the direction of the object. In Patent Document 3, the captured image and the image model are used to authenticate the captured object, and when there is an object that could not be authenticated, the image of the unrecognizable object is searched and acquired via the network. , Image data is generated from the acquired image, and the object name of the object is recognized based on the generated image data.

JP-A-2017-165345 JP-A-2015-172934 Japanese Unexamined Patent Publication No. 2018-169746

However, in the prior art, the following approaches for improving the accuracy of object recognition by utilizing the characteristics of the in-vehicle camera image have not been studied.

That is, in general, in object recognition from an image, recognition is performed using parameters of a model learned in advance, but at this time, the closer to the object, the larger and clearer the image of the object on the image is, so the recognition accuracy is high. On the contrary, the recognition accuracy is low at a distance. Here, considering the characteristics of the in-vehicle camera image, by shooting an object around the vehicle while driving on the roadway, the same object is continuously shot from a situation where it is close to the camera to a situation where it is far away. It is normal to obtain a good image. Even if recognition succeeds in a close-up situation and recognition fails in a continuous distant situation on the image, there is a possibility that an object to be recognized in the distance exists and appears on the image. Information that is high can be obtained. It is also possible to collect a large amount of such images in a connected car system.

Therefore, an approach to improve the recognition accuracy (detection accuracy) of a distant object by learning the object recognition using the in-vehicle camera image having such characteristics can be considered. There was a problem that the approach was not made. This is not limited to the in-vehicle camera image, but the same applies to an image having the same characteristics, that is, an image taken while moving by a camera installed in a moving means such as a robot or a drone.

In view of the above-mentioned problems of the prior art, the present invention can learn to improve the detection accuracy of a distant object in object recognition by utilizing an image taken while moving by a camera provided in a moving means. It is intended to provide possible learning devices and programs.

In order to achieve the above object, the present invention is a learning device, which is a detection success section in which it is determined that a predetermined object is continuously detected from an image taken while moving by a camera provided in a moving means. , Apply the predetermined object recognition method to each frame of the video under the existing model with the detection failure section that is adjacent to the relevant section and it is determined that the predetermined object is being photographed but not continuously detected. The success image set in which the detection area of the predetermined target is linked to each frame of the detection success section and the region in which the predetermined target should be detected are estimated and linked as the estimation area in each frame of the detection failure section. By learning and evaluating the model of the predetermined object recognition method by using the failure image set, the preparation unit for preparing the failure image set, and the success image set and the image set selected from the failure image set as correct answers. It is characterized by including a learning unit for obtaining a model obtained by improving the existing model, and is also characterized in that it is a program that causes a computer to function as the learning device.

According to the present invention, by associating a region in which a predetermined target should be detected with each frame of the detection failure section, learning is performed using an image in a state where the target is distant, which could not be detected by the existing model, as a correct answer. This makes it possible to obtain an improved model of the existing model.

It is a functional block diagram of the learning device and the detection device which concerns on one Embodiment. It is a schematic diagram of the system configuration of a connected car. It is a functional block diagram of the preparation part which concerns on one Embodiment. It is a figure which shows the schematic example of the learning data prepared in the preparation part. It is a flowchart of learning by the learning department which concerns on one Embodiment. It is a figure which shows the typical example in which the model and the learning data are updated by repeating the flow of FIG. 5 as corresponding to the example of FIG. It is a figure which shows the example of the hardware composition in a general computer device.

FIG. 1 is a functional block diagram of a learning device and a detection device according to an embodiment. The learning device 1 includes a preparation unit 2 and a learning unit 3, and the detection device 10 includes a detection processing unit 11. The overall processing contents of these are as follows. The learning device 1 reads the video acquired by the in-vehicle camera in the preparation unit 2, and prepares the learning data regarding the target (for example, a road sign) whose type is specified by the manual work of the administrator or the like using this video. Then, by learning using this learning data in the learning unit 3, an improved model obtained by improving an existing model having a low recognition accuracy of a distant object is obtained. In the detection device 10, the detection processing unit 11 performs image recognition using this improved model to obtain the detection result of the target in the image in a form in which the detection accuracy at a distance is improved as compared with the case where the existing model is used. be able to. The processing content of the detection processing unit 11 is the same as that of the detection unit 21 of FIG. 3 which will be described later. (The detection unit 21 and the detection processing unit 11 differ only in the model used, and perform the same detection processing on the input video or image.)

FIG. 2 is a schematic diagram of a connected car system. In the system SY, there are multiple connected cars CC1, CC2, CC3 (three cars as an example here, but generally any number), each equipped with in-vehicle cameras C1, C2, C3 and running. The images are acquired, and these images are transmitted on the network NW and collected by the server SV. In the learning device 1, learning data can be obtained by using the video collected in the server SV in this way, and the server SV may function as the learning device 1. On the other hand, the mode of acquiring the image by the in-vehicle camera in the learning device 1 is not limited to this. For example, the video taken by the in-vehicle camera is recorded on the spot on a physical storage medium (for example, a semiconductor memory such as SSD or an optical disk such as DVD) without transmission on the network NW, and is manually recorded. The video may be read from this storage medium through the work.

Hereinafter, the preparation unit 2 and the learning unit 3 will be described as details of the processing by the learning device 1.

FIG. 3 is a functional block diagram of the preparation unit 2 according to the embodiment. The preparation unit 2 includes a detection unit 21, an identification unit 22, and an estimation unit 23. The detection unit 21 reads the in-vehicle camera image that can be acquired in various modes described with reference to FIG. 2, detects a predetermined type of target specified by the administrator or the like from each frame of this image, and each of the images. The frame is associated with the range occupied by the target as the detection result, and the image is output to the identification unit 22. Here, even for a frame for which the target has not been detected, the detection result is obtained by associating the fact that the detection range does not exist (the fact that the detection has failed), and the frame is output to the identification unit 22.

The detection unit 21 uses an existing model (learned parameter) of the object recognition method that has already been learned and constructed when detecting an object specified from each frame of the video. Object recognition methods include object recognition by machine learning such as SVM (support vector machine) and object recognition by deep learning such as various CNNs (convolutional neural networks) (for example, SSD, YOLOv3, Mask R-CNN, etc.). , Any existing method may be used, and the detection unit 21 can perform target detection from each frame using learned parameters (weights of each layer of the network in the case of object recognition by deep learning). The type of detection target specified to the detection unit 21 by the administrator or the like is selected from those that have been learned in advance so that this type of target is detected by the object recognition method and have parameters as a learning result. And specify.

In the object recognition method, by designating the type of a predetermined object to be recognized, the range occupied by the object from the image (for example, a rectangular bounding box) and the degree corresponding to the object (likeness of the object). Score) and can be obtained. In the detection unit 21, by applying the object recognition method to each frame, if the score of the object-likeness is equal to or higher than the threshold value for determination, the range is linked to the frame as if the detection was successful, and the score is this threshold value. If it is less than, the detection result can be obtained without associating the range with the frame as if the detection failed.

In the identification unit 22, a detection success section in which it is determined that the target detection is continuously successful on the frame time from the image added to each frame obtained by the detection unit 21 and a section adjacent to the detection success section. Therefore, the detection failure section that is continuously determined to have failed in the target detection on the frame time is identified, and the detection success section and the detection failure section of the discrimination result are output to the estimation unit 23. The output detection success section and detection failure section are images as data, and the information of the target area as the detection result by the detection unit 21 is associated with each frame. ..

The identification unit 22 can identify the detection success section and the detection failure section as follows. First, in the video obtained from the detection unit 21, a section determined to have succeeded in detecting the target continuously for a certain period of time (for example, 5 seconds or the like) is identified as a detection success section. Here, since the target may not be detected for a short time due to the influence of sudden noise or the like, the undetected part within a short time of a predetermined threshold value is missing in the middle of the same detection success section. It may be deleted and handled as data, and may not be used by the estimation unit 23 and the learning unit 3 on the rear side of the identification unit 22.

For example, if a section that has been successfully detected for 5 seconds or more as a threshold is a detection success section, and an undetected section (detection failure) of 0.2 seconds or less is treated as missing data, detection is successful for 2.9 seconds continuously. If there is a first section, a second section in which detection fails for 0.1 seconds in succession, and a third section in which detection succeeds in succession for 3 seconds in succession, the first section and The third section is one detection success section (total length 2.9 seconds + 3 seconds = 5.9 seconds with a judgment threshold of 5 seconds or more), and detection fails only for a short time (0.1 seconds with a threshold of 0.2 seconds or less) in the middle. The second section may be treated as being deleted as missing data.

Further, in the determination of the continuity between frames within the section for identifying the detection successful section, it is determined that the movement of the detection result region by the detection unit 21 between the frames adjacent in time is equal to or less than the threshold value. It may be imposed as a condition. The determination that this region movement is below the threshold value may be determined by the amount of movement of the representative position of the region (center of gravity, etc.), or the overlap of the positions of the regions of the detection result between adjacent frames is a certain value or more. It may be judged by.

After identifying the detection successful section as described above, the identification unit 22 next identifies the section having a predetermined length adjacent to the detection successful section as the detection failure section. For the sake of explanation, the frame time of the video is t (t = 1,2,3, ...), the frame at time t is F (t), and the section within the range of time t1 ≤ t ≤ t2 is the detection success section R1 = It is assumed that it is identified as {F (t) | t1 ≤ t ≤ t2}. (Also, in the following description examples, the frame time will be described as an integer, that is, as a frame number.) The identification unit 22 has a predetermined length L with respect to this detection success section R1 and is adjacent to the future side. The section R2 _[future] or the section R2 _[past] having a predetermined length L and adjacent on the past side is identified as the corresponding detection failure section R2. According to the above definition, the section R2 _[future] and the section R2 _[past] can be written as a frame set of video as follows.
R2 _[future] = {F (t) | t2 + 1 ≤ t ≤ t2 + 1 + L}… (1)
R2 _[Past] = {F (t) | t1-1-L ≤ t ≤ t1-1}… (2)

In the identification unit 22, the size S (t) (area, vertical or horizontal) of the area where the target is detected at each time t on the detection successful interval R1 = {F (t) | t1 ≤ t ≤ t2}. Investigate the behavior of time change of size S (t) such as length), and if the behavior is such that size S (t) increases as time t progresses, the interval R2 _[past] on the past side is used. If the behavior is such that the size S (t) decreases as the time t advances, the interval R2 _[future] on the future side is the detection failure interval R2. Can be identified as a thing.

This identification can be summarized in one rule: That is, of any of both ends t1 and t2 on the time axis of the successful detection interval R1, as the direction toward either end (t1 or t2) on the interval R1, the detection region in the frame F (t) The detection failure interval R2 is identified as being adjacent to the end on the side where it is determined that the size S (t) is decreasing.

That is, as a typical example of the shooting situation of the image of the in-vehicle camera, it is assumed that the in-vehicle camera is installed facing either the front or the rear of the vehicle, and the vehicle is moving forward or backward on the roadway in 2 × 2 = 4. However, in any case, by the above method, the detection failure in the situation where the target is photographed in the distance corresponds to the detection success section R1 in the situation where the object is photographed in close proximity. The interval R2 can be estimated.

For example, if an in-vehicle camera is installed to photograph the front of the vehicle and the vehicle is moving forward on the roadway, an object (road sign, etc.) that is in front of the camera and is photographed by the camera and is stationary near the roadway is It is always taken closer to the camera as time goes by, and when it gets closer than a certain distance, it disappears from the shooting range (angle of view of the camera) and moves to the rear of the vehicle so that it cannot be seen from the camera. At this time, the size of the object approaches the camera as time goes by, and shows a behavior of increasing. In this case, with respect to the detection success section R1 of t1 ≤ t ≤ t2, on the past side of time t <t1, the target exists in front of the vehicle but is too far away, so the image is taken but the detection fails. Yes, the time t = t1 is the situation where the detection was successful for the first time when the image was taken at a certain size when it came close to a certain extent, and the time t = t2 after that was the situation when the subject was sufficiently close to the camera. This is the situation just before disappearing from the shooting range. That is, the past section R2 _[past] can be defined as the detection failure section R2 from the increasing behavior of the size S (t).

Similarly, in other situations, the detection failure section can be determined as the past side or the future side of the detection success section from the increase / decrease of the size S (t). In addition, the target is not a stationary target such as a road sign, but is frequently overtaken or overtaken by the own vehicle at a substantially constant relative speed (that is, overtaken or overtaken by each other) like other vehicles running. Similarly, in the case of a relatively moving object (not a situation), the detection failure section can be determined as the past side or the future side of the detection success section from the increase / decrease of the size S (t).

When the shooting situation with the in-vehicle camera (for example, the situation where the vehicle is shooting while moving forward with the front camera) is known, and the increase / decrease judgment is performed without performing the increase / decrease judgment of the size S (t). The detection failure section may be defined on the past side or the future side of the detection success section by the same method as above.

The identification unit 22 has identified that the detection failure section R2 exists on either the past side or the future side of the detection success section R1, but at this point, it is determined by the equation (1) or (2). If there is no frame F (t) group that has failed to detect the target continuously with at least a certain length L, it is treated as if the detection success section R1 and the detection failure section R2 as a pair could not be detected. You can do it. (In this case, the detection success section R1 and the detection failure section R2 that could be detected as a pair at another location on the video may be processed by the estimation unit 23 on the rear side of the identification unit 22.) Here. In the determination of the existence of the frame F (t) group that has failed to detect the target continuously with a fixed length L, the part where the detection failed for a short time in the above-mentioned detection success section is treated as missing data. A similar technique may be applied. That is, if the detection fails almost continuously, but the detection succeeds suddenly for a short time during that period, the data may be deleted and handled as missing data in the middle of the continuous detection failure section.

The estimation unit 23 is detected by the detection unit 21 for each frame of the detection success section using the detection success section and the detection failure section corresponding to each other on the time axis obtained by the identification unit 22. Learning data is obtained by estimating and associating the area where the target should be detected for each frame of the detection failure section by using the information of the target area, and outputting it to the learning unit 3. That is, the output learning data is composed of the detection success section and the detection failure section corresponding to each other, and each frame of the detection success section is associated with the target area detected by the detection unit 21, and the detection failure section. Each frame of is associated with the area where the target should be detected estimated by the estimation unit 23 (however, because it is far away, it corresponds to the area that was not detected by the detection unit 21 using the existing model). ..

The estimation unit 23 estimates the region in which the target should be detected for each frame of the detection failure section as follows. As an explanatory example, the detection success section is set to the above-mentioned section R1 = {F (t) | t1 ≤ t ≤ t2}, and the target area detected in each frame F (t) is included in the rectangle B1. Let it be (t). In addition, it is assumed that the detection failure interval is the interval R2 _[past] = {F (t) | t1-1-L ≤ t ≤ t1-1} of the length L on the past side, and it is detected in each frame. Let the area to be the rectangle B2 (t). (Although it will be explained on the past side, the rectangle B2 (t) can be estimated in the same way even if the detection failure section is the section R2 _[future] on the future side.)

In one embodiment, the time-series data of the position and size of the rectangle B1 (t) (t1 ≤ t ≤ t2) on the future side in the image frame F (t) is fitted by a predetermined function of time. (For example, performing linear fitting as a linear function) By predicting the position and size of the rectangle B2 (t) in the range of t1-1-L ≤ t ≤ t1-1 on the past side with this fitting function, The rectangle B2 (t) may be estimated.

In one embodiment, as additional information in the in-vehicle camera image read and used by the detection unit 21, information on the camera position C (t) at each time t when the image is taken (three-dimensional coordinate C (three-dimensional coordinate C as real-world world coordinates)). Assuming that the information in t), including the information in the direction of the camera optical axis) is associated in advance, the rectangular B2 (t) is estimated using the position information C (t) of this camera. May be good. Therefore, the calculation for converting "world coordinates (X, Y, Z) ⇔ camera coordinates (x, y, z) ⇔ image coordinates (u, v)" may be performed as follows. It is assumed that the camera parameters for these coordinate transformations are known. The camera position C (t) information can be obtained on the vehicle side using any existing positioning method such as GPS (Global Positioning System) and a directional sensor (direction sensor that acquires the direction of the camera optical axis). The frame time of the video and the positioning time may be matched and acquired.

First, regarding the rectangle B1 (t) (t1 ≤ t ≤ t2) on the future side, the stereo is obtained after seeking the same point correspondence within the rectangle between the adjacent frame images F (t) and F (t + 1). By applying an existing method such as parallax, the rectangular range b1 (t) occupied by the rectangle B1 (t) in the 3D camera coordinate system is obtained. (Here, on the premise that the target is composed of a plane shape such as a road sign, the rectangular range b1 (t) may be obtained to surround this plane shape.)

Next, on the premise that the target is a road sign or the like and is stationary in the real world, the rectangular range b (a certain range b that does not depend on the time t) occupied by the world coordinates is obtained. Specifically, using the camera position C (t) (world coordinates), a rectangle b1'(t) obtained by converting the rectangular range b1 (t) of the camera coordinates obtained with reference to this camera position into world coordinates is obtained. Then, the rectangle range b of the world coordinates may be obtained as the average value of this rectangle b1'(t) at t1 ≤ t ≤ t2. (Ideally, the rectangle b1'(t) should not change with time as a stationary position and a fixed size, but since there is noise etc., it is calculated as a time average.)

Finally, the rectangular range b (target stationary position and range) of the world coordinates is set in each of the detection failure sections R2 _[past] = {F (t) | t1-1-L ≤ t ≤ t1-1} on the past side. At time t, the camera position C (t) (world coordinates) is used to convert to a rectangular range b2 (t) in camera coordinates, which is further converted to a range within the frame image, and this conversion range is framed. As a rectangle surrounded in the image, the rectangle B2 (t) at the estimated position can be obtained.

FIG. 4 is a diagram showing a schematic example of the learning data prepared by the above preparation unit 2. In FIG. 4, the detection success interval R1 composed of 16 frames F (17) to F (32) at times t = 17 to 32 and the corresponding 16 frames at times t = 1 to 16 on the past side corresponding to the detection success interval R1. An example is shown in which the training data SD (0) is composed of the detection failure interval R2 _[past] composed of F (1) to F (16).

In the training data SD (0), each frame F (t) (t = 17 to 32) of the detection success section R1 is associated with the area B1 (t) that was successfully detected by the detection unit 21, and the detection failure section. The region B2 (t) estimated by the estimation unit 23 is associated with each frame F (t) (t = 1 to 16) of R2 _[past] . In FIG. 4, the frame F (in the case of t = 11,19,29 as an example of a part of the total time t = 1 to 32 in the region B1 (t) or B2 (t) on the frame F (t). 11) The estimated area B2 (11) on the frame F (19), the detection area B1 (19) on the frame F (19), and the detection area B1 (29) on the frame F (29) are schematic illustrations of road signs in the road image. It is shown as an area of. In these illustrations, it is shown that the road sign is getting closer to the camera side as the time t advances. The example of the learning data SD (0) in FIG. 4 will be referred to as a common explanatory example in the following as appropriate.

The learning unit 3 obtains an improved model of the existing model used in the detection unit 21 by performing learning using the learning data prepared by the preparation unit 2 as described above. This improved model is an improvement in that it can detect objects farther away than the existing model.

FIG. 5 is a flowchart of learning by the learning unit 3 according to the embodiment. In step S0, the existing model used in the detection unit 21 and the training data obtained from the preparation unit 2 are set as the initial value M (0) of the training parameter (model) and the initial value SD (0) of the training data, respectively. Proceed to step S1.

Hereinafter, as shown in FIG. 5, there are a process of looping in steps S1 to S6 (large loop process) and a process of further looping in steps S1 to S3 (small loop process). In explaining, the number of times of the former large loop processing at that time is referred to as i (i = 1,2,3 ...). At the time from step S0 to step S1, this number of times i corresponds to the first time, and i = 1. It is assumed that this number i is added by 1 to become the next value i + 1 when returning from step S6 to step S1, and the value is updated.

In step S1, the training image set, the verification image set, and the evaluation image set are selected from the learning data SD (i-1) at the present time (the number of repetitions i), and then the process proceeds to step S2. At this time, it may be randomly selected so as to satisfy the following conditions (1) and (2).

Condition (1) ... The training image set, the verification image set, and the evaluation image set are set as a predetermined number of n1, n2, and n3 image sets, respectively, so as not to overlap each other, that is, the same image is the training image set. , Select from the images of the training data SD (i-1) so that they do not overlap with two or more of the verification image set and the evaluation image set.

Condition (2) ... The training image set, the verification image set, and the evaluation image set each include a failure image by a certain ratio r (0 <r <1), and the remaining ratio (1-r) includes a success image. elect. Here, since the training data SD (i-1) is composed of the success image set D1 (i-1) and the failure image set D2 (i-1), the success images are randomly selected from the former (1-r). ) * n1, (1-r) * n2 and (1-r) * n3 are selected, and only r * n1, r * n2 and r * n3 are randomly selected from the latter, respectively. By doing so, a training image set, a verification image set, and an evaluation image set may be obtained. (Here, the selection is random, but according to the condition (1), the images already selected may be excluded from the subsequent selection.) The training image set, the verification image set, and the evaluation image set fail at a fixed rate r. Instead of including images, the images may be selected so as to include failed images at proportions r1, r2, and r3 that are not necessarily equal to each other.

The success image set D1 (0) and the failure image set D2 (0) in the training data SD (0) corresponding to the number of repetitions i = 1 (first time) in FIG. 5 are the image set of the detection success section and the detection failure, respectively. It is assumed that the image set of the section is set, and these are also set as the initial values in step S0.

In step S2, training is performed using the training image set and the verification image set selected in step S1, the model obtained by the training is evaluated using the evaluation image set selected in step S1, and then the process proceeds to step S3. move on. When the model is deep learning such as CNN, this learning and evaluation may be performed as follows.

(Learning)… Learning is performed by the existing method. That is, the weight of each layer as a network parameter is adjusted (trained) a predetermined number of times by a gradient method or the like using a training image set, and the obtained parameter is used as a model to be used with the detection unit 21 by the verification image set. One epoch is to detect and verify by the same method. By repeating this one epoch process a predetermined number of times or until it is determined that the verification result (detection accuracy) by the verification image set has converged, the finally obtained parameter is used as a model obtained by learning. ..

Here, the above learning method is an existing method, but in the present embodiment, the existing method is particularly applied with the following treatment. That is, both the training image set and the verification image set include success images and failure images. In the success image, as usual, the area where the target is detected in the associated frame is treated as the detection result for this target and treated as the correct answer at the time of learning. On the other hand, in the failed image, the region estimated by the estimation unit 23 is treated as a correct answer at the time of learning as a detection result for this target.

The training image set and the verification image set collectively constitute a learning image set used for learning (training and verification).

(Evaluation) ... The learned parameters (models) are evaluated using the evaluation image set in the same manner as the handling of the success image and the failure image in the training image set and the verification image set described above. That is, in the evaluation image set, the success image has the target detection area in the associated frame as the correct answer, and the failure image has the area estimated by the estimation unit 23 as the correct answer, and is detected from the evaluation image set by the learned parameters. Detection is performed by the same method as in Part 21 and the detection accuracy is evaluated.

In step S3, the detection accuracy of the model evaluated in step S2 is compared with the detection accuracy of the model M (i-1) at the present time (the number of repetitions i), and if it is improved, the process proceeds to step S4. If it does not improve, return to step S1. The detection accuracy may be evaluated by an F value or the like, and whether or not it is improved may be determined by a threshold value judgment for the increment of the evaluation value, or even if the evaluation value is slightly incremented (positive increment). If there is, it may be determined that the improvement is achieved. Further, the detection accuracy of the current model M (i-1) to be compared may be calculated from a common evaluation image set (selected in step S1) in the same manner as the evaluation in step S2.

In step S4, the model for which the improvement judgment was obtained in step S3 is the current model M (i) (= M (i +)) as a comparison target in the next iterative process (i + 1th time) at the present time (ith time). After updating the model by setting 1-1)), proceed to step S5.

In step S5, it is determined whether or not a convergence test has been obtained for model update, and if it is obtained, the process proceeds to step S7, and if not, the process proceeds to step S6. This convergence test may be determined to have converged when the number of repetitions i at the present time reaches a certain number of times, or immediately before the latest current model M (i) obtained by updating in step S4. It may be determined that the degree of improvement with respect to the model M (i-1) has converged because it has become smaller in the threshold value determination.

In step S7, M (i) is selected as the best model from the obtained series of models M (0), M (1),…, M (i-1), M (i) by the learning unit 3. It is output as the final result (improved model of FIG. 1), and the flow of FIG. 5 ends.

In step S6, the training data SD (i-1) at the present time (ith iteration process) is updated and the learning data SD (i) (= SD (i + 1-)) used in the next iteration process (i + 1th time). After obtaining 1)), return to step S1. This update is an update of the distinction between the successful image set D1 (i-1) and the failed image set D2 (i-1) in the training data SD (i-1), and the successful image set D1 (i) and the failure. The training data SD (i) consisting of the image set D2 (i) is obtained. Specifically, the modified image set D3 (i-1) ⊂ D2 (i-1), which is a part of the failed image set D2 (i-1), is changed to a successful image (to be treated as a successful image). By doing), the update is performed. That is, the updated successful image set D1 (i) is the unupdated successful image set D1 (i-1) with the modified image set D3 (i-1) added (the union of these), and is updated. The failed image set D2 (i) is the failed image set D2 (i-1) before the update, excluding the modified image set D3 (i-1) (the difference between them). If the operation that takes the union and the difference set is expressed by + and-, it can be written by the set formula as follows.
D1 (i) = D1 (i-1) + D3 (i-1)
D2 (i) = D2 (i-1)-D3 (i-1)

Here, the modified image set D3 (i-1), which is the target for changing from the failed image to the successful image, learns the model M (i) whose improvement judgment was obtained in step S3 and updated in step S4 in step S2. And all or part of the failed images included in the training image set, the verification image set, or the evaluation image set used in the evaluation. (All of the failed images included in the training, verification, and evaluation image set may be used as the modified image set D3 (i-1).) If it is a part, the modified image set D3 (i-) is selected by randomly selecting a part. 1) may be obtained, or the modified image set D3 (i-1) may be obtained by selecting a predetermined number in order from the side closest to the detection successful interval R1 (initial successful image set D1 (0)). May be obtained.

When the failed image is changed to the successful image, the target area estimated by the estimation unit 23 with respect to the original failed image is the area detected by the detection unit 21 in the changed successful image. It may be regarded as a success image in the subsequent steps in the iterative process of the flow of FIG. 5 (steps after the changed and updated step S6).

FIG. 6 is a diagram showing a schematic example in which the model M (i-1) and the learning data SD (i-1) are updated by repeating the flow of FIG. 5 as corresponding to the example of FIG. In FIG. 6, as in FIG. 4, the success image is shown as a white frame and the failure image is shown as a gray frame. By repeating the flow of FIG. 5 three times with i = 1,2,3, the model is shown. Is updated as "M (0)-> M (1)-> M (2)-> M (3)", and the training data is also "SD (0)-> SD (1)-> SD (2)-> correspondingly. An example is shown that is updated with "SD (3)".

In FIG. 6, in the initial training data SD (0), as described in FIG. 4, the frames F (1) to F (16) are the failed image set D2 (0), and the frames F (17) to F ( 32) is the successful image set D1 (0). Frames F (14) and F (16) are changed to successful images and handled when the model M (1) and training data SD (1) are updated with i = 1 (first iteration). become. Similarly, at i = 2 (second iteration), frames F (12) and F (13) are treated as successful images, and at i = 3 (third iteration), frames F (8) and F. (10) will be treated as a successful image. The update of these success image sets and failure image sets is as follows when written in the set formula.

D1 (1) = D1 (0) + {F (14), F (16)}, D2 (1) = D2 (0)-{F (14), F (16)}… (at i = 1) update)
D1 (2) = D1 (1) + {F (12), F (13)}, D2 (2) = D2 (1)-{F (12), F (13)}… (at i = 2) update)
D1 (3) = D1 (2) + {F (8), F (10)}, D2 (3) = D2 (2)-{F (8), F (10)}… (at i = 3) update)

In the learning unit 3, according to the flow of FIG. 5, the learning (training and verification) and this evaluation are performed by estimating the correct area and treating the failed image that could not be detected by the existing model because it is far away. It is repeated for the correctly selected correct image, and when the detection accuracy of the model is improved by the evaluation, the treatment of the corresponding failed image is changed to the successful image, and the learning and the evaluation are continued in the same manner. As a result, as schematically shown in FIG. 6, the number of processes i is increased by setting the models that can detect the failed image that could not be detected at a distance as M (1), M (2), M (3). It can be detected even on the distant side and can be updated and acquired as an improved model.

It should be noted that some failed images may be in a state that cannot contribute to the improvement of distant recognition accuracy, but the failed image in such a state is obtained by the small loop processing in S1 to S3 in FIG. When a certain number or more are selected, a negative judgment is obtained in step S3 as a result, and it is waited for a failed image that can contribute to the improvement of the accuracy of distant recognition to be randomly selected in step S1 again. It is also possible to exclude such a failed image by the method of the following supplementary explanation (3).

Hereinafter, supplementary explanations will be given regarding additional embodiments and the like.

(1) In step S1 (repetition processing i-th time) of FIG. 5, the learning unit 3 does not select from the entire failed image set D2 (i-1) as a target for selecting training, detection, and evaluation images. The frame number may be selected from the upper predetermined number on the side closer to the successful image set D1 (i-1).

For example, in the example of FIG. 6, when selecting from the top four, i = 1 from all of the failed image sets D2 (0) = F (1) to F (16) for the first training data SD (0). Not among the top four F (16), F (15), F (14), F (13) near the success image set D1 (0) = F (17) to F (32). You may be elected from. Similarly, all of the failed image sets D2 (1), D2 (2), D2 (3) in the i = 2,3,4th training data SD (1), SD (2) and SD (3) ( As mentioned above, not from the frame image displayed in gray in FIG. 6), but from the side whose frame number is closer to the successful image set (in the case of the example of FIG. 6, the side at a later time), the top 4 below each. You may choose from the individual.
F (15), F (13), F (12), F (11) for the failed image set D2 (1)
F (15), F (11), F (10), F (9) for the failed image set D2 (2)
F (15), F (11), F (9), F (7) for the failed image set D2 (3)

In this way, when the number of repetitions i is set, the failed image is selected from within the upper predetermined number on the side where the frame number is close to the successful image set D1 (i-1), so that the target exists in the distance. For learning and evaluation, priority is given to the failed image that is assumed to be the one with the largest size of the object being photographed by being as close to the camera as possible in the entire failed image set. By selecting and using it, it is possible to avoid using an image in which the target is photographed in an extremely small size from the size that can be detected by the detection accuracy of the current model M (i-1) for learning and evaluation. Be expected. This is expected to increase the possibility of promptly obtaining a positive judgment result in the model accuracy improvement judgment in step S3, and to speed up the improvement of the model accuracy by the flow of FIG.

(2) In the above description, as in the schematic example of FIG. 4, the training data SD (0) includes a success image set with one detection success section, a failure image set with one detection failure section corresponding to the detection success section, and a failure image set. The case of being composed of is described as an example. In order to increase the number of images of the training data SD (0), the preparation unit 2 reads one or more images, and for the specified target of the same type, two or more corresponding detection success sections and detection failure sections. The training data SD (0) (success image set D1 (0) and failure image set D2 (0)) may be obtained as being composed of images.

In this case, when the method of the above supplementary explanation (1) is applied, each frame (failed image) of the detection failure section is framed at the end of the corresponding detection success section (the end on the side adjacent to the detection failure section). The failure image may be selected from the upper predetermined number on the side where the time difference is small by associating the time difference with.

(3) In the update process in step S6 by the learning unit 3, the following additional process may be further performed. As a premise, in the learning unit 3, each image in the failed image set was selected as one of training, detection, and evaluation image in step S1, but the number of times a negative judgment was made in the corresponding step S3 (learning). Alternatively, the model selected as an image for evaluation, but the obtained model does not improve the detection accuracy) shall be counted and recorded. Then, in step S6, the learning unit 3 counts and records the above-mentioned counts among the images of the failed image set as additional processing after performing the process of changing a part of the images of the failed image set to the successful images. If the number of selected times exceeds the threshold value, it may be deleted from the failed image set so as to be excluded from the selection target in the subsequent step S1. This threshold value may be a threshold value that increases according to the number of times i (or the number of small loop processes in steps S1 to S3) of the large loop processing (loop processing in steps S1 to S6) of the flow of FIG.

The following can be given as a schematic example of this additional processing. For example, the failed image F (15) in FIG. 6 is on the side close to the successful image set (initial detection successful interval R1), but even if the training data is updated as SD (1) to SD (3), the failed image Since it has not been changed to a successful image, there is a high possibility that it will not contribute to the improvement of the detection accuracy of the model (for example, the shooting condition of the target is bad). At that point, the number of counts exceeds the threshold and it is possible to remove it from the failed image set.

(4) The image read by the preparation unit 2 and used for learning by the learning device 1 has been described as an in-vehicle camera image with reference to FIG. 2, but any image having the same characteristics is handled by the learning device 1. It is possible. That is, it is possible for the learning device 1 to handle an image taken while moving on a road or the like by a camera provided in an arbitrary moving means such as a robot or a drone.

(5) The following is also possible as a modification of the flowchart of FIG. 5 by the learning unit 3. That is, steps S5 and S6 may be omitted, and the flow may be terminated after proceeding from step S4 to step S7. In this case, the number of repetitions i = 1 as the large loop processing already described is only, and only the small loop processing (steps S1 to S3) can be repeated, and the learning device 1 uses the existing model M (0). The improved model M (1) will be output.

(6) In the above description, when the preparation unit 2 (of which the identification unit 22) identifies the detection success section and the detection failure section adjacent thereto, the detection success section and the detection failure section are continuous. Although it is identified as a thing, it may be possible to identify both adjacent sections and output them to the estimation unit 23 as discontinuous ones. That is, one or more frames on the image exist between the detection success section and the detection failure section and are discontinuous with each other, but they are identified as being adjacent to each other because they have a front-to-back relationship on the time axis. You may try to do it. At this time, in the same manner as described above, the identification unit 22 detects the detection success section and the detection failure section as being continuous and adjacent to each other, and then from this continuous point, the detection success section side and / Alternatively, after deleting a predetermined number of frames (set in advance by the administrator or the like) on the detection failure section side, the detection success section and the detection failure section can be obtained as discontinuous and adjacent to each other.

(7) The current model M (i-1) is a model with the upper limit accuracy that can be learned by the training data or when the preparation unit 2 cannot obtain appropriate training data due to reasons such as the video used is not appropriate. ), Even if the small loop processing of steps S1 to S3 is repeated in the learning unit 3, it is possible that the improvement judgment of the model cannot be obtained in step S3. Therefore, an upper limit threshold value for the number of times of this small loop processing may be set in each i times of the large loop processing in steps S1 to S6, and the learning process may be terminated when the upper limit is reached. When the upper limit is reached with i ≧ 2, the obtained best model M (i-1) may be used as the output of the learning unit 3. When the upper limit is reached at i = 1, an error message such as the possibility that the training data may not be appropriate may be output.

(8) FIG. 7 is a diagram showing an example of a hardware configuration in a general computer device 70. The learning device 1 and the detection device 10 can each be realized as one or more computer devices 70 having such a configuration. The computer device 70 is a CPU (central processing unit) 71 that executes a predetermined instruction, and a dedicated processor 72 (GPU (graphic calculation device)) that executes a part or all of the execution instructions of the CPU 71 on behalf of the CPU 71 or in cooperation with the CPU 71. And deep learning dedicated processor, etc.), RAM73 as the main storage device that provides a work area for the CPU71 and the dedicated processor 72, ROM74 as the auxiliary storage device, communication interface 75, display 76, mouse, keyboard, touch panel, etc. It includes an input interface 77 that accepts data, and a bus BS for exchanging data between them.

Each part of the learning device 1 and the detection device 10 can be realized by a CPU 71 and / or a dedicated processor 72 that reads and executes a predetermined program corresponding to the function of each part from the ROM 74. Here, when the display-related processing is performed, the display 76 further operates in conjunction with the display 76, and when the communication-related processing related to data transmission / reception is performed, the communication interface 75 further operates in conjunction with the display 76.

1 ... learning device, 2 ... preparation unit, 3 ... learning unit, 21 ... detection unit, 22 ... identification unit, 23 ... estimation unit

Claims (13)

A detection success section in which it is determined that a predetermined object is continuously detected from an image taken while moving by a camera provided in the moving means, and a detection success section adjacent to the section, in which a predetermined object is photographed continuously. The detection failure section, which is determined not to be detected, is detected by applying a predetermined object recognition method to each frame of the video under the existing model.
A successful image set in which a detection area of a predetermined target is linked to each frame of a detection success section, and a failure image set in which a region in which a predetermined target should be detected is estimated and linked as an estimation area in each frame of a detection failure section. , The preparation department to prepare, and
A learning unit that obtains an improved model of the existing model by learning and evaluating a model of the predetermined object recognition method using the success image set and the image set selected from the failure image set as correct answers. , A learning device characterized by being provided with. In the preparation unit, the predetermined target is photographed in the detection failure section, but the predetermined object is farther from the camera and the size photographed on the frame is smaller than that in the detection success section. The learning device according to claim 1, wherein the learning device is detected as a section that is not detected in. In the preparation unit, the time change of the size of the detection area of the predetermined target in the detection success section is obtained, and the side of both ends on the time axis of the detection success section, which is determined to have a smaller size of the obtained time change, is set. The learning device according to claim 1 or 2, wherein an adjacent detection failure interval is defined. In the preparation unit, the position and size of the region to be detected in each frame of the detection failure section are obtained by predicting from the time series data of the position and size of the detection area associated with each frame of the detection success section. The learning device according to any one of claims 1 to 3, wherein the estimation region to be linked is obtained. Each frame of the video is associated with the position information of the camera in the world coordinate system at the time of shooting.
In the preparation section, under the assumption that the predetermined object is stationary in the world coordinate system, the detection area associated with each frame of the successful image set and the camera at the time of shooting each frame are used using known camera parameters. Based on the position information, the rest position information in the world coordinate system of the predetermined target is estimated, and further,
A claim characterized by obtaining an estimated region in which the region where the predetermined object should be detected is estimated based on the estimated static position information and the camera position information at the time of shooting each frame of the failed image set. Item 6. The learning device according to any one of Items 1 to 3. The learning unit selects an image from the success image set and the failure image set so as to include at least one image of the success image set and at least one image of the failure image set, so that the learning image set and the evaluation image Each set is determined, the model is obtained by learning with the training image set, and the model is evaluated with the evaluation image set.
The invention according to any one of claims 1 to 5, wherein the selection, learning, and evaluation are repeated until the evaluation of the obtained model is determined to be improved from the evaluation of the existing model. Learning device. The learning device according to claim 6, wherein the learning unit selects an image from a predetermined number whose frame time is closer to the frame time of the successful image set when selecting an image from the failed image set. If it is determined that the evaluation of the obtained model is improved from the evaluation of the existing model, the learning unit further
Changed the treatment of all or part of the images belonging to the failure image set included in the training image set or evaluation image set used to train the obtained model as belonging to the success image set instead of the failure image set. The learning device according to claim 6 or 7, wherein the existing model is replaced with the obtained model, and then the selection, learning, and evaluation are repeated. In the learning unit, the estimated area associated with the image when it was treated as belonging to the failed image set is used as the detection area associated with the image when the treatment is changed as belonging to the successful image set. The learning device according to claim 8, wherein the learning device is adopted. The learning unit counts the number of times that each image in the failed image set is selected for the learning image set or the evaluation image set, but the evaluation of the obtained model is not improved, and the number of times sets a threshold value. The learning device according to claim 8 or 9, wherein the exceeded image is deleted from the failed image set. The preparation unit according to claim 1 to 10, wherein the detection success section and the detection failure section adjacent to each other are detected as continuous sections on the video or discontinuous sections on the video. The learning device described in any. In the learning unit, a model improved stepwise is obtained by repeating learning and evaluation, and each time the improved model is obtained, a part of the image of the failed image set is updated as belonging to the successful image set. The learning device according to any one of claims 1 to 11, wherein learning and evaluation are continued. A program characterized in that the computer functions as the learning device according to any one of claims 1 to 12.

Images