JP7383954B2 - Learning data collection device - Google Patents

Description

The present invention relates to a learning data collection device.

Patent Document 1 discloses a road obstacle detection device that uses a trained model obtained by machine learning to detect road obstacles in images captured by an on-vehicle camera.

Japanese Patent Application Publication No. 2018-194912

In the technology of detecting road obstacles in an image using a trained model as described in Patent Document 1, in order to improve the detection accuracy of road obstacles, a learning model is used to update the trained model. It is preferable to be able to collect data efficiently. However, Patent Document 1 does not take into account efficient collection of training data for trained models.

The present invention has been made in consideration of the above facts, and an object of the present invention is to efficiently collect training data for a trained model in order to improve the accuracy of detecting road obstacles in images.

The learning data collection device according to claim 1 includes: an obstacle detection unit that detects road obstacles in an image using a trained model that has been trained in advance; and an object detection unit that detects an object in the image. , among the images of a plurality of frames taken by an on-vehicle camera, which are between two images in which a road obstacle is detected by the obstacle detection unit, the road obstacle is detected in an area where the road obstacle is estimated to exist. and an output unit that outputs an image in which no road obstruction is detected and an object is detected by the object detection unit as learning data for the learned model.

According to the learning data collection device according to the first aspect, a road obstacle in an image is detected using a learned model learned in advance, and an object in the image is detected. Then, among the images of multiple frames taken by the in-vehicle camera, no road obstacle is detected in the area where the road obstacle is estimated to exist between the two images in which the road obstacle was detected. , and the image in which the object is detected is output as training data for the trained model. Therefore, it is possible to efficiently collect training data for a trained model to improve the accuracy of detecting road obstacles in images.

According to the present invention, it is possible to efficiently collect learning data for a trained model to improve the accuracy of detecting road obstacles in images.

FIG. 1 is a block diagram showing an example of the configuration of a learning data collection system. FIG. 2 is a block diagram showing an example of a hardware configuration of a control device. FIG. 3 is a diagram for explaining a trained model. FIG. 3 is a diagram for explaining an approximate expression for estimating the position of an object in an image. It is a figure which shows an example of the measurement data for calculating an approximate formula. FIG. 2 is a block diagram showing an example of a functional configuration of a control device. FIG. 3 is a diagram for explaining learning data collection processing according to the embodiment. It is a figure which shows an example of the image collected as data for learning. 3 is a flowchart illustrating an example of image storage processing. It is a flowchart which shows an example of data collection processing for learning. FIG. 7 is a diagram for explaining a learning data collection process according to a modification.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

First, with reference to FIG. 1, the configuration of a learning data collection system 10 according to the present embodiment will be described. As shown in FIG. 1, the learning data collection system 10 includes an information processing device 16 and a control device 14 mounted on a vehicle 12. The control device 14 and the information processing device 16 are each connected to a network N, and are capable of transmitting and receiving information via the network N. An example of the control device 14 is an ECU (Electronic Control Unit), and an example of the information processing device 16 is a server computer.

Next, with reference to FIG. 2, the hardware configuration of the control device 14 according to this embodiment will be described. As shown in FIG. 2, the control device 14 includes a CPU (Central Processing Unit) 20, a memory 21 as a temporary storage area, and a nonvolatile storage section 22. Further, the control device 14 includes a network I/F (InterFace) 23 connected to the network N by wireless communication, and an input/output I/F 24. A vehicle speed sensor 26 and an on-vehicle camera 27 are connected to the input/output I/F 24 . The CPU 20 , memory 21 , storage section 22 , network I/F 23 , and input/output I/F 24 are connected to the bus 25 . The control device 14 is an example of a learning data collection device according to the disclosed technology.

The storage unit 22 is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. A learning data collection program 30 is stored in the storage unit 22 as a storage medium. The CPU 20 reads out the learning data collection program 30 from the storage unit 22, expands it into the memory 21, and executes the expanded learning data collection program 30. The storage unit 22 also stores a learned model 32 and approximate formula data 34.

Vehicle speed sensor 26 detects the vehicle speed of vehicle 12 and outputs the detected vehicle speed to control device 14 . The on-vehicle camera 27 is mounted in the cabin of the vehicle 12 and outputs an image obtained by photographing the front of the vehicle 12 to the control device 14.

The learned model 32 is a model learned in advance by machine learning by the information processing device 16, and is used to detect road obstacles in the input image. The control device 14 acquires the learned model 32 from the information processing device 16 and stores the acquired learned model 32 in the storage unit 22 . As an example, as shown in FIG. 3, an image photographed by an on-vehicle camera 27 is input to the learned model 32 according to the present embodiment. The trained model 32 also outputs the probability P _m (m is an integer from 1 to M) of belonging to each of the M semantic labels for each pixel of the input image.

The semantic label here is a label that represents an object in an image. Examples of semantic labels include sky, roads (e.g., paved roads and white lines, etc.), vehicles (e.g., cars, trucks, and motorcycles, etc.), nature (e.g., mountains, forests, roadside trees, etc.), and man-made. Examples include buildings (for example, streetlights, iron poles, guardrails, etc.), road obstacles, and the like.

Approximate formula data 34 is data representing an approximate formula used to estimate the position of an object in an image one frame before. The approximate expression represented by the approximate expression data 34 is obtained in advance through an experiment using the vehicle 12, for example.

Specifically, as shown in FIG. 4, first, an object is placed on the road, and while the vehicle 12 is driven at a constant speed in a direction approaching the object, the in-vehicle camera 27 takes pictures at a predetermined frame rate. The amount of movement of the object in the image taken by the vehicle-mounted camera 27 is measured. In the following, the horizontal direction of the image will be referred to as the x-axis direction, and the vertical direction will be referred to as the y-axis direction. In addition, in the following, the position of the object in the image is defined as the position of the lower edge of the object in the image, and the position of the object is defined as the origin at a predetermined reference position of the image (for example, the position of the upper left edge of the image). Expressed in y coordinate. Note that the position of the object in the image is not limited to the position of the lower end of the object, and for example, the position of the center of gravity of the object may be used.

FIG. 5 shows an example of the position of the object in the n-1th frame image, the position of the object in the nth frame image, and the vehicle speed at the time of measurement, which are measured in this way. In the example of FIG. 5, while the vehicle 12 is running at vehicle speeds of 50 km/h and 100 km/h, 6 frames (that is, n=2 to 6) of images are captured by the on-vehicle camera 27 at 10 fps (frames per second). It shows the position of the object in each image when it was taken.

From the above measurement results, an approximate expression expressed by the following equation (1) is calculated by an approximation method such as the least squares method. In equation (1), a ₁ and a ₂ are coefficients, and b is a constant. Furthermore, in equation (1), x ₁ is a variable representing the position of the object in the n-th frame image, x ₂ is a variable representing the vehicle speed, and y is the position of the object in the n-1 frame image. This is a variable that represents
y=a ₁ ×x ₁ +a ₂ ×x ₂ +b...(1)

The approximate equation actually calculated from the measurement results in FIG. 5 is shown in equation (2) below.
y=0.555449×x ₁ -0.0795×x ₂ +9.024482...(2)

The approximate formula data 34 is data representing the approximate formula calculated as described above. Therefore, using the approximate expression represented by the approximate expression data 34, the position of the object in the n-1st frame image can be estimated from the position of the object in the nth frame image and the vehicle speed. Note that in this embodiment, an example will be described in which the frame rate of the vehicle-mounted camera 27 is a fixed value, but the frame rate is not limited to this. For example, if the user can change the set value of the frame rate of the in-vehicle camera 27, an approximate expression may be calculated in advance for each settable frame rate.

Next, with reference to FIG. 6, the functional configuration of the control device 14 according to this embodiment will be described. As shown in FIG. 6, the control device 14 includes an acquisition section 40, an obstacle detection section 42, an object detection section 44, a determination section 46, and an output section 48. By executing the learning data collection program 30 stored in the storage unit 22 by the CPU 20 of the control device 14, the acquisition unit 40, obstacle detection unit 42, object detection unit 44, determination unit 46, and output shown in FIG. 48.

The acquisition unit 40 acquires the vehicle speed detected by the vehicle speed sensor 26. Further, the acquisition unit 40 acquires an image photographed by the in-vehicle camera 27.

The obstacle detection unit 42 uses the learned model 32 to detect road obstacles in the image acquired by the acquisition unit 40 . Specifically, the obstacle detection unit 42 inputs the image acquired by the acquisition unit 40 to the learned model 32. In response to this input, the trained model 32 outputs a probability P _m of belonging to each of the M semantic labels for each pixel of the input image.

Further, the obstacle detection unit 42 divides the image acquired by the acquisition unit 40 into N local areas S _n (n is an integer from 1 to N). This division process is also called superpixelization process. Each local area is a continuous area, and the feature amounts of each point inside are similar to each other. As the feature amount, color, brightness, edge strength, texture, etc. can be used. The local area can also be expressed as an area that does not straddle the boundary between the foreground and the background.

The obstacle detection unit 42 determines i (i is an integer from 1 to N) of the image based on the local area S _n obtained by dividing the image and the probability P _m output from the learned model 32. )-th local area S _i , the road obstacle likelihood L _i is calculated. Then, the obstacle detection unit 42 detects the road obstacle in the image acquired by the acquisition unit 40 based on the calculated likelihood of the road obstacle L _i . Details of the road obstacle detection process using the learned model 32 by the obstacle detection unit 42 described above are disclosed in Japanese Patent Application Laid-Open No. 2018-194912, so further detailed explanation will be omitted.

The object detection unit 44 detects an object in the image acquired by the acquisition unit 40 by performing a known object detection process on the image acquired by the acquisition unit 40. Examples of this object detection processing include Faster R-CNN (Regions with Convolutional Neural Networks), YOLO (You Only Look Once), and SSD (Single Shot Multibox Detector). That is, the object detection unit 44 detects the object in the image acquired by the acquisition unit 40 without considering whether it is an obstacle on the road. On the other hand, the aforementioned obstacle detection section 42 detects road obstacles among the objects in the image acquired by the acquisition section 40.

The determining unit 46 performs the following determination on an image between two images in which an obstacle on the road is detected by the obstacle detecting unit 42 with respect to a plurality of consecutive frames of images acquired by the acquiring unit 40. The determining unit 46 determines that in the image between the two images, there is an image in which no road obstacle is detected within the area where the road obstacle is estimated to exist, and an object is detected by the object detection unit 44. Determine whether or not.

With reference to FIG. 7, details of the determination processing by the determination unit 46 will be described. As shown in FIG. 7, for the sake of ease of explanation, three consecutive frames of images will be taken as an example, and in order to distinguish between each frame image, each frame image will be numbered 1 to 3. I will explain it by attaching it.

Further, "y coordinate" in FIG. 7 indicates the detection results of road obstacles by the obstacle detection unit 42 and the detection results of objects by the object detection unit 44 for images with frame numbers 1 to 3. Specifically, the value of the "y coordinate" of the "road obstacle" in FIG. 7 indicates the y coordinate of the lower end of the road obstacle detected by the obstacle detection unit 42. Further, the numerical value of "y coordinate" of "object" in FIG. 7 indicates the y coordinate of the lower end of the object detected by the object detection unit 44. In addition, a location where the "y coordinate" in FIG. 7 is "-" indicates that no road obstacle or object was detected for the image of that frame number. That is, in the example of FIG. 7, images with frame numbers 1 and 3 indicate that the obstacle detection section 42 has detected an obstacle on the road, and the object detection section 44 has detected an object. Furthermore, in the example of FIG. 7, the image with the frame number 2 indicates that the obstacle detection section 42 did not detect an obstacle on the road, and the object detection section 44 detected an object.

The determination unit 46 uses the y-coordinate of the road obstacle and the vehicle speed acquired by the acquisition unit 40 at the time of photographing the image to determine the above (( 1) According to the equation, an estimated value of the y-coordinate of the road obstacle in the image one frame before is calculated. Similarly, for the image in which the object is detected, the determination unit 46 uses the y-coordinate of the object and the vehicle speed acquired by the acquisition unit 40 at the time of photographing the image to determine one frame according to the above equation (1). Compute an estimate of the y-coordinate of the object in the previous image. The "y coordinate of one frame before" in the example of FIG. 7 represents the estimated value of the y coordinate calculated in this way.

Further, the determination unit 46 determines that the absolute value of the difference between the "y coordinate of one frame before" and the "y coordinate" of the image one frame before is a predetermined threshold value for the image in which the road obstacle has been detected. If it is less than or equal to TH1 (for example, "1"), it is determined that there is a correspondence between the image and the image one frame before. Similarly, for the image in which the object is detected, if the absolute value of the difference between the "y-coordinate of one frame before" and the "y-coordinate" of the image one frame before is less than or equal to the threshold value TH1, , it is determined that there is a correspondence between that image and the image one frame before. “Compatibility/Presence/Presence” in the example of FIG. 7 represents the presence/absence of correspondence determined in this way. That is, in the example of FIG. 7, there is no correspondence between the images of each frame for obstacles on the road, and there is correspondence for objects between the images with frame numbers 1 and 2, and between the images with frame numbers 2 and 3. It represents that. In the example of FIG. 7, the determining unit 46 determines that the objects detected from the images of three consecutive frames are the same object. Note that if the object detection unit 44 can also detect the object name of the detected object, the determination unit 46 determines that not only the “y coordinates” of the objects correspond but also that the detected object names are the same. In some cases, it may be determined that they are the same object.

Further, for each frame image, if the absolute value of the difference between the y-coordinate of the road obstacle and the y-coordinate of the object is less than or equal to a predetermined threshold TH2 (for example, "2"), the determination unit 46 determines that It is determined that the obstacle and the object correspond. That is, in the example of FIG. 7, the determination unit 46 determines that the detected road obstacle and object correspond to each other in the image with frame number 1 and the image with frame number 3. For the image with frame number 2, the determination unit 46 determines that no road obstacle is detected in the area corresponding to the detected object, that is, the area where the road obstacle is estimated to exist. .

Through the above processing, the determination unit 46 determines whether the image between two images in which a road obstacle has been detected (in the example of FIG. 7, the image with frame number 1 and the image with frame number 3) is (In the example of FIG. 7, the image with frame number 2), there is an image in which no road obstacle is detected and an object is detected by the object detection unit 44 within the area where the road obstacle is estimated to exist. Determine whether or not to do so. Note that the determination unit 46 may use four or more images as images of a plurality of consecutive frames. Furthermore, it is considered that the slower the vehicle speed of the vehicle 12, the greater the number of images showing the same object. Therefore, the determination unit 46 may use a larger number of images as images of a plurality of consecutive frames as the vehicle speed of the vehicle 12 is lower.

The output unit 48 outputs the image determined by the determination unit 46 to exist between the two images as learning data for the trained model 32. In the present embodiment, the output unit 48 outputs the image determined by the determination unit 46 to exist between the two images to the information processing device 16 via the network N as learning data for the trained model 32. (i.e. send).

An image determined by the determination unit 46 to exist between the above two images has a relatively high possibility that the detected object is an obstacle on the road, but the obstacle detection unit 42 determines that the detected object is an obstacle on the road. This is an image that was not detected as an object. This image is the image of frame n shown in FIG. 8 as a specific example. In the example of FIG. 8, in the images of frames n-1 and n+1, an object instructing a lane change (an object surrounded by a rectangular broken line) is detected as a road obstacle and object. On the other hand, in the image of frame n, an object instructing a lane change is detected as an object, but is not detected as a road obstacle. Therefore, by further learning and updating the trained model 32 using such images, the accuracy of detecting road obstacles in images using the trained model 32 is improved.

Next, the operation of the control device 14 according to this embodiment will be explained with reference to FIGS. 9 and 10. Note that FIG. 9 is a flowchart showing an example of the flow of image saving processing executed by the CPU 20 of the control device 14. Further, FIG. 10 is a flowchart showing an example of the flow of the learning data collection process executed by the CPU 20 of the control device 14. The image storage process shown in FIG. 9 and the learning data collection process shown in FIG. 10 are executed by the CPU 20 executing the learning data collection program 30 stored in the storage unit 22. The image storage process shown in FIG. 9 is executed, for example, at every time interval of shooting according to the frame rate of the vehicle-mounted camera 27. Further, the learning data collection process shown in FIG. 10 is executed at regular timing, such as once every hour. Note that the learning data collection process shown in FIG. 10 may be executed each time a predetermined number (for example, 100 images) of images are stored in the storage unit 22 by the image storage process shown in FIG. 27 may be executed each time an image is taken.

In step S10 of FIG. 9, the acquisition unit 40 acquires an image photographed by the vehicle-mounted camera 27. In step S12, the acquisition unit 40 acquires the vehicle speed detected by the vehicle speed sensor 26. In step S14, the acquisition unit 40 stores the image acquired in step S10 and the vehicle speed acquired in step S12 in the storage unit 22 in association with each other. When the process of step S14 ends, the image saving process ends.

In step S20 of FIG. 10, the obstacle detection unit 42 uses the learned model 32 to detect road obstacles in all images stored in the storage unit 22 through the image storage process described above. do. In step S22, the object detection unit 44 performs a known object detection process on all images stored in the storage unit 22 through the image storage process described above, so that the images stored in the storage unit 22 are not stored in the storage unit 22. Detect objects in all images.

In step S24, the determination unit 46 determines, as described above, that the obstacle detection unit 42 detects an obstacle on the road with respect to consecutive frames of all images stored in the storage unit 22 through the image storage process described above. The following determination is made on the image between the two images obtained. The determining unit 46 determines that in the image between the two images, there is an image in which no road obstacle is detected within the area where the road obstacle is estimated to exist, and an object is detected by the object detection unit 44. Determine whether or not. The determination unit 46 performs the above determination for all combinations of consecutive frames (for example, three frames) in all images stored in the storage unit 22 through the image storage process described above. If this determination is negative, the learning data collection process ends, and if this determination is affirmative, the process moves to step S26.

In step S26, the output unit 48 outputs the image determined to exist between the above two images by the processing in step S24 to the information processing device 16 as learning data for the trained model 32, as described above. . When the process of step S26 is finished, the learning data collection process is finished.

The image transmitted as learning data from the control device 14 through the process of step S26 is received by the information processing device 16. The user assigns a semantic label of "road obstacle" to a road obstacle that is not detected by the obstacle detection unit 42 in the image received by the information processing device 16. The information processing device 16 updates the trained model 32 stored in its own storage unit by training it using images to which semantic labels have been added by the user. The information processing device 16 transmits the updated learned model 32 to the control device 14. The control device 14 updates the learned model 32 in the storage unit 22 with the new learned model 32 transmitted from the information processing device 16. This improves the accuracy of detecting obstacles on the road using the learned model 32.

As explained above, according to the present embodiment, for images of multiple frames, among images between two images in which a road obstacle is detected, a road obstacle is detected in an area where a road obstacle is estimated to exist. Images in which no object is detected and in which the object is detected are output as learning data for the trained model 32. Therefore, it is possible to efficiently collect learning data for the trained model 32 to improve the accuracy of detecting road obstacles in images.

Note that in the above embodiment, among images between two images in which a road obstacle is detected, no road obstacle is detected in an area where a road obstacle is estimated to exist, and an object is detected. Although a case has been described in which an image is output as learning data for the learned model 32, the present invention is not limited to this. For example, as shown in FIG. 11, among the images between two images in which an object is detected, an image in which no object is detected in the area where the object is estimated to exist and an obstacle on the road is detected. , the object detection unit 44 may output it as learning data of a trained model used for object detection. In this case, the image with frame number 2 in FIG. 11 is collected as learning data. Note that each item in FIG. 11 is the same as in FIG. 7.

Further, in the embodiment described above, the control device 14 may include either the obstacle detection section 42 or the object detection section 44. In this case, when the detection result by any of the functional units included in the control device 14 is unstable, an example is illustrated in which an image in which no object or road obstacle is detected is output as learning data. Further, the object detection unit 44 may further detect the object name. In this case, if the object detected by the object detection unit 44 in each of the images of a plurality of consecutive frames is a corresponding object, but the object name is different, the object detection unit 44 uses the image of each frame to detect the object. An example is a form in which the data is output as learning data of a trained model to be used.

Further, in the above embodiment, a case has been described in which the amount of movement of the object in the y-axis direction is used to estimate the position of the object in the image one frame before, but the present invention is not limited to this. For example, the amount of movement of the object in the x-axis direction may be added to the estimation of the position of the object in the image one frame before. In this case, the amount of movement of the object in the y-axis direction and the x-axis direction can be calculated from, for example, the yaw rate and the vehicle speed.

Further, the processing performed by the CPU 20 in the above embodiment has been described as software processing performed by executing a program, but it is also possible to use GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). ) etc. may be performed by hardware. Furthermore, the processing performed by the CPU 20 may be performed using a combination of both software and hardware. Further, the learning data collection program 30 stored in the storage unit 22 may be stored in various storage media and distributed.

Further, the present invention is not limited to the above embodiments, and it goes without saying that various modifications can be made in addition to the above embodiments without departing from the spirit thereof.

10 Learning data collection system 12 Vehicle 14 Control device (learning data collection device)
16 Information processing device 20 CPU (obstacle detection section, object detection section, output section)
30 Learning data collection program 32 Learned model 40 Acquisition unit 42 Obstacle detection unit 44 Object detection unit 46 Determination unit 48 Output unit

Claims (7)

includes an obstacle detection unit that detects obstacles on the road using the updated trained model;
The updated trained model detects road obstacles in the images and objects in the images using a trained model trained in advance for multiple frames of images taken by an on-vehicle camera. Among the images between the two images in which the road obstacle is detected, an image in which the road obstacle is not detected and the object is detected in an area where the road obstacle is estimated to exist, A control device that is updated using learning data . Detect road obstacles in images using a trained model that has been trained in advance, and
detecting an object in the image;
Of a plurality of frames of images taken by an in-vehicle camera, the road obstacle is detected within an area where the road obstacle is estimated to exist, among images between two images in which the road obstacle is detected. and updating the trained model using the image in which the object is detected as training data for the trained model.
A method for updating trained models in which processing is performed by a computer. an obstacle detection unit that detects road obstacles in an image using a trained model that has been trained in advance;
an object detection unit that detects an object in the image;
Of a plurality of frames of images taken by an in-vehicle camera, the road obstacle is detected within an area where the road obstacle is estimated to exist, among images between two images in which the road obstacle is detected. an output unit that outputs an image in which the object is detected by the object detection unit as learning data for the learned model;
A learning data collection device equipped with an obstacle detection unit that detects road obstacles in an image using a trained model that has been trained in advance;
an object detection unit that detects an object in the image;
Regarding multiple frames of images taken by an in-vehicle camera, the object is not detected within the area where the object is estimated to exist among the images between the two images in which the road obstacle is detected, and the object is an output unit that outputs an image in which a road obstacle is detected as learning data for the learned model;
A learning data collection device equipped with The obstacle detection unit detects a road obstacle in the i-th local region S _i of the image based on the local region S _n obtained by dividing the image and the probability P _m output from the learned model. The learning data collection device according to claim 3 or 4 , wherein the road obstacle is detected by calculating the object-likeness L _i . 3. The slower the vehicle speed of the vehicle equipped with the in-vehicle camera is, the larger the number of consecutive frames is used to obtain the image to be used as the learning data from among the images between the two images. - The data collection device for learning according to any one of claims 5 to 6. Among multiple frames of images taken by an on-vehicle camera, the road obstacle is not detected within an area where the road obstacle is estimated to exist among the images between two images in which the road obstacle is detected. , and an output unit that outputs an image in which the object is detected as data of the road obstacle;
Obstacle data collection device with.