JP2021107981A - Teacher data generation device - Google Patents

Abstract

To improve recognition performance when learning a photographed image and teacher data as a set.SOLUTION: A teacher data generation device includes means (130) for storing a background CG model space previously created by a three-dimensional point cloud and a camera video image, means (102) for recognizing a foreground object in video photographed by a camera disposed at a road side, and creating a CG model by arranging a CAD model corresponding to the foreground object in the background CG model space, and means (103) for calculating similarity between the CG model and an original flame image, and generating teacher data from the generated CG model on the basis of the similarity.SELECTED DRAWING: Figure 1

Description

The present invention relates to a teacher data generator, and more particularly to a teacher data generator that improves recognition performance between a live-action image and a CG image.

Currently, various developments are underway to realize an automatic driving system and a driving support system for vehicles. For example, in such a system, the outside world information around the own vehicle such as obstacles and moving objects around the own vehicle is recognized, and traveling control is performed according to the situation around the outside world of the own vehicle. Therefore, false detection or non-detection in recognition of external information is a serious safety problem. In order to solve the problems of false detection and undetection in the recognition of external world information, deep learning (machine learning) is used in which the features are gradually learned from the image data of the external world information. In order to improve the recognition performance by deep learning, a large number of highly realistic samples are required. Patent Document 1 discloses that the number of learning samples is increased and the recognition rate is improved by generating an image very similar to a live-action image by CG. With such CG, it is possible to create a wide variety of teacher data.

Japanese Unexamined Patent Publication No. 2018-060511

However, in reality, even a highly realistic CG image that is very similar to the live-action image has a problem that the recognition performance deteriorates when applied to the real image due to the domain shift between the CG image and the live-action image. There is a point.

The present invention has been made in view of such circumstances, and provides a teacher data generation device for improving recognition performance between a live-action image and a CG image.

The teacher data generation device according to the present invention includes means for storing a background CG model space created in advance by a three-dimensional point cloud and a camera image, and
A means for creating a CG model by recognizing a foreground object of an image taken by a camera installed on the roadside and arranging a CAD model corresponding to the foreground object in the background CG model space.
A means for calculating the degree of similarity between the CG model and the original frame image,
A means for generating teacher data from the created CG model based on the similarity, and
Is provided.

According to the present invention, when the teacher data is increased from the CG model, more precise teacher data can be generated by judging the validity of the teacher data.

According to the present invention, the recognition performance when learning a live-action image and CG teacher data as a set is improved.

It is a schematic block diagram which shows the structure of the learning apparatus which concerns on this Embodiment. It is a figure explaining the neural network learning which concerns on this Embodiment. It is a block diagram which shows the structure of the learning apparatus which concerns on this embodiment. It is a flowchart of background CG model creation. It is a flowchart of a frame CG model creation process. It is a flowchart of a frame CG model creation process. It is a flowchart of the teacher data creation process by the teacher data creation unit 302, the validity determination process by the validity determination unit 304, and the manual operation correction process by the manual operation correction unit 305. It is a flowchart of the teacher data creation process by the teacher data creation unit 302, the validity determination process by the validity determination unit 304, and the manual operation correction process by the manual operation correction unit 305. It is a figure which shows various teacher data. It is a flowchart of the training process by the network training unit 306.

First, an outline of the teacher data generation device according to the present invention will be described. In the present invention, a background CG model space is created in advance using a three-dimensional point cloud (point cloud) and a camera image. It recognizes the foreground object (for example, car, motorcycle, person, etc.) of the image taken by the camera installed on the roadside, arranges the corresponding CAD model in the background CG model space, and semi-automatically sets the CG model that matches the image. Create with. Highly accurate teacher data is created by automatically outputting various highly relevant teacher data from the created CG model. As a result, it is possible to reduce the man-hours for annotation to identify and classify a large amount of data and create a correct label. Further, the recognition performance can be improved by learning the live-action image and the CG teacher data as a set without using the CG image at the time of learning.

Until now, in deep learning, the annotation work of manually creating teacher data required a large amount of man-hours. In addition, distance sensors and LiDAR used for detecting objects on the road are expensive and expensive to introduce, and it is difficult to accurately create distance and three-dimensional teacher data. A large amount of teacher data can be easily created by CG, but even if it is a photorealistic CG image, if it is learned only with CG teacher data or mixed with teacher data of actual data, the recognition performance deteriorates at the time of application. was there. GAN (Generative Adversarial Nets) conversion can reduce the domain shift between CG images and live-action images, but it is not a complete solution. By the method of the present invention, distances and three-dimensional teacher data that are difficult to actually measure and create can be created semi-automatically, and cost and man-hours can be expected to be reduced. In addition, improvement in recognition performance can be expected as compared with the conventional mixed learning method with CG.

Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments. In addition, the following description and drawings have been simplified as appropriate to clarify the description.

FIG. 1 is a schematic block diagram showing a configuration of a learning device according to the present embodiment. FIG. 2 is a diagram illustrating neural network learning according to the present embodiment.

The learning device includes an arithmetic processing unit such as a CPU, and also has a function as a functional arithmetic unit that executes each of the subdivided processes. Specifically, the learning device includes a roadside camera moving image input unit 101, a teacher data creation unit 102, a precision teacher data creation unit 103, and a learning unit 104. Further, the learning device includes a foreground object CAD model database 120 and a background CG model database 130. The teacher data creation unit 102, the precision teacher data creation unit 103, and the background CG model database 130 also function as a teacher data generation device, which is one of the feature parts of the present invention.

The roadside camera moving image input unit 101 inputs a moving image taken by a sensor means such as a camera installed on the roadside as a live-action moving image.

The foreground object CAD model database 120 stores a CAD model of a template of a foreground object (for example, a car, a motorcycle, a person, etc.) that may come and go on the road. The background CG model database 130 stores background CAD models such as roads and peripheral structures in which roadside cameras are installed.

The teacher data creation unit 102 arranges a foreground object such as a car on the background CAD model, creates a CG model space corresponding to the live-action video input from the roadside camera video input unit 101, and automatically performs various teacher data TDs. Create with.

Examples of various teacher data TDs created include, for example, 2D, 3D Bounding Box, Semantic, Instance Segmentation, Depth Map, and cloud. Points (Cloud Point), class name, attribute information, etc. can be mentioned.

The precision teacher data creation unit 103 corrects incomplete teacher data that could not be automatically created due to reasons such as undetection due to hiding, and creates highly accurate teacher data PTD. Further, the precision teacher data creation unit 103 determines the validity of the teacher data TD and creates a highly accurate teacher data PTD. Although the details will be described later, the precision teacher data creation unit 103 calculates the similarity between the created CG model and the original frame image, and generates the precision teacher data PTD from the created CG model based on the similarity. ..

The learning unit 104 trains the neural network by setting the live-action image from the roadside camera and the teacher data created by CG. In this specification, detailed description of known neural networks, deep learning, and the like will be omitted.

FIG. 3 is a block diagram showing a configuration of a learning device according to an embodiment of the present invention. The learning device includes a teacher data generation device which is one of the feature parts of the present invention.

The high-density point cloud acquisition unit 311 acquires a high-density point cloud using MMS (Mobile Mapping System), LiDAR (Light Detection and Ranging), or the like. The camera image input unit 312 shoots a camera image at the same time as acquiring the point cloud, and inputs the camera image data.

The roadside camera video input unit 301 inputs video captured by a camera installed on the roadside. The roadside camera installation information database 310 stores installation information (for example, latitude, longitude, altitude, installation angle) of the roadside camera.

The background CG model database 330 stores the CAD models of roads and their surrounding structures (for example, traffic lights, signs, etc.) and the CG model space in which they are arranged.

The foreground object CAD model database 320 stores CAD models of foreground objects such as cars, motorcycles, and people who come and go on the road.

The teacher data database 340 stores a live-action image and CG teacher data as a set for each frame. The teacher CG model database 350 stores the CG model space corresponding to each frame.

The learning result database 360 stores a network model (learned model) trained at any time using the background CG model creation unit 313.

The background CG model creation unit 313 creates a CAD model of the road and its surrounding structures from the high-density point cloud and the camera image.

The frame CG model creation unit 303 creates a CG model space corresponding to each frame from the roadside camera moving image. The teacher data creation unit 302 creates various teacher data corresponding to each frame. The validity determination unit 304 confirms the accuracy of the created teacher data and determines the validity. The manual correction unit 305 manually corrects the CG model that failed to be automatically generated.

The network training unit 306 trains various networks (for example, neural networks) using the created teacher data.

FIG. 4 is a flowchart of the background CG model creation process by the background CG model creation unit 313.

In step S40l, the point cloud from the "high-density point cloud acquisition unit 311" and the camera image from the "camera image input unit 312" are taken as arguments. At the same time as measuring a high-density point cloud with LiDAR, an image is taken with a camera. It is assumed that the LiDAR and the camera are calibrated, and the acquired point cloud and the camera image are synchronized frame by frame.

In step S402, a point cloud (foreground object) other than the background such as a vehicle or a pedestrian is removed. In step S403, a class attribute is assigned to each pixel of the frame image by using a method such as semantic segmentation.

In step S404, the point cloud is mapped to the frame image and the semantic segmentation image. In step S405, color information and class attributes are added to each point cloud from the mapping result. This creates a colored, class-attributed point cloud.

The following steps S406 to S408 are repeated for each class (road, building, tree, etc.) classified as a background.

In step S406, the point cloud of the target class is clustered. In step S407, a three-dimensional CAD model is created (converted) from the clustered point cloud. In step S408, the created CAD model is arranged in the global CG model space, and the created background CG model is stored in the above-mentioned "background CG model database 330".

5A and 5B are flowcharts of the frame CG model creation process by the frame CG model creation unit 303.
In the process of inferring by deep learning, if an accurate network can be created from the "learning result database 360", it is replaced with an updated trained model as appropriate.

In step S501, images and point clouds for each frame are sequentially acquired from the “roadside camera moving image input unit 301” and the “roadside LiDAR log 504”. At this time, it is possible to create a CG model more accurately by installing LiDAR on the roadside.

In step S502, it is confirmed whether or not the CG model corresponding to the previous frame image exists. If the corresponding CG model exists (YES in step S502), the CG model corresponding to the previous frame is acquired from the "teacher CG model database 350" (step S503). On the other hand, if the corresponding CG model does not exist (NO in step S502), the rough installation position is grasped from the "roadside camera installation information database 310", and the CG model around the camera is acquired from the "background CG model database 330". (Step S504). In addition, the camera parameters in the CG model are set from the "roadside camera installation information database 310".

In step S505, foreground objects such as a car, a motorcycle, and a person existing in the frame are detected (recognized). If the CG model corresponding to the previous frame can be acquired, that information may also be referred to. Further, a method of object detection by deep learning may be used. The detection accuracy may be improved by comparing the results of several networks.

The following steps S506 to S514 are repeated for the detected foreground object. In step S506, the detailed attributes of the target object are estimated. For example, the vehicle type of a car and a motorcycle may be determined, and the gender, age, and body shape of a person may be estimated. In step S507, a CAD model of a similar foreground object is acquired from the "foreground object CAD model database 320" according to the attributes estimated in step S506. In the case of the human class, it is necessary to acquire a CAD model of a body shape with a high degree of similarity. In step S508, the distance of the target object from the camera and the posture of the target object (rotation matrix R and translation vector t) are estimated.

In step S509, it is determined whether or not the target object is in a class (car, motorcycle, etc.) that can be regarded as a rigid body.

When the target object is of the human class (NO in step S509, that is, when it is not a rigid body), the following processing is performed. The three-dimensional position of the key point of the joint (the point that seems to be characteristic from the image) is estimated (step S510). The CAD model is deformed (adjusted) so as to match the estimated key point (step S511). In step S512, the CAD model of the target object (person) is arranged in the CG space using the transformation matrices (R and t) estimated in step S508.

On the other hand, when the target object is a rigid body class (car, motorcycle, etc.) (YES in step S509), the process directly proceeds to step S512, and the target object (car, t) is used by using the transformation matrix (R and t) estimated in step S508. A CAD model (such as a motorcycle) is placed in the CG space.

In step S513, an RGB image of the corresponding region is rendered from the CG model. The feature amounts of the edges and corners are compared between the CG image and the actual image, and the similarity is calculated. In step S514, if there is actual data of the point cloud, the point cloud in that area is cut out. The point cloud of the corresponding model area is output from CG, the point cloud from actual data is compared with the point cloud from CG, and the degree of similarity is calculated.

In step S515, it is determined whether or not the CAD model is accurately arranged in the CG space by using the comparison result in step S513 and step S514. If the CAD model is not accurately arranged (NO in step S515), the process proceeds to step S516. In step S516, the transformation matrix (rotation matrix R and translation vector t) is optimized so that step S513 and step S514 are minimized. After that, the process returns to step S512 and the process is repeated.

On the other hand, when the CAD model is accurately arranged in the CG space (YES in step S515), the process ends.

6A and 6B are flowcharts of the teacher data creation process by the teacher data creation unit 302, the validity determination process by the validity determination unit 304, and the manual correction process by the manual correction unit 305.

In step S60l, the teacher data creation unit 302 receives the CG model corresponding to the frame image from the “frame CG model creation unit 303” and outputs various teacher data. As shown in FIG. 7, various teacher data include a two-dimensional bounding box (2D Bounding Box), a semantic segmentation (Semantic Segmentation), an instance segmentation (Instance Segmentation), a depth map (Depth Map), and a three-dimensional bounding box (3D bounding box). 3D Bounding Box), Point Cloud, class name (car, bike, person, etc.), attribute information (vehicle type, gender, age, etc.).

In step S602, the CG image 61 is rendered.

In steps S603 to S606, the validity determination unit 304 determines the validity of the created CG teacher data. Specifically, in step S603, feature quantities such as edges and corners are extracted from the original frame image 62 and the CG image 61, respectively, and compared to calculate the degree of similarity. In step S604, the original frame image 62 is divided into pixel values for each class by semantic segmentation. Further, in step S605, the similarity between the segmentation image created in step S604 and the CG semantic segmentation teacher image is compared. Further, in step S606, the actually measured point cloud 63 and the CG point cloud are compared, and the degree of similarity is calculated.

When these similarities are high (similar) (that is, when YES in step S603, step S605, and step S606), in step S607, it is determined that the teacher data is valid. After that, the original frame image and the CG teacher data are stored in the "teacher data database 340". The CG model is stored in the "teacher CG model database 350". The data to be stored may be only the difference information from the previous frame in order to reduce the capacity.

On the other hand, when these similarities are not similar (that is, when NO in step S603, step S605, and step S606), if the difference between the various similarities compared is lower than the fighting value (NO in step S608). ), Manually make corrections (step S609). Specifically, in step S609, the arrangement of the existing CAD model is manually adjusted, the CAD model of the undetected object is arranged, and the frame CG model is modified.

If it is determined that the correction is not possible in step S608 (YES in step S608), the process ends without storing the CG image in the teacher data database 340.

As described above, according to the present embodiment, highly accurate teacher data can be generated by extracting highly valid teacher data. In addition, it is difficult to accurately create teacher data such as distance, 3D bounding box, and point cloud, but by making such a validity judgment, these teacher data can also be used.

FIG. 8 is a flowchart of the training process by the network training unit 306.
After accumulating some teacher data, various networks are trained. In step S80l, teacher data (CG teacher data) is acquired from the teacher data database 340, and the live-action image and the CG teacher data are set as a set to learn various networks. In step S802, the trained model is evaluated. The evaluation result and the trained network model are stored in the training result database 360. In step S803, the parameters are optimized. The parameter optimization process may be collectively calculated by batch process or the like. In this way, the recognition performance can be improved by learning the live-action image and the CG teacher data as a set without using the CG image at the time of learning. Moreover, since there is a limit to the reduction of domain shift by GAN conversion, the method shown in the present invention is effective.

In the above example, the program can be stored and supplied to a computer using various types of non-transitory computer readable medium. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible discs, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W, DVD (Digital Versatile Disc), BD (Blu-ray (registered trademark) Disc), semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM ( Random Access Memory)) is included. The program may also be supplied to the computer by various types of transient computer readable medium. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

The present invention is not limited to the above embodiment, and can be appropriately modified without departing from the spirit.

61 CG image 62 Frame image 63 Frame point cloud 101 Roadside camera video input unit 102 Teacher data creation unit 103 Precision teacher data creation unit 104 Learning unit 120 Foreground object CAD model database 130 Background CG model database 301 Roadside camera video input unit 302 Teacher data Creation unit 303 Frame CG model creation unit 304 Validity judgment unit 305 Manual work correction unit 306 Network training unit 310 Roadside camera installation information database 311 High-density point cloud acquisition unit 312 Camera image input unit 313 Background CG model creation unit 320 Foreground object CAD Model database 330 Background CG model database 340 Teacher data database 350 Teacher CG model database 360 Learning result database 504 Roadside LiDAR log TD Teacher data PTD Precision teacher data

Claims (1)

A means to store the background CG model space created in advance by the 3D point cloud and the camera image,
A means for creating a CG model by recognizing a foreground object of an image taken by a camera installed on the roadside and arranging a CAD model corresponding to the foreground object in the background CG model space.
A means for calculating the degree of similarity between the CG model and the original frame image,
A means for generating teacher data from the created CG model based on the similarity, and
A teacher data generator.

Images