TWI789267B - Method of using two-dimensional image to automatically create ground truth data required for training three-dimensional pointnet - Google Patents

Abstract

A method of using a two-dimensional image to automatically create ground truth data required for training a three-dimensional PointNet includes the following steps: using an automated segmentation technology to distinguish a plurality of objects shown on the two-dimensional image; generating a significant number of texture maps applicable to the objects; using YOLOv3 (You Only Look Once v3) to identify the texture maps to produce a plurality of corresponding object identifications; and combining the texture maps with a plurality of 3D triangular grids, whereby to obtain the object identification corresponding to each grid point of each of the 3D triangular grids, creating ground truth data required for training the three-dimensional PointNet. Once the three-dimensional PointNet has completed training, the object identification corresponding to another 3D triangular grid can be obtained by learning the three-dimensional PointNet.

Description

Use 2D images to automatically generate the truth needed to train 3D point cloud learning networks factual method

The present invention is related to the training of a 3D point cloud learning network (PointNet); in particular, it refers to a method for automatically generating ground truth required for training a 3D point cloud learning network by using 2D images.

In the latest technological developments in the fields of autonomous driving, virtual reality (VR), augmented reality (AR) and mixed reality (mixed reality, WR), 3D simulation systems have become the mainstay of computer vision. research trends. In order to realize these applications, the identification of object materials is a very important part, but to identify object materials in the 3D model database, it is often limited by incomplete data, and it may even be necessary to rebuild the database. If there is a technology that can automatically detect and classify various objects in the 3D model, it can be quite helpful for expanding the amount of data applicable to various situations.

The current practice is mainly to classify and judge unknown objects by learning a 3D point cloud learning network that has been trained. However, training a 3D point cloud learning network requires a large amount of real data, but the collection and identification of real data is time-consuming and labor-intensive, which virtually limits the training and application of a 3D point cloud learning network.

In view of this, the present invention provides a method for automatically generating real data required for training a 3D point cloud learning network using a 2D image, which can effectively and automatically generate a large amount of real data for training a 3D point cloud with a 2D image Learn how to use the Internet.

The present invention provides a method for using two-dimensional images to automatically generate real data required for training a three-dimensional point cloud learning network, comprising the following steps: A. using an automatic cutting technology to cut a plurality of objects presented on a two-dimensional image; B. Generate a large number of texture maps suitable for these objects; C. Use the third version of YOLO (You Only Look Once v3) neural network model technology to identify these texture maps and generate corresponding plural object identifiers; and D . Combining these texture maps with a plurality of three-dimensional triangular meshes, so as to obtain the corresponding object identifiers on each point of each mesh, and then generate a plurality of real data.

In one embodiment, the two-dimensional image is in color, and the automatic cutting technology described in step A first converts the two-dimensional image in color into a grayscale image, then performs binarization, and then converts the two-dimensional image into a grayscale image. The objects are tagged, whereby the objects are detected and cut.

In one embodiment, the two-dimensional image is in color, and the automatic cutting technique described in step A is to first convert the two-dimensional image in color into a grayscale image, and then adjust it by means of histogram equalization The contrast of the gray scale of the image is then binarized, and the noise of these objects is removed by erosion and blurring, so as to detect and cut these objects.

In one embodiment, if the length or width of any object after cutting exceeds 4000 pixels, the object is converted into a grayscale image and binarized, and then opened to remove the object. The noise of objects, thereby detecting and cutting the plurality of objects contained in these objects.

In one embodiment, the blurring system adopts Gaussian blur (gaussian blur), and its filter is expressed as follows:

Where σ is the standard deviation of the normal distribution, and u and v are the blur radii of different dimensions.

In one embodiment, the erosion is to perform a convolution operation on the black and white image using a filter with a fixed size.

In one embodiment, step B is to apply actions such as RGB channel arrangement, color, and flipping different angles to the objects cut in step A to cause various changes, and to adjust the background of the two-dimensional image. A large number of texture maps are generated by randomly combining the objects and the 2D image.

In one embodiment, the arrangement of the RGB channels is to change the arrangement order of the RGB channels without changing the RGB values.

In one embodiment, the 3D point cloud learning network is a file format developed by Wavefront Technology for the animation tool Advanced Visualizer.

In one embodiment, the 3D point cloud learning network is multiplied by the transformation matrix of the neural network T-Net before extracting the point cloud features, wherein the transformation matrix is processed by a convolutional layer and a fully connected layer After that, a 3×3 or 64×64 matrix is obtained.

Through the method of using 2D images to automatically generate the real data required for training the 3D point cloud learning network proposed by the present invention, a large amount of real data can be generated, which is helpful for training the 3D point cloud learning network and greatly reduces related costs.

S1, S2, S3, S4: steps

Fig. 1 is a flowchart of an embodiment of the present invention, illustrating each step of a method for automatically generating real data required for training a 3D point cloud learning network using 2D images; Fig. 2(a) to Fig. 2(h) are various Figure 3 is a schematic flow chart of one of the automatic cutting techniques (simple auto parts cutting method) used in the present invention; Fig. 4 is another automatic cutting technology (detailed auto parts cutting method) used in the present invention ) of the flow chart; Fig. 5 is a schematic flow chart of the method of generating a large number of texture maps in the present invention; Fig. 6 (a) to Fig. 6 (c) are schematic diagrams of one type of texture maps generated in the present invention; Fig. 7 (a) to Fig. 7(c) is a schematic diagram of another type of texture map generated by the present invention; FIG. 8 is a schematic diagram of the results obtained by the present invention using YOLOv3 technology to detect parts; FIG. 9 is a schematic diagram of a point cloud model with category labels; FIG. 10 is a schematic diagram of the present invention Figure 11 is a schematic diagram of the structure of the point cloud learning network of the present invention; and Figure 12 is the classification result of the present invention for auto parts.

In order to illustrate the present invention more clearly, preferred embodiments are given and detailed descriptions are given below in conjunction with drawings. In the following paragraphs of the description, we will use a three-dimensional model of a car as an example for illustration, but this is not a limitation of the present invention. The method proposed by the present invention can be applied in the 3D simulation system of other objects in principle, and can be used to identify the type of objects in the simulation system.

Before starting to explain the technical content of the present invention, here is a brief introduction to the basic concept of the three-dimensional model to help understand the present invention. Generally speaking, the shape of a 3D model simulates the surface or solid of an object through a triangular grid. If the facet of the triangular grid is smaller, the resolution of the surface of the formed object will be higher. The higher the height, the more details of the shape of the object can be displayed. The facet of a triangular mesh is composed of three vertex coordinates and a unit normal vector, and in order to present color or material, a texture image is usually applied to the triangular mesh, that is, a two-dimensional plane image . More specifically, the coordinates of the three vertices of the triangle mesh will correspond to the UV coordinates of the texture, so that the model and the texture can be combined to make the object model present a more realistic object appearance.

In short, the data stored in the 3D model include all the triangular meshes it contains, the 3D space data of each mesh vertex, the facet vector, the text data describing the facet, and the texture map relationship of the 2D plane (generally expressed as UV Coordinate representation), and the correlation information among the triangular meshes. There are many commonly used data formats for 3D models. The present invention adopts the point cloud format, which has the advantages of easy format conversion, small amount of stored data, and short computer processing time.

Please refer to FIG. 1 , which is a flowchart of an embodiment of the present invention, showing various steps of a method for generating real data required for training a 3D point cloud learning network using 2D images. The first step S1 is to use an automatic cutting technology to cut a plurality of objects presented on a two-dimensional image. In this example, the 2D image is a 2D car texture map, and the plurality of objects are the various car components, parts or parts contained in the 2D image (for simplicity of description, hereinafter referred to as "parts"). In practice, the parts contained in the car texture map are closely arranged and irregular, and the way the map is presented varies from creator to artist. norms and standards. The parts in the picture and the background have large differences and similar textures, and the arrangement of the parts Depending on the quantity, the arrangement is tight or the edges overlap, and there are also maps with inconspicuous outlines of parts. Figure 2(a) to Figure 2(h) show the actual car texture maps with different styles.

The present invention proposes two different automatic cutting technologies according to different characteristics of textures. The first automatic cutting technology is for textures with large color differences between parts and background, which is called simple auto parts cutting method in this manual; the second automatic cutting technology is for textures with small color differences between parts and background, which needs to be used The more detailed processing is referred to as the detailed auto parts cutting method in this specification. The two automated cutting technologies are described below respectively.

First, the simple car parts cutting method converts the colored car texture map into a grayscale image (gray image), and then performs binarization (binarization), and finally performs labeling of the parts, and the parts can be detected and cut. The processing flow is shown in Figure 3.

As for the detailed auto parts cutting method, please refer to Figure 4. Firstly, the color map image is replaced with the background color, and then converted into a grayscale image, and then the contrast of the image grayscale is adjusted by histogram equalization. Histogram equalization will make the overall color scale distribution of the image more uniform, enhance the contrast of the image color scale, and improve the identification of parts and backgrounds. Then do binarization, and use erosion and blurring to remove the noise of the parts, and finally carry out labeling of the parts, and then the parts can be detected and cut.

For the blurring described here, the present invention adopts the method of Gaussian blur (gaussian blur), and its filter is expressed as follows:

Where σ is the standard deviation of the normal distribution, u and v are the blur radii of different dimensions. The erosion described here is to use a filter with a fixed size to perform a convolution operation on a black and white image.

It is worth mentioning that, in order to filter the problem of some overlapping parts, resulting in the cutting of large objects with more than two parts, the meticulous auto parts cutting method adopted in the present invention will also count the dimensions of all parts. If the cut parts are long Or if the width exceeds 4000 pixels, the part image will be converted into a grayscale image, and then the grayscale image will be binarized, and the noise between the parts will be removed by opening operation, and then the parts will be cut. , so that independent parts can be cut out.

Please refer to FIG. 1 , after the objects (ie auto parts) are cut out from the 2D image, the next step S2 will generate a large number of texture maps suitable for these objects. In short, for these cut objects, the present invention applies actions such as RGB channel arrangement, color, and flipping different angles to cause various changes. In addition, the present invention also makes different changes to the map background, and then randomly combines various parts and backgrounds to generate a large number of texture maps.

In more detail, by changing the order of the RGB channels, it is possible to combine parts of multiple colors. The way to change the channel is to change the arrangement order of the RGB channels without changing the RGB value, and a total of 6 arrangement orders (RGB, RBG, GRB, GBR, BGR, BRG) can be generated, so that 6 different colors can be generated. Texture map image. As for the rotation of the object, the angle is 90 degrees as the unit, and the rotation range of each object is 0 to 270 degrees. In this example, the background of the 2D image uses white, gray and black as the background colors of the new texture map.

In addition, the present invention may also adjust the shapes of these objects according to the situation, make changes on the basis of similar appearance, and meet the conditions of the characteristics of the object type. For example, adding an additional small object to one of the objects, or extracting some features from the other object to divide it into objects, thereby generating more objects.

Due to the irregular arrangement of the parts of the texture map, the angle and direction of placement are not fixed, the number of types of parts is also different, and the background color of the map is also quite diverse. Therefore, the present invention generates a large number of automobile texture maps based on the design and arrangement of the original texture maps, and the production process is shown in FIG. 5 . In detail, first randomly select parts in all categories, and each time a fixed selection of 8 to 10 parts. Then the rotated parts are arranged in a non-fixed pitch, and finally all kinds of parts are randomly arranged and placed to generate a large number of car texture maps.

The present invention takes the number of parts types, parts types and RGB channel sorting as the goal of generating texture maps, and generates texture training sets in 4 different ways, which are called "Type-1", "Type-2", and "Type-1" respectively. Type-3" and "Type-4" map training sets, the combination of which is generated is shown in Table (1). The number of part types in the table is the number of its part types in the generated map, which is divided into "all types" and "only one type". Part types are combined by "original part" or "newly generated part". RGB channel refers to "grayscale object image" and "arrangement and combination of all color channels". Type-1 is a new texture map composed of all parts types, original parts and 6 color images. Type-2 is a new texture map consisting of all part types, newly generated parts, and grayscale images. Type-3 is a new texture map composed of all parts types, newly generated parts and 6 color images. Type-4 is a new texture map consisting of all single part types, newly generated parts, and 6 color images.

Please refer to FIG. 1 again. Then, step S3 adopts the third version of YOLO (You Only Look Once v3) neural network model technology to identify the texture maps and generate a plurality of corresponding object identifiers. More specifically, the third version of the YOLO-like neural network model technology is an object detection algorithm based on a convolutional neural network. In addition to using Darknet53 to extract features, this algorithm also introduces a feature pyramid network (feature Pyramid network, FPN) technology to improve the problem of inability to detect small objects. In the present invention, the convolutional neural network architecture of Darknet53 consists of 53 convolutional layers, 5 residual layers, a global average pooling layer, and a fully connected layer (fully connected layer), the stride of the convolutional layer of this architecture is 2; as for the concept of multi-scale detection of the feature pyramid network, the upsampling layer uses the nearest neighbor method. The YOLOv3 architecture contains 75 layers of convolutional layers, 4 route layers, 2 upsampling layers (upsample) and 23 Shortcut layers. The result after detection is shown in Figure 8. The position and type of the part obtained after detection are marked in the area of the object on the map in the form of a mask, and the label of the type is marked in this area as a part category information for .

Next, in step S4 shown in FIG. 1 , these texture maps are combined with a plurality of three-dimensional triangular meshes, thereby obtaining object identifiers corresponding to each point of each mesh, and generating a plurality of real data. As mentioned above, the 3D data format used in the present invention is point cloud, which is composed of a large number of point coordinates to form the surface of a 3D object, usually scanned by cameras and lasers to obtain the surface grid structure , and then only consider the vertex data of the mesh, the above data set composed of these points is called a point cloud. The point cloud can be converted into other types of data formats to reduce the overall amount of stored data and reduce the time for computer processing. Therefore, the present invention converts the 3D voxel model into point cloud data, and uses the point cloud learning network (PointNet) framework as the framework for classifying parts on the 3D voxel model point cloud.

The present invention classifies and labels the parts of the two-dimensional automobile texture map, and then according to the vertex (U, V) coordinate values on the three-dimensional voxel model, the part labels corresponding to the coordinates on the two-dimensional texture map make the three-dimensional point cloud model The points above have label information of part categories, as shown in Fig. 9, the points in different colors in the figure represent different part categories. Therefore, in order to achieve the automatic part classification of the three-dimensional voxel model, the present invention inputs the three-dimensional voxel model with the part category label into the three-dimensional neural network model framework for training, and learns the characteristics of different types of parts, so as to predict the three-dimensional point cloud in the future Classification of parts of the model. The 3D point cloud model is composed of many triangular mesh facets, and each triangular mesh facet has three vertices. Part label on point cloud The present invention will learn the network model through the point cloud, and the part label trained by this model is obtained from the two-dimensional texture map.

The 3D point cloud model used in the present invention is a file format developed by Wavefront Technology. This format is developed for the animation tool Advanced Visualizer. It is also a file format of geometric figures. It is currently used by many 3D graphics software. Its sub-file name is "obj". This file format is a format for storing simple data of 3D geometric figures. The arrangement format in each file includes data such as "point", "texture map", "normal vector", and "surface". The present invention only needs two data of "point" and "texture map", "point" records the point coordinates, and "texture map" records the coordinates corresponding to the texture map.

Firstly, after obtaining the coordinates of the points on the 3D point cloud model (the file whose extension is "obj"), then save the point coordinates as a file in general text format (the file whose extension is "pts"). Then open the corresponding texture map (the file whose extension is "mtl"), and search for the part label value at the position, and then save the "part label value" on the point as a general text format file (the extension name is "seg"), its data conversion method and process are shown in Figure 10. Finally, the "pts" and "seg" data are converted into three-dimensional models, so as to be used as training data for the PointNet model, that is, the real data required for training.

After the real data is generated, it can be used to train a 3D point cloud learning network. Since the format of a point cloud or grid is not conventional, most researchers convert such data into a regular 3D voxel grid or collection of images before feeding it to a deep learning network architecture. However, this transformation of the data representation will fail to reveal the natural invariance properties of the data. The 3D point cloud model is one of the important types of geometric data structures, and its data has three characteristics: "rotation invariance", "interrelationship" and "disorder". Existing point cloud models are task-specific and are designed and constructed manually. Points on a point cloud are usually codes of statistical properties, which have the property of transformation invariance. Therefore, the present invention adopts 3D point cloud to represent different input forms of 3D geometry, and uses the point cloud network as the deep learning network architecture of the 3D object classification model.

Before the point cloud learning network architecture extracts point cloud features, it must be multiplied by the transformation matrix of the small neural network T-Net and the point cloud data, so that the data of the point cloud are aligned to preserve the invariant characteristics of point cloud rotation . The transformation matrix of T-Net is processed by a convolutional layer and a fully connected layer to obtain a 3×3 or 64×64 matrix.

A 3D model is composed of points, and there is "interrelationship" between all points in space. In the point cloud learning network architecture, semantic segmentation combines the local and global feature information of the classification to make effective use of the spatial information between points. "Irregularity" refers to inputting point cloud data of different sorts into the model structure, and the point cloud data will not be affected. Although the input of point cloud data is disordered, the convolutional layer is sequential. Therefore, after convolution and feature acquisition, there is still a sequential relationship between points in the point cloud. In view of the disordered characteristics of point clouds, the point cloud learning network model uses the "max pooling layer" to extract its features, so that the point cloud remains disordered.

The architecture diagram of the point cloud learning network is shown in Figure 11. Its architecture is divided into two architectures: classification and segmentation. This model first performs a classification structure, taking n×3 as input, n is the number of sampling points of the point cloud, and 3 is the three-dimensional coordinate value of the x, y, and z axes. First, the small neural network T-Net generates a 3×3 conversion matrix, and then converts the input coordinate values to align the coordinate values of the points. Then, the features of the point cloud are obtained through the multi-layer perceptron with shared weights, and then the transformation matrix of 64×64 size is generated by T-Net to align the points, and then the features of the points are obtained by the multi-layer perceptron, so that each point will get 1024 Dimensional feature vector, and then obtain the feature of the whole domain through the maximum pooling, and finally obtain the category score (score) of the point by a multi-layer perceptron. After completing the processing of the classification structure, the processing of the segmentation structure is carried out. The structure combines the 64-dimensional regional feature vector obtained in the classification structure with the feature vector of the entire domain obtained by the maximum pooling, and finally through the multi-layer perceptron with shared weights. , the part category score of each point can be obtained. The classification results of auto parts are shown in Fig. 12.

To sum up, since the present invention performs object segmentation and classification on two-dimensional images through traditional image processing technology and convolutional neural network, it can reduce the cost of labeling scene object categories. In addition, the present invention randomly combines and generates a large number of texture maps by changing the characteristics of the object such as shape, color, rotation, and background color system, which can replace the work of manually marking the object category of the 3D model. In addition, the present invention uses two-dimensional and three-dimensional deep learning networks to classify objects of three-dimensional models, and uses the classification results of texture maps as an auxiliary use. It is not necessary to manually check files and process classification one by one, so manpower and time costs can be reduced.

The above description is only a preferred embodiment of the present invention, and all equivalent method changes made by applying the description of the present invention and the scope of the patent application should be included in the scope of the patent of the present invention.

S1, S2, S3, S4: steps

Claims (10)

A method of using two-dimensional images to automatically generate the ground truth required for training a three-dimensional point cloud learning network (PointNet), which is implemented by a computer. The method includes the following steps: A. receiving a plurality of two-dimensional images, Each of these two-dimensional images contains a plurality of objects, and an automatic cutting technology is used to cut these objects presented on each of these two-dimensional images; B. For the objects that have been cut in step A, apply RGB Various changes are caused by channel arrangement, color, and flipping different angles, and different changes are adjusted to the background of the two-dimensional image. By randomly combining these objects and the two-dimensional image, a large number of textures suitable for these objects are generated. texture maps, the number of these texture maps is greater than the number of these two-dimensional images; C. Use the third version of YOLO (You Only Look Once v3) neural network model technology to identify these texture maps and generate corresponding plural Object identification; and D. Combining these texture maps with a plurality of three-dimensional triangular meshes, thereby obtaining object identifications corresponding to each point of each of the meshes, and then generating complex numbers for training a three-dimensional point cloud learning network factual information. The method for using two-dimensional images as described in claim 1 to generate real data required for training a three-dimensional point cloud learning network, wherein the two-dimensional images are in color, and the automatic cutting technology described in step A is to first convert the color The two-dimensional image is converted into a grayscale image, and then binarization is performed, and then the objects are labeled, so as to detect and cut the objects. The method of using two-dimensional images as described in claim 1 to generate real data required for training three-dimensional point cloud learning networks, wherein the two-dimensional images are in color, and the steps described in step A The automatic cutting technology first converts the color two-dimensional image into a grayscale image, then adjusts the grayscale contrast of the image by means of histogram equalization, and then performs binarization processing, and uses erosion and blur Remove the noise of these objects, thereby detect these objects and cut them. As described in claim 3, the method for using two-dimensional images to generate real data required for training three-dimensional point cloud learning networks, wherein, if the length or width of any object after cutting exceeds 4000 pixels, then the object is re- Convert to a grayscale image and perform binarization processing, and then perform an opening operation (opening) to remove the noise of the object, so as to detect and cut the multiple objects contained in the objects. As described in claim 3, the method of using two-dimensional images to generate the real data required for training the three-dimensional point cloud learning network, wherein the blurring system adopts the method of Gaussian blur (gaussian blur), and its filter is expressed as follows:

Where σ is the standard deviation of the normal distribution, u and v are the blur radii of different dimensions. The method for generating real data required for training a 3D point cloud learning network using a 2D image as described in Claim 3, wherein the erosion is to use a filter with a fixed size to perform a convolution operation on a black and white image. The method for using two-dimensional images to generate real data required for training a three-dimensional point cloud learning network as described in claim 1, wherein the RGB channel arrangement is to change the arrangement order of the RGB channels without changing the RGB values. The method of using two-dimensional images to generate real data required for training three-dimensional point cloud learning network as described in claim item 7, wherein changing the arrangement order of RGB channels includes RGB, RBG, GRB, GBR, BGR, and BRG. As described in Claim 1, the method of using two-dimensional images to generate real data required for training a three-dimensional point cloud learning network, wherein the three-dimensional point cloud learning network is a file format developed by Wavefront Technology for the animation tool Advanced Visualizer. As described in claim 9, the method of using two-dimensional images to generate the real data required for training a three-dimensional point cloud learning network, wherein the three-dimensional point cloud learning network is first combined with the neural network T- before extracting point cloud features Net's transformation matrix is multiplied, where the transformation matrix is processed by a convolutional layer and a fully connected layer to obtain a 3×3 or 64×64 matrix.

Images