KR20240048762A - Method and apparatus for 3d object recognition and pose estimation based on graph convolutional network - Google Patents

Abstract

The present invention is a 3D object recognition and pose estimation method performed in a 3D object recognition and pose estimation device, comprising the steps of receiving a point cloud as an input, constructing a graph from the point cloud, and adding GCN (GCN) to the graph. It includes detecting an object and estimating its pose by repeatedly applying a Graph Convolutional Network, and determining a bounding box of the detected object.

Description

GCN-based 3D object recognition and pose estimation method and device {METHOD AND APPARATUS FOR 3D OBJECT RECOGNITION AND POSE ESTIMATION BASED ON GRAPH CONVOLUTIONAL NETWORK}

The present invention relates to 3D object recognition and pose estimation technology. Based on GCN (Graph Convolutional Network), 3D objects are detected by structuring points in a point cloud into a graph and 9 degrees of freedom. ; DOF) relates to a 3D object recognition and pose estimation method.

Computer vision is one of the computer science research fields that studies aspects of machine vision. Recently, deep learning-based 3D object detection methods using 3D sensors have been actively proposed in various fields such as autonomous driving and robotics. With the rapid development of 3D acquisition technology, there are various types of 3D scanners, LiDAR, and RGB-D cameras (for example, Kinect, RealSense, or Apple depth cameras) to acquire 3D information, such as Space and object geometric shape or scale information can be obtained from information collected by a 3D sensor. A 3D sensor uses infrared rays to find depth information between an object and the sensor through changes in the waveform or time difference of the wavelength reflected by the object. Accurate spatial characteristics can be obtained through depth information obtained from a 3D sensor, and can be converted into spatial information including color information using the camera's RGB data. This spatial information can be represented as a point cloud, one of the three-dimensional data structures.

A point cloud is primitive point information representing three-dimensional information, and has information such as color, reflectivity, or transparency as well as coordinate information. Existing 3D information was used by converting it into a 3D mesh model using polygons to reduce data processing, but with the development of hardware performance such as GPU, 3D point clouds can now be processed directly. Recently, deep learning has been used to efficiently process point clouds, and in particular, research in the field of object detection is being actively conducted for use in the fields of unmanned vehicles, drones, or augmented reality. A method of processing point clouds using deep learning for object detection includes a 3D CNN (Convolution Neural Network) method that processes spatial information implicitly. CNN is a deep learning model widely used in the image field and has the characteristic of efficiently gathering and processing image features. However, in the case of 3D, there is a problem that a large amount of computation occurs due to the application of CNN filters when implying spatial features, and 3D coordinate information of points is lost during the implication process. Therefore, research is continuously being conducted on defining the most suitable point cloud representation for deep learning and how to process it.

Korean Patent Publication No. 10-2022-0095091

The purpose of the present invention is to represent a point cloud as a graph based on GCN and use a structured adjacency matrix and feature matrix to obtain the class, bounding box, size, and pose of the object to which each vertex belongs. 3D object recognition and pose The purpose is to provide estimation methods and devices.

The purpose of the present invention is to apply KAT (Keypoint Attention Mechanism), which sorts coordinates according to the characteristics of neighboring points based on GAT (Graph Attention Network), to estimate the poses of multiple objects with a single shot and reduce the transformation variance of GCN. The purpose is to provide a 3D object recognition and pose estimation method and device.

The purpose of the present invention is to overcome the rotation error by estimating the nine degrees of freedom (i.e., three coordinates, three translations, and three rotations) of the object through quaternion rotation and to estimate the IoU rotated around all axes. The goal is to provide a 3D object recognition and pose estimation method and device that calculates (Intersection over Union).

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

The first aspect of the present invention for achieving the above object is a 3D object recognition and pose estimation method performed in a 3D object recognition and pose estimation device, comprising the steps of receiving a point cloud as an input, the point cloud It includes constructing a graph from, detecting an object and estimating its pose by repeatedly applying GCN (Graph Convolutional Network) to the graph, and determining a bounding box of the detected object.

Preferably, the step of constructing the graph includes determining key points that will become vertices of the graph in the point cloud, connecting the key points to structure the graph, and generating an adjacency matrix and a feature matrix from the structured graph. It may include steps.

Preferably, the step of determining the key point includes calculating the size of the space based on the maximum coordinate value of the point cloud, dividing the space by dividing the size of the space by a preset voxel size, and dividing the space into the point cloud. Dividing each coordinate by the voxel size to set a voxel index and indicating the position of the voxel corresponding to the voxel index, and randomly determining a key point among points included in the same voxel based on the voxel index, , the voxel size can be set according to the size of the 3D object to be detected.

Preferably, the step of structuring the graph includes addressing the key points in a determined order, and constructing a graph by connecting the key points through a sphere-based nearest neighbor search algorithm set to a size that can include the size of the object. It may include a step of generating an edge list indicating a connection relationship between key points based on the coordinate information and addressing information of the key point.

Preferably, the step of generating the adjacency matrix and the feature matrix includes adding a loop connecting the keypoint itself to the edge list to generate an original adjacency matrix indicating the connection state between keypoints, and the edge list It may include generating a degree matrix representing the number of edges of each key point based on and generating a relative adjacency matrix representing the connection state and relative distance information between key points through the difference between the degree matrix and the original adjacency matrix. .

Preferably, the step of generating the adjacency matrix and the feature matrix includes generating a feature matrix based on feature information of the key point, wherein the feature information includes coordinate information of the key point, RGB information of the key point, or key point information. It may correspond to the coordinate information of the relative adjacency matrix for .

Preferably, the step of detecting the object and estimating the pose includes multiplying the relative adjacency matrix and the feature matrix to generate a relative feature matrix reflecting relative characteristics between the key points, and It may include obtaining a new feature matrix by applying color information to a multi-layer neural network (Multi-Layer Perceptrons (MLP)).

Preferably, the step of detecting the object and estimating the pose includes repeating an operation of multiplying the new feature matrix and the original adjacency matrix a certain number of times to obtain an operation output value in which information on adjacent key points is aggregated, the operation calculation It may include applying the value to a multi-layer neural network to convert it into a final calculated value from which information about the order of key point features is removed, and predicting the class and bounding box of the object based on the final calculated value.

Preferably, the step of obtaining the operation output value includes obtaining a result value by multiplying the new feature matrix and the original adjacency matrix, applying a skip connection to the result value, and the skip connection. After applying , the step of multiplying the new feature matrix and the original adjacency matrix to obtain a result value and re-performing the step of applying a skip connection may be included.

Preferably, the step of predicting the class and bounding box includes applying the final calculated value to a multi-layer neural network to generate a predicted value for each class, and applying the predicted value to Softmax to generate a probability value for each class. It may include converting to , and predicting bounding boxes as many as the number of classes.

Preferably, the step of determining the bounding box includes performing quaternion rotation on the bounding box predicted for the detected object, and when there are multiple predicted bounding boxes, Non-Maximum Suppression (NMS) It may include determining a bounding box for the object using .

Preferably, the step of learning the GCN may be further included.

Preferably, before the step of learning the GCN, generating a class label by assigning a class index, a background index, or a non-consideration index to the corresponding object according to the position of the key point constituting the graph with respect to the bounding box, and Generating a box label based on the center coordinates (x, y, z), width, depth, height, pitch, roll, and yaw of the bounding box, The center coordinate corresponds to a vector value calculated through the difference between the center of the bounding box and the key point, the width, depth, and height are normalized values between 0 and 1, and the pitch, roll, and yaw are quaternion rotations. It may correspond to a value converted through .

Preferably, the step of learning the GCN includes calculating cross entropy based on the distribution between the actual class of the object and the class predicted for the object, for class learning for detecting the object, and the object In order to learn the bounding box for estimating the pose, a step of calculating by applying a Huber loss function to the actual bounding box of the object and the bounding box predicted for the object may be included.

The second aspect of the present invention for achieving the above object is a 3D object recognition and pose estimation device, comprising a data input module that receives a point cloud, a graph conversion module that constructs a graph from the point cloud, and a GCN in the graph. It includes a deep learning performance module that detects an object and estimates its pose by repeated application, and a box management module that determines a bounding box of the detected object.

A third aspect of the present invention for achieving the above object is characterized in that, in a computer program stored in a computer-readable medium, when an instruction of the computer program is executed, a data matching method is performed.

As described above, according to the present invention, it is possible to obtain the class, bounding box, size, and pose of a 3D object from a point cloud with a single shooting.

In addition, by designing a GCN model for efficient feature implication, the learning and prediction layers are simplified compared to GNN-based systems to improve speed, and by introducing a relative adjacency matrix to reflect relative characteristics, accuracy is also improved.

In addition, the efficiency of memory use is increased by reusing the graph structure in each process of aggregating the point features of the point cloud itself and its neighbors, making learning and prediction possible not only on the GPU but also on the CPU.

1 is a block diagram of a 3D object recognition and pose estimation device according to a preferred embodiment of the present invention.
Figure 2 is a block diagram of a graph transformation module according to an embodiment.
Figure 3 is a flowchart of a 3D object recognition and pose estimation method according to an embodiment.
Figure 4 is an example diagram for explaining a relative adjacency matrix according to an embodiment.
Figure 5 is an example diagram for explaining a new feature matrix according to an embodiment.
Figure 6 is an example diagram for explaining Intersection over Union (IoU) according to an embodiment.

Hereinafter, the advantages and features of the present invention and methods for achieving them will become clear with reference to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification. “And/or” includes each and every combination of one or more of the mentioned items.

Although first, second, etc. are used to describe various elements, elements and/or sections, it is understood that these elements, elements and/or sections are not limited by these terms. These terms are merely used to distinguish one element, component or section from other elements, elements or sections. Accordingly, it goes without saying that the first element, first element, or first section mentioned below may also be a second element, second element, or second section within the technical spirit of the present invention.

In addition, identification codes (e.g., a, b, c, etc.) for each step are used for convenience of explanation. The identification codes do not explain the order of each step, and each step is clearly specified in the context. Unless the order is specified, it may occur differently from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” refers to the presence of one or more other components, steps, operations and/or elements. or does not rule out addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Additionally, in describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. The terms described below are defined in consideration of functions in the embodiments of the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

1 is a block diagram of a 3D object recognition and pose estimation device according to a preferred embodiment of the present invention.

Referring to FIG. 1, the 3D object recognition and pose estimation device 100 includes a data input module 110, a graph transformation module 120, a box management module 130, a deep learning performance module 140, and a visualization module ( 150), and a control module 160.

The data input module 110 receives data corresponding to a point cloud. Preferably, the subject providing the point cloud data can be applied in a variety of ways, and there are no restrictions on the method of providing and receiving data, so it will not be described in detail here.

The graph conversion module 120 configures the point cloud into a graph. Preferably, the graph transformation module 120 extracts key points on a voxel basis to efficiently encode a point cloud in an adjacent graph of a fixed radius and aggregate features of adjacent points using a new graph representation. And the extracted keypoints can be connected to form an adjacency matrix representing the connection relationship and a feature matrix representing the feature information of the keypoints. More specifically, referring to FIG. 2, the graph transformation module 120 includes a keypoint extractor 210 that extracts keypoints based on voxels, and an edge extractor that creates an edge list by forming connections between keypoints. A generator 220, an adjacency matrix converter 230 that converts the edge list into a sparse adjacency matrix, a feature matrix generator 240 that generates a feature matrix containing feature information of each key point, and finally It may include a tensor conversion unit 250 that converts the generated matrix into a tensor that can be used in Pytorch, an open source machine learning library.

The box management module 130 converts the actual bounding box of the 3D object into learning data, preprocesses it, and determines the bounding box for the recognized 3D object. That is, the box management module 130 includes a preprocessing algorithm and a normalizing algorithm for converting the actual bounding box into learning data, and may include a Non-Maximum Suppression (NMS) algorithm for determining the bounding box.

The deep learning performance module 140 is composed of a GCN layer and repeatedly applies GCN to the graph to detect a 3D object and estimate its pose. GCN (Graph Neural Networks) is a neural network that learns the representation of nodes or edges through message passing between nodes while maintaining the graph representation. GCN (Graph Convolutional Network), a representative GNN method, is an extension of the existing CNN (Convolutional Neural Network) and can process data supported by graphs. In the present invention, we propose a one-stage GCN pipeline that detects 3D objects and estimates their pose by structuring the distribution of irregular points into a graph. The one-stage method is a method in which the classification and localization problems of detecting and classifying objects are simultaneously solved. According to the present invention, detection and pose estimation of a 3D object from a point cloud can be performed simultaneously.

Preferably, the deep learning performance module 140 applies a Keypoint Attention Mechanism (KAT) that aggregates the relative features between each point to estimate the class and pose of the object to which the key point of the graph belongs and 9 of multiple object poses. Degrees of freedom (DOF) estimates can be designed. Here, KAT is an application of GAT (Graph Attention Network) to key points. GAT efficiently performs attention-based operations on graph-structured data and sets different weights for neighboring points. By specifying it, it can be applied to graph nodes that have a different degree of freedom than the number of neighbors the node has.

Visualization module 150 provides visualization. Preferably, the visualization module 150 can visualize the predicted bounding box through the deep learning performance module 140 so that it can be output on the display, and the visualization method can be performed by a person skilled in the art of the present invention. Since it can be easily performed, it will not be described in detail.

The control module 160 controls the operations and data flow of the data input module 110, graph conversion module 120, box management module 130, deep learning performance module 140, and visualization module 150. , learning and visualization can be implemented through options. That is, the control module 160 can control learning to be performed according to the user's selection or prediction (i.e., testing) based on the learned weight values, and visualization can be performed through the visualization module 150. You can control the visualization to be performed or not performed.

In one embodiment, the 3D object recognition and pose estimation device 100 increases the reusability of program sources through modularization of the configuration according to each role and allows the preprocessing process, learning process, and prediction process to be performed efficiently. . Preferably, the preprocessing process can be performed equally in both the learning and prediction processes, the learning process is a process for learning a deep learning algorithm based on the results of the preprocessing process, and the prediction process is a process for training the stored model after the learning has sufficiently progressed. This is the process of recognizing a 3D object and estimating its pose based on the results of loading and preprocessing.

Hereinafter, operations performed through each component of the 3D object recognition and pose estimation device 100 will be described in detail with reference to FIG. 3. Each step to be described with reference to FIG. 3 is described as being performed by a different configuration, but is not limited thereto, and depending on the embodiment, at least some of the steps may be performed in the same or different configurations.

Figure 3 is a flowchart of a 3D object recognition and pose estimation method according to an embodiment.

Referring to FIG. 3, the data input module 110 receives a point cloud (step S310). For example, a point cloud for N points is a set of It can be defined as, where is a three-dimensional coordinate that represents the properties of the point and status value It is a point that has all the length vectors corresponding to .

The graph conversion module 120 constructs a graph from the point cloud (step S320). Preferably, the graph transformation module 120 You can construct a graph G=(P,E) by using as a vertex and connecting neighboring points within a fixed radius, that is, a preset radius. Here, E corresponds to the neighboring points that are connected within the radius. Since a point cloud generally consists of tens of thousands of points, constructing a graph with all points as vertices can result in a significant computational burden. Therefore, a graph can be constructed using the down-sampled point cloud, and in the present invention, a graph can be constructed by extracting key points.

Preferably, the keypoint extractor 210 can determine a keypoint that will be the vertex of the graph in the point cloud. More specifically, the keypoint extractor 210 may calculate the size of the space based on the maximum spatial coordinates of the point cloud and divide the space by dividing the size of the space by a preset voxel size. Here, since the number of keypoints is determined according to the voxel size, the voxel size can be set according to the size of the 3D object to be detected. The key point extractor 210 may divide each coordinate of the point cloud by the voxel size to set a voxel index and represent each point as the position of a voxel corresponding to the voxel index. Here, the voxel index may correspond to the quotient obtained by dividing each coordinate of the point cloud by the voxel size. Next, the keypoint extractor 210 may randomly determine a keypoint among points included in the same voxel based on the voxel index.

Preferably, when a key point is determined through the key point extractor 210, the edge generator 220 can connect the key points to structure the graph. More specifically, the edge generator 220 addresses keypoints according to the order in which the keypoints are determined by the keypoint extractor 210. In other words, key points are numbered and indexed starting from 0 according to the order in which they were determined. Next, the edge generator 220 may construct a graph by connecting the keypoints through a sphere-based nearest neighbor search algorithm (k-Nearest Neighbors) set to a size that can include the size of the 3D object. Additionally, the edge generator 220 may generate an edge list indicating the connection relationship between key points centered on each key point based on the coordinate information and addressing information of the key point.

Preferably, the adjacency matrix conversion unit 230 generates an adjacency matrix from a structured graph. In other words, the adjacency matrix conversion unit 230 converts the adjacency matrix based on the edge list, and the point cloud structured as a graph is not a connection of all key points, but rather a connection of key points within a certain range, so the adjacency matrix is in the form of a sparse matrix. It is expressed.

More specifically, the adjacency matrix converter 230 adds a loop connecting the keypoint itself to the edge list so that the keypoint's own characteristics can be included when calculating feature matrices and matrix multiplication later, thereby indicating the connection status between keypoints. The original adj matrix can be created. The original adjacency matrix generated in this way can indicate the connection state between points, but does not contain information about the relative distance from the key point, so the adjacency matrix converter 230 generates an order matrix indicating the number of edges of each key point based on the edge list. You can create a relative adjacency matrix representing the connection status and relative distance information between key points through the difference between the degree matrix and the original adjacency matrix. In other words, a relative adjacency matrix is used to reflect relative information between points in the GCN structure. The present invention can apply a structure based on GAT (Graph Attention Networks) that uses attention to learn the relative weight between two nodes connected in message passing, and GAT is applied to the node pair i and j. It is not explained in detail because it can be easily understood by a person skilled in the art of the present invention to calculate the attention to detail. In the present invention, the GAT-based structure applied to aggregate the relative characteristics between each point is called KAT (Keypoint Attention Mechanism), and KAT aligns coordinates according to the characteristics of neighboring points to reduce the transformation variance of GCN. Hereinafter, KAT will be described in more detail with reference to FIG. 4.

Preferably, referring to FIG. 4, as an example diagram for explaining a relative adjacency matrix generated by applying KAT, the adjacency matrix conversion unit 230 sorts the coordinates of neighboring points according to structural features instead of the center coordinates of the key point. By applying KAT (Keypoint Attention Mechanism) and taking a negative number to the matrix obtained by applying KAT, the connection relationship of points is expressed as a positive number and the number of edges is expressed as a negative number, which can be used as a relative adjacency matrix. Here, the relative adjacency matrix can be expressed as relative coordinates between adjacent points by subtracting the center coordinate feature of the key point by the number of neighboring points connected to the key point, rather than adding the center coordinate feature of the key point as is when calculating the feature matrix and matrix multiplication. Using the state of its neighbors, the state of the key point is modified by [Equation 1] below, and the value obtained by [Equation 1] is similar to each vector value with direction and size.

[Equation 1]

Here, X is the (x, y, z) coordinate of the point, X _pk is the coordinate of the kth neighboring point _, and

The feature matrix generator 240 generates a feature matrix from a structured graph. Preferably, the feature matrix generator 240 may generate a feature matrix based on the feature information of the key point, and the feature information may be generated by applying the key point's RGB information, coordinate (x, y, z) information, and KAT. It may contain new coordinate information updated to the modified state, that is, KAT.

The deep learning performance module 140 repeatedly applies GCN to the graph to detect the object and estimate the pose (step S330). First, the graph is a set of nodes and a set of edges ( , ), where the node and the main line between silver node Autumn node It is a binary value indicating whether it is related to . The connections between nodes are directed, so edges is a reciprocal trunk may be different from , and each node has a c-dimensional feature vector, that is, It can be expressed as GCN is an extension of CNN that operates on a graph representation instead of a regular image grid, where the input graph Given a GCN operation is the given node neighbor of in Aggregates the functions of nodes. GCN operations Updates the value of a given node by aggregating features between adjacent nodes as shown in [Equation 2], which mirrors the way a convolutional filter aggregates nearby pixels and aggregates the features of nearby nodes. Unlike CNN, GCN does not apply different weights to different neighbors.

[Equation 2]

More specifically, the deep learning performance module 140 multiplies the relative adjacency matrix generated by the adjacency matrix conversion unit 230 and the feature matrix generated by the feature matrix generator 240 to create a relative feature matrix reflecting the relative information between key points. A new, higher-dimensional feature matrix can be obtained by generating a relative feature matrix and applying the color information (RGB information) of key points to a multi-layer neural network (Multi-Layer Perceptrons; MLP). Preferably, the deep learning performance module 140 may multiply a relative adjacency matrix and a feature matrix including coordinate information of key points to generate a relative feature matrix reflecting relative characteristics between key points. For example, referring to FIG. 5, which is an example diagram for explaining a relative feature matrix, it shows an example of an adjacency matrix configuration and an operation with a feature matrix, and the deep learning performance module 140 uses the relative adjacency matrix of (a). By multiplying the rows and columns of the feature matrix in (b), a relative feature matrix reflecting relative characteristics can be generated as in (c).

The deep learning performance module 140 may repeat the operation of multiplying the new feature matrix and the original adjacency matrix a certain number of times to obtain an operation output value in which information on adjacent key points is aggregated. Preferably, the deep learning performance module 140 obtains a result value by multiplying the new feature matrix and the original adjacency matrix, applies skip connection to the result value, and then multiplies the original adjacency matrix and the new feature matrix again. The result can be obtained. That is, the deep learning performance module 140 may repeat the process of applying the skin connection to the result of multiplying the new feature matrix and the original adjacency matrix a certain number of times. For example, the GCN operation can be performed three times, and thus information up to the third neighbor of the graph can be aggregated. Here, the skip connection adds previous features to new features created through the graph convolution process, solving the problem that performance weakens as the number of operations for features disappearing through differentiation and the adjacency of the graph increases during the backpropagation process. It is applied for. Preferably, the deep learning performance module 140 connects the information of the previous layer and the next layer to use the information of the previous layer, adds features of the original point, and spatial information lost during down-sampling of key point extraction. To recover, features can be combined from the contracting path to the expanding path.

The deep learning performance module 140 applies the calculation output value to a multi-layer neural network to convert it into a final output value from which information about the order of key point features is removed, and predicts the class and bounding box of the 3D object based on the final output value. You can. That is, the deep learning performance module 140 reads out and generalizes the features aggregated through GCN and then estimates values for the class and bounding box through the fully connected layer. Preferably, the deep learning performance module 140 may apply the final calculated value to a multi-layer neural network to generate a predicted value for each class and apply the predicted value to Softmax to convert it into a probability value for each class.

Preferably, the deep learning performance module 140 can predict bounding boxes as many as the number of predicted classes. Here, the value of the predicted bounding box is the center coordinates (Xc, Yc, Zc), width (W), depth (L), and height (H) information indicating the size of the box, and pitch indicating the degree of rotation of the box. Since it consists of roll and yaw values, prediction of the bounding box means estimation of the object pose.

The box management module 130 determines the bounding box of the detected object (step S340). Preferably, the box management module 130 loads the stored model after sufficient learning of the GCN has progressed through the deep learning performance module 140, and when the class and bounding box of the object are predicted, the result of the predicted bounding box is converted to the original. It can be converted on a point basis. That is, the box management module 30 predicts the bounding box or performs box encoding to convert it into a label corresponding to a key point and normalize it before learning progresses, so the bounding box is transformed through the deep learning performance module 140. Once predicted, box decoding is performed in reverse order of the normalization process in box encoding.

Preferably, when the bounding box predicted for each class through the deep learning performance module 140 is for the same object, that is, when a plurality of bounding boxes are predicted for the same object, the box management module 130 It is possible to perform NMS (Non-Maximum Suppresion) to determine the final bounding box by removing overlapping bounding boxes and making the predicted bounding boxes into one common bounding box that is most suitable for the object. The method of determining the bounding box through NMS can be easily performed by a person skilled in the art of the present invention, so it will not be described in detail.

Preferably, the box management module 130 may perform quaternion rotation on predicted bounding boxes before performing NMS. Quaternion rotation is a method used to solve the case of gimbal lock, which occurs when two rotation axes overlap in certain situations when using Euler angles and the directionality of one axis is lost. In other words, the quaternion matrix expresses the rotation of each axis by introducing the concept of complex numbers, so the gimbal lock phenomenon that occurs in Euler rotation, It solves shortcomings such as instability that occurs when the size of increases and an increase in the amount of computation. A quaternion is a rotation matrix corresponding to R _q shown in [Equation 3] and is a vector with 4 components, 3 of which are imaginary parts and 1 of which is a real part, and can be defined as q= w+xi+yj+zi there is.

[Equation 3]

Preferably, the visualization module 150 can visualize the bounding box determined by the box management module 130. Here, whether to perform visualization through the visualization module 150 can be performed by the user's selection, and this can be implemented as an option regarding whether to perform visualization through the control module 160.

Below, we describe the configuration performed during learning to predict classes and bounding boxes through GCN. Preferably, prediction and learning of classes and bounding boxes can be selectively performed by the user and implemented as a prediction and learning option in the control module 160.

In one embodiment, the deep learning performance module 140 may classify objects and backgrounds to learn classes. Preferably, the deep learning performance module 140 may receive the actual class of the object as input and convert the actual class into matrix data consisting of integer values through a one-hot encoding process. The deep learning performance module 140 calculates the predicted value for each class in [Equation 4]. After passing softmax to make it a value between 0 and 1, pass log, and calculate the categorical cross-entropy error of the class using the correct answer label and prediction so that the greater the uncertainty of the prediction, the larger the error. Calculate . Cross entropy is the actual answer during training and predicted value It can be calculated based on the distribution between the predicted class and the actual class among key points. Due to the nature of object detection, the distribution of the background class is larger than the distribution of the object class, so efficient learning is difficult when simply calculating the error, so cross entropy is used.

[Equation 4]

In one embodiment, the deep learning performance module 140 may perform learning to increase the accuracy of the bounding box through the correct answer label instead of the predicted value for the bounding box. Since the bounding box must be predicted as an exact value rather than a probability prediction, the Huber loss function, which is not sensitive to outliers, can be used. Huber loss function in [Equation 5] is the correct answer and predict The absolute value of the error is If it is smaller than L2 error (Least Square Error) is used. For larger areas, L1 errors (Least Absolute Deviations) are used. By combining the L1 error and L2 error, differentiation is possible in all sections, making it easy to use in the backpropagation process.

[Equation 5]

Preferably, before learning to predict classes and bounding boxes is performed in the deep learning performance module 140, the box management module 130 creates a keypoint label through the graph transformation module 120. You can create and normalize the bounding box. More specifically, the box management module 130 assigns an index of the corresponding class to keypoints within the bounding box and assigns a background index to keypoints outside the bounding box. Here, the index is increased by 1 for each class of the object, starting with 0, which means the background, and the last index is Don't Care, which means that if the object is partially cut off severely, it is not distinguished by any object during learning. Can be divided into classes. The box management module 130 may generate a class label corresponding to the index of the class and then generate a box label of the bounding box to be predicted. The bounding box consists of center coordinates (Xc, Yc, Zc), width (W), depth (L), and height (H) information indicating the size of the box, and pitch, roll, and yaw values indicating the degree of rotation of the box. . Here, the pitch, roll, and yaw values are values converted through quaternion rotation, and the corresponding 9 values can be converted into box labels corresponding to the bounding box to which the key point belongs to be used as predicted correct answer labels. The box management module 130 uses a vector value calculated through the difference between the center of the bounding box and the key point as the center coordinate of the bounding box, and sets the standard bounding representing the object for normalization of the width, depth, and height of the bounding box. Take the log of the value divided by the size of the box and normalize it to have a value between 0 and 1. Here, the average value of the class size of the object is used as the size of the standard bounding box. The class labels and box labels generated in this way can be converted to Pytorch tensors and used for learning, and normalization can be performed in the same way during prediction.

In one embodiment, the box management module 130 may obtain an Intersection over Union (IoU) based on the predicted bounding box and the actual bounding box to evaluate the accuracy of the predicted bounding box through step S330. IoU is a widely used evaluation metric for computer vision tasks such as object detection, segmentation, or tracking, and measures how close the predicted bounding box is to the actual one. Preferably, the box management module 130 may use the predicted bounding box and the actual bounding box as input to calculate the intersection volume of the two bounding boxes through IoU.

More specifically, in 3D object detection, the exact 3D IoU value of a 3D bounding box can be calculated using a three-part algorithm. First, the box management module 130 calculates the intersection between the faces of two bounding boxes using the Sutherland-Hodgman polygon clipping algorithm. For example, if x represents a determined bounding box and y represents an actual bounding box, the box management module 130 uses the inverse transformation of the bounding box x to calculate the intersection between the two bounding boxes x and y. Convert all boxes. The transformed bounding box x is axis-aligned and centered about the origin, while the bounding box y is translated into the coordinate system of bounding box x, its orientation is maintained, and its volume remains invariant in the rigid body transformation. The box management module 130 can calculate intersection points in this new coordinate system and estimate volumes from the transformed intersection points, and when that coordinate system is used, it is more efficient and simpler for bounding boxes because each surface is perpendicular to one of the coordinate axes. Polygonal clipping is possible. Next, the box management module 130 cuts each side of the bounding box y, which is a convex polygon, with respect to the axis-aligned bounding box x using a polygon clipping algorithm. To determine the rendered environment, each world polygon is clipped to the camera frustum, and clipping is performed using the powerful Sutherland-Hodgman algorithm. At this time, to cut the polygon with respect to the plane, the edges of the plane are extended to virtually infinity, each edge of the polygon is iterated clockwise, and whether that edge intersects a face of the axis-aligned bounding box x is determined. And for each vertex of the bounding box y, it is checked whether there is a vertex within the bounding box x, and the corresponding vertex is also added to the intersecting vertices. For example, as shown in (a) of FIG. 6, the box management module 130 can clip a polygon to a bounding box to calculate the intersection of each side, and the overall process of exchanging the bounding boxes x and y Repeat. Second, the box management module 130 calculates the intersection volume using the convex hull algorithm of all truncated polygons. For example, the volume of the intersection can be calculated by the convex hull of all truncated polygons, as shown in green in Figure 6(b). Third, based on the intersection and combined volumes of the two bounding boxes, the IoU as shown in (c) of FIG. 6 can be calculated. Preferably, after the IoU is calculated, the box management module 130 can perform NMS on a plurality of bounding boxes to determine one bounding box, which is the same as the process performed in step S340 during prediction, so detailed It is not explained clearly.

Meanwhile, the steps of the method or algorithm described in relation to the embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

The components of the present invention may be implemented as a program (or application) and stored in a medium in order to be executed in conjunction with a hardware computer. Components of the invention may be implemented as software programming or software elements, and similarly, embodiments may include various algorithms implemented as combinations of data structures, processes, routines or other programming constructs, such as C, C++, , may be implemented in a programming or scripting language such as Java, assembler, etc. Functional aspects may be implemented as algorithms running on one or more processors.

Although preferred embodiments of the three-dimensional object recognition and pose estimation method and device according to the present invention have been described above, the present invention is not limited thereto and may be modified in various ways within the scope of the claims, detailed description of the invention, and accompanying drawings. It is possible to implement it with modifications, and this also belongs to the present invention.

100: 3D object recognition and pose estimation device
110: data input module
120: Graph transformation module
130: Box management module
140: Deep learning performance module
150: Visualization module
160: control module
210: Keypoint extraction unit
220: Edge creation unit
230: Adjacency matrix conversion unit
240: Feature matrix generator
250: Tensor conversion unit

Claims (16)

In a 3D object recognition and pose estimation method performed in a 3D object recognition and pose estimation device,
Step of receiving point cloud input;
Constructing a graph from the point cloud;
Detecting an object and estimating its pose by repeatedly applying a Graph Convolutional Network (GCN) to the graph; and
A 3D object recognition and pose estimation method comprising determining a bounding box of the detected object.
The method of claim 1, wherein the step of constructing the graph includes:
determining a key point that will become a vertex of a graph in the point cloud;
Connecting the key points and structuring them into a graph; and
A 3D object recognition and pose estimation method comprising generating an adjacency matrix and a feature matrix from the structured graph.
The method of claim 2, wherein determining the key point comprises:
calculating the size of the space based on the maximum coordinates of the point cloud;
dividing the space by dividing the size of the space by a preset voxel size;
Setting a voxel index by dividing each coordinate of the point cloud by the voxel size and indicating the position of the voxel corresponding to the voxel index; and
A step of randomly determining a key point among points included in the same voxel based on the voxel index,
A 3D object recognition and pose estimation method, wherein the voxel size is set according to the size of the 3D object to be detected.
The method of claim 2, wherein the step of structuring into a graph comprises:
Addressing the key points according to a determined order;
Constructing a graph by connecting the key points through a sphere-based nearest neighbor search algorithm set to a size that can include the size of the object; and
A three-dimensional object recognition and pose estimation method comprising generating an edge list indicating a connection relationship between key points based on coordinate information and addressing information of the key points.
The method of claim 4, wherein generating the adjacency matrix and feature matrix comprises:
generating an original adjacency matrix representing the connection state between keypoints by adding a loop connecting the keypoints themselves to the edge list; and
Generating a degree matrix representing the number of edges of each key point based on the edge list, and generating a relative adjacency matrix representing the connection state and relative distance information between key points through the difference between the degree matrix and the original adjacency matrix. A 3D object recognition and pose estimation method characterized by:
The method of claim 5, wherein generating the adjacency matrix and feature matrix comprises:
Including generating a feature matrix based on the feature information of the key point,
The feature information corresponds to coordinate information of a key point, RGB information of a key point, or coordinate information of a relative adjacency matrix for a key point.
The method of claim 6, wherein detecting the object and estimating the pose comprises:
multiplying the relative adjacency matrix and the feature matrix to generate a relative feature matrix reflecting relative characteristics between the keypoints; and
A three-dimensional object recognition and pose estimation method comprising the step of applying the relative feature matrix and the color information of the key points to a multi-layer neural network (Multi-Layer Perceptrons; MLP) to obtain a new feature matrix.
The method of claim 7, wherein detecting the object and estimating the pose comprises:
repeating an operation of multiplying the new feature matrix and the original adjacency matrix a specific number of times to obtain an operation output value in which information on adjacent key points is aggregated;
Applying the calculated calculation value to a multi-layer neural network to convert it into a final calculated value from which information about the order of key point features is removed; and
A 3D object recognition and pose estimation method comprising predicting a class and bounding box of the object based on the final calculated value.
The method of claim 8, wherein the step of obtaining the calculated calculation value includes,
obtaining a result by multiplying the new feature matrix and the original adjacency matrix;
Applying a skip connection to the result value; and
After applying the skip connection, obtaining a result by multiplying the new feature matrix and the original adjacency matrix and re-performing the step of applying the skip connection.
The method of claim 8, wherein predicting the class and bounding box comprises:
generating a predicted value for each class by applying the final calculated value to a multi-layer neural network;
Converting the predicted value to a probability value for each class by applying Softmax; and
A 3D object recognition and pose estimation method comprising predicting bounding boxes equal to the number of classes.
The method of claim 1, wherein determining the bounding box comprises:
performing quaternion rotation on a bounding box predicted for the detected object; and
When there are a plurality of predicted bounding boxes, a 3D object recognition and pose estimation method comprising determining a bounding box for the object using Non-Maximum Suppression (NMS).
According to paragraph 1,
A 3D object recognition and pose estimation method further comprising the step of learning the GCN.
The method of claim 12, before learning the GCN,
generating a class label by assigning a class index, a background index, or a non-consideration index to the corresponding object according to the position of the key point constituting the graph with respect to the bounding box; and
Generating a box label based on the center coordinates (x, y, z), width, depth, height, pitch, roll, and yaw of the bounding box,
The center coordinates correspond to vector values calculated through the difference between the center of the bounding box and the key point, the width, depth, and height are normalized values between 0 and 1, and the pitch, roll, and yaw are quaternion rotations. A 3D object recognition and pose estimation method, characterized in that it corresponds to the value converted through .
The method of claim 12, wherein the step of learning the GCN includes:
For class learning to detect the object, calculating cross entropy based on the distribution between the actual class of the object and the predicted class for the object; and
In order to learn a bounding box for estimating the pose of the object, a step of calculating by applying a Huber loss function to the actual bounding box of the object and the bounding box predicted for the object. 3D object recognition and pose estimation method.
Data input module that receives point cloud input;
a graph conversion module that constructs a graph from the point cloud;
A deep learning performance module that detects objects and estimates poses by repeatedly applying GCN to the graph; and
A 3D object recognition and pose estimation device including a box management module that determines a bounding box of the detected object.
In a computer program stored on a computer-readable medium,
A computer program stored in a computer-readable medium, wherein when the command of the computer program is executed, the method according to any one of claims 1 to 14 is performed.