KR20230106057A - Method and apparatus for 6 degree of freedom pose estimation using artifical neural network - Google Patents

Abstract

A method and apparatus for estimating a six-degree-of-freedom posture performed by a computing device are disclosed.
A method for estimating a 6-DOF posture, performed by a computing device according to an embodiment, includes obtaining a first image of a first frame and a second image of a second frame, and converting the first image and the second image to an artificial neural network. generating a feature map from the first image and the second image by using a feature extraction network of the artificial neural network, combining 6 tokens corresponding to the 6 degrees of freedom and the feature map. Generating a set, generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combined set, the dimension of the artificial neural network The step of inputting the input vector to a reduction network and the step of determining the six degrees of freedom from the output of the dimensionality reduction network may be included.

Description

Method and apparatus for estimating 6 degree of freedom posture using artificial neural network

Hereinafter, a method and apparatus for estimating a six-degree-of-freedom posture using an artificial neural network are disclosed. According to embodiments, a method for learning an artificial neural network for estimating a camera pose including six degrees of freedom and a method and apparatus for estimating a camera pose using the learned artificial neural network are disclosed.

Simultaneous Localization and Mapping (SLAM) technology is an important element technology for robot operation and autonomous driving. In order to implement Odometry-related technologies for controlling autonomous vehicles such as SLAM, various sensors are used. Among them, a method of estimating 6 degrees of freedom (DoF) using only a monocular camera is called Monocular Visual-Odometry (VO). It is said. VO using this monocular camera has a relatively simple structure than a method using multiple cameras or auxiliary sensors (inertial sensors, LIDAR), and has the advantage of being easy to handle data and economical.

Recently, as the computational speed of computers has risen dramatically, deep learning has been applied to achieve greater performance improvement. For example, among the methods of predicting depth information through one image using a convolutional network, a network with a coarse-to-fine structure is constructed and the depth value obtained through the depth sensor is the true value (Ground Truth, GT). However, it has the disadvantage of being expensive to obtain GT depth information in a real environment. To solve this cost problem, a completely unsupervised learning method that learns without GT depth information has been proposed. This method is a method of estimating depth information and relative camera posture from consecutive frames, synthesizing a new image from the two information, and learning through the difference between the real image and the synthesized image. However, this method has a problem of accumulation of errors for images with a long camera pose estimation network due to scale-ambiguity, which is a fundamental problem of the VO method using a monocular camera. In order to solve this scale ambiguity problem, a geometric consistency loss function (Geometry Consistency Loss) has been proposed. The geometric coherence loss function is a function that normalizes the difference between the depth values of two consecutive images estimated through the depth information estimation network and minimizes the difference between the two values.

David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Neural Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 Jia-Wang Bian, Zhichao Li, Naiyan Wang, and Huangying Zhan. Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video. In Neural Information Processing Systems (NIPS), 2019

The object of at least one embodiment disclosed below is to provide a method for training an artificial neural network to estimate 6 degrees of freedom using only a monocular camera and solving the problem of high cost and accumulation of errors in the prior art. to be At least one embodiment aims to provide a method for estimating 6 degrees of freedom using a trained artificial neural network.

A method for estimating a 6-DOF posture, performed by a computing device according to an embodiment, includes obtaining a first image of a first frame and a second image of a second frame, and converting the first image and the second image to an artificial neural network. generating a feature map from the first image and the second image by using a feature extraction network of the artificial neural network, combining 6 tokens corresponding to the 6 degrees of freedom and the feature map. Generating a set, generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combined set, the dimension of the artificial neural network The step of inputting the input vector to a reduction network and the step of determining the six degrees of freedom from the output of the dimensionality reduction network may be included.

The feature extraction network may extract features using a 6-channel image obtained by combining the 3-channel image of the first image and the 3-channel image of the second image in a channel direction.

The step of generating a combined set using the six tokens and the feature map is the step of reconstructing the generated feature map into a set of patches having a constant size, combining the set of patches and the six tokens to perform the combination. It may include generating a set.

Each of the six tokens may have the same size as each of the patches.

The combination set may include n+6 patches, and n may be the number of patches included in the feature map.

The position embedding vector may be a vector having the same size as the patches.

The input vector may be determined based on the sum of element vectors constituting the combination set and position embedding vectors.

The dimensionality reduction network may reduce the size of patches constituting the input vector by repeatedly performing a self-attention operation on the input vector.

The step of determining the 6 degrees of freedom includes extracting patches corresponding to the 6 tokens from among patches of the input vector whose size is reduced by the dimensionality reduction network, and performing average pooling for each patch using the extracted patches. pooling) to estimate 6 degrees of freedom.

According to an embodiment, a learning method of an artificial neural network for position estimation with 6 degrees of freedom for location recognition and map construction, performed by a computing device, obtains a first image of a first frame and a second image of a second frame. estimating a first depth map of the first image and a second depth map of the second image, inputting the first image and the second image to the artificial neural network and outputting 6 degree-of-freedom information. calculating a transformation matrix between the first frame and the second frame based on the 6 degree of freedom information and outputting a loss function based on the first depth map, the second depth map, and the transformation matrix; Calculating a value and updating the artificial neural network based on the output value of the loss function.

The step of outputting the 6 degree of freedom information includes inputting the first image and the second image to an artificial neural network, and generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network. generating a combination set using the six tokens corresponding to the six degrees of freedom and the feature map; generating a position embedding vector reflecting positional characteristics of patches included in the feature map; Generating an input vector using an embedding vector and the combination set, inputting the input vector to a dimensionality reduction network of the artificial neural network, and determining the six degrees of freedom from an output of the dimensionality reduction network. there is.

After the 6 tokens and the position embedding vector are set to predetermined initial values, the output value of the loss function may be updated to decrease.

The output of the loss function is a first auxiliary function for adding all the differences between the first frame and the third frame obtained by reconstructing the second frame using the transformation matrix and the first depth map for effective pixels satisfying a predetermined condition. It can be determined based on the output.

The first auxiliary function may further include a term representing a structural similarity index between the first frame and the third frame with respect to the effective pixel.

According to another embodiment, the first auxiliary function may include a third depth map obtained by reconstructing the first depth map using the transformation matrix and a fourth depth map reconstructed so that the second depth map is positioned on the same pixel grid as the first depth map. It may be a function calculated by multiplying the normalization function obtained by dividing the difference between the third depth map and the fourth depth map by the sum of the third depth map and the fourth depth map.

The structural similarity index may be based on comparison of luminance, contrast, and structure between pixels.

The predetermined condition may be that a difference between the first frame and the third frame is smaller than a difference between the first frame and the second frame.

According to an embodiment, the output of the loss function may be determined by further considering an output of a second auxiliary function that adds all products of the spatial gradient component of the first frame and the gradient component of the first depth map for all pixels in space. can

According to another embodiment, the output of the loss function may be determined by further considering the output of a third auxiliary function that adds the normalization function to all effective pixels.

Generating a combined set using the six tokens corresponding to the six degrees of freedom and the feature map includes dividing the generated feature map into non-overlapping patches of a constant size, and generating a patch set composed of the patches. generating 6 tokens having the same size as the patches and corresponding to the 6 degrees of freedom, and generating the combined set by combining the patch set and the 6 tokens.

The combination set may include n+6 patches, and n may be the number of patches included in the feature map.

The step of determining the 6 degrees of freedom includes extracting patches corresponding to the 6 tokens from among patches of the input vector whose size has been reduced by the dimensionality reduction network, and calculating average pooling for each patch using the extracted patches. and estimating 6 degrees of freedom.

The computing device may include a processor, and the processor may perform steps of obtaining a first image of a first frame and a second image of a second frame; inputting the first and second images to an artificial neural network; generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network; generating a combination set using 6 tokens corresponding to the 6 degrees of freedom and the feature map; generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combination set; inputting the input vector to a dimensionality reduction network of the artificial neural network; and determining the six degrees of freedom from the output of the dimensionality reduction network.

According to one embodiment, a computing device is disclosed. The disclosed computing device includes a processor, the processor comprising: acquiring a first image of a first frame and a second image of a second frame; inputting the first image and the second image to an artificial neural network; Generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network, and generating a combination set using 6 tokens corresponding to the 6 degrees of freedom and the feature map. , generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combination set, the input to the dimensionality reduction network of the artificial neural network. The step of inputting a vector and the step of determining the six degrees of freedom from the output of the dimensionality reduction network may be performed.

According to one embodiment, a computing device is disclosed. The disclosed computing device includes a processor, and the processor includes steps of acquiring a first image of a first frame and a second image of a second frame, a first depth map of the first image and a second depth of the second image. Estimating a map, inputting the first image and the second image to the artificial neural network and outputting 6 degree of freedom information, based on the 6 degree of freedom information, between the first frame and the second frame. Calculating a transformation matrix; and calculating an output value of a loss function based on the first depth map, the second depth map, and the transformation matrix, and updating the artificial neural network based on the output value of the loss function. The outputting of the degree information may include inputting the first image and the second image to an artificial neural network; Generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network, and generating a combination set using 6 tokens corresponding to the 6 degrees of freedom and the feature map. , generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combination set, the input to the dimensionality reduction network of the artificial neural network. It may include inputting a vector and determining the six degrees of freedom from the output of the dimensionality reduction network.

Embodiments of the present invention may provide a method and apparatus for estimating a six-degree-of-freedom posture using an artificial neural network.

Embodiments may provide an unsupervised learning method for estimating a posture using only a monocular camera using a 6 DOF posture estimation method using an artificial neural network.

Embodiments have a simple structure using a monocular camera without an additional sensor, and thus can provide a method for estimating a 6DOF posture that is easy to handle and economical. This can reduce costs.

In addition, the embodiments may provide a method for estimating a 6DOF posture with improved performance compared to conventional convolutional neural networks using a hybrid artificial neural network.

In addition, the embodiments may solve the problem of accumulation of errors through a configuration using a loss function and provide a method for estimating an accurate camera posture.

Effects of the present invention are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

The accompanying drawings for use in describing the embodiments of the present invention are only some of the embodiments of the present invention, and for those of ordinary skill in the art (hereinafter referred to as "ordinary technicians") Other drawings may be obtained on the basis of these drawings without additional effort leading up to the invention.
1A to 1C are flowcharts illustrating a method for estimating a 6-DOF posture according to an exemplary embodiment.
2A to 2B are flowcharts illustrating a learning method of an artificial neural network according to an embodiment.
3 is a diagram for explaining the structure of a six degree of freedom posture estimation method.
4 is a diagram for explaining a hybrid structure composed of a feature extraction network and a dimensionality reduction network according to an embodiment.
5 is an exemplary diagram of learning data for learning an artificial neural network for 6-DOF posture estimation according to an embodiment.
6 is an exemplary view showing qualitative test results for a method for estimating a 6 degree of freedom posture according to an embodiment.
7 is a block diagram illustrating a device for estimating a 6 degree of freedom posture according to an exemplary embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of the present invention refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced in order to make the objects, technical solutions and advantages of the present invention clear. These embodiments are described in detail to enable those skilled in the art to practice the present invention.

Throughout the description and claims of the present invention, the word 'comprise' and variations thereof are not intended to exclude other technical features, additions, components or steps. In addition, 'one' or 'one' is used to mean more than one, and 'another' is limited to at least two or more.

In addition, terms such as 'first' and 'second' of the present invention are intended to distinguish one component from another, and the scope of rights is limited by these terms unless understood to indicate an order. is not For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected to the other element, but other elements may intervene. On the other hand, when an element is referred to as being “directly connected” to another element, it should be understood that no intervening elements exist. Meanwhile, other expressions describing the relationship between components, ie, “between” and “directly between” or “adjacent to” and “directly adjacent to” should be interpreted similarly.

In each step, identification codes (eg, a, b, c, etc.) are used for convenience of description, and identification codes do not explain the order of each step unless they inevitably result in logic, and each The steps may occur out of the order specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

Other objects, advantages and characteristics of the present invention will appear to those skilled in the art, in part from this description and in part from practice of the invention. The examples and drawings below are provided as examples and are not intended to limit the invention. Accordingly, details disclosed herein with respect to a particular structure or function are not to be construed in a limiting sense, but are merely representative and provide guidance for those skilled in the art to variously practice the present invention with any detailed structures substantially suitable. It should be interpreted as basic data.

Moreover, the present invention covers all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment in another embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

In this specification, unless otherwise indicated or clearly contradicted by context, terms referred to in the singular encompass the plural unless the context requires otherwise. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present invention.

The present invention relates to a 6 degree of freedom (DoF) pose estimation method for position recognition and map construction, and estimates the pose of a camera including 6 degrees of freedom by inputting an image obtained using only a monocular camera to an artificial neural network. may be the way to do it. According to an embodiment of the present invention, the six degrees of freedom may include position (x, y, z) change information and tilt and rotation (θ_Pitch, φ_Roll, ψ_Yaw) information of the camera between two different frames.

According to an embodiment of the present invention, a methodology for estimating a 6-DOF posture by acquiring images having a plurality of different frames and inputting the images to an artificial neural network is disclosed. The 6DOF posture estimation method according to an embodiment of the present invention has a relatively simpler structure than a method using multiple cameras or auxiliary sensors (inertial sensors, LIDAR), and can provide an easy-to-handle and economical means for handling data. Through this, it can be used to implement odometry-related technologies for controlling autonomous vehicles such as location recognition and map construction.

1A to 1C are flowcharts illustrating a method for estimating a 6-DOF posture according to an exemplary embodiment.

According to an embodiment, a method for estimating a 6 degree of freedom posture may include determining the 6 degrees of freedom by inputting an image acquired according to the method shown in FIG. 1A to an artificial neural network. Specifically, the method for estimating a 6 degree of freedom posture performed by a computing device includes obtaining a first image of a first frame and a second image of a second frame (S110), and the first image and the second image are converted to an artificial neural network. (S120), generating a feature map from the first image and the second image using the feature extraction network of the artificial neural network (S130), the six tokens corresponding to the six degrees of freedom and the Generating a combined set using the feature map (S140), generating a positional embedding vector reflecting positional characteristics of patches included in the feature map, and generating an input vector using the positional embedding vector and the combined set It may include generating (S150), inputting the input vector to the dimensionality reduction network of the artificial neural network (S160), and determining the six degrees of freedom from the output of the dimensionality reduction network (S170).

In step S110, the computing device may obtain a first image of the first frame and a second image of the second frame. The first image and the second image may be images of different viewpoints acquired in adjacent frames while a single camera moves. According to an embodiment, the first image may include a first frame. According to an embodiment, the first image and the second image may be color images having 3 channels. That is, the first image and the second image may be 3-channel RGB images. However, the embodiment is not limited thereto. For example, the color corresponding to each channel may be set differently within a range easily changeable by a person skilled in the art.

In step S120, the computing device may input the first image and the second image to the artificial neural network. According to an embodiment, an artificial neural network may have a hybrid structure composed of a feature extraction network and a dimensionality reduction network.

According to an embodiment, the feature extraction network may be based on a ResNet model to which a Convolutional Neural Network (CNN) is applied. The convolutional neural network is an artificial neural network model using convolution, and it is specialized in processing multi-dimensional array data and can clearly detect features of images. In the convolutional neural network, features of an image can be detected while repeatedly performing a convolution operation using a filter. Specifically, features can be extracted from the convolution layer and pooling layer of the convolutional neural network, and classification can be determined from the fully-connected layer.

Residual neural network (hereinafter, ResNet) is a model that applies convolutional neural networks. It solves the problem of deteriorating performance as neural networks are accumulated more deeply by adding the output of the convolutional layer to the input used in the previous layer to prevent loss of features. can ResNet only increases the amount of addition operations, but has no significant effect on the learning parameters, so deep models with a larger number of layers can be learned faster.

According to one embodiment, the feature extraction network may be based on ResNet18 with 18 layers. In the case of ResNet18, it can be composed of a total of 18 layers, including one convolutional layer at the beginning, four layers each from layers 1 to 4, and finally one fully connected layer. According to another embodiment, the feature extraction network may be based on ResNet50 or ResNet152. If you count the layers in ResNet50, first of all, a bottleneck block consists of a total of three convolutional layers, 1x1, 3x3, and 1x1, and considering layers1 to 4 in this layer, 3 * (3 + 4 + 6 +3), 48 may be a model composed of a total of 50 layers each having one convolutional layer and one fully connected layer. In the above description, examples of the structure of the neural network have been specifically described, but the scope of the present invention is not limited thereto. The structure of the neural network may be changed within a range easily conceivable by a person skilled in the art in order to achieve the purpose of performing a function of extracting feature maps of the first image and the second image.

According to an embodiment, the feature extraction network may output a feature using a 6-channel image obtained by combining the 3-channel image of the first image and the 3-channel image of the second image in the channel direction. Exemplarily, the computing device may extract features of the combined 6-channel image by inputting a 6-channel image obtained by combining two different RGB images in a channel direction to a feature extraction network.

According to one embodiment, the dimension reduction network may be based on a dimension reduction vision transformer network. The dimension reduction network may use a method of reducing the dimension of the hidden layer output after each encoder layer using a linear projection.

According to another embodiment, the dimension reduction network is composed of Layer Norm, Multihead Self-Attention, Multilayer Perceptron (MLP), and Dim-reduction MLP, and has a configuration in which the dimension is gradually reduced by repeating the attention operation as many as the number of L layers. can

In step S130, the computing device may generate a feature map from the first image and the second image using a feature extraction network. Specifically, the computing device may apply a filter to a 6-channel image obtained by combining 3-channel RGB images to obtain a convolutional result value, and may generate a feature map using the obtained result value.

More specifically, referring to FIG. 1B , the computing device may reconstruct the generated feature map into a set of patches having a certain size (S141), and combine the patch set and the six tokens to generate the combined set. (S142).

According to an embodiment, in step S141, the computing device may divide the feature map into non-overlapping patches of a certain size in order to perform an attention operation for each patch on the 3D feature map. The computing device may generate a set of patches having divided patches as elements.

According to an embodiment, the computing device may generate 6 tokens having the same size as the patches generated using the feature maps of the first image and the second image. The generated 6 tokens may be learnable parameters corresponding to each of the 6 degrees of freedom in order to obtain the 6 degrees of freedom. That is, as will be described later, the 6 tokens can be updated while the computing device trains the artificial neural network. The computing device may create a combined set by combining the generated 6 tokens and patches.

According to an embodiment, in the case of patches reconstructed as a set of patches generated by dividing a feature map extracted from a convolutional neural network, location information may be lost. Therefore, in order to consider location information, in step S150, the computing device may generate a location embedding vector that reflects locational characteristics of patches included in the feature map. The computing device may generate an input vector by adding the generated position embedding vector and elements of the combination set.

In step S160, the computing device may input the input vector to the dimensionality reduction network. The computing device may gradually reduce the dimensionality of the patches constituting the input vector while repeating the self-attention operation as many as the number of L layers through the dimensionality reduction network.

Self attention may be an operation that calculates a degree of similarity using a dot product of vectors. The computing device may calculate the similarity of each vector by using a dot product of vectors, and may express it as a probability by using softmax. Softmax may be a function that makes the range of each component greater than or equal to 0 and less than or equal to 1 and makes the total sum equal to 1. The computing device may obtain a weight using the similarity of the vectors obtained in this way. In summary, self-attention may be an operation that calculates a relationship between components in an input vector. That is, when the computing device receives an input vector, it can reduce the dimensionality of the patches by outputting a highly related component and repeating this process.

According to an embodiment, the computing device may extract patches output according to the input of 6 tokens from among the patches of the input vector whose size is reduced by the dimensionality reduction network (S171), and use the extracted patches to obtain an average per patch. Six degrees of freedom can be estimated by calculating pooling (S172). Average pooling is a method of downsampling by averaging each patch, and the amount of calculation can be reduced because the use of parameters is not required during calculation.

2A to 2B are flowcharts illustrating a learning method of an artificial neural network according to an embodiment.

Referring to FIG. 2A , a learning method of an artificial neural network for position estimation with 6 degrees of freedom for location recognition and map construction, performed by a computing device, obtains a first image of a first frame and a second image of a second frame. step (S210), estimating a first depth map of the first image and a second depth map of the second image (S220), inputting the first image and the second image to the artificial neural network to obtain 6 Outputting DOF information (S230), calculating a transformation matrix between the first frame and the second frame based on the 6 DOF information (S240), and the first depth map and the second depth. The method may include calculating an output value of a loss function based on the map and the transformation matrix and updating the artificial neural network based on the output value of the loss function.

The computing device may perform completely unsupervised learning in which it learns without actual depth information. To this end, the computing device may estimate a depth map and a relative camera posture from successive frames, synthesize a new image using the depth map, and learn the difference between the synthesized image and the actual image.

Specifically, the computing device may acquire a first image of a first frame and a second image of a second frame in step S210. In operation S220, the computing device may estimate a first depth map by inputting the obtained first image to a depth estimation network. The first depth map may include depth information of the first image. Also, the computing device may estimate the second depth map by inputting the second image to the depth estimation network.

According to an embodiment, the depth estimation network may be based on a U-net structure having an encoder-decoder form. By using the U-net structure, the overlapping ratio of the patches that recognize images is small, so the inefficiency of the sliding window method, which was widely used in the existing segmentation model, can be solved. In particular, in case of using the U-net structure, the part verified in the previous patch may not be repeatedly verified in the next patch, so the efficiency of calculation and the speed can be improved. The encoder part of the U-net structure can be composed of a typical convolutional neural network. Illustratively, the encoder portion of the depth estimation network may be based on the Resnet50 model. The decoder part of the U-net structure may be composed of upsampling and convolutional neural networks. Illustratively, the decoder portion of the depth estimation network may be based on the DispResnet model.

According to an embodiment, the computing device may output a 1-channel depth map by inputting a 3-channel image to a depth estimation network. The output 1-channel depth map may have the same size as the input 3-channel image. In addition, the 1-channel depth map may represent a depth value for each pixel of a 3-channel image.

In step 230, the computing device may output 6DOF information by inputting the first image and the second image to the artificial neural network. The 6 degree of freedom information may be information about a change in camera posture between the first image and the second image. According to an embodiment of the present invention, the six degrees of freedom may include position (x, y, z) change information and tilt and rotation (θ, φ, ψ) information of the camera between the first image and the second image.

Specifically, the computing device may input the first image and the second image to the artificial neural network (S231), and generate a feature map from the first image and the second image using the feature extraction network of the artificial neural network (S232). . In addition, the computing device may generate a combination set using the generated feature map and tokens corresponding to the six degrees of freedom (S233), and generate a location embedding vector that reflects the locational characteristics of the feature map to generate the location embedding vector. An input vector may be generated using the combination set generated by and (S234). The computing device may determine 6 degrees of freedom from the output of the dimensionality reduction network by inputting an input vector to the dimensionality reduction network of the artificial neural network (S235).

According to an embodiment, the generated 6 tokens may have the same size as patches included in the feature map. In step S233, the computing device may create a combination set by combining the six tokens and the patches included in the feature map. Tokens combined with patches can acquire global information from each patch as learning progresses, and can play a role in estimating and classifying the class of an image through the acquired information.

In step S240, the computing device may calculate a transformation matrix between the first frame and the second frame based on the 6DOF information output in step S230. The transformation matrix may include information about movement and rotation of the camera between the first frame and the second frame. The transformation matrix can express the change in position according to the movement of the camera as (x, y, z) components, and the change according to the rotation of the camera can be expressed as (θ, φ, ψ) components.

In operation S250, the computing device may calculate an output value of the loss function based on the first depth map, the second depth map, and the transformation matrix. For example, the computing device may multiply the second frame by the transformation matrix and generate the third frame based on the depth information of the first depth map. According to an embodiment, if the third frame generated based on the transformation matrix and the first depth map is the same as the first frame, the loss function may be null.

According to an embodiment, the loss function may be determined by a combination of first to third auxiliary functions described later. The loss function may include all of the first to third auxiliary functions. As another example, the loss function may include only some of the first to third auxiliary functions. The first auxiliary function may be calculated based on the transformation matrix and the difference between the third frame and the first frame. For example, the first auxiliary function can be expressed as Equation 1.

에서의

의 영상평면으로 성공적으로 투영된 유효한 점을 나타내고,

는 점의 개수를 나타내고,

는 제1 프레임을 나타내고,

는 제3 프레임을 나타내고,

는 제2 프레임을 나타낸다.In Equation 1, V is

in

represents a valid point successfully projected onto the image plane of

represents the number of points,

represents the first frame,

represents a third frame,

represents the second frame.

According to an embodiment, the first auxiliary function may be a function calculated on the assumption that a fixed scene is captured using a moving camera. However, with this assumption, the network can operate to have an infinite distance value for a moving object or a scene where the camera is still.

According to another embodiment, the first auxiliary function based on Equation 1 may not be able to reflect the loss for the problem that the pixel brightness value between two frames is not constant because the amount of light changes in real time in a real environment. . Due to this loss, the computing device may not be able to learn normally. To solve this problem, the first auxiliary function may further include a term indicating a structural similarity index between the first frame and the third frame with respect to the effective pixel. The structural similarity index may be based on comparison of luminance, contrast, and structure between pixels. The first auxiliary function including the structural similarity index can be expressed as Equation 2.

에서의

의 영상평면으로 성공적으로 투영된 유효한 점을 나타내고,

는 점의 개수를 나타내고,

는 제1 프레임을 나타내고,

는 제3 프레임을 나타내고,

는 제2 프레임을 나타낸다. SSIM은 영상 간의 픽셀 밝기 값을 정규화 해주는 함수로써,

는 SSIM 함수를 통해

와

간의 요소별 유사성을 계산한 것을 나타낸다. 수학식 2에서 는 직접 설정이 가능한 하이퍼 파라미터(Hyper parameter)이다.In Equation 2, V is

in

represents a valid point successfully projected onto the image plane of

represents the number of points,

represents the first frame,

represents a third frame,

represents the second frame. SSIM is a function that normalizes pixel brightness values between images.

through the SSIM function

and

It shows the calculation of the similarity of each element between the liver. In Equation 2, is a hyper parameter that can be directly set.

According to an embodiment, the computing device may calculate a difference between the first depth map and the second depth map based on Equation 3.

는 제1 프레임의 제1 깊이맵

를 변환행렬을 이용해 재구성한 제3 프레임의 제3 깊이맵을 나타내고,

는 제2 깊이맵

을 보간법을 이용해 재구성한 제4 깊이맵을 나타낸다. 즉, 제1 깊이맵과 제2 깊이맵의 차이는 제3 깊이맵과 제4 깊이맵 간의 차이를 정규화 한 함수로 나타낼 수 있다. 정규화한 함수를 이용함으로써, 수학식3은 제1 깊이맵과 제2 깊이맵의 차이의 절대 값을 이용하는 것보다 더 동등한 깊이 정보 차이를 반영할 수 있다. 추가적으로, 함수의 결과 값이 0과 1 사이로 변환되기 때문에 최적화 과정에서 수치적으로 안정된 학습이 가능할 수 있다. in Equation 3

Is the first depth map of the first frame

Represents a third depth map of a third frame reconstructed using a transformation matrix,

is the second depth map

represents a fourth depth map reconstructed using the interpolation method. That is, the difference between the first depth map and the second depth map may be expressed as a normalized function of the difference between the third depth map and the fourth depth map. By using the normalized function, Equation 3 may reflect a more equal depth information difference than using the absolute value of the difference between the first depth map and the second depth map. Additionally, since the resulting value of the function is converted between 0 and 1, numerically stable learning may be possible during the optimization process.

According to another embodiment, the computing device may acquire an image of a scene in which moving objects or obscured objects are present. In this case, the computing device may not be able to smoothly learn. In order to solve this problem, the computing device uses a first auxiliary function to reconstruct a third depth map obtained by reconstructing the first depth map into the transformation matrix and a second depth map reconstructed so that they are located on the same pixel grid as the first depth map. The fourth depth map may be multiplied by a normalization function obtained by dividing a difference between the third depth map and the fourth depth map by the sum of the third depth map and the fourth depth map. Specifically, the first auxiliary function may be calculated based on Equation 4.

를 나타내고,

는 수학식 3을 기초로 하는 제1 깊이맵과 제2 깊이맵의 차이를 나타내는 함수이다. 또한, 수학식 4에서 V는

에서의

의 영상평면으로 성공적으로 투영된 유효한 점을 나타내고,

는 점의 개수를 나타낸다.in Equation 4

represents,

Is a function representing a difference between the first depth map and the second depth map based on Equation 3. Also, in Equation 4, V is

in

represents a valid point successfully projected onto the image plane of

represents the number of points.

According to another embodiment, the computing device may obtain an image captured by an object moving at the same speed as the camera or by a stationary camera. In this case, the computing device may calculate the first auxiliary function only when the difference between the first frame and the third frame is smaller than the difference between the first frame and the second frame. Specifically, the first auxiliary function may be calculated based on Equation 5.

를 나타내고,

는 수학식 3을 기초로 하는 제1 깊이맵과 제2 깊이맵의 차이를 나타내는 함수이다. 수학식 5에서

으로,은 제1 프레임이 제2 깊이맵과 변환행렬을 기반으로 재구성된 영상이고

는 아이버슨 괄호(Iverson bracket)를 나타낸다. 즉,

는 이진 마스크로 작용하여 컴퓨팅 장치가 제1 보조함수를 소정 조건을 만족하는 유효 픽셀에 대하여 계산하도록 한다. 소정 조건은 상기 제1 프레임과 제3 프레임에서의 차이가 제1 프레임과 제2 프레임에서의 차이보다 적은 픽셀이다. 또한, 수학식 5에서 V는

에서의

의 영상평면으로 성공적으로 투영된 유효한 점을 나타내고,

는 점의 개수를 나타낸다.in Equation 5

represents,

Is a function representing a difference between the first depth map and the second depth map based on Equation 3. in Equation 5

, is an image in which the first frame is reconstructed based on the second depth map and the transformation matrix,

denotes Iverson brackets. in other words,

acts as a binary mask to allow the computing device to calculate the first auxiliary function for effective pixels that satisfy a predetermined condition. A predetermined condition is that a difference between the first frame and the third frame is smaller than a difference between the first frame and the second frame. Also, in Equation 5, V is

in

represents a valid point successfully projected onto the image plane of

represents the number of points.

According to another embodiment, when a computing device acquires a low-quality image or an image having a uniform area, it may be difficult to learn because accurate information is not obtained from the obtained image. The second auxiliary function may reflect the loss of such a low quality image or an image having a uniform area. The second auxiliary function may be calculated by adding all products of the spatial gradient component of the first frame and the gradient component of the first depth map for all pixels in space. Specifically, the computing device may calculate the second auxiliary function based on Equation 6.

는 제1 프레임을 나타내고,

는 제1 깊이맵을 나타내고 ∇는 공간상의 방향 미분을 나타낸다.in Equation 6

represents the first frame,

denotes the first depth map and ∇ denotes the directional differential in space.

According to another embodiment, since the computing device acquires an image captured using a monocular camera, when using an image obtained using a monocular camera, errors may accumulate in the artificial neural network due to scale-ambiguity. can The computing device may calculate a third auxiliary function that minimizes a difference between two values by normalizing a difference between depth information values of consecutive images to compensate for a loss due to scale ambiguity. The computing device may be capable of estimating the camera posture by uniformly adjusting the scale of depth information between frames acquired through the third auxiliary function. Specifically, the third auxiliary function may be calculated based on Equation 7.

에서의

의 영상평면으로 성공적으로 투영된 유효한 점을 나타내고,

는 점의 개수를 나타내고,

는 수학식 3을 기초로 하는 제1 깊이맵과 제2 깊이맵의 차이를 나타내는 함수이다.In Equation 7, V is

in

represents a valid point successfully projected onto the image plane of

represents the number of points,

Is a function representing a difference between the first depth map and the second depth map based on Equation 3.

According to another embodiment, the computing device may obtain an image captured by an object moving at the same speed as the camera or by a stationary camera. In this case, the computing device may calculate the third auxiliary function only when the difference between the first frame and the third frame is smaller than the difference between the first frame and the second frame. Specifically, the third auxiliary function may be calculated based on Equation 8.

에서

의 영상평면으로 성공적으로 투영된 유효한 점을 나타내고,

는 점의 개수를 나태내고

는 수학식 3을 기초로 하는 제1 깊이맵과 제2 깊이맵의 차이를 나타내는 함수이다. 수학식 8에서

으로,

은 제1 프레임이 제2 깊이맵과 변환행렬을 기반으로 재구성된 영상이고

는 아이버슨 괄호(Iverson bracket)를 나타낸다. 즉,

는 이진 마스크로 작용하여 컴퓨팅 장치가 제1 보조함수를 소정 조건을 만족하는 유효 픽셀에 대하여 계산하도록 한다. 소정 조건은 상기 제1 프레임과 제3 프레임에서의 차이가 제1 프레임과 제2 프레임에서의 차이보다 적은 픽셀이다.In Equation 8, V is

at

represents a valid point successfully projected onto the image plane of

represents the number of points

Is a function representing a difference between the first depth map and the second depth map based on Equation 3. in Equation 8

by,

Is an image in which the first frame is reconstructed based on the second depth map and the transformation matrix.

denotes Iverson brackets. in other words,

acts as a binary mask to allow the computing device to calculate the first auxiliary function for effective pixels that satisfy a predetermined condition. A predetermined condition is that a difference between the first frame and the third frame is smaller than a difference between the first frame and the second frame.

According to an embodiment, the loss function may be expressed as the sum of the first to third auxiliary functions. The computing device may calculate the loss function based on Equation 9.

는 수학식 5를 기초로 하는 제1 보조함수,

는 수학식 6을 기초로 하는 제2 보조함수,

는 수학식 8을 기초로 하는 제3 보조함수이고, α,β,γ는 제1 내지 제3 보조함수의 비율을 설정하는 하이퍼 파라미터이다.in Equation 9

Is a first auxiliary function based on Equation 5,

Is a second auxiliary function based on Equation 6,

Is a third auxiliary function based on Equation 8, and α, β, and γ are hyperparameters for setting ratios of the first to third auxiliary functions.

According to one embodiment, the computing device may generate 6 tokens corresponding to 6 degrees of freedom and set them to a predetermined initial value. After the six tokens corresponding to the six degrees of freedom are set to predetermined initial values, the computing device may update the output value of the loss function to decrease as learning progresses.

3 is a diagram for explaining the overall network structure of the 6 degree of freedom posture estimation method.

According to an embodiment, referring to FIG. 3 , the computing device may generate a first depth map 311 by obtaining a first frame 310 and inputting the first frame 310 to a depth estimation network 330 . Also, the computing device may generate the second depth map 322 by acquiring the second frame 320 and inputting the second frame 320 to the depth estimation network 330 .

According to an embodiment, the depth estimation network may be based on a ResNet model to which a Convolutional Neural Network (CNN) is applied. Specifically, the depth estimation encoder part based on the U-net structure may be composed of a typical convolutional neural network. Illustratively, the encoder portion of the depth estimation network may be based on the Resnet50 model. The decoder part of the depth estimation network may consist of upsampling and convolutional neural networks. Illustratively, the decoder portion of the depth estimation network may be based on the DispResnet model.

Referring to FIG. 3 , the computing device may determine six degrees of freedom by combining the first frame and the second frame and inputting the input to the artificial neural network 350 . The artificial neural network 350 may be composed of a feature extraction network and a dimensionality reduction network. The feature extraction network may be based on a ResNet model to which a Convolutional Neural Network (CNN) is applied. The dimension reduction network may be based on a dimension reduction vision transformer network.

According to an embodiment, the computing device may learn using the depth maps 311 and 321 of the first frame and the second frame. Through learning, the computing device may obtain a loss function, and may update six tokens corresponding to six degrees of freedom and a position embedding vector so that an output value of the loss function becomes small. The computing device may determine 6 degrees of freedom by inputting the image 340 synthesized with the first frame and the second frame to the artificial neural network 350 using the updated 6 tokens and the position embedding vector.

4 is a diagram for explaining a hybrid structure composed of a feature extraction network and a dimensionality reduction network according to an embodiment.

Referring to FIG. 4A , the artificial neural network may have a hybrid structure composed of a feature extraction network 420 and a dimensionality reduction network 430 . The computing device may synthesize the first frame and the second frame, thereby obtaining a 6-channel image 410 obtained by combining the 3-channel image of the first image and the 3-channel image of the second image in the channel direction. The computing device may input the obtained 6-channel image 410 to the feature extraction network 420 . The feature extraction network may be based on a ResNet model to which a Convolutional Neural Network (CNN) is applied.

According to an embodiment, the computing device may apply a channel-specific filter to the 6-channel image 410 using the feature extraction network 420 to obtain a convolutional result value, and use the obtained result value to obtain a feature map. can create The computing device may reconstruct the generated feature map into a set of patches 422 having a constant size, and generate 6 tokens 423 having the same size as the patches. The six tokens 423 may be learnable parameters corresponding to each of the six degrees of freedom in order to obtain the six degrees of freedom. The computing device may create a combined set by combining the generated 6 tokens 423 and patches. The computing device may generate an input vector by adding the location embedding vector including the location information of the patches 422 and the elements of the combination set.

Referring to FIG. 4A , a computing device may input an input vector to a dimensionality reduction network 430 . The dimension reduction network may include a dimension reduction vision transformer network. The dimension reduction network 430 is composed of Layer Norm, Multihead Self-Attention, Multilayer Perceptron (MLP), and Dim-reduction MLP, and may have a configuration in which the dimension is gradually reduced by repeating the self-attention operation by the number of L layers. The computing device may extract patches 440 output according to the input of 6 tokens among the patches of the input vector whose size has been reduced by the dimensionality reduction network 430, and patch using the extracted patches 440. Six degrees of freedom can be estimated by calculating the per-average pooling.

4B is a diagram for explaining a method of generating an input vector 426 using a feature map 421 according to an embodiment.

According to an embodiment, the computing device may generate a feature map 421 of the 6-channel image 410 using the feature extraction network 420 . Specifically, the computing device may apply a filter according to a channel to a 6-channel image obtained by combining 3-channel RGB images to obtain a convolutional result value, and may generate a feature map 421 using the obtained result value. The computing device may divide the feature map 421 into non-overlapping patches 422 of a certain size and reconstruct the 3D feature map 421 into a set of patches 422 in order to perform an attention operation for each patch. .

Referring to FIG. 4B , the computing device may generate six tokens 423 having the same size as the patches generated using feature maps of the first and second images. The generated 6 tokens 423 may be learnable parameters corresponding to each of the 6 degrees of freedom in order to obtain the 6 degrees of freedom. The computing device may generate a combination set 424 by combining the generated six tokens 423 and patches.

Referring to FIG. 4B , patches reconstructed as a set of patches 422 generated by dividing a feature map 421 extracted from a convolutional neural network may be in a state in which location information is lost. Therefore, in order to consider the location information, the computing device may generate a location embedding vector 425 reflecting locational characteristics of the patches 422 included in the feature map 421 . The computing device may generate the input vector 426 by adding the generated position embedding vector 425 and elements of the combination set 424 . The computing device may input the input vectors into dimensionality reduction network 430 .

According to an embodiment, when the number of patches included in the feature map 421 is n, the combination set 424 may include n+6 patches.

According to an embodiment, the computing device may extract patches 440 corresponding to the six tokens 423 from among patches of the input vector whose size is reduced by the dimensionality reduction network 430 . The computing device may estimate 6 degrees of freedom by calculating average pooling for each patch using the extracted patches 440 .

5 is an exemplary diagram of learning data for learning an artificial neural network for 6-DOF posture estimation according to an embodiment.

According to an embodiment, the computing device may obtain learning data by using a total of four cameras (two black and white cameras and two color cameras), a rotating laser scanner, and a GPS/IMU device in the vehicle. As disclosed in FIG. 5 , learning data may be images captured in various environments such as a residential area, a campus, and a road in Karlsruhe, Germany.

According to an embodiment, the computing device may perform learning using images that have undergone preprocessing such as segmentation labels, object detection labels, and object tracking labels. Illustratively, the training image may be an image to which preprocessing is performed at a resolution of 832*256 and random resizing, cropping, flipping, etc. are applied using a data augmentation technique (Augmentation). Specifically, the computing device uses the ADAMW optimization technique, the learning rate is 10 ^-4 , the hyperparameters of the optical loss function are λ _i =0.15,λ _s =0.85 , the hyperparameters of the total loss function α=1.0,β = 0.1, γ = 0.5 to proceed with learning.

6 is an exemplary view showing qualitative test results for a method for estimating a 6 degree of freedom posture according to an embodiment.

Referring to FIG. 6 , the computing device may perform two experiments with different amounts of learning data. The first experiment (K) is an experiment using a total of 8,146 sheets of data, and the second experiment (CS+K) is an experiment using a total of 65,688 sheets of data. The computing device is an artificial neural network in the second experiment (CS+K) using more data. It can be seen that the performance of

As can be seen from the test results in Table 1, when using the artificial neural network according to one embodiment (lines 3 and 4), the translation error (translation error, t _err ) is higher than when only the existing synthetic ball network is used (lines 1 and 2). , Rotation Error (r _err ), and Absolute Trajectory Error (ATE) were all lower.

Method Seq.09 Seq. 10 t _err r _err ATE t _err r _err ATE Prior art (K) 12.571 3.339 56.832 10.054 4.911 20.706 Prior Art (CS+K) 7.605 2.188 15.083 10.070 4.626 20.343 Ours(K) 7.323 2.087 31.040 8.102 3.667 15.325 Ours (CS+K) 4.363 1.322 13.932 6.608 2.986 12.356

7 is a block diagram illustrating a device for estimating a 6 degree of freedom posture according to an exemplary embodiment.

Referring to FIG. 7 , a computing device 700 according to an embodiment includes a processor 720 . The error injection attack device 700 may further include a memory 730 and a communication unit 710. The processor 720, the memory 730, and the communication unit 710 may communicate with each other through a communication bus (not shown).

The processor 720 may control a series of operations of the computing device 700 described above. More specifically, the processor 720 inputs the first image and the second image to the artificial neural network, generates a feature map from the first image and the second image using a feature extraction network of the artificial neural network, and A combination set is generated using 6 tokens corresponding to degrees of freedom and the feature map, a location embedding vector reflecting positional characteristics of patches included in the feature map is generated, and the location embedding vector and the combination set are generated. It is possible to generate an input vector, input the input vector to the dimensionality reduction network of the artificial neural network, and determine the six degrees of freedom from the output of the dimensionality reduction network.

Memory 730 may be volatile memory or non-volatile memory.

In addition, the processor 720 may execute a program and control the computing device 700 . Program codes executed by the processor 720 may be stored in the memory 730 . The computing device 700 may be connected to an external device (eg, a personal computer or network) through an input/output device (not shown) and exchange data. The error injection attack device 700 may be mounted on a server.

Based on the description of the above embodiments, a person skilled in the art can understand that the methods and/or processes of the present invention, and the steps thereof, can be realized with hardware, software, or any combination of hardware and software suitable for a particular application. point can be clearly understood. Furthermore, the objects of the technical solution of the present invention or parts contributing to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-readable recording medium. The machine-readable recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the machine-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROM, DVD, and Blu-ray, and magneto-optical media such as floptical disks. (magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include stored and compiled or interpreted for execution on any one of the foregoing devices, as well as a heterogeneous combination of processors, processor architectures, or different combinations of hardware and software, or any other machine capable of executing program instructions. Machine code, This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter or the like.

Therefore, in one aspect according to the present invention, when the above-described methods and combinations thereof are performed by one or more computing devices, the methods and combinations of methods may be implemented as executable code that performs each step. In another aspect, the method may be implemented as systems performing the steps, the methods may be distributed in several ways across devices or all functions may be integrated into one dedicated, stand-alone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, TPU coupled to a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, and external devices and signals It may include an input/output unit capable of sending and receiving. In addition, the hardware device may include a keyboard, mouse, and other external input devices for receiving commands written by developers.

In the above, the present invention has been described by specific details such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Various modifications and variations can be made from these descriptions by those of ordinary skill in the art to which the present invention belongs.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments and should not be determined, and not only the claims to be described later, but also all modifications equivalent or equivalent to these claims fall within the scope of the spirit of the present invention. will do it

Such equivalent or equivalent modifications will include, for example, logically equivalent methods that can produce the same results as those performed by the method according to the present invention, the spirit and scope of the present invention. should not be limited by the above examples, and should be understood in the broadest sense permitted by law.

Claims (16)

In the six degree of freedom posture estimation method, performed by a computing device,
obtaining a first image of a first frame and a second image of a second frame;
inputting the first and second images to an artificial neural network;
generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network;
generating a combination set using 6 tokens corresponding to the 6 degrees of freedom and the feature map;
generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combination set;
inputting the input vector to a dimensionality reduction network of the artificial neural network; and
6 degrees of freedom posture estimation method comprising determining the 6 degrees of freedom from the output of the dimensionality reduction network.
According to claim 1,
The feature extraction network,
A six-degree-of-freedom posture estimation method for extracting features using a six-channel image obtained by combining the three-channel image of the first image and the three-channel image of the second image in a channel direction.
According to claim 1,
Generating a combination set using the six tokens and the feature map,
reconstructing the generated feature map into a set of patches having a constant size; and
combining the set of patches and the six tokens to generate the combined set;
Each of the six tokens has the same size as each of the patches.
According to claim 3,
The combination set,
A method for estimating a 6 degree of freedom pose comprising n+6 patches, wherein n is the number of patches included in the feature map.
According to claim 1,
The position embedding vector is a vector having the same size as the patches,
The input vector is,
A method for estimating a six-degree-of-freedom pose determined based on the sum of element vectors and position embedding vectors constituting the combination set.
According to claim 1,
The dimensionality reduction network,
A method for estimating a six-degree-of-freedom posture in which a size of patches constituting the input vector is reduced by repeatedly performing a self-attention operation on the input vector.
According to claim 1,
In the step of determining the six degrees of freedom,
extracting patches corresponding to the six tokens from among patches of an input vector whose size is reduced by the dimensionality reduction network; and
and estimating 6 degrees of freedom by calculating average pooling for each patch using the extracted patches.
In the learning method of an artificial neural network for estimating a six-degree-of-freedom posture, performed by a computing device,
obtaining a first image of a first frame and a second image of a second frame;
estimating a first depth map of the first image and a second depth map of the second image;
outputting six degree of freedom information by inputting the first image and the second image to the artificial neural network;
calculating a transformation matrix between the first frame and the second frame based on the six degree of freedom information; and
Calculating an output value of a loss function based on the first depth map, the second depth map, and the transformation matrix, and updating the artificial neural network based on the output value of the loss function,
In the step of outputting the six degree of freedom information,
inputting the first and second images to an artificial neural network;
generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network;
generating a combination set using 6 tokens corresponding to the 6 degrees of freedom and the feature map;
generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combination set;
inputting the input vector to a dimensionality reduction network of the artificial neural network; and
An artificial neural network learning method comprising the step of determining the six degrees of freedom from the output of the dimensionality reduction network.
According to claim 8,
The 6 tokens and the location embedding vector,
An artificial neural network learning method in which an output value of the loss function is updated to become smaller after being set to predetermined initial values.
According to claim 8,
The output of the loss function is a first auxiliary function for adding all the differences between the first frame and the third frame obtained by reconstructing the second frame using the transformation matrix and the first depth map for effective pixels satisfying a predetermined condition. determined based on the output;
The first auxiliary function,
Further comprising a term indicating a structural similarity index between the first frame and the third frame with respect to the effective pixel;
A third depth map and a fourth depth map for a third depth map obtained by reconstructing the first depth map with the transformation matrix and a fourth depth map reconstructed such that the second depth map is positioned on the same pixel grid as the first depth map. A function calculated by multiplying the normalization function divided by the sum of the third depth map and the fourth depth map,
The structural similarity index is based on comparison of luminance, contrast, and structure between pixels,
The predetermined condition is that the difference between the first frame and the third frame is less pixels than the difference between the first frame and the second frame.
According to claim 10,
The output of the loss function is determined by further considering the output of a second auxiliary function that adds all products of the spatial gradient component of the first frame and the gradient component of the first depth map for all pixels in space.
According to claim 11,
The output of the loss function is determined by further considering the output of a third auxiliary function for adding the normalization function to all of the effective pixels.
According to claim 8,
Generating a combination set using the 6 tokens corresponding to the 6 degrees of freedom and the feature map,
dividing the generated feature map into non-overlapping patches of a constant size;
generating a patch set composed of the patches;
generating 6 tokens having the same size as the patches and corresponding to the 6 degrees of freedom; and
combining the patch set and the six tokens to generate the combined set;
The combination set,
It includes n+6 patches, wherein n is the number of patches included in the feature map.
According to claim 8,
In the step of determining the six degrees of freedom,
extracting patches corresponding to the six tokens from among patches of an input vector whose size is reduced by the dimensionality reduction network; and
An artificial neural network learning method comprising the step of estimating 6 degrees of freedom by calculating average pooling for each patch using the extracted patches.
In a computing device,
contains a processor;
obtaining, by the processor, a first image of a first frame and a second image of a second frame; inputting the first and second images to an artificial neural network; generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network; generating a combination set using 6 tokens corresponding to the 6 degrees of freedom and the feature map; generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combination set; inputting the input vector to a dimensionality reduction network of the artificial neural network; and determining the six degrees of freedom from the output of the dimensionality reduction network.
In a computing device,
contains a processor;
obtaining, by the processor, a first image of a first frame and a second image of a second frame; estimating a first depth map of the first image and a second depth map of the second image; outputting six degree of freedom information by inputting the first image and the second image to the artificial neural network; calculating a transformation matrix between the first frame and the second frame based on the six degree of freedom information; and calculating an output value of a loss function based on the first depth map, the second depth map, and the transformation matrix, and updating the artificial neural network based on the output value of the loss function,
In the step of outputting the six degree of freedom information,
inputting the first and second images to an artificial neural network;
generating a feature map from the first image and the second image using a feature extraction network of the artificial neural network;
generating a combination set using 6 tokens corresponding to the 6 degrees of freedom and the feature map;
generating a positional embedding vector reflecting positional characteristics of the patches included in the feature map, and generating an input vector using the positional embedding vector and the combination set;
inputting the input vector to a dimensionality reduction network of the artificial neural network; and
and determining the six degrees of freedom from the output of the dimensionality reduction network.

Images