KR102620475B1 - Method and apparatus for performing a re-topology using deep learning algorithms based on images - Google Patents

Abstract

The present invention relates to a method and device for performing retopology by squareling a plurality of subpolygons in an image based on a topology prediction result using a deep learning algorithm based on the image. Specifically, the present invention uses partial images randomly cut from a plurality of training images when a deep learning model learns on training images in which topology prediction was performed with a plurality of subpolygons based on edges and vertices. A configuration that learns the prediction of, removes the decoder from the prediction model of the learned topology, and connects a small number of randomly initialized layers behind the convolutional layers in the encoder to create a retopology prediction model based on the prediction of the FINE TUNING topology. , A configuration that performs learning of a topology prediction-based retopology prediction model based on training images for which retopology has been completed with the topology prediction results recorded, and images are generated using a retopology prediction model based on the learned topology prediction. A method including a configuration for performing topology prediction-based retopology prediction based on partial images received as input and cut out according to a set order, an electronic device configured to perform the method, and a computer program configured to perform the method.

Description

Method and device for performing retopology using deep learning algorithm based on images {METHOD AND APPARATUS FOR PERFORMING A RE-TOPOLOGY USING DEEP LEARNING ALGORITHMS BASED ON IMAGES}

The present invention relates to a method and device for performing re-topology based on an image by rectangulating a plurality of subpolygons in the image. Specifically, the present invention relates to a method and device for performing re-topology by squareling a plurality of subpolygons in an image based on a topology prediction result using a deep learning algorithm based on the image. .

Re-topology is an important step in 3D (3-dimensional) modeling and refers to the process of reconstructing an existing complex mesh structure into a simpler and more efficient structure. This process helps improve the quality of 3D models and optimize performance in animation, rendering, game engines, and more.

To convert an image into a 3D video, you first need to convert the image into a 3D model. The 3D models created from this process generally have a high level of detail, but their structures can be complex and inefficient. These models are difficult to use for rendering or animation. To solve this problem, a process of re-topology is necessary.

However, retopology is a very time-consuming task. In particular, the task of manually converting each sub-polygon of the model into a square is very complicated and difficult. For this reason, deep learning algorithms are needed to automate this process.

Deep learning algorithms can predict existing 3D models and learn how to convert each subpolygon in the image into an efficient square structure. The deep learning algorithm learned in this way can automatically perform the retopology process when converting a new image into a 3D model. This can greatly simplify 3D modeling tasks and save time and effort.

To improve this problem, a technology for learning retopology from an image to a quad mesh using a deep learning algorithm has been previously developed. However, this technology consists of convolution operations based on CNN (Convolutional Neural Network), U-net, etc. Convolution is good at extracting local features, but it is difficult to extract overall features, so when an image with a field of view (FOV) that is too large is input, the performance of feature extraction and segmentation is poor.

In order to learn a deep learning model that generates topology prediction-based retopology information for an image, a number of training images for which topology prediction-based retopology has been completed must be secured for each image. However, there is a problem that it is not realistically easy to secure a training image in which retopology has been completed while the topology prediction result remains recorded for the image. After training the deep learning model on the topology prediction results made up of multiple subpolygons in the image, the resulting value is set as the initial value for the layer that predicts topology for multiple subpolygons in the image, and the identified topology If you add a layer that performs retopology based on the prediction results and then perform learning with training images containing only the retopology results without any records of topology analysis, you can use the deep learning model as a topology prediction-based retopology. Sufficient data can be obtained to train to perform. It is possible to secure a large number of training images in which topology prediction was performed with multiple subpolygons based on the edges and vertices of each image, and it is also possible to secure a large number of training images for which retopology has been completed without any existing topology prediction results remaining. Because.

Publication Patent No. 10-2023-0047042 (Retopology method and device)

Based on the above-described discussion, various embodiments of the present invention use a deep learning algorithm based on the image to square a plurality of subpolygons in the image according to the topology prediction result, thereby achieving retopology. Provides a method and device for performing.

In addition, various embodiments of the present invention train a deep learning model to predict topology for a plurality of subpolygons in an image, and use the result as an initial value for a layer that predicts topology for a plurality of subpolygons in the image. Then, a deep learning model is created by adding a layer that performs retopology based on the prediction results of the identified topology, and then learning with training images containing only the retopology results without any records of topology analysis. Provides a method and device for performing topology prediction-based retopology.

In addition, various embodiments of the present invention learn topology prediction using training images in which topology prediction is performed with a plurality of subpolygons based on the edges and vertices of each image for a deep learning model, and record the topology analysis. Provides a method and apparatus for performing topology prediction-based retopology prediction using a deep learning model by learning retopology prediction on a quad mesh using only training images containing retopology results.

In addition, various embodiments of the present invention use partial images randomly cut from a plurality of training images when a deep learning model learns on training images whose topology has been predicted with a plurality of subpolygons based on edges and vertices. A configuration that learns the prediction of the topology, removes the decoder from the prediction model of the learned topology, and connects a small number of randomly initialized layers behind the convolutional layers in the encoder to create a retopology prediction model based on the prediction of the fine-tuned topology. A configuration that performs learning of a topology prediction-based retopology prediction model based on training images in which the topology prediction result has been completed without recording the topology prediction result, and the final retopology to a quad mesh has been completed. A method comprising receiving an image using a topology prediction-based retopology prediction model and performing topology prediction-based retopology prediction based on the cut partial images according to a set order, and an electronic device configured to perform the method , providing a computer program configured to perform the method.

In addition, various embodiments of the present invention provide a method and apparatus for performing retopology prediction after accurate identification of the topology by learning a deep learning model with high performance in feature extraction even when an image with a field of view (FOV) that is too large is input. provides.

According to various embodiments of the present invention, in a method of operating an electronic device including an input device, a memory, a processor, and an output device, topology prediction and analysis are performed for a learning model stored in the memory based on a plurality of test images. performing topology learning by the processor; Receiving information about an input image through the input device; extracting edges and vertices from the input image by the processor; generating, by the processor, a prediction result of a topology consisting of a plurality of subpolygons in the input image using the learning model based on information about the corners and vertices of the input image; performing re-topology by the processor by squareling the plurality of subpolygons in the input image using the learning model; A method is provided including outputting, by the output device, a visualized wireframe for the input image based on a result of the retopology.

According to various embodiments of the present invention, in an electronic device, the electronic device includes an input device, a memory, a processor, and an output device, and performs a method of operating the electronic device according to various embodiments of the present invention. An electronic device configured to do so is provided.

According to various embodiments of the present invention, a computer program configured to perform a method of operating an electronic device according to various embodiments of the present invention, and recorded on a computer-readable storage medium, is provided.

Various embodiments of the present invention provide a method and device for performing re-topology by squareling a plurality of subpolygons in the image according to the topology prediction result using a deep learning algorithm based on the image. can do.

In addition, various embodiments of the present invention train a deep learning model to predict topology for a plurality of subpolygons in an image, and use the result as an initial value for a layer that predicts topology for a plurality of subpolygons in the image. Then, a layer that performs retopology is added based on the prediction results of the identified topology, and then learning is performed with training images that only contain the retopology results to the final quad mesh without the topology prediction results. By doing so, it is possible to provide a method and device for performing topology prediction-based retopology using a deep learning model.

In addition, various embodiments of the present invention learn topology prediction using training images in which topology prediction is performed with a plurality of subpolygons based on the edges and vertices of each image for a deep learning model, and record the topology analysis. By learning retopology prediction on a quad mesh with only training images containing retopology results, a method and apparatus for performing topology prediction-based retopology prediction using a deep learning model can be provided. .

In addition, various embodiments of the present invention use partial images randomly cut from a plurality of training images when a deep learning model learns on training images whose topology has been predicted with a plurality of subpolygons based on edges and vertices. A configuration that learns the prediction of the topology, removes the decoder from the prediction model of the learned topology, and connects a small number of randomly initialized layers behind the convolutional layers in the encoder to create a retopology prediction model based on the prediction of the fine-tuned topology. Configuration that generates, configuration that performs learning of a topology prediction-based retopology prediction model based on training images for which retopology has been completed with the topology prediction results recorded, using a retopology prediction model based on the learned topology prediction A method including a configuration for receiving an image as an input and performing topology prediction-based retopology prediction based on the cut partial images according to a set order, and an apparatus configured to perform the method can be provided.

In addition, various embodiments of the present invention provide a method and apparatus for performing retopology prediction after accurate identification of the topology by learning a deep learning model with high performance in feature extraction even when an image with a field of view (FOV) that is too large is input. can be provided.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 shows an overview of the operation of an electronic device according to various embodiments of the present invention.
Figure 2 shows the configuration of an electronic device according to various embodiments of the present invention.
Figure 3 illustrates a method of operating an electronic device according to various embodiments of the present invention.
Figure 4 shows the structure of a multi-layer perceptron (MLP) for implementing a deep learning algorithm according to various embodiments of the present invention.
Figure 5 illustrates a learning process for predicting a topology consisting of a plurality of subpolygons in an input image based on a deep learning algorithm according to various embodiments of the present invention.
Figure 6 shows a process for predicting a topology consisting of a plurality of subpolygons in an input image based on a deep learning algorithm according to various embodiments of the present invention.
Figure 7 shows a process for predicting a topology consisting of a plurality of subpolygons in an input image based on a deep learning algorithm according to various embodiments of the present invention.
Figure 8 shows topology prediction-based retopology prediction by adding a plurality of layers to the layers corresponding to the encoder of the model that has completed learning to identify the topology consisting of a plurality of subpolygons in the input image according to various embodiments of the present invention. The process of creating a model is shown.
FIG. 9 illustrates an example of a retopology prediction model based on topology prediction based on convolution neural networks (CNN) according to various embodiments of the present invention.
FIG. 10 shows an example of an image in which a topology composed of a plurality of subpolygons in an input image is analyzed and retopology based on topology analysis is performed according to various embodiments of the present invention.
11 to 13 illustrate an example of an output image of a wireframe visualized after performing topology prediction-based retopology according to various embodiments of the present invention.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. The present invention may be implemented in many different forms and is not limited to the embodiments described herein.

The present invention relates to a method and device for performing retopology by squareling a plurality of subpolygons in an image based on a topology prediction result using a deep learning algorithm based on the image. Specifically, the present invention uses partial images randomly cut from a plurality of training images when a deep learning model learns on training images in which topology prediction was performed with a plurality of subpolygons based on edges and vertices. A configuration that learns the prediction of, removes the decoder from the prediction model of the learned topology, and connects a small number of randomly initialized layers behind the convolutional layers in the encoder to create a retopology prediction model based on the prediction of the FINE TUNING topology. , A configuration that performs learning of a topology prediction-based retopology prediction model based on training images for which retopology has been completed with the topology prediction results recorded, and images are generated using a retopology prediction model based on the learned topology prediction. A method including a configuration for performing topology prediction-based retopology prediction based on partial images received as input and cut out according to a set order, an electronic device configured to perform the method, and a computer program configured to perform the method.

1 shows an overview of the operation of an electronic device according to various embodiments of the present invention.

Referring to FIG. 1, the electronic device 120 according to various embodiments of the present invention operates in a structure that receives an image 110 on which topology analysis has not been performed and outputs a retopology-based wireframe image 130.

In this specification, topology analysis and topology prediction can be used interchangeably. Additionally, in this specification, retopology analysis and retopology prediction can be used interchangeably.

Figure 2 shows the configuration of an electronic device according to various embodiments of the present invention.

Referring to FIG. 2 , the electronic device 120 includes a memory 121, a processor 122, an input device 123, and an output device 124.

The memory 121 is connected to the memory 121, the processor 122, the input device 123, and the output device 124, and can store information input through the input device 123. In addition, the memory 121 is connected to the processor 122 and can store data such as basic programs for operation of the processor 122, application programs, setting information, and information generated by operations of the processor 122. . The memory 121 may be comprised of volatile memory, non-volatile memory, or a combination of volatile memory and non-volatile memory. Additionally, the memory 121 may provide stored data according to the request of the processor 122.

The processor 122 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 122 controls overall operations of the electronic device 120. For example, the processor 122 writes and reads data into the memory 121. Additionally, the processor 122 receives information through the input device 123. Additionally, the processor 122 outputs information through the output device 140. Processor 122 may include at least one processor.

The input device 123 is connected to the processor 122 and can input information, etc. The input device 123 may include a touch display, keypad, keyboard, information input module, etc. According to one embodiment, the electronic device 120 may further include a transceiver, and the input device 123 may input information received from another device connected to a wired/wireless communication network through the transceiver.

The output device 124 is connected to the processor 122 and can output information in the form of video/audio, etc. The output device 124 may include a display, speaker, information output module, etc. According to one embodiment, the electronic device 120 may further include a transceiver, and the output device 124 may transmit and output information, etc. to another device connected to a wired/wireless communication network through the transceiver.

Although not shown in FIG. 2, according to one embodiment, the electronic device 120 may further include a transceiver. The transceiver is coupled to the processor 122 and transmits and/or receives signals. All or part of a transceiver may be referred to as a transmitter, receiver, or transceiver. The transceiver is a wired access system and a wireless access system, including the IEEE (institute of electrical and electronics engineers) 802.xx system, IEEE Wi-Fi system, 3rd generation partnership project (3GPP) system, 3GPP LTE (long term evolution) system, and 3GPP 5G. It can support at least one of various wireless communication standards such as NR (new radio) system, 3GPP2 system, and Bluetooth.

Figure 3 illustrates a method of operating an electronic device according to various embodiments of the present invention.

In the embodiment of Figure 3, the electronic device includes an input device, a memory, at least one processor, and an output device.

Referring to FIG. 3, in step S301, the electronic device performs topology prediction and retopology learning on the learning model stored in the memory by the processor based on a plurality of test images for which topology analysis has not been performed. do.

In step S302, the electronic device receives information about the input image through the input device.

In step S303, the electronic device generates, by the processor, a prediction result of a topology consisting of a plurality of subpolygons in the input image using the learning model from the input image.

In step S304, the electronic device uses the learning model to square the plurality of subpolygons in the input image and performs re-topology by the processor.

In step S305, the electronic device outputs a visualized wireframe for the input image through the output device based on the result of the retopology.

According to various embodiments of the present invention, step S301 of performing topology prediction and retopology learning for the learning model stored in the memory based on the plurality of test images by the processor, topology analysis Receiving a plurality of first test images that have not been performed by the input device; receiving, by the input device, a plurality of second test images in which a topology prediction consisting of a plurality of sub-polygons based on corners and vertices has been performed on the plurality of first test images; determining, by the processor, a plurality of first partial test images at a plurality of random locations within the first test image; determining, by the processor, the plurality of second partial test images at a plurality of locations corresponding to the plurality of first partial test images within the second test image; performing learning to predict a topology of a first learning model stored in the memory based on the plurality of first partial test images and the plurality of second test images by the processor; generating the second learning model by the processor by removing a decoder of the first learning model on which learning was performed and combining a plurality of predetermined layers with an encoder of the first learning model; A plurality of third test images in which topology prediction and retopology were not performed, and a plurality of fourth test images in which retopology was performed by squarening a plurality of subpolygons in the plurality of test images. receiving input by the input device; It may include performing, by the processor, prediction of a topology and learning of a retopology of the second learning model based on the plurality of third test images and the plurality of fourth test images. The learning model may correspond to the second learning model.

According to various embodiments of the present invention, the second learning model divides the input image into a plurality of partial images and then performs topology prediction using the plurality of sub-polygons. It can be configured to generate a retopology image on which retopology has been performed by rectangulating based on the .

According to various embodiments of the present invention, the encoder of the first learning model is composed of N first parameters, the plurality of layers are composed of M second parameters, and N and M are integers greater than 0. You can. The second learning model may be composed of N first parameters and M second parameters. The N first parameters may have values according to the results of training of the second learning model as initial values. The M second parameters may have random values close to 0 as initial values.

According to various embodiments of the present invention, N, the number of the first parameters, may be 9 times or more than M, the number of the second parameters.

According to various embodiments of the present invention, the plurality of first 3D partial test images and the plurality of second partial test images may have the same size and structure. The number of the plurality of first partial test images and the plurality of second partial test images may correspond to a random number greater than or equal to a set number. The plurality of first test images and the plurality of second test images may have the same size and structure.

According to various embodiments of the present invention, neighboring second partial test images among the plurality of second partial test images may overlap each other or a gap may exist between them.

At this time, neighboring second partial test images among the second partial test images overlap each other or have a gap for the following reasons.

At this time, the overlap or gap may be expressed intentionally or may be due to an error.

When overlaps and gaps occur due to errors:

i) Modeling errors: Mistakes by software users may create gaps or overlaps between meshes, which may be due to incomplete merging, incorrect scaling, inaccurate rotation, etc.

ii) Data loss or errors: File conversion, corruption, or data loss can cause mesh problems.

iii) Complex geometry: In the modeling process of complex shapes, gaps or overlaps may occur due to the limitations of automatic generation tools.

iv) Scan errors: 3D scan data can often be incomplete or inaccurate, which can lead to overlaps or gaps.

When intentionally creating overlaps and gaps:

i) Complex object: In the case of a complex object consisting of several independent parts, there may be gaps between each part. For example, there may be a gap between the robot's arms and body.

ii) Matrix/Arrangement: When the same mesh is copied and arranged multiple times during the modeling process, spatial gaps may exist between these copies.

iii) Temporary overlap: Specific parts of the model being worked on can be temporarily overlapped. This is mainly to visualize how the two parts should be connected, and there is no such overlap in the finished model.

iv) Game development: In game development, meshes are often overlapped. For example, when a character wears clothes, the character's body and clothes actually overlap. This is to allow the two meshes to move and interact independently of each other, which is a normal overlap.

In this way, cases where overlaps and gaps are generated due to errors or overlaps and gaps are intentionally created are distinguished, and the damaged portion of the damaged mesh model where errors are overlapped or gaps are created is restored, gaps between meshes are filled, etc. work becomes possible.

According to various embodiments of the present invention, performing training of the first learning model stored in the memory based on the plurality of first partial test images and the plurality of second partial test images includes: The first learning method is based on one first partial test image among the first partial test images and one second partial test image corresponding to the one first partial test image among the plurality of second partial test images. The step of performing model learning may include a step of being performed repeatedly. Whenever training of the first learning model is performed, the first learning model is based on a different first partial test image and a different second partial test image among the plurality of first partial test images. Learning can be performed.

According to various embodiments of the present invention, in an electronic device, the electronic device includes an input device, a memory, a processor, and an output device, and the processor is configured to perform the method of operating the electronic device according to the embodiment of FIG. 3. A terminal is provided.

According to various embodiments of the present invention, a computer program configured to perform the method of operating an electronic device according to the embodiment of FIG. 3 and recorded on a computer-readable storage medium is provided.

Figure 4 shows the structure of a multi-layer perceptron (MLP) for implementing a deep learning algorithm according to various embodiments of the present invention.

The first learning model and the second learning model according to various embodiments of the present invention may be configured with an MLP structure. The first learning model and the second learning model can be implemented in various architectures such as CNN (Convolutional Neural Network), DenseNet, U-net, GoogLeNet, and Generative adversarial network. For example, when the first learning model and the second learning model are implemented as CNNs, the first learning model and the second learning model perform a learning process of adjusting the connection weights of the artificial neural network using learning data.

Deep learning is one of the technologies that has recently emerged in the field of machine learning. It is a neural network composed of multiple hidden layers and multiple hidden units included in them. When basic features (low level features) are input to a deep learning model, these basic features pass through multiple hidden layers and are transformed into high level features that can better explain the problem to be predicted. Because this process does not require an expert's prior knowledge or intuition, subjective factors in feature extraction can be eliminated, and a model with higher generalization ability can be developed. Furthermore, deep learning has the advantage of being able to form the final model through a simpler process compared to existing machine learning theories because feature extraction and model construction are comprised of one set.

A multi-layer perceptron (MLP) is a type of artificial neural network (ANN) with multiple nodes based on deep learning. Each node uses a non-linear activation function with neurons similar to the connection patterns of animals. This non-linear property allows inseparable data to be linearly separated.

Referring to FIG. 4, the artificial neural network 400 of the MLP model according to various embodiments of the present invention includes an input layer 410, a plurality of hidden layers 430, and an output layer. )(450).

Input data, such as a plurality of factors related to an image, are input to the nodes of the input layer 410. Here, a plurality of factors 411 related to the image correspond to basic features (low level features) of the deep learning model.

At the nodes of the hidden layer 430, calculations are made based on the input parameters. The hidden layer 430 is a layer in which units defined by a plurality of nodes formed by combining a plurality of factors 411 related to the image are stored. The hidden layer 430 may be composed of a plurality of hidden layers as shown in FIG. 4.

For example, when the hidden layer 430 is composed of a first hidden layer 431 and a second hidden layer 433, the first hidden layer 431 contains a plurality of factors related to the image, which is the lowest feature ( It is a layer in which first units 432 defined as a plurality of nodes formed by combining 411 are stored, and the first unit 432 corresponds to the upper characteristics of a plurality of factors related to the image. The second hidden layer 433 is a layer in which second units 434, defined as a plurality of nodes formed by combining the first units of the first hidden layer 431, are stored, and the second unit 434 is the 1 Corresponds to the upper feature of unit 432.

The nodes of the output layer 450 display the calculated prediction results. The output layer 450 may be provided with a plurality of prediction result units 451. Specifically, the plurality of prediction result units 451 may be composed of two units: a True unit and a False unit. Specifically, a true unit is a prediction result unit that means that the image is related to a specific feature, and a false unit is a prediction result unit that means that the image is not related to a specific feature.

Each weight is assigned to the connection between the second units 434 included in the second hidden layer 433, which is the last layer of the hidden layer 430, and the prediction result units 451. Based on these weights, it is predicted whether a particular feature of the image is related or not.

For example, if any one of the second units 434 predicts that the image will be related to a specific feature, it is connected to the true unit and the false unit, respectively. A weight having a positive value for the connection with the true unit will be assigned, and a negative weight will be assigned to connections with false units. Conversely, if any one of the second units 434 predicts that the image is not related to a specific feature, it is connected to the true unit and the false unit, respectively, and the connection with the true unit has a negative weight. A positive weight will be assigned to connections with false units.

A plurality of connection lines will be formed between the plurality of second units 434 and the true unit. If the total sum of the plurality of connection lines has a positive value, the plurality of factors 411 related to the image in the input layer 410 will be predicted as factors related to the image with a specific feature. According to one embodiment, whether such an image is related to a specific feature may be predicted by comparing the total sum of a plurality of connection lines with a preset value.

The artificial neural network 400 of the MLP model learns by adjusting learning parameters. According to one embodiment, the learning parameters include at least one of weight and variance. Learning parameters are iteratively adjusted through an optimization algorithm called gradient descent. Each time a prediction result is calculated from a given data sample (forward propagation), the network's performance is evaluated through a loss function that measures the prediction error. Each learning parameter of the artificial neural network 400 is adjusted by increasing it slightly in the direction of minimizing the value of the loss function, and this process is called back-propagation.

Figure 5 illustrates a learning process for predicting a topology consisting of a plurality of subpolygons in an input image based on a deep learning algorithm according to various embodiments of the present invention.

Figure 5 is shown as an example of U-net, but the first learning model and the second learning model according to various embodiments of the present invention include various models such as CNN (Convolutional Neural Network), DenseNet, GoogLeNet, and Generative adversarial network in addition to U-net. It can be implemented as an architecture.

Referring to Figure 5, the upper part shows the process of finding the parameters of the U-net using learning data, and the lower part displays the topology prediction results in a plurality of subpolygons using the parameters of the U-net learned through learning. indicates the process provided.

The first learning model can be applied whether the input image is a 2-dimensional image or a 3-dimensional image. The operating principle is to divide the input image into partial images of the same size, perform topology prediction using multiple subpolygons, and generate one overall output image from each partial image represented by multiple subpolygons. It can be equally applied to video or 3D video.

Referring to the upper part of FIG. 5, U-net parameters, for example, N parameters, w ₁ , ..., w _N can be determined through learning using data with correct answers. During learning, U-net parameters are determined so that cropped image patches are input at random positions and 3D segmentation patches are output at the same positions. At this time, random cropping is performed many times (at least several hundred times) to minimize missed areas in space during learning. If the crop size is 10x10x10, the U-net obtained after learning is completed has a structure that receives a 10x10x10 image as input and outputs 10x10x10 segmentation.

Referring to the lower part of FIG. 5, when providing the actual prediction result, an image without topology analysis into multiple subpolygons, for example, an image of size 200x200x100, is regularly divided into several partial images of size 10x10x10 (total 20x20x10 = 4000), a 10x10x10 segmentation patch can be obtained by inputting each 10x10x10-sized partial image into the already learned U-net model, and a total of 4000 segmentation patches obtained repeatedly in this way are concatenated to create the same size as the original image. You can obtain segmentation results with a size of 200x200x100.

U-net is composed of an encoder and a decoder. The operation called convolution, which is the core of the encoder (i.e. left wing) of U-net, is good at extracting local features, but is difficult to extract global features. There is a problem. If the test image is learned at once without cropping as shown in Figure 5, for example, if a 200x200 original image is input to the U-net for learning without cropping, the U-net learned in this way When providing actual prediction results, when an image with a too large FOV (field of view) is input, the performance of performing segmentation may deteriorate.

Figure 6 shows a process for predicting a topology consisting of a plurality of subpolygons in an input image based on a deep learning algorithm according to various embodiments of the present invention.

Specifically, FIG. 6 illustrates an operation process of a first learning model for identifying a topology from an image into a plurality of subpolygons in various embodiments of the present invention.

The first learning model can be trained using learning data that identifies the topology of various types of images with a plurality of subpolygons. For example, when trying to create an artificial intelligence model that divides an image into a plurality of subpolygons, the first learning model can be trained using training data in which each part of the training image is divided into a plurality of subpolygons. .

When the first learning model that has completed training receives an image for which topology analysis has not been performed, it outputs a result of identifying the topology of the image with a plurality of subpolygons.

Figure 7 shows a process for predicting a topology consisting of a plurality of subpolygons in an input image based on a deep learning algorithm according to various embodiments of the present invention.

Specifically, FIG. 7 shows the structure of a first learning model for identifying a topology composed of a plurality of subpolygons from an image in various embodiments of the present invention.

Figure 7 exemplarily shows the structure of a learning process for identifying a topology composed of a plurality of subpolygons from an image using a vision transformer.

In Figure 7, H, W, and D represent the horizontal, vertical, and depth sizes of the input image.

In FIG. 7, Z3, Z6, Z9, and Z12 refer to the output tensor for each layer of the convolution-based encoder. For example, Z3 means a 4-dimensional tensor with a size of H/16

In machine learning, a tensor is a mathematical entity that can be represented as a multidimensional array of numeric values. It is the basic data structure used to represent inputs, outputs, and intermediate computations in machine learning algorithms. Tensors can have a varying number of dimensions, called ranks. For example, a scalar (a single value) is a rank 0 tensor, a vector (an array of values) is a rank 1 tensor, and a matrix (2D array of values) is a rank 2 tensor. Tensors are used in various types of machine learning algorithms, including deep learning, convolutional neural networks, and recurrent neural networks. It is also used to represent data in many other scientific fields such as physics and engineering. A tensor is a multidimensional array of numeric values used to represent data in machine learning algorithms.

Referring to FIG. 7, the first learning model divides the input image into a specific set size, extracts features for each divided partial image, and performs topology prediction that divides the divided image into a plurality of subpolygons. Each divided area is displayed as a plurality of subpolygons, and the partial images on which topology prediction is displayed for each of the plurality of subpolygons are combined to generate and output one output image on which topology prediction has been performed.

The first learning model can be applied whether the input image is a 2-dimensional image or a 3-dimensional image. The operating principle is to divide the input image into partial images of the same size, perform topology prediction using multiple subpolygons, and generate one overall output image from each partial image represented by multiple subpolygons. It can be equally applied to video or 3D video.

In the structure of the first learning model in Figure 7, the left side corresponds to the encoder and the right side corresponds to the decoder.

The encoder of the exemplary first learning model in FIG. 7 is shown as Vision Transformer (ViT), which corresponds to a transformer-based architecture. However, ViT is only an example, and various architectures may be applied to the encoder of the first learning model. For example, U-Net, which belongs to CNN, may be used as an encoder for the first learning model according to various embodiments of the present invention.

In an encoder-decoder architecture, the encoder is responsible for processing the input data and creating a compressed representation, which is then passed to the decoder to produce the output. The choice of encoder architecture depends on the nature of the input data and the task at hand.

Below are some examples of encoder architectures that can be used in an encoder-decoder architecture.

Convolutional neural networks (CNN): CNNs are commonly used as encoders in image and video processing tasks. It is designed to process input data with a grid-like structure, such as images, and can learn a hierarchical representation of the input data by successively applying convolution and pooling operations.

Recurrent neural networks (RNN): RNNs are typically used as encoders in natural language processing tasks where the input data is a sequence of words. RNNs can learn to process sequences of variable length and capture dependencies between different parts of the sequence.

Transformer-based architectures: Transformer-based architectures, such as the Transformer used in the original Transformer architecture for natural language processing, can also be used as encoders. These architectures can use self-attention mechanisms to process input data and capture long-range dependencies between different parts of the input.

Autoencoders: Autoencoders are a type of neural network that can be used in an encoder-decoder architecture where the encoder and decoder are trained together to reconstruct the input data. Autoencoders can learn compressed representations of input data, which can be useful for tasks such as image compression or anomaly detection.

Overall, the choice of encoder architecture depends on the nature of the input data and the task at hand; different encoder architectures may be better suited for different types of input data and tasks.

A schematic description of each encoder configuration in the example first learning model of Figure 7 is as follows. Vision Transformer (ViT) is a popular deep learning architecture for computer vision tasks that uses a self-attention mechanism to process input images. The following are the core components of ViT:

Embedded patches: The input image is segmented into non-overlapping patches and then linearly projected onto a low-dimensional feature space using a learnable linear projection layer. This transformation allows the network to operate on local features of the image while maintaining the global context.

Multi-head attention: Embedded patches are processed by a self-attention mechanism that learns to pay attention to important areas of the image. The attention mechanism is applied multiple times with different sets of learned weights (i.e. heads), which allows the network to capture different patterns and dependencies in the input.

Normalization: Layer normalization is applied to the output of each attention layer and MLP to stabilize the learning process and improve the generalization performance of the network.

Multi-layer perceptron (MLP): The output of the attention mechanism is passed through a multi-layer perceptron (MLP) that applies a non-linear transformation to each patch representation separately.

Overall, the ViT architecture uses these components to transform an input image into a series of patch representations, which are then processed by a series of attention and MLP layers to produce the final output.

Figure 8 shows topology prediction-based retopology prediction by adding a plurality of layers to the layers corresponding to the encoder of the model that has completed learning to identify the topology consisting of a plurality of subpolygons in the input image according to various embodiments of the present invention. The process of creating a model is shown.

Referring to FIG. 8, as shown in FIG. 7, the decoder is discarded and only the encoder is separated from the first learning model learned to identify the topology consisting of a plurality of subpolygons in the input image from the image, and a classification head (classification head) is placed behind the encoder. Multiple fully connected layers (FC layer, FC layer) called head, c-head are attached.

If the encoder separated from the first learning model is ViT in FIG. 8, the MLP and softmax at the bottom of FIG. 8 correspond to c-head.

Softmax is a mathematical function used in the classification head of a neural network to convert a real vector into a probability distribution. It is commonly used as the final activation function in the output layer of a neural network when the task is multi-class classification. The Softmax function takes a vector of real numbers as input and normalizes it so that the output becomes a probability distribution for possible classes. Specifically, the softmax function maps each input value to a value between 0 and 1 and then normalizes the values so that the sum is 1. The output of the softmax function is a probability distribution over the possible classes, and each element of the output vector represents the probability of the input belonging to that class. In the context of neural networks, the softmax function is typically applied to the output of the last layer of the network to generate a probability distribution for the class the network is trying to predict. The class with the highest probability is selected as the predicted class.

FIG. 9 illustrates an example of a retopology prediction model based on topology prediction based on convolution neural networks (CNN) according to various embodiments of the present invention.

Specifically, Figure 9 shows an example of a second learning model.

Referring to FIG. 9, the decoder is discarded and only the encoder is separated from the first learning model learned to identify the topology consisting of a plurality of subpolygons in the input image from the image as shown in FIG. 7, and a classification head (classification head, c) is placed behind the encoder. Multiple fully connected layers (FC layer, FC layer) called -head are attached.

If the encoder separated from the first learning model is not ViT in Figure 8 but a CNN of U-Net as shown in Figure 9, convolution layer 1 (CONV 1) to convolution layer 5 (CONV 5) is U-Net It corresponds to the encoder separated from , and FC layer 6, FC layer 7, and softmax correspond to c-head.

According to various embodiments of the present invention,

(1) In the embodiment of FIG. 5, an encoder-decoder model for predicting the topology of a subpolygon is trained using the topology of the subpolygon corresponding to the original image. The final parameter values of the encoder of the first learning model that has completed training may be referred to as w1, w2, ..., wN.

(2) Remove only the encoder from the first learning model that has completed training and add a few layers to create a new structure for the second learning model. Let the number of parameters of the additional layer be M, and N>>M. According to one embodiment, N may correspond to 9 times or more than M. At this point, the second learning model has not yet been trained. In other words, the values of N+M parameters constituting the second learning model have not yet been determined.

(3) To learn the second learning model (i.e., to determine N+M parameter values), gradient descent is applied using the original image and retopology information corresponding to the image. Gradient descent is a method that sets the initial values of model parameters appropriately and gets closer to the correct value as iterations occur. At this time, in normal cases, the initial values of all parameters of the model to which gradient descent is applied are set to random values near 0, but in various embodiments of the present invention, among the N+M parameters constituting the second learning model For N parameters, (1) the w1, w2, ..., wN values obtained from the learning results of the first learning model are set as initial values. Among the N+M parameters constituting the second learning model, the initial values of the remaining M parameters are set to initial values near 0 as usual. The initial value near 0 may be a random value among peripheral values close to 0 that are designated based on a peripheral range value preset based on 0.

Afterwards, the second learning model is confirmed by learning so that the initial value set using gradient descent converges to the correct value.

Various embodiments of the present invention are based on the learning result value of identifying the topology from an image to a plurality of subpolygons in learning a deep learning model that acquires retopology information from an image to a quad mesh. Apply tuning. The concept of fine-tuning is to identify the topology with multiple subpolygons (task 1) and use the learned and acquired w1, w2,..., wN values to predict retopology into a quad mesh. This means that it is used as the initial value when learning with (task2).

In the encoder + c-head model, the encoder model parameters to be obtained are called e1, e2, ..., eN for convenience (total N), and the model parameters of c-head are c1, c2, ..., cM ( A total of M) can be said. N>>M because the encoder is much larger than the c-head. According to one embodiment, N may correspond to 9 times or more than M.

If fine-tuning is not considered and all N+M parameters are set to initial values near 0, retopology information from the image to the quad mesh is obtained for the second learning model (encoder + c-head model). If you learn to acquire it, you will go through the following process.

[Learning method without fine-tuning]

(1) Random initialize a total of N+M (N encoders + M c-heads) parameter values to values around 0 (for example, e1=0.01, e2 = -0.001, ... c1 =0.002, c2 = 0.012... )

(2) Parameter values are continuously updated through an iterative algorithm called gradient descent. The data to be used at this time has nothing to do with topology prediction, and only uses classified learning data related to retopology into a quad mesh. For example, when an image is input, N+M parameters are updated so that the retopology prediction result can be output correctly.

In contrast, the learning process proposed in the present invention is as follows.

[Learning method proposed by the present invention]

(1) M c-head parameters, which are much smaller than N encoder parameters, use random initialization as above. The N encoder parameters, which are much more than the M c-head parameters, take the learned result values and use them as initial values to identify a topology composed of multiple subpolygons.

(2) The N+M parameter values in the second learning model are updated using gradient descent based on the learning data classified in relation to the retopology results. At this time, the N encoder parameters are additionally updated using the values obtained through learning another task (i.e., identification of a topology consisting of multiple subpolygons) as initial values, so it is called fine-tuning (rather than new learning). The method of using learning results from other tasks is generally called transfer learning.

[Advantages of the learning method proposed in the present invention]

(1) If you use a learning method that does not consider fine-tuning, all parameters must be updated from random initial values, so a lot of labeled data is needed. Using the method proposed in the present invention, learning is possible even with less training data because the N parameters, which correspond to more than 90%, but not all, of the parameters are learned from fairly good values. It is difficult to obtain a lot of training data for the second learning model because it must be obtained from training images that have been retopologically converted to a quad mesh. The reason for the 'pretty good value' is that if the topology was well predicted with multiple subpolygons based on the edges and vertices of each image, it is proof that the encoder extracted the features of the image well.

(2) The learning data that should be used to learn topology prediction with multiple subpolygons based on the edges and vertices of each image, which is the source task of transfer learning, is from images in which retopology to a quad mesh has not been performed. Since it is obtainable, you can easily secure a large amount. If we compare transfer learning to using the knowledge gained through studying mathematics to study physics, while a lot of mathematics study materials can be easily obtained, physics study materials are limited.

FIG. 10 shows an example of an image in which a topology composed of a plurality of subpolygons in an input image is analyzed and retopology based on topology analysis is performed according to various embodiments of the present invention.

Referring to Figure 10, the mesh types of (1) solid and (2) surface are shown for each row. Column by column, we represent (a) the original object, (b) all triangles, (c) triangles on the edges only, (d) no triangles, the entire grid, and (e) no triangles, the quad tree.

The state identified as a plurality of subpolygons before being retopologized into a quad mesh may correspond to a state such as (b) or (c).

Regarding (b) or (c) of the triangular mesh, (d) or (e) is a form in which retopology was performed with a square mesh (quad mesh).

Retopology to a quad mesh offers several advantages:

(1) Ease of animation: Quad mesh is ideal for animation. Quad provides more predictable deformations and more natural results, especially on complex surfaces such as skins. Additionally, Quad mesh can effectively control the flow of the model through edge loops and edge rings.

(2) Ease of UV mapping: Quad mesh is better suited for UV mapping. Quads map naturally to rectangular texture pixels, which minimizes texture distortion and makes texture painting easier. UV is a term used in 3D modeling and refers to a coordinate system that references the 2D texture map of a 3D object. UV is not really an abbreviation, we use UV to represent 2D texture space because we already use XYZ in 3D space. UV mapping is the process of applying a 2D image (texture) to the surface of a 3D model. It is used to enhance the detail of a 3D model by adding color, pattern, texture, etc. to it. The UV coordinate system defines where this 2D image should be applied to the 3D model.

(3) Ease of modeling: Quad mesh is more suitable for modeling. Quad allows you to easily add or remove model shapes through edge loops. Additionally, quads help make the surface of the model smoother.

(4) Subdivision surface: Quad mesh is ideal for modeling subdivision surfaces. This is a technique that creates additional geometry to increase the detail of the model. Quads provide more uniform and predictable results in this process. In 3D computer graphics, “geometry” refers to the basic shape or structure that makes up a 3D model or scene. It generally consists of basic elements such as points, lines, and faces in 3D space. These basic elements are connected together to form complex 3D shapes. For example, the geometry of a 3D model includes all of the model's vertices, edges, and faces. These factors determine the shape, size, and location of the model. Additionally, they may contain additional information such as UV mapping, which is used to determine the model's surface texture, color, material, etc. Therefore, in 3D modeling, “geometry” can be thought of as a term representing the “skeleton” or “structure” of the model.

(5) Tool compatibility: Most 3D modeling and animation tools prefer quad mesh. This is because Quad provides more flexible and predictable results thanks to the advantages mentioned above.

Because of these advantages, many 3D artists and engineers retopologize complex triangular meshes into quad meshes. However, this task is time consuming and can be very difficult for complex models. For this reason, various research and technologies are being developed to automate or simplify this process.

11 to 13 illustrate an example of an output image of a wireframe visualized after performing topology prediction-based retopology according to various embodiments of the present invention.

Referring to Figures 11 to 13, an original image, a lipology wireframe image, and a lipology rendering image are shown. After performing topology prediction-based quad mesh retopology according to various embodiments of the present invention from the original image, a visualized wireframe output image is generated. In addition, a lipology rendering image is created by performing rendering in a post-process based on the lipology wireframe.

The process of creating a wireframe from a quad mesh is typically performed using 3D modeling software. This process works similarly in most 3D modeling software, here are the typical steps:

(1) Load 3D model: First, load the 3D model consisting of quad mesh into 3D modeling software.

(2) Change view mode: Most 3D modeling software provides various view modes. Among these, select 'Wireframe' or 'Wireframe View' mode. This mode displays all polygons in the 3D model as lines only, allowing you to view the wireframe of the model.

(3) Wireframe extraction (optional): Some 3D modeling software provides the ability to extract wireframes as separate objects. Using this feature, you can manipulate wireframes independently or export them to other formats.

Wireframes represent the basic structure of a 3D model and help understand the flow and topology of the design. However, the wireframe itself does not include surface details or textures, so additional work is required to represent this information. According to various embodiments of the present invention, the memory can store 3D modeling software and generate a wireframe based on the result of retopology into a quad mesh.

Quad mesh is very useful for creating wireframes. This is because quad mesh offers several important advantages:

(1) Predictable structure: Quad mesh has a predictable and consistent structure because each polygon has four vertices. This simplifies wireframe creation and makes the results cleaner.

(2) Edge Loop and Edge Ring: Quad mesh can easily create edge loops and edge rings. This is very useful for expressing the flow and structure of a 3D model.

(3) Subdivision surface: Quad mesh is ideal for modeling subdivision surfaces. This is a technique that creates additional geometry to increase the detail of the model. Quads provide more uniform and predictable results in this process.

Edge Flow is a concept used in 3D modeling and digital sculpting processes. This refers to the flow or arrangement of polygons that make up the surface of a 3D model. Edge flow plays an important role in determining the appearance and details of the model. A well-designed edge flow helps express the shape of the model naturally and accurately, and is useful when animating or adding texture to the model. For example, in the case of human models, edge flow helps express the movement of joints and muscles naturally. Additionally, edge flow is used in animation to create detailed expressions such as skin folds, wrinkles, and curves. When designing an edge flow, the geometry of the model and the connectivity between other parts must be considered. In other words, consideration must be given to ensure that all polygons are naturally connected and the surface is formed smoothly. This allows natural movement when the model is transformed or animated. Edge flow is mainly used for visual representation in 3D modeling software, and plays an important role for designers or artists in the modeling process. Edge flow determines the detail and expressiveness of a model, so a well-designed edge flow can improve the quality and appearance of the final result.

When implementing embodiments of the present invention using hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), and programmable logic devices (PLDs) configured to perform the present invention. , FPGAs (field programmable gate arrays), etc. may be provided in the processor of the present invention.

Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable medium. Additionally, the data structure used in the above-described method can be recorded on a computer-readable storage medium through various means. Program storage devices, which may be used to describe a storage device containing executable computer code for performing various methods of the present invention, should not be understood to include transient objects such as carrier waves or signals. do. The computer-readable storage media includes storage media such as magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical readable media (eg, CD-ROM, DVD, etc.).

The embodiments described above combine the components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. Additionally, it is possible to configure an embodiment of the present invention by combining some components and/or features. The order of operations described in embodiments of the invention may be changed. Some features or features of one embodiment may be included in another embodiment or may be replaced with corresponding features or features of another embodiment. It is obvious that claims that do not have an explicit reference relationship in the patent claims can be combined to form an embodiment or included as a new claim through amendment after filing.

It will be clear to those skilled in the art that the present invention can be embodied in other forms without departing from the technical spirit and essential features of the present invention. Accordingly, the above embodiments should be considered in all respects as illustrative rather than restrictive. The scope of rights of the present invention should be determined by reasonable interpretation of the appended claims and all possible changes within the equivalent scope of the present invention.

110: Image 120: Electronic device
121: memory 122: processor
123: input device 124: output device
130: Wireframe image 400: Artificial neural network
410: Input layer 411: Multiple arguments
430: Hidden layer 431: First hidden layer
432: first unit 433: second hidden layer
434: second unit 450: output layer
451: Prediction result unit

Claims (10)

In a method of operating an electronic device including an input device, memory, processor, and output device,
performing, by the processor, topology prediction and retopology learning for a learning model stored in the memory based on a plurality of test images for which topology analysis has not been performed;
Receiving information about an input image through the input device;
generating, by the processor, a prediction result of a topology consisting of a plurality of subpolygons in the input image using the learning model from the input image;
performing re-topology by the processor by squareling the plurality of subpolygons in the input image using the learning model;
Outputting a visualized wireframe for the input image based on the result of the retopology by the output device,
The step of performing topology prediction and retopology learning on the learning model stored in the memory based on the plurality of test images by the processor,
Receiving a plurality of first test images on which topology analysis has not been performed by the input device;
receiving, by the input device, a plurality of second test images in which a topology prediction consisting of a plurality of sub-polygons based on corners and vertices has been performed on the plurality of first test images;
determining, by the processor, a plurality of first partial test images at a plurality of random locations within the first test image;
determining, by the processor, the plurality of second partial test images at a plurality of locations corresponding to the plurality of first partial test images within the second test image;
performing learning to predict a topology of a first learning model stored in the memory based on the plurality of first partial test images and the plurality of second test images by the processor;
generating a second learning model by the processor by removing a decoder of the first learning model on which learning was performed and combining a plurality of predetermined layers with an encoder of the first learning model;
A plurality of third test images in which topology prediction and retopology were not performed, and a plurality of fourth test images in which retopology was performed by squarening a plurality of subpolygons in the plurality of test images. receiving input by the input device;
A step of performing, by the processor, prediction of a topology and learning of a retopology of the second learning model based on the plurality of third test images and the plurality of fourth test images,
Neighboring second partial test images among the plurality of second partial test images overlap each other or have a gap between them,
The learning model corresponds to the second learning model, and is characterized in that it restores damaged parts of a damaged mesh model that overlaps with an error or a gap is created, or fills the gap between meshes,
method.
delete According to claim 1,
The second learning model is,
After dividing the input image into a plurality of partial images, topology prediction is performed using the plurality of subpolygons,
Configured to generate a retopology image in which retopology has been performed by squared based on the plurality of subpolygons,
method.
According to claim 1,
The encoder of the first learning model consists of N first parameters, the plurality of layers consists of M second parameters, N and M are integers greater than 0,
The second learning model is composed of N first parameters and M second parameters,
The N first parameters have values according to the results of learning of the second learning model as initial values,
The M second parameters have random values among peripheral values specified based on a peripheral range value preset with 0 as the reference,
method.
According to clause 4,
N, the number of the first parameters, is 9 times more than M, the number of the second parameters,
method.
According to claim 1,
The plurality of first 3D partial test images and the plurality of second partial test images are configured with the same set size and structure,
The number of the plurality of first partial test images and the plurality of second partial test images corresponds to a random number greater than or equal to a set number,
The plurality of first test images and the plurality of second test images have the same size and structure,
method.
delete According to claim 1,
The step of performing training of the first learning model stored in the memory based on the plurality of first partial test images and the plurality of second partial test images includes:
Based on one first partial test image among the plurality of first partial test images and one second partial test image corresponding to the one first partial test image among the plurality of second partial test images A step of performing learning of the first learning model is repeatedly performed,
Whenever training of the first learning model is performed, the first learning model is based on a different first partial test image and a different second partial test image among the plurality of first partial test images. where learning is performed,
method.
In electronic devices,
The electronic device includes an input device, a memory, a processor, and an output device,
The processor is configured to perform the method according to any one of claims 1, 3 to 6, and 8,
Electronic devices.
A computer program configured to perform the method according to any one of claims 1, 3 to 6, and 8, and recorded on a computer-readable storage medium.

Images