KR102245220B1 - Apparatus for reconstructing 3d model from 2d images based on deep-learning and method thereof - Google Patents

Abstract

Provided are a device for reconstructing a 3D model from a 2D image based on deep-learning and a method thereof. According to some embodiments of the present disclosure, the device for reconstructing the 3D model includes: a memory which stores one or more instructions; and a processor which obtains a 2D input image of a target object by executing the one or more stored instructions, and reconstructs a 3D voxel model of the target object from the input image through a deep-learning network. The device for reconstructing the 3D model may improve the accuracy of 3D model restoration by using the deep-learning network.

Description

Apparatus and method for restoring a 3D model from a 2D image based on deep learning {APPARATUS FOR RECONSTRUCTING 3D MODEL FROM 2D IMAGES BASED ON DEEP-LEARNING AND METHOD THEREOF}

The present disclosure relates to an apparatus and method for reconstructing a 3D model from a 2D image based on deep learning. In more detail, a device capable of reconstructing a three-dimensional voxel model from a two-dimensional image of a target object using a deep-learning network, and a method performed in the device. About.

Recently, as interest in augmented reality and virtual reality is increasing, research on a technology related to a 3D stereoscopic image is actively and in progress. In addition, as part of this research, research on a technology for restoring a three-dimensional image or model from a two-dimensional image (hereinafter, "three-dimensional restoration technology") is also being conducted.

So far, several 3D restoration techniques have been proposed. For example, a technique for detecting a contour point in a given two-dimensional image based on a difference in contrast and restoring a three-dimensional image based on an estimated depth value of the detected contour point has been proposed (refer to Patent Document 1). In addition, a technique for restoring a three-dimensional voxel model by using a plurality of images captured from different angles has also been proposed.

However, the proposed techniques have a problem in that 3D reconstruction accuracy is poor, or when only a single-view image is given, a 3D voxel model cannot be reconstructed.

Korean Patent Registration No. 10-1725166 (published on April 12, 2017)

The technical problem to be solved through some embodiments of the present disclosure is to restore a three-dimensional voxel model from a two-dimensional single-view image of a target object using a deep-learning network. It relates to a device capable of and a method to be performed on the device.

Another technical problem to be solved through some embodiments of the present disclosure is an apparatus capable of reconstructing a three-dimensional voxel model from a two-dimensional multi-view image of a target object using a deep learning network, and It relates to how it is performed on the device.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, an apparatus for restoring a 3D model from a 2D image according to some embodiments of the present disclosure includes a memory storing one or more instructions, and executing the stored one or more instructions. And a processor for acquiring a two-dimensional input image for a target object and reconstructing a three-dimensional voxel model for the target object from the input image through a deep learning network. At this time, the deep learning network includes a restorer for restoring and outputting the voxel model from the input image, and a discriminator for discriminating an actual image and a fake image, and the restorer is between the restored voxel model and the correct voxel model. It may be learned through a first error calculated based on the difference and a second error calculated based on a result of determination of the discriminator for the restored voxel model.

In some embodiments, the reconstructor is further learned based on a third error, but the third error is a two-dimensional projection image and the input image generated through a projection operation on the restored voxel model. It can be calculated based on the difference of.

In some embodiments, the reconstructor may include an encoder for extracting features of the target object from the input image and a decoder for generating the voxel model by decoding the extracted features.

In some embodiments, the input image is a multi-view image including a plurality of images corresponding to different views of the target object, and the reconstructor is a plurality of encoders, a plurality of decoders, and an aggregator. And a refiner, wherein the plurality of encoders receive the multi-view image and extract a plurality of features of the target object, and the plurality of decoders decode the plurality of features to obtain a plurality of volume features ( volume feature), the aggregator aggregating the plurality of volume features, and the refiner is configured as a neural network to restore the voxel model from the aggregated volume features.

An apparatus for restoring a 3D model from a 2D image according to some embodiments of the present disclosure for solving the above technical problem includes a memory storing one or more instructions, and executing the one or more instructions. , Receiving a multi-view image of a target object, extracting two-dimensional depth information from the multi-view image, generating volume data for the target object based on the depth information, and deep learning It may include a processor for restoring a three-dimensional voxel model of the target object from the volume data through a network.

A method of restoring a 3D model from a deep learning-based 2D image according to some embodiments of the present disclosure to solve the above technical problem is a method performed in a computing device, and includes a 2D input image for a target object. And restoring a three-dimensional voxel model of the target object from the input image through a deep learning network.

A method of restoring a 3D model from a deep learning-based 2D image according to some other embodiments of the present disclosure for solving the above technical problem is a method performed in a computing device, and includes a multiview of a target object. -view) receiving an image, extracting two-dimensional depth information from the multi-view image, generating volume data for the target object based on the depth information, and the volume data through a deep learning network The method may include restoring a 3D voxel model for the target object.

A computer program according to some embodiments of the present disclosure for solving the above-described technical problem is combined with a computing device to obtain a two-dimensional input image for a target object, and from the input image through a deep learning network. It may be stored in a computer-readable recording medium to execute the step of restoring a three-dimensional voxel model of the target object.

A computer program according to some other embodiments of the present disclosure for solving the above-described technical problem is combined with a computing device to receive a multi-view image of a target object, from the multi-view image. Extracting two-dimensional depth information, generating volume data for the target object based on the depth information, and a three-dimensional voxel model for the target object from the volume data through a deep learning network It may be stored in a computer-readable recording medium to perform the step of restoring.

According to some embodiments of the present disclosure described above, a three-dimensional voxel model may be restored from a two-dimensional image of a target object through a deep-learning network. By doing so, the reconstruction accuracy for the 3D voxel model can be greatly improved.

In addition, even when only a single-view image is given, a 3D voxel model can be reconstructed with high accuracy.

In addition, as the deep learning network is further learned through various errors such as an outline error, a projection error, a discrimination error, and a classification error, the restoration performance of the deep learning network may be further improved.

In addition, rough 3D shape information of the target object can be estimated from depth information of a multi-view image of the target object, and a 3D voxel model from 3D shape information estimated through a deep learning network This can be accurately restored.

In addition, by using a deep learning network having a feature accumulation structure (e.g. RNN, aggregator), reconstruction accuracy for a 3D voxel model can be further improved.

The effects according to the technical idea of the present disclosure are not limited to the above-mentioned effects, and other effects that are not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is an exemplary diagram illustrating an apparatus for reconstructing a 3D model from a 2D image and input/output data thereof according to some embodiments of the present disclosure.
2 is an exemplary flowchart illustrating a method of reconstructing a 3D model from a 2D image based on deep learning according to the first embodiment of the present disclosure.
3 to 6 are exemplary diagrams for explaining a structure and a learning method of a deep learning network according to some embodiments of the present disclosure.
7 and 8 are exemplary diagrams for explaining a method of reconstructing a 3D model from a 2D image based on deep learning according to a second embodiment of the present disclosure.
9 is an exemplary diagram for describing a method for extracting depth information according to some embodiments of the present disclosure.
10 is an exemplary diagram for explaining a method of reconstructing a 3D model from a 2D image based on deep learning according to a third embodiment of the present disclosure.
11 is an exemplary diagram for describing a method of generating a multi-view image according to some embodiments of the present disclosure.
12 illustrates an exemplary computing device capable of implementing an apparatus for reconstructing a 3D model according to some embodiments of the present disclosure.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Advantages and features of the present disclosure, and a method of achieving them will be apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the technical idea of the present disclosure is not limited to the following embodiments, but may be implemented in various different forms, and only the following embodiments complete the technical idea of the present disclosure, and in the technical field to which the present disclosure pertains. It is provided to completely inform the scope of the present disclosure to those of ordinary skill in the art, and the technical idea of the present disclosure is only defined by the scope of the claims.

In adding reference numerals to elements of each drawing, it should be noted that the same elements are assigned the same numerals as possible, even if they are indicated on different drawings. In addition, in describing the present disclosure, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present disclosure, a detailed description thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which this disclosure belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically. The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present disclosure. In this specification, the singular form also includes the plural form unless specifically stated in the phrase.

In addition, in describing the constituent elements of the present disclosure, terms such as first, second, A, B, (a), (b) may be used. These terms are only for distinguishing the component from other components, and the nature, order, or order of the component is not limited by the term. When a component is described as being "connected", "coupled" or "connected" to another component, the component may be directly connected or connected to that other component, but another component between each component It will be understood that elements may be “connected”, “coupled” or “connected”.

As used in this disclosure, "comprises" and/or "comprising" refers to the referenced components, steps, actions, and/or elements of one or more other elements, steps, actions and/or elements. It does not exclude presence or addition.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram illustrating an apparatus 10 for reconstructing a 3D model from a 2D image and input/output data thereof according to some embodiments of the present disclosure.

As shown in FIG. 1, the 3D model restoration apparatus 10 is a computing device capable of reconstructing a 3D voxel model 13 from a 2D input image 11 for a target object (eg airplane). I can. That is, the 3D model restoration apparatus 10 may receive a 2D image 11 of a target object, restore it to a 3D voxel model 13 and output it. In this case, the input image 11 may be a single-view image as shown, or may be a multi-view image in some cases. Hereinafter, for convenience of description, the 3D model restoration apparatus 10 will be abbreviated as “restore apparatus 10”.

The computing device may be a notebook, a desktop, a laptop, etc., but is not limited thereto, and may include all types of devices equipped with a computing function.

In various embodiments of the present disclosure, the reconstruction apparatus 10 may reconstruct the 3D voxel model 13 from the 2D input image 11 using a deep-learning network. Accordingly, the reconstruction accuracy of the 3D voxel model can be greatly improved, and the present embodiment will be described in detail later with reference to the accompanying drawings in FIG. 2.

Meanwhile, although FIG. 1 illustrates that the restoration device 10 is implemented as a single computing device, the restoration device 10 may be implemented as a plurality of computing devices. For example, a first function of the restoration device 10 may be implemented in a first computing device, and a second function may be implemented in a second computing device. Alternatively, a specific function of the restoration device 10 may be implemented in a plurality of computing devices.

So far, the restoration apparatus 10 according to some embodiments of the present disclosure has been schematically described with reference to FIG. 1. Hereinafter, a method for restoring a 3D model from a 2D image based on deep learning in the restoration apparatus 10 illustrated in FIG. 1 (hereinafter, “a method for restoring a 3D model based on deep learning”) will be described in detail. .

Each step of the 3D model restoration method to be described below may be implemented with one or more instructions executed by the processor of the computing device, and it is understood that it is performed by the restoration device 10 when the subject of a specific operation is omitted. Can be. However, in some cases, some steps of the 3D model restoration method may be performed in another computing device.

First, a deep learning-based 3D model restoration method according to a first embodiment of the present disclosure will be described with reference to FIGS. 2 to 6.

2 is an exemplary flowchart illustrating a deep learning-based 3D model restoration method according to the first embodiment of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some steps may be added or deleted as necessary.

As shown in FIG. 2, the present embodiment may begin in step S100 of training a deep learning network using training data. The training data (set) may be composed of, for example, a two-dimensional image (set) of a target object and a three-dimensional correct voxel model (set).

In this step S100, the structure and/or learning method of the deep learning network may be designed in various ways, which may vary according to embodiments.

In some embodiments, as illustrated in FIG. 3, the deep learning network reconstructor 20 capable of reconstructing a 3D voxel model from a 2D image in an end-to-end manner. ) Can be included. The reconstructor 20 is configured to include, for example, an encoder 21 for extracting features 23 for a target object and a decoder 25 for generating a voxel model 33 by decoding the extracted features 23 Can be. However, the present invention is not limited thereto, and the restorer 20 may be formed of another type of neural network structure. Both the encoder 21 and the decoder 25 may be composed of a neural network. For example, the encoder 21 may be composed of CNN (Convolutional Neural Networks; eg, one or more convolutional layers), and the decoder (21) may be composed of CNN or DCNN (DeConvolutional Neural Networks; eg, one or more deconvolutional layers). However, it is not limited thereto. In addition, the feature 23 of the target object may be, for example, one or more feature maps in two dimensions or three dimensions. However, it is not limited thereto.

In the example of FIG. 3, the reconstructor 20 may be trained by backpropagating an error or loss (hereinafter “voxel error”) based on a difference between the restored voxel model 33 and the correct voxel model (not shown). have. That is, the weight (ie, weight parameter) of the restorer 20 may be updated in a direction in which the error is minimized. Those skilled in the art will be sufficiently familiar with such an error back-propagation technique, and a detailed description thereof will be omitted.

In addition, in some embodiments, as illustrated in FIG. 4, a projection error 39 may be further used to learn the reconstructor 20. That is, the weight of the reconstructor 20 may be updated in a direction in which the voxel error 37 and the projection error 39 are minimized. Here, the projection error 39 is a two-dimensional image 41 (hereinafter “projection image”) generated by applying a projection operation to the restored voxel model 33 and a two-dimensional image 31 input to the reconstructor 20. ) Can be calculated based on the difference. At this time, the loss function used to calculate the error may be set to any function. In this embodiment, the projection image 41 may be generated by performing a projection operation to have the same or similar view as the input image 31, but the scope of the present disclosure is not limited thereto. According to the present embodiment, by further using the projection error 39 to learn the restorer 20, the performance of the restorer 20 may be further improved.

In addition, in some embodiments, an outline error may be further used to learn the restorer 20. For example, the weight of the reconstructor 20 may be updated in a direction in which the voxel error 37 and the outline error are minimized. Here, the outline error is calculated based on the difference between the outline extracted from the restored voxel model 33 and the outline extracted from the correct voxel model 35, or the outline and the input image 31 extracted from the projection image 41 It can be calculated based on the difference between the outlines extracted from. According to the present embodiment, by further using the outline error to learn the restorer 20, the performance of the restorer 20 may be further improved. For example, since the outline error reflects the restoration error of the shape (or shape) of the target object, the restorer 20 having further learned the outline error can more accurately restore the shape (or shape) of the target object. .

In addition, in some embodiments, as shown in FIG. 5, the deep learning network may be configured to further include a discriminator 40, and the restorer 20 may determine the discriminator 40 It may be learned by further using the discrimination error 43 calculated based on the result. That is, the weight of the reconstructor 20 may be updated in a direction in which the voxel error 37 and the discrimination error 43 are minimized. Here, the discriminator 40 may be a neural network (eg CNN) trained to determine an actual voxel model (eg correct answer voxel model 35, etc.) and a fake voxel model (eg restored voxel model 33), and the discrimination error 43 ) May be calculated to have a smaller value as the restored voxel model 33 is closer to the actual voxel model. The discriminator 40 and the restorer 20 may be learned to be hostile to each other, but the scope of the present disclosure is not limited thereto. According to the present embodiment, by further using the discrimination error 43 to learn the restorer 20, the performance of the restorer 20 may be further improved. For example, the reconstructor 20 may restore the input 2D image (e.g. 31) into a 3D voxel model closer to the actual.

In addition, in some embodiments, as shown in FIG. 6, the deep learning network may be configured to further include a classifier 50 for classifying a class of a target object, and a restorer 20 ) May be further learned by using the classification error 49 calculated based on the classification result of the classifier 50. That is, the weight of the reconstructor 20 may be updated in a direction in which the voxel error 37 and the classification error 49 are minimized. In this embodiment, the reconstructor 20 may receive the 2D image 31 and class information 45 (e.g. airplane) of the target object, restore the voxel model 33, and output it. In addition, the classifier 50 may be a neural network (eg CNN) trained to classify the class of the target object included in the projected image 41 or the restored voxel model 33, and the classification error 49 is the classifier 50 ) May be calculated based on the difference between the predicted class information 47 and the correct answer class information 45. According to the present embodiment, by further using the classification error 49 to learn the restorer 20, the performance of the restorer 20 may be further improved. For example, the restorer 20 may restore a voxel model that better matches the class of the target object.

The embodiments described so far with reference to FIGS. 3 to 6 may be combined in various forms. For example, the deep learning network may be configured to include both the discriminator 40 and the classifier 50, and the reconstructor 20 includes a projection error 39, a discrimination error 43, and a classification error 49. ) Can also be used.

This will be described with reference to FIG. 2 again.

Steps S120 and S140 below may be understood as a process of utilizing the deep learning network learned through step S100. Hereinafter, steps S120 and S140 will be described.

In step S120, a two-dimensional original image of the target object may be obtained.

In step S140, the two-dimensional original image may be reconstructed into a three-dimensional voxel model through the deep learning network. For example, the restoration device 10 may input the original image to the deep learning network and obtain a 3D voxel model restored from the deep learning network (see FIG. 3 ).

So far, a deep learning-based 3D model restoration method according to the first embodiment of the present disclosure has been described with reference to FIGS. 2 to 6. According to the above-described method, a three-dimensional voxel model can be restored from a two-dimensional image in an end-to-end manner by using a deep learning network. This method can guarantee high restoration accuracy without requiring a complex algorithm as in the prior art. I can. In addition, by learning using various errors such as outline error, projection error (eg 39), discrimination error (eg 43), classification error (eg 49) in addition to voxel error (eg 37), the restoration performance of the deep learning network is improved. It can be further improved. In addition, there is an advantage that a voxel model can be reconstructed using only a single view image.

Hereinafter, a deep learning-based 3D model restoration method according to a second embodiment of the present disclosure will be described with reference to FIGS. 7 to 9. The second embodiment relates to a method of further improving the reconstruction accuracy of a 3D model by using a multi-view image.

7 is an exemplary flowchart illustrating a deep learning-based 3D model restoration method according to a second exemplary embodiment of the present disclosure, and FIG. 8 is an exemplary diagram for further explaining this. Hereinafter, it will be described with reference to FIGS. 7 and 8 together.

In step S200, a two-dimensional multi-view image 61 of a target object (e.g. automobile) may be obtained. The multi-view image 61 may include a plurality of 2D images (e.g. 61-1, 61-2) of target objects corresponding to different views. For example, the multi-view image 61 may include a first image 61-1 corresponding to a first view and a second image 61-2 corresponding to a second view.

In step S220, two-dimensional depth information 63 may be extracted from the multi-view image 61. For example, the first depth information 63-1 is extracted from the first image 61-1 constituting the multi-view image 61, and the second depth information 63-2 is extracted from the second image 61-2. ) Can be extracted. Here, the depth information may be a two-dimensional depth map, but is not limited thereto.

In this step S220, the method of extracting the depth information 63 may be various, and any method may be used. For example, the depth information 63 may be extracted through a depth information extraction algorithm widely known in the art. In some embodiments, depth information may be extracted using a Generative Adversarial Networks (GAN) module (model), which will be further described below with reference to FIG. 9.

9, the GAN module receives a two-dimensional image 71 (eg RGB image) and generates a two-dimensional depth map 75 corresponding thereto, a generator 73, and an actual image. It may include a discriminator 79 to determine the fake image.

As shown, the generator 73 may include an encoder that extracts features from a 2D image and a decoder that decodes the extracted features to generate a depth map 75. However, the scope of the present disclosure is not limited thereto. In addition, both the encoder and the decoder may be composed of a neural network. For example, the encoder may be composed of a CNN, and the decoder may be composed of a CNN or DCNN. However, it is not limited thereto.

Next, the discriminator 79 may be trained to determine an actual depth map (e.g. depth map 77) and a fake depth map (e.g. depth map 75). The discriminator 79 may be composed of, for example, a CNN, but is not limited thereto.

Hostile learning may be performed between the generator 73 and the discriminator 79, and as the hostile learning is performed, the generator 73 can more elaborately generate the depth map 75 corresponding to the input image 71. There will be. If you are a practitioner in the relevant technical field, you will be sufficiently familiar with the hostile learning technique, and a detailed description thereof will be omitted.

This will be described with reference to FIGS. 7 and 8 again.

In step S240, volume data 65 for the target object may be generated based on the extracted depth information 63. For example, the volume data 65 may be generated based on first depth information (e.g. 63-1) corresponding to the first view and second depth information (e.g. 63-2) corresponding to the second view. In some cases, volume data 65 may be generated further based on the multi-view image 61.

The volume data 65 may mean data representing a three-dimensional shape of a target object, and specific data types (formats) may be designed in various ways. For example, the volume data 65 may be voxel data representing rough 3D shape information on a target object. As another example, the volume data 65 may be a three-dimensional depth map representing three-dimensional shape information of a target object. Alternatively, the volume data 65 may be a two-dimensional depth map.

In step S260, the 3D voxel model 67 may be restored from the volume data 65 through the deep learning network 60. For example, the restoration device 10 may input the volume data 65 to the deep learning network 60 and obtain a 3D voxel model 67 restored from the deep learning network 60.

The deep learning network 60 may be a model trained to restore the voxel model 67 from the volume data 65. For example, the deep learning network 60 may have an encoder-decoder structure (refer to FIG. 3), and may be constructed by learning training data composed of volume data for a target object and a correct voxel model. However, it is not limited thereto. The deep learning network 60 may be constructed in a manner similar to that described with reference to FIGS. 3 to 6.

So far, a deep learning-based 3D model restoration method according to a second embodiment of the present disclosure has been described with reference to FIGS. 7 to 9. According to the above-described method, rough 3D shape information (ie, volume data) of the target object can be estimated from depth information (eg 63) of the multi-view image (eg 65) of the target object, and the deep learning network The 3D voxel model (eg 67) can be accurately reconstructed from the 3D shape information estimated through (eg 60).

Hereinafter, a deep learning-based 3D model restoration method according to a third embodiment of the present disclosure will be described with reference to FIG. 10. The second embodiment also relates to a method of further improving the reconstruction accuracy of a 3D model by using a multi-view image.

10 is an exemplary diagram for explaining a deep learning-based 3D model restoration method according to a third embodiment of the present disclosure. In particular, FIG. 10 shows a detailed structure of a deep learning network.

The restoration apparatus 10 may reconstruct the 3D voxel model 99 from the 2D multi-view image 91 of the target object in an end-to-end manner through the deep learning network illustrated in FIG. 10. Such a deep learning network can be constructed by learning training data composed of a multi-view image (eg 91) and a correct voxel model (eg 99), and the learning method is performed in the manner described with reference to FIGS. 3 to 6, for example. Can be. In order to exclude redundant descriptions, a description of the learning method will be omitted, and description will be continued focusing on the structure and operation principle of the deep learning network according to the present embodiment.

As shown, the deep learning network includes a plurality of encoders 81-1 to 81-n, a recurrent neural network (RNN) module 83, a plurality of decoders 85-1 to 85-n, and an aggregator ( 87) and a refiner 89. Hereinafter, each component of the deep learning network will be described.

A plurality of encoders (81-1 to 81-n) is a feature (eg feature map) of the target object from the multi-view image (91) consisting of a plurality of images (91-1 to 91-n) corresponding to different views Can be extracted. For example, the first encoder 81-1 may receive the first image 91-1 and extract a first feature of the target object from the first image 91-1, and the second encoder 81- 2) receives the second image 91-2 and extracts a second feature of the target object from the second image 91-2. The encoder (e.g. 81-1) may be configured with a CNN, for example, but the scope of the present disclosure is not limited thereto.

Next, the RNN module 83 is located between a plurality of encoders (81-1 to 81-n) and a plurality of decoders (85-1 to 85-n), and a plurality of RNN units (83-1 to 83-n) ). The plurality of RNN units 83-1 to 83-n perform a predetermined neural network operation on the feature data extracted by the plurality of encoders 81-1 to 81-n, and connect the result of the operation to the connected RNN unit (eg 83-2) and the decoder (eg 85-1). For example, the first RNN unit 83-1 performs a neural network operation on the first feature extracted by the first encoder 81-1, and the calculation result is transmitted to the second RNN unit 83-1 and the first It can be provided to the decoder 85-1. In addition, the second RNN unit 83-2 performs a neural network operation on the second feature extracted by the second encoder 81-2, and the RNN unit and the second decoder 85-2 are connected to the result of the operation. Can be provided to.

The RNN module 83 may perform a role of enhancing (elaborating) features of a target object by accumulating features extracted by the plurality of encoders 81-1 to 81-n along the connection direction. For example, the second RNN unit 83-2 accumulates features extracted from the first encoder 81-1 and the second encoder 81-2 to provide a more advanced feature to the second decoder 85-2. Can provide. Accordingly, the second decoder 85-2 can generate the volume feature 95-2 that is more sophisticated than the first decoder 85-1, and the last decoder 85-n is the most sophisticated volume feature 95 -n) can be created. In order to maximize this effect of the RNN module 83, the input multi-view image 91 is composed of a plurality of images having sequential views (eg, a plurality of images having sequential views in a clockwise or counterclockwise direction). It may be desirable to be.

10 illustrates that the RNN module 83 is made of a long short-term memory (LSTM) neural network as an example, but the RNN module 83 may be made of a neural network of another structure having a cyclic structure. In addition, although FIG. 10 illustrates that the RNN units (e.g. 83-1) are connected in one direction, the RNN units (e.g. 83-1) may be connected in both directions.

Next, the plurality of decoders 85-1 to 85-n may generate a plurality of volume features 95-1 to 95-n by decoding an operation result of the RNN module 83. For example, the first decoder 85-1 may generate the first volume feature 95-1 by decoding the operation result of the first RNN unit 83-1, and the second decoder 85-2 The second volume feature 95-2 may be generated by decoding an operation result of the second RNN unit 83-2.

The volume feature (eg 95-1) may indicate shape information on a target object or mean feature data containing geometric characteristics, and the specific data type is determined according to the design method of the decoder (eg 85-1). It can be different. For example, the volume feature (e.g. 95-1) may be voxel data representing rough shape information on a target object, but is not limited thereto.

The decoder (e.g. 85-1) may be composed of, for example, CNN or DCNN, but the scope of the present disclosure is not limited thereto.

Next, the aggregator 87 aggregates the plurality of volume features 95-1 to 95-n generated by the plurality of decoders 85-1 to 85-n to generate a target volume feature 97. ) Can be created. However, a method in which the aggregator 87 aggregates the plurality of volume features 95-1 to 95-n may vary according to exemplary embodiments.

In some embodiments, the aggregator 87 may aggregate the plurality of volume features 95-1 to 95-n with equal weights. For example, the aggregator 87 may generate the target volume feature 97 by performing a weighted sum (or weighted average) operation on the plurality of volume features 95-1 to 95-n with equal weights.

In some other embodiments, the aggregator 87 may aggregate a plurality of volume features 95-1 to 95-n with a differential weight. For example, the aggregator 87 gives a higher weight to a more sophisticated volume feature (eg, gives the second volume feature 95-2 a higher weight than the first volume feature 95-1, and gives the nth volume feature 95-n a higher weight). The highest weight is assigned) and the target volume feature 97 may be generated by performing a weighted sum (or weighted average) operation according to the assigned weight. In this case, a more elaborate target volume feature 97 may be generated to improve the performance of the deep learning network.

Meanwhile, the aggregator 87 may be implemented as a simple arithmetic module that does not include a learnable parameter, or may be implemented as a neural network such as a fully connected layer. When the aggregator 87 is implemented as a neural network, after the values of the plurality of volume features 95-1 to 95-n are amplified or decreased based on the assigned weight, the plurality of volume features 95-1 to 95 -n) may be input to the aggregator 87.

Next, the refiner 89 may restore the 3D voxel model 99 of the target object from the target volume feature 97. The refiner 89 is composed of a neural network, and may perform a role of more elaborately reconstructing the voxel model 99 by refining the target volume feature 97.

As shown, the refiner 89 may be configured in an encoder-decoder structure. However, the scope of the present disclosure is not limited thereto, and the refiner 89 may be configured with a neural network having a different structure.

Meanwhile, the structure of the deep learning network illustrated in FIG. 10 may be modified according to embodiments. For example, the deep learning network may be configured with an encoder-decoder structure that does not include the RNN module 83. As another example, the deep learning network may be configured such that depth information (e.g. 93-1) of the target object is not input to the decoder (e.g. 85-1).

So far, a deep learning-based 3D model restoration method according to a third embodiment of the present disclosure has been described with reference to FIG. 10. As described above, by configuring the deep learning network to have a feature accumulation structure (e.g. RNN 83, aggregator 87), the reconstruction performance of the deep learning network can be further improved.

The second and third embodiments described so far require a multi-view image for a target object, and this may act as a kind of limitation. Hereinafter, a method of alleviating these restrictions through the GAN module will be described with reference to FIG. 11.

11 is an exemplary diagram for describing a method of generating a multi-view image according to some embodiments of the present disclosure. Hereinafter, it will be described with reference to FIG. 11.

The present embodiment relates to a method of generating a multi-view image from a single-view image using a GAN module having a function of translating an image of a first domain into an image of a second domain. In this case, the image of the first domain may be an image corresponding to the first view, and the image of the second domain may be an image corresponding to the second view.

As shown, the GAN module according to the embodiment may include a first generator 101, a second generator 102, a first discriminator 103 and a second discriminator 104.

The first generator 101 may be a module that converts the image 105-1 of the first domain into a fake image 105-2 of the second domain. In addition, the second discriminator 104 may be a module that determines the actual image 106-1 and the fake image 105-2 of the second domain. The first generator 101 and the second discriminator 104 may be complementarily learned through hostile learning.

In addition, the first generator 101 may be further learned using _{a first consistency loss (L CONST A).} The first consistency error (L _CONST A) is calculated based on the difference between the actual image 105-1 input to the first generator 101 and the fake image 105-3 converted through the second generator 102. Can be. In this case, the fake image 105-3 may refer to an image obtained by converting the fake image 105-2 converted through the first generator 101 into an original domain through the second generator 102 again. By learning the first consistency error L _CONST A, the first generator 101 can accurately perform image conversion even if the training data is not composed of an image pair. The first consistency error L _CONST A may be calculated based on, for example, a Euclidian distance, a cosine similarity, and the like, but the scope of the present disclosure is not limited to the examples listed above.

Next, the second generator 102 may be a module that converts the image 106-1 of the second domain into a fake image 106-2 of the first domain. In addition, the first discriminator 103 may be a module that determines the actual image 105-1 and the fake image 106-2 of the first domain. The second generator 102 and the first discriminator 103 may also be complementarily learned through hostile learning.

In addition, the second generator 102 may be further learned using _{the second consistency error (L CONST B).} The learning method of the second generator 102 is similar to that of the first generator 101, so further description will be omitted.

When the GAN module is built through hostile learning, image conversion may be performed through the first generator 101 or the second generator 102. For example, an image of a first domain (ie, an image corresponding to a first view) may be converted into an image of a second domain (ie, an image corresponding to a second view) through the first generator 101. Alternatively, an image of the second domain (ie, an image corresponding to the second view) may be converted into an image of the first domain (ie, an image corresponding to the first view) through the second generator 102. Therefore, if one or more GAN modules having the illustrated structure are used, a multi-view image can be easily generated from a single-view image of a target object. For example, if only the first image corresponding to the first view is given, the first image may be converted into a second image corresponding to the second view through the first GAN module, and the first image ( Alternatively, the second image) may be converted into a third image corresponding to the third view. In addition, a multi-view image of the target object may be configured with the first to third images.

Meanwhile, the GAN module may be implemented in a structure different from that illustrated in the example of FIG. 11. For example, the GAN module is applied to GANs of different structures with image conversion functions such as Disco-GAN, UNIT (Unsupervised Image-to-Image Translation), MUNIT (Multimodal Unsupervised Image-to-Image Translation), starGAN, fusionGan, etc. It may be implemented on the basis of.

So far, a method of generating a multi-view image according to some embodiments of the present disclosure has been described with reference to FIG. 11. According to the above-described method, a multi-view image for a target object can be easily configured by using a GAN module having an image conversion function.

Hereinafter, an exemplary computing device 1000 capable of implementing the restoration device 10 according to some embodiments of the present disclosure will be described with reference to FIG. 12.

12 is an exemplary hardware configuration diagram illustrating the computing device 1000.

As shown in FIG. 12, the computing device 1000 is a memory for loading a computer program executed by one or more processors 1100, bus 1300, communication interface 1400, and processor 1100. 1200) and a storage 1500 for storing the computer program 1600. However, only the components related to the embodiment of the present disclosure are shown in FIG. 12. Accordingly, those of ordinary skill in the art to which the present disclosure pertains can recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 12. That is, the computing device 1000 may further include various components in addition to the components illustrated in FIG. 12. In addition, in some cases, the computing device 1000 may be implemented in a form in which some of the components illustrated in FIG. 12 are omitted.

The processor 1100 controls the overall operation of each component of the computing device 1000. The processor 1100 includes at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any type of processor well known in the technical field of the present disclosure. It can be configured to include. Also, the processor 1100 may perform an operation on at least one application or program for executing the operation/method according to the embodiments of the present disclosure. The computing device 1000 may include one or more processors.

The memory 1200 stores various types of data, commands, and/or information. The memory 1200 may load one or more programs 1600 from the storage 1500 in order to execute an operation/method according to embodiments of the present disclosure. The memory 1200 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

The bus 1300 provides a communication function between components of the computing device 1000. The bus 1300 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The communication interface 1400 supports wired/wireless Internet communication of the computing device 1000. In addition, the communication interface 1400 may support various communication methods other than Internet communication. To this end, the communication interface 1400 may be configured to include a communication module well known in the art. In some cases, the communication interface 1400 may be omitted.

The storage 1500 may non-temporarily store the one or more programs 1600. The storage 1500 is a nonvolatile memory such as a ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, etc. It may be configured to include any known computer-readable recording medium.

The computer program 1600 may include one or more instructions that when loaded into the memory 1200 cause the processor 1100 to perform an operation/method according to various embodiments of the present disclosure. That is, the processor 1100 may perform operations/methods according to various embodiments of the present disclosure by executing the one or more instructions.

For example, the computer program 1600 may include instructions to perform an operation of receiving a two-dimensional image of a target object and an operation of restoring a three-dimensional voxel model from a two-dimensional image input through a deep learning network. I can. In this case, the restoration device 10 according to some embodiments of the present disclosure may be implemented through the computing device 1000.

So far, an exemplary computing device 1000 capable of implementing the restoration device 10 according to some embodiments of the present disclosure has been described with reference to FIG. 12.

The technical idea of the present disclosure described with reference to FIGS. 1 to 12 so far may be implemented as computer-readable codes on a computer-readable medium. The computer-readable recording medium is, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). I can. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

In the above, even if all the components constituting the embodiments of the present disclosure are described as being combined into one or operating in combination, the technical idea of the present disclosure is not necessarily limited to these embodiments. That is, as long as it is within the scope of the object of the present disclosure, one or more of the components may be selectively combined and operated.

Although the operations are illustrated in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order shown or in a sequential order, or all illustrated operations must be executed to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the above-described embodiments should not be understood as necessarily requiring such separation, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is.

Although the embodiments of the present disclosure have been described with reference to the accompanying drawings above, those of ordinary skill in the art to which the present disclosure pertains may implement the present disclosure in other specific forms without changing the technical idea or essential features. I can understand that there is. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the technical ideas defined by the present disclosure.

Claims (9)

A memory that stores one or more instructions; And
By executing the stored one or more instructions,
Acquire a two-dimensional input image for the target object,
Including a processor for restoring a three-dimensional voxel model of the target object from the input image through a deep learning network,
The deep learning network includes a restorer configured to receive the input image and restore and output the voxel model and a discriminator to determine an actual image and a fake image,
The restorer includes a first error calculated based on a difference between the restored voxel model and a correct voxel model, a second error calculated based on a determination result of the discriminator for the restored voxel model, and the restored voxel model Learning through a third error calculated based on a difference between a two-dimensional projection image generated through a projection operation for and the input image,
A device that restores a 3D model from a 2D image.
The method of claim 1,
The restorer is further learned based on the fourth error,
The fourth error is calculated based on a difference between an outline extracted from the restored voxel model and an outline extracted from the correct voxel model,
A device that restores a 3D model from a 2D image.
The method of claim 1,
The deep learning network further includes a classifier for classifying a class of the target object,
The restorer receives the correct answer class information of the target object and the input image, restores the voxel model, and outputs it,
The restorer is further learned based on a fourth error,
The fourth error is calculated through a process of obtaining prediction class information of the projection image through the classifier and comparing the prediction class information with the correct answer class information,
A device that restores a 3D model from a 2D image.
The method of claim 1,
The input image is a multi-view image including a plurality of images corresponding to different views of the target object,
The restorer includes a plurality of encoders, a plurality of decoders, an aggregator and a refiner,
A first encoder among the plurality of encoders receives a first image corresponding to a first view from among the multi-view images and extracts a first feature of the target object, and a second encoder among the plurality of encoders receives the multi-view image. Receive a second image corresponding to a second view from among the images and extract a second feature of the target object,
A first decoder among the plurality of decoders generates a first volume feature by performing a decoding operation based on the first feature, and a second decoder among the plurality of decoders performs a decoding operation based on the second feature. To create a second volume feature,
The aggregator aggregates a plurality of volume features including the first volume feature and the second volume feature,
The refiner is composed of a neural network to restore the voxel model from the aggregated volume feature,
A device that restores a 3D model from a 2D image.
The method of claim 4,
The first decoder further receives and decodes depth information on the first image to generate the first volume feature,
The second decoder further receives and decodes depth information on the second image to generate the second volume feature,
The deep learning network further includes a Generative Adversarial Networks (GAN) module for converting an image corresponding to the first view into an image corresponding to the second view,
The processor,
Configuring the multi-view image by converting the first image into the second image through the GAN module,
A device that restores a 3D model from a 2D image.
The method of claim 4,
The restorer further includes a recurrent neural networks (RNN) module located between the plurality of encoders and the plurality of decoders,
The RNN module includes a first RNN unit located between the first encoder and the first decoder, and a second RNN unit located between the second encoder and the second decoder,
The first RNN unit performs a first neural network operation on the first feature, and provides a result of the first neural network operation to the second RNN unit and the first decoder,
The second RNN unit performs a second neural network operation on the second feature and the result of the first neural network operation, and provides the second neural network operation result to the second decoder,
The first decoder generates the first volume feature by decoding a result of the first neural network operation,
The second decoder generates the second volume feature by decoding a result of the second neural network operation,
A device that restores a 3D model from a 2D image.
A memory that stores one or more instructions; And
By executing the one or more instructions,
Receive a multi-view image of the target object,
Extracting two-dimensional depth information from the multi-view image,
Generate volume data for the target object based on the depth information,
Including a processor for receiving the volume data through a deep learning network, restoring and outputting a three-dimensional voxel model for the target object,
The deep learning network includes a first error calculated based on a difference between the restored voxel model and the correct voxel model, and a two-dimensional projection image and the input image generated through a projection operation on the restored voxel model. Learned through the second error calculated based on the difference with
A device that restores a 3D model from a 2D image.
The method of claim 7,
The deep learning network includes an encoder for extracting features of the target object from the volume data, a decoder for generating the voxel model by decoding the extracted features, and a discriminator for determining an actual voxel model and a fake voxel model. Including,
The encoder and the decoder are trained through the first error, the second error, and a third error calculated based on the determination result of the discriminator for the restored voxel model,
A device that restores a 3D model from a 2D image.
The method of claim 7,
The deep learning network includes an encoder for extracting features of the target object from the volume data and a decoder for generating the voxel model by decoding the extracted features,
The multi-view image includes a first image corresponding to a first view of the target object and a second image corresponding to a second view,
The processor,
Generating first depth information from the first image through a Generative Adversarial Networks (GAN) module,
Generating second depth information from the second image through the GAN module,
A device that restores a 3D model from a 2D image.

Images