KR102412373B1 - Apparatus and method for generating 3D data using multi-view X-ray data - Google Patents

Abstract

A generating device is provided. The generating device may include a generator configured to generate a three-dimensional image of the object when a two-dimensional image of the object is input by using the machine-learned generation model.

Description

Apparatus and method for generating 3D data using multi-view X-ray data

The present invention relates to an apparatus and method for generating a three-dimensional image using two-dimensional images taken from multiple angles.

3D X-ray technology is being introduced to accurately grasp the inside of an object.

Various algorithms for identifying and processing the inside of various objects using 3D X-ray technology can be applied. Some of these algorithms can be created through machine learning. At this time, it will be of great help if the existing machine learning model learned in the existing 2D X-ray technology stage is applied to the 3D X-ray technology as it is.

However, existing methodologies for generating 3D data from multi-view 2D (two-dimensional) data, such as lateral two-dimensional data and planar two-dimensional data, have disadvantages in that data for multiple views is required and inaccurate. Due to these problems, it is practically difficult to convert the previously learned 2D X-ray processing model into a 3D X-ray processing model.

Korean Patent Application Laid-Open No. 2020-0100190 discloses a technique for determining pixel coordinates for image conversion and a memory address for storing the converted image data.

Korean Patent Publication No. 2020-0100190

An object of the present invention is to provide a generating apparatus and a generating method capable of accurately generating a three-dimensional image using a plurality of two-dimensional images of multiple viewpoints.

The generating apparatus of the present invention may include a generator configured to generate a three-dimensional image of the object when a two-dimensional image of the object is input by using the machine-learned generation model.

The generating method of the present invention comprises: a projection step of generating a plurality of two-dimensional second images in which the first image is projected from multiple angles when a three-dimensional first image is input; a generating step of generating a three-dimensional third image when the plurality of second images are input by using the machine-learned generative model; Comparing the first image and the third image, calculating a loss (loss) in units of pixels, and a correction step of learning the generation model in a direction in which the loss is minimized; may include.

According to the present invention, a multi-view 2D X-ray image can be accurately converted into a 3D X-ray image.

Through this, an environment in which various image processing models learned and generated based on an existing two-dimensional image can be grafted onto a three-dimensional image can be provided.

In addition, when the generating apparatus of the present invention is applied to an existing system for analyzing a two-dimensional image, a three-dimensional image may be acquired and analyzed together with the two-dimensional image. Accordingly, a 3D image analysis may be added as an auxiliary analysis to the analysis of the 2D image.

1 is a block diagram showing a generating device of the present invention.
Fig. 2 is a schematic diagram showing the operation of the generating device.
3 is a flowchart illustrating a method for generating according to the present invention.
4 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present specification, duplicate descriptions of the same components will be omitted.

Also, in this specification, when it is said that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but other elements in the middle It should be understood that there may be On the other hand, in this specification, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that another element does not exist in the middle.

In addition, the terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention.

Also, in this specification, the singular expression may include the plural expression unless the context clearly dictates otherwise.

Also, in this specification, terms such as 'include' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more It is to be understood that the existence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof, is not precluded in advance.

Also in this specification, the term 'and/or' includes a combination of a plurality of described items or any item of a plurality of described items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

1 is a block diagram showing a generating apparatus 100 of the present invention. 2 is a schematic diagram illustrating an operation of the generating device 100 .

The generating apparatus 100 illustrated in FIG. 1 may include a projecting unit 110 , a generating unit 130 , and a correcting unit 150 .

The generator 130 may generate a three-dimensional image of the object when a two-dimensional image of the object is input by using the machine-learned generation model.

The generative model may generate a 3D image of the specific target by using a plurality of 2D images obtained by photographing the specific target from various angles (multi-viewpoint). Many errors can be involved in this process. In order to improve the effectiveness of a technology for converting a two-dimensional image into a three-dimensional image, the error needs to be minimized. For this, a large amount of training data for the generative model is required.

Numerous training data may exist. However, when accuracy is considered, the amount of actual training data may drop sharply.

Therefore, it is necessary to generate a large amount of learning data satisfying the accuracy of the setting range.

The projection unit 110 may be used to obtain training data for the generative model.

When the three-dimensional first image 10 is input, the projection unit 110 may generate a plurality of two-dimensional second images 20 in which the first image 10 is projected from various angles.

For example, the projection unit 110 may rotate the first image 10 at various angles with the center of the three-dimensional first image 10 as the rotation center. For example, in the virtual space formed by the x-axis, y-axis, and z-axis orthogonal to each other, the projection unit 110 variously displays the first image 10 based on at least one of the x-axis, y-axis, and z-axis. can be rotated The projection unit 110 may capture a two-dimensional image of the first image 10 viewed from the set position whenever the first image 10 is rotated by a set angle. The two-dimensional image captured in this way may correspond to the second image 20 .

According to the projection unit 110 , a plurality of 2D learning images for the generation unit 130 may be generated using the 3D image.

The plurality of second images 20 obtained from the single first image 10 may include all of a front image, a side image, and a planar image of the first image 10 . The second image 20 basically includes a front image, a side image, and a planar image, and may additionally include other images of various angles.

When a plurality of second images 20 obtained from a single three-dimensional image are input, the generator 130 may generate the three-dimensional third image 30 by using the generation model.

According to the projection unit 110 and the generation unit 130 , the second image 20 is generated based on the first image 10 , and the third image 30 is generated based on the second image 20 . may have been created. According to this, the first image 10 and the third image 30 corresponding to the three-dimensional image should be the same, but in reality, this is not the case. In other words, if three-dimensional A is made into two-dimensional B, and then two-dimensional B is made three-dimensional, it should be A, but in reality, C different from A can be created. In this case, the generation model needs to be corrected so that C generated by the generation unit 130 tracks A.

The compensator 150 may compare the first image 10 and the third image 30 .

The corrector 150 may correct the generated model so that the third image 30 follows the first image 10 . For example, the compensator 150 may calculate a loss in units of pixels by comparing the first image 10 and the third image 30 . The compensator 150 may train the generative model in a direction in which the corresponding loss is minimized.

For example, the compensator 150 compensates for the loss in units of pixels for all coordinates in the image by squaring the error value between each pixel value of the first image 10 and each pixel value of the third image 30 . can be calculated. In this case, the pixel value may include a discrete value indicating the color, contrast, or other property of the pixel.

The correction unit 150 may calculate a mean squared error by averaging the losses calculated for all coordinates.

The corrector 150 may apply the mean square error to the generated model using an error back propagation method. The compensator 150 may train the generative model in a direction in which the mean square error is minimized.

According to the generating apparatus 100 of the present invention, the object restoration ability of the generating unit 130, that is, the ability to accurately convert a two-dimensional image into a three-dimensional image, is improved through the projection unit 110 and the correction unit 150 . can be

According to the generator 130 capable of reliably converting a two-dimensional image into a three-dimensional image, a method for applying various deep learning models learned on an existing two-dimensional image to a three-dimensional image will be realized. can

As an example, the two-dimensional second image 20 may be input to a specific deep learning model learned from various existing two-dimensional images. When a normal two-dimensional image is input to a specific deep learning model, a two-dimensional effect image to which various effects are added can be generated based on the existing learning results.

The generator 130 may obtain the plurality of second images 20 and output data of the deep learning model.

The generator 130 may restore the first image 10 by using the plurality of second images 20 . The generator 130 may add output data to the restored first image 10 . The third image 30 generated by the generator 130 may include the first image 10 restored using the second image 20 . At this time, since the restored first image 10 may not be completely the same as the original first image 10, it was named as the third image 30 in the sense that it is partly different from the first image 10 previously. to ventilate The generator 130 may receive the second image 20 as an input and add output data output from the deep learning model to the newly restored first image 10 (corresponding to the third image 30 ).

According to this embodiment, in order to use a deep learning model that is difficult to apply to a 3D image, the 3D image may be converted into a 2D image by the projection unit 110 once. A deep learning model may be normally applied to the two-dimensional image converted by the projection unit 110 . The application result, processing result, output result, etc. of the deep learning model may be reflected in the two-dimensional image. The generator 130 may restore the two-dimensional image in which the output result of the deep learning model is reflected to the original first image 10 . In this case, the restored first image 10 may be in a state in which the application result of the deep learning model is reflected.

According to this embodiment, various deep learning models learned using an existing two-dimensional image may be used for a three-dimensional image.

3 is a flow chart showing a generating method of the present invention.

The generating method shown in FIG. 3 may be performed by the generating apparatus 100 of FIG. 1 .

The generating method may include a transparent step (S510), a generating step (S520), and a correction step (S530).

In the transparent step S510 , when the three-dimensional first image 10 is input, a plurality of two-dimensional second images 20 in which the first image 10 is projected from various angles may be generated. The projecting step ( S510 ) may be performed by the projection unit 110 .

In the generating step ( S520 ), a three-dimensional third image 30 may be generated when a plurality of second images 20 are input by using the machine-learned generative model. The generating step ( S520 ) may be performed by the generating unit 130 .

In the correction step ( S530 ), the first image 10 and the third image 30 may be compared to calculate a loss in units of pixels, and the generation model may be trained in a direction in which the loss is minimized. The correction step ( S530 ) may be performed by the correction unit 150 . According to the corrector 150 , the third image 30 may become similar to the first image 10 as learning progresses, and may have the status of the restored first image 10 .

An artificial intelligence model (generative model) may receive a plurality of two-dimensional images from multiple viewpoints and output one three-dimensional image. The learning data acquisition method acquires a plurality of 2D images by projecting one 3D (three-dimensional) X-RAY scan data based on each viewpoint (including a front image, a side image, and a plane image). The artificial intelligence model can learn with the goal of reconstructing the original 3D image as naturally as possible by receiving the acquired 2D image data. Artificial intelligence that has been trained can generate 3D stereoscopic data from multi-view 2D input data.

The projection unit 110 secures a 3D image through 3D X-RAY. A plurality of 2D images are obtained by performing 2D projection with respect to each viewpoint (eg, xy plane, yz plane, etc.) with respect to the acquired 3D image.

The generator 130 may create a generative model-based artificial intelligence in which a plurality of 2D images are input and 3D images are output.

The generator 130 receives a plurality of 2D images as inputs of artificial intelligence and generates one 3D image.

The compensator 150 compares the created 3D image (the third image 30, the restored first image 10) and the original 3D image (the original first image 10) to obtain a pixel-unit loss (Loss) , and train the generative model in a direction that minimizes the loss. Here, the loss function uses a mean squared error. For example, if the pixel value at the (1,5,3) position of the original data is 0.3, and the pixel value at the (1,5,3) position of the data generated through the model is 0.5, the loss of pixel unit occurred here (Squared Error) becomes (0.3-0.5) ² , and in this way, the loss in pixels for all coordinates is obtained, and the mean squared error is measured by averaging them.

The correction unit 150 learns to minimize the error by applying the error back-propagation method to the artificial intelligence with the loss measured in the above manner.

4 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 4 may be a device described herein (eg, the generating device 100 , etc.).

In the embodiment of FIG. 4 , the computing device TN100 may include at least one processor TN110 , a transceiver device TN120 , and a memory TN130 . In addition, the computing device TN100 may further include a storage device TN140 , an input interface device TN150 , an output interface device TN160 , and the like. Components included in the computing device TN100 may be connected by a bus TN170 to communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to an embodiment of the present invention are performed. The processor TN110 may be configured to implement procedures, functions, and methods described in connection with an embodiment of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may be configured as at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory TN130 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver TN120 may transmit or receive a wired signal or a wireless signal. The transceiver TN120 may be connected to a network to perform communication.

On the other hand, the embodiment of the present invention is not implemented only through the apparatus and/or method described so far, and a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded may be implemented. And, such an implementation can be easily implemented by those skilled in the art from the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also presented. It belongs to the scope of the invention.

10...First image 20...Second image
30...third image 100...generating device
110...projection unit 130...creation unit
150...correction unit

Claims (8)

A generator that generates a three-dimensional image of the object when a two-dimensional image of the object is input by using the machine-learned generation model;
When a three-dimensional first image is input, a projection unit for generating a plurality of two-dimensional second images in which the first image is projected from various angles is provided;
When the plurality of second images are input, the generating unit generates a three-dimensional third image by using the generation model,
A correction unit is provided to compare the first image and the third image to calculate a loss in units of pixels,
The compensator trains the generative model in a direction in which the loss is minimized.
delete delete delete A generator that generates a three-dimensional image of the object when a two-dimensional image of the object is input by using the machine-learned generation model;
When a three-dimensional first image is input, a projection unit for generating a plurality of two-dimensional second images in which the first image is projected from various angles is provided;
When the plurality of second images are input, the generating unit generates a three-dimensional third image by using the generation model,
A correction unit is provided for calculating the loss in units of pixels for all coordinates in a manner of squaring an error value between each pixel value of the first image and each pixel value of the third image;
The correction unit calculates a mean squared error by averaging the losses calculated for all coordinates,
The compensator applies the mean square error to the generative model using an error back propagation method, and trains the generative model in a direction in which the mean square error is minimized.
The method of claim 1,
The plurality of second images includes all of a front image, a side image, and a planar image of the first image.
A generator that generates a three-dimensional image of the object when a two-dimensional image of the object is input by using the machine-learned generation model;
When a three-dimensional first image is input, a projection unit for generating a plurality of two-dimensional second images in which the first image is projected from various angles is provided;
The second image is input to the deep learning model trained as a two-dimensional image,
The generating unit obtains the plurality of second images and output data of the deep learning model,
The generator restores the first image by using the plurality of second images, and adds the output data to the restored first image.
A generating method performed by a generating device, comprising:
a projection step of generating a plurality of two-dimensional second images in which the first image is projected from multiple angles when a three-dimensional first image is input;
a generating step of generating a three-dimensional third image when the plurality of second images are input using a machine-learned generating model;
a correction step of calculating a loss in units of pixels by comparing the first image with the third image, and learning the generative model in a direction in which the loss is minimized;
A creation method comprising

Images