WO2021100906A1

WO2021100906A1 - Method for displaying virtual x-ray image by using deep neural network

Info

Publication number: WO2021100906A1
Application number: PCT/KR2019/015980
Authority: WO
Inventors: 오주영
Original assignee: 오주영
Priority date: 2019-11-20
Filing date: 2019-11-20
Publication date: 2021-05-27

Abstract

A method for displaying a virtual X-ray image by using a deep neural network is disclosed. In one embodiment, disclosed is a method for displaying a virtual X-ray image by using a deep neural network, comprising the steps of: acquiring a learning image database having a camera image and an X-ray image that are matched by location on a human body; generating an X-ray image estimation model by repetitively performing deep learning on camera images and X-ray images through the learning image database; estimating, on the basis of a camera image of a real patient or a practice subject, the X-ray image with the highest similarity through the estimation model; and synthesizing the estimated X-ray image with the camera image of the real patient or the practice subject as a predicted X-ray image of the camera image of the real patient or the practice subject, thereby displaying the synthesized image.

Description

Virtual X-ray image display method using deep neural network

The present invention relates to a method of displaying a virtual X-ray image using a deep neural network.

Radiography is an essential medical technique that can quickly and reliably grasp a patient's condition, but it is also a dangerous technique that inflicts certain harm to the human body when used. Therefore, accurate and safe radiography and accurate results are the biggest task of radiologists. However, since the skill of the radiologist's imaging technique inevitably can only be obtained through repeated failures and re-shoots (fail and repeat) for a number of patients, etc., damage to the public due to repeated re-shooting occurs. do. Accordingly, national institutions centered on the Ministry of Food and Drug Safety are striving to reduce the amount of medical radiation exposure.

The present invention provides a method for displaying a virtual X-ray image using a deep neural network.

The method for displaying a virtual X-ray image using a deep neural network according to the present invention is a method for displaying a virtual X-ray image using a deep neural network, the method comprising: acquiring a training image database having camera images and X-ray images matched for each location of a human body; Generating an X-ray image estimation model by repeatedly deeply learning the camera image and the X-ray image through the training image database; Estimating an X-ray image with the highest similarity from the camera image of an actual patient or trainee through the estimation model; And synthesizing and displaying the estimated X-ray image as a predicted X-ray image of the camera image of the actual patient or the trainee, on the camera image of the real patient or the trainee.

In addition, the obtaining of the training image database includes correcting a camera image captured by a main camera located at least one side of a camera device other than an X-ray tube or an X-ray imaging device of the x-ray imaging device based on the x-ray image of the x-ray imaging device. step; Matching the X-ray image and the camera image; And storing the X-ray image and the camera image as a pair in the training image database.

In addition, the obtaining of the learning image database includes modeling the 3D image of the human body and the shape of the bone, and positioning the 3D image and the shape of the bone in a radiography posture and direction to obtain the X-ray image and the camera image. I can.

In addition, in obtaining the learning image database, the 3D image of the human body and the shape of the bone are modeled, and the 3D image of the human body and the shape of the bone are input to a Generative Adversarial Network (GANs) algorithm to It is possible to change the style to a style close to the image and the X-ray image of, and learn in-depth one-to-one with the image of the human body and the X-ray image that are close to the reality in which the style is changed.

In addition, the step of synthesizing and displaying the X-ray image matched with the selected camera image as a predicted X-ray image on the camera image of the actual patient or the subject is correcting the selected camera image based on the camera image of the actual patient or trainee. , Correcting and displaying the matched X-ray image to match the correction of the selected camera image.

In addition, the step of synthesizing and displaying the X-ray image matched with the selected camera image as a predicted X-ray image to the camera image of the actual patient or the trainee is to match the thermal image of the auxiliary camera with the camera image of the actual patient or trainee. Thus, it may be to perform division of the human body region.

In addition, if there is a change in the color above the reference value, it is determined that the actual patient or the trainee is wearing clothes, and if there is only a change less than the reference value, it is determined that the actual patient or the trainee is not wearing clothes, and the learning It may be to extract and display an X-ray image of a corresponding situation from an image database.

The virtual X-ray generation and display method using the deep neural network according to the present invention can be generated by estimating and generating an X-ray image according to an estimation algorithm learned in advance through a learning image database, and by predicting and displaying the X-ray image, there is no exposure to radiation. The patient may check the posture and position for radiography before imaging, or the trainee may be allowed to repeatedly practice X-ray imaging without radiation exposure.

In addition, the method of generating and displaying virtual X-rays using a deep neural network according to the present invention is to change the style of the body image to a real body image and an X-ray image through a generative hostile neural network, and change the style of the body image and the X-ray image. By learning, you can display an image similar to the real thing.

1 is a flowchart illustrating a method of displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

2 is a flowchart showing detailed steps of acquiring a training image database in a method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

3A is a schematic diagram illustrating a photographing equipment used in a method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

3B is a conceptual diagram illustrating a process of performing correction of a camera image in a method of displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

3C and 3D are conceptual diagrams illustrating a step of matching a camera image and a radiation image to acquire a training image in a method of displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

3E is a conceptual diagram illustrating a process of creating a speculative model through learning by applying a generative adversarial neural network to a 3D model in a method of displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

4A to 4D are conceptual diagrams illustrating a step of generating and displaying a predictable radiographic image of a camera image of an actual patient in a method of displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

5 is a conceptual diagram illustrating a step of displaying a guide line with a camera image of an actual patient in a method of displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

The preferred embodiments of the present invention will be described in detail with reference to the drawings to the extent that those of ordinary skill in the art can easily implement the present invention.

1 is a flowchart of a method and method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention. 2 is a flowchart illustrating detailed steps of acquiring a training image database in a method and method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

First, referring to FIG. 1, a method and method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention include obtaining a learning image database (S1), estimating an X-ray image through a deep learning model (S2), and predicting It may include an X-ray image display step (S3).

Meanwhile, referring to FIGS. 1 and 2 together, the learning image database acquisition step (S1) includes a camera image generation and pre-processing step (S11), a camera image and an X-ray image matching step (S12), and a camera/X-ray image pair storage. It may be accomplished including step S13.

3A is a schematic diagram illustrating a method of displaying a virtual X-ray image using a deep neural network and photographing equipment used in the method according to an embodiment of the present invention. 3B is a schematic diagram illustrating a method for displaying a virtual X-ray image using a deep neural network and photographing equipment used in the method according to an embodiment of the present invention. 3B is a conceptual diagram illustrating a process of performing distortion error correction of a camera image in a method and method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention. 3C and 3D are conceptual diagrams illustrating a step of matching a camera image and an X-ray image to obtain a training image in a method and method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention. 3E is a conceptual diagram illustrating a process of changing a style and performing learning by applying a generative hostile neural network to a 3D model in a method of displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

First, the camera image correction step (S11) of the learning image database acquisition step (S1) may start with acquiring a camera image and an X-ray image together through the photographing equipment of FIG. 3A. Here, as shown in FIG. 3A, the camera image may be photographed through a main camera or an auxiliary camera (or sensor) located on the side of the X-ray photographing apparatus. In addition, the main camera or the auxiliary camera may use a visible light or a thermal imaging camera for photographing, or, in some cases, a TOF camera or an ultrasonic sensor. In addition, it is possible to extract the training image from a separate database that has already secured camera images and x-ray images.

Meanwhile, the X-ray image may be photographed through an X-ray imaging device located at the center of the equipment. However, since the main camera and the auxiliary camera are different in angles from the actual X-ray photographing apparatus, distortion may occur in the captured camera image. In this case, as shown in FIG. 3B, the actual camera image is compared with the X-ray image, and the shape of the camera image is corrected based on the X-ray image, so that correction to reduce the distortion error may be performed.

In addition, as shown in FIG. 3C, the entire size of the training image to be obtained is cut out to the size of a cassette (square shape), because the training image is limited to a cassette-sized area for learning. i) When using a cassette, the image can be cut and saved by recognizing the size and shape of the cassette using edge detection or shape recognition, and ii) not using a cassette or completely removing the cassette with the body. If it is not covered, the light area of the X-ray irradiation field (when shooting, the area is set by emitting a square light on the area), the value set according to the existing imaging method, and the area value calculated back to the area of the X-ray image Alternatively, the cassette position and area can be calculated by manual setting.

In addition, the resulting final post-processed human body image may be matched with the captured X-ray image, and the final image may be stored.

And, as shown in FIG. 3D, after matching the camera image and the X-ray image according to the position and posture of each human body, filtering into an image having excellent feature points, learning is performed as a post-processed image, or in case Accordingly, when learning is performed through post-processed images processed using an original visible light image, an auxiliary camera (thermal image), and an auxiliary camera (thermal image), it is possible to obtain a final training image database.

Meanwhile, in the step S1 of acquiring the learning image database, a method of obtaining an image by modeling and rendering a 3D image of a human body and a model of a bone may be used in addition to the image through actual radiation irradiation. It is possible to acquire a larger amount of training image data than to acquire an image by repeating actual photographing by making a posture for radiography through a 3D image of the human body and a model of a bone, adjusting the position, and matching the X-ray image according to it. have. In addition, it is possible to repeatedly process and store the above 3D image and bone model by body type (age, race, sex, height, weight) based on the existing actual patient data. In addition, it is possible to perform running (deep learning) while changing the location, angle, and size of images stored in the database.

On the other hand, as shown in Fig. 3e, the learning image database acquisition step (S1) is performed through Generative Adversarial Networks (GANs) before learning the 3D image of the human body and the 3D bone image. The style can be changed as an X-ray image. Specifically, a generative adversarial neural network is known as a deep learning algorithm that generates and learns data through competition between a generator and a discriminator. In addition, when the 3D image of the human body and the image of the 3D bone mentioned above are input to the algorithm, an image of an actual human body and an actual X-ray image having a similar shape but with a changed style may be generated. Accordingly, in the learning image database acquisition step (S1), an image of a human body and an X-ray image whose style has been changed through a generative hostile neural network are generated, and matched 1:1 through this. Through this, if deep learning is performed again through a generative adversarial neural network algorithm or a convolutional neural network, a picture close to the real can be generated.

4A to 4D are conceptual diagrams illustrating a step of matching predictable X-ray images with a camera image of an actual patient in a method and method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention. 5 is a conceptual diagram illustrating a step of displaying a guide line with a camera image of an actual patient in a method and method for displaying a virtual X-ray image using a deep neural network according to an embodiment of the present invention.

Referring to FIGS. 4A to 4D together with FIG. 1, the step of estimating an x-ray image through a deep learning model (S2) is a radiography of an actual patient or a camera photographed in the radiography practice of the trainee in the training process using the corresponding equipment. This is a step of inferring an X-ray image from the image through the deep learning model. Here, since the camera image acquires only a visible ray image, since X-rays are not used, the patient or the trainee may not be exposed to radiation. In addition, as shown in FIG. 4A, an image is acquired and processed in accordance with an existing learning method and a guessing algorithm through edge detection or a thermal image camera screen in the camera image of the patient or the trainee, and similar X-ray images are obtained from the database. I can guess.

In addition, the predicted X-ray image display step (S3) is a step of displaying an X-ray image matched with the searched camera image on the screen. Here, the position of the patient's or the subject's human body may be different in position when compared with the human body (or 3D image) at the time of capturing a camera image for acquiring the database. In this case, as shown in FIGS. 4B and 4C, the camera image of the database is compared with the camera image of the actual patient or the trainee, and the camera image of the patient or the trainee is learned before estimating the X-ray image through the deep learning model. It is possible to transform it to fit the camera image. This is based on an image registration algorithm that calculates the error distance of pixel values between two images and improves the error by transforming the image in a direction to reduce the error. In addition, an X-ray image matching the camera image of the database may also be deformed accordingly, and as a result, an expected X-ray image corresponding to the camera image of an actual patient or trainee may be generated.

On the other hand, the camera image of the main camera and the thermal image of the auxiliary camera may help to determine the human body position of an actual patient or trainee. Specifically, it is possible to measure the temperature of the skin through the thermal image of the secondary camera and visualize it as a color, and to perform the segmentation of the human body more accurately by coloring the image of the main camera obtained through visible light. I can.

In addition, if there is no sudden change above the reference value in the color displayed through visible light, it is determined that the actual patient or the trainee is not wearing clothes for the human body, and the model of the existing database can be used as it is. On the other hand, if there is a sudden change in the color of which the change is more than the reference value, the step of calculating the ratio of the range of the clothing part from the actually photographed image may be performed. And if the ratio of clothes is less than the reference value, the clothes part is removed and the prediction model is applied only to the rest of the human body. On the other hand, when the ratio of clothes is greater than or equal to the reference value, it may be possible to search and apply a prediction model that assumes the presence of clothes from a database defined in advance.

Meanwhile, referring to FIG. 5, in addition to the predicted X-ray image, a line for guiding a position suitable for radiography may be further displayed for a camera image of an actual patient or a trainee. Here, the guide line is generated from the camera image stored in the database, and is displayed as a line that is distinct from the actual image, so that the patient or the trainee may change the position of his or her body accordingly. In addition, the degree of similarity to the area to be additionally photographed is measured, and if the similarity is low, the similarity score is displayed to the patient or the trainee, indicating that the area to be photographed is not the target area or the posture is incorrect. Or, you can reconfirm with the trainee.

Further, in this case, when the region is not a photographing target region, it is possible to determine (recognize) a location of the current photographing region in an existing image database, and propose a location of the photographing target region as a display.

What has been described above is only one embodiment for implementing the method of displaying a virtual X-ray image using a deep neural network according to the present invention, and the present invention is not limited to the above embodiment, as claimed in the claims below. Without departing from the gist of the present invention, anyone of ordinary skill in the field to which the present invention pertains will have the technical spirit of the present invention to the extent that various changes can be implemented.

Claims

A virtual X-ray image display method using a deep neural network,

Obtaining a training image database having camera images and X-ray images matched for each location of the human body;

Generating an X-ray image estimation model by repeatedly deeply learning the camera image and the X-ray image through the training image database;

Estimating an X-ray image with the highest similarity from the camera image of an actual patient or trainee through the estimation model; And

A virtual X-ray image display method using a deep neural network comprising the step of synthesizing and displaying the estimated X-ray image as a predicted X-ray image of the camera image of the real patient or the trainee.
The method of claim 1,

The step of obtaining the training image database

Correcting a camera image captured by a main camera located at least on one side of a camera device other than an X-ray tube of the X-ray imaging device or an X-ray imaging device based on an X-ray image of the X-ray imaging device;

Matching the X-ray image and the camera image; And

A method of displaying a virtual X-ray image using a deep neural network comprising the step of storing in the training image database as a pair of the X-ray image and the camera image.
The method of claim 1,

The step of obtaining the training image database

A virtual X-ray image display method using a deep neural network to obtain the X-ray image and camera image by modeling the 3D image of the human body and the shape of the bone, and positioning the 3D image and the shape of the bone in a radiography posture and direction.
The method of claim 1,

The step of obtaining the training image database

Model the 3D image of the human body and the shape of the bone, input the 3D image of the human body and the shape of the bone to the Generative Adversarial Network (GANs) algorithm, and change the style to a style close to the actual human body image and X-ray image. ,

A method of displaying a virtual X-ray image using a deep neural network for deep learning of the human body image and the X-ray image in which the style has been changed, which is close to the real one.
The method of claim 1,

The step of synthesizing and displaying the X-ray image matched with the selected camera image as a predicted X-ray image on the camera image of the real patient or the subject

A virtual X-ray image display method using a deep neural network for correcting the selected camera image based on the camera image of the actual patient or the trainee, and correcting and displaying the matched X-ray image to match the correction of the selected camera image.
The method of claim 1,

The step of synthesizing and displaying the X-ray image matched with the selected camera image as a predicted X-ray image on the camera image of the real patient or the subject

A method of displaying a virtual X-ray image using a deep neural network for performing region division of a human body by matching a thermal image of an auxiliary camera with the camera image of the actual patient or the trainee.
The method of claim 4,

If there is a change in the color beyond the reference value, it is determined that the actual patient or the trainee is wearing clothes, and if there is only a change less than the reference value, it is determined that the actual patient or the trainee is not wearing clothes, and the learning image database A virtual X-ray image display method using a deep neural network that extracts and displays an X-ray image of a corresponding situation from