KR102488676B1 - Method and apparatus for improving z-axis resolution of ct images based on deep learning - Google Patents

Abstract

An apparatus and method for improving z-axis resolution of a CT image based on deep learning are provided. The method comprises the steps of: preprocessing the CT image including m slices (where m is a natural number); generating a sagittal image from which n slices are removed from the preprocessed CT image as an input image (where n is a natural number smaller than m); performing a deep learning-based first operation using the input image to newly generate the removed n slices; and performing a deep learning-based second operation on a cross-sectional image of the newly generated n slices to improve the resolution of the newly generated n slices. Accordingly, inter-slice resolution of a facial CT image is improved through a 2D convolutional neural network-based network.

Description

Method and apparatus for improving Z-axis resolution of CT images based on deep learning {METHOD AND APPARATUS FOR IMPROVING Z-AXIS RESOLUTION OF CT IMAGES BASED ON DEEP LEARNING}

The present invention relates to a method and apparatus for improving Z-axis resolution of CT images based on deep learning.

The orbital bone plays an important role in protecting the eyeball and nerves and ensuring that the eyeball is positioned correctly. The orbital bone has a four-sided pyramidal structure including the orbital roof, orbital lateral wall, orbital medial wall, and orbital floor, and is composed of cortical bone, a dense tissue. bone), low-density tissue, trabecular bone, and thin bone. At this time, since the medial and inferior walls of the orbit are composed of thin bones, they are easily ossified even with a small impact, so in order to rebuild the fractured orbit, the orbital bone on the opposite side that is not fractured in the CT (computed tomography) image is divided and divided. After mirroring the result to create a 3D model, a patient-specific bone plate is manufactured.

However, because the slice thickness of the orbital bone in the facial CT image is 2 to 3 times larger than the pixel spacing due to the specific nature of the image acquisition process, aliasing occurs when reconstructed into a 3D model, or There is a problem in that an empty space is generated due to disconnection in the z-axis direction in thin bones.

Therefore, it is necessary to improve the covariance resolution in the z-axis direction before segmenting the orbital bone and reconstructing it into a 3D model.

Conventionally, various methods have been used to improve inter-slice resolution in medical images, but among the conventional methods, the interpolation-based method has a simple calculation formula and is fast, but cannot accurately restore detailed parts or blurs the boundary of an object. There are limitations that appear. In addition, in the case of the deep learning-based method, using only the spatial information of the coronal (or sagittal) plane of the image, the mid-sliced image has an unclear boundary from the transverse plane to the sagittal plane (or coronal plane), and some tissue parts are broken. There is a limit to losing.

Registered Patent Publication No. 10-2155760, 2020.09.08.

An object to be solved by the present invention is to provide a method for improving inter-slice resolution of a facial CT image through a network based on a two-dimensional convolutional neural network.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A deep learning-based Z-axis resolution improvement method of a CT image performed by an apparatus according to an aspect of the present invention for solving the above problems includes performing preprocessing on the CT image including m slices (where m is a natural number), generating a sagittal image from which n slices are removed from the preprocessed CT image as an input image (where n is a natural number smaller than m), and performing a deep learning-based first operation using the input image. The step of newly generating the removed n slices and performing a deep learning-based second operation on the cross-sectional images of the newly created n slices to improve the resolution of the newly created n slices. include

In the present invention, n is a half value of m, and the removed n slices may be even-numbered slices among the m slices.

In the present invention, in the preprocessing step, intensity rescaling of the CT image may be performed, and the rescaled CT image may be cropped so that only the region of interest to be examined remains.

In the present invention, the input image may include a label mask in which bones included in the preprocessed CT image are displayed, and the label mask may be a binary image in which bones are indicated by 1 and non-bone portions are indicated by 0. there is.

In the present invention, in the step of newly generating the removed n slices, the newly generated n slices are newly generated through the first operation, and the newly generated n slices are included to improve inter-slice resolution. A sagittal image is output, and the first operation uses a convolution operation through a convolution layer, an up-sampling operation through an up-sampling layer, and a loss function in the sagittal plane. may contain operations.

In the present invention, the loss function in the sagittal plane is a loss function related to error minimization of the brightness value between the image output by the upsampling operation and the preprocessed CT image, the sagittal image with improved inter-slice resolution, A loss function related to error minimization of the brightness value between the preprocessed CT images and a loss related to error minimization between the gradient value between the preprocessed CT image and the sagittal image with improved inter-slice resolution using the sagittal image of the label mask. It can be calculated as a sum of functions.

In the present invention, the second calculation includes an operation using a loss function in a cross-section, and the loss function in the cross-section corresponds to a brightness value between the newly generated cross-section image of n slices and the preprocessed CT image. of the error minimization related loss function, the error minimization related loss function for the gradient value between the preprocessed CT image and the newly generated cross sectional image of the n slices using the cross sectional image of the label mask, and the newly generated n slices It can be calculated as the sum of loss functions related to minimizing the difference between the cross-sectional image and the feature map of the preprocessed CT image.

An apparatus for improving the Z-axis resolution of a CT image based on deep learning according to another aspect of the present invention for solving the above problems is a communication unit, a memory having at least one process for improving the resolution of the CT image, and the process and a processor operating according to the above process, wherein the processor performs pre-processing on the CT image including m slices (where m is a natural number) based on the process, and n slices are obtained from the pre-processed CT image. The removed sagittal plane image is generated as an input image (where n is a natural number smaller than m), and a deep learning-based first operation is performed using the input image to newly generate the removed n slices, and the newly generated The resolution of the newly created n slices is improved by performing a deep learning-based second operation on the cross-sectional images of the n slices.

In the present invention, n is a half value of m, and the removed n slices may be even-numbered slices among the m slices.

In the present invention, the processor may perform preprocessing by performing intensity rescaling of the brightness value of the CT image and cropping the rescaled CT image so that only the region of interest to be examined remains.

In the present invention, the input image may include a label mask in which bones included in the preprocessed CT image are displayed, and the label mask may be a binary image in which bones are indicated by 1 and non-bone portions are indicated by 0. there is.

In the present invention, the processor newly generates the removed n slices through the first operation, outputs a sagittal plane image in which resolution between slices is improved by including the newly generated n slices, and Operation 1 may include a convolution operation through a convolution layer, an up-sampling operation through an up-sampling layer, and an operation using a loss function in a sagittal plane.

In the present invention, the loss function in the sagittal plane is a loss function related to error minimization of the brightness value between the image output by the upsampling operation and the preprocessed CT image, the sagittal image with improved inter-slice resolution, A loss function related to error minimization of the brightness value between the preprocessed CT images and a loss related to error minimization between the gradient value between the preprocessed CT image and the sagittal image with improved inter-slice resolution using the sagittal image of the label mask. It can be calculated as a sum of functions.

In the present invention, the second calculation includes an operation using a loss function in a cross-section, and the loss function in the cross-section corresponds to a brightness value between the newly generated cross-section image of n slices and the preprocessed CT image. of the error minimization related loss function, the error minimization related loss function for the gradient value between the preprocessed CT image and the newly generated cross sectional image of the n slices using the cross sectional image of the label mask, and the newly generated n slices It can be calculated as the sum of loss functions related to minimizing the difference between the cross-sectional image and the feature map of the preprocessed CT image.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, it is possible to generate a mid-slice image with clear thin bone information as well as cortical bone by considering boundary information of the orbital region in the sagittal plane through an orbital bone edge-aware loss function. In addition, the quality of the intermediate slice image can be improved by additionally considering the feature map difference between the original image and the intermediate slice image extracted through a pre-trained image feature extraction network as a loss function.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram for explaining an apparatus for improving Z-axis resolution of a CT image based on deep learning according to the present invention.
2 is a flowchart of a method for improving Z-axis resolution of CT images based on deep learning according to the present invention.
3 is a diagram for explaining data pre-processing according to the present invention.
4 is a diagram for explaining a first operation for newly generating an intermediate slice according to the present invention.
5 is a diagram for explaining a second operation for improving the resolution of a newly generated intermediate slice according to the present invention.
6A and 6B are diagrams for explaining a sagittal plane image with improved resolution between slices generated through OB-Net according to the present invention.
7A and 7B are views for explaining cross-sectional images with improved inter-slice resolution generated through OB-Net according to the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

The spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe a component's correlation with other components. Spatially relative terms should be understood as including different orientations of elements in use or operation in addition to the orientations shown in the drawings. For example, if you flip a component that is shown in a drawing, a component described as "below" or "beneath" another component will be placed "above" the other component. can Thus, the exemplary term “below” may include directions of both below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Prior to the description, the meaning of the terms used in this specification will be briefly described. However, it should be noted that the description of terms is intended to help the understanding of the present specification, and is not used in the sense of limiting the technical spirit of the present invention unless explicitly described as limiting the present invention.

In this specification, 'device' includes all various devices capable of providing results to users by performing calculation processing. For example, the devices may be in the form of computers and mobile terminals. The computer may be in the form of a server receiving a request from a client and processing information. The mobile terminal includes a mobile phone, a smart phone, a personal digital assistants (PDA), a portable multimedia player (PMP), a navigation device, a notebook PC, a slate PC, a tablet PC, and an ultrabook. ), a wearable device (eg, a watch type terminal (smartwatch), a glass type terminal (smart glass), a head mounted display (HMD)), and the like may be included.

In this specification, an orbital rbital rbital boneoneone-network (OB-Net) is a network based on a two-dimensional convolutional neural network that improves inter-slice resolution of a computed tomography (CT) image, and generates an intermediate slice image in the sagittal plane ( first operation) and improving the quality of the intermediate slice image in the cross section (second operation). In the first step, a low-resolution image of the sagittal plane and a label mask of the region of interest (eg, the orbital region) of the original image are input, and an intermediate slice image having a clear boundary of the region of interest is generated. In the second step, in order to solve the problem that bones are broken or unclear in the mid-slice image created only with the cross-sectional information of the sagittal plane, the mid-slice image created in the sagittal plane is converted into a cross-section, and the difference in brightness value from the original image and the region of interest are analyzed. By minimizing the error by considering the gradient value difference and the feature map difference, a middle slice image with an accurate region of interest is generated even in a cross-section.

The OB-Net may be trained to generate an image of the excluded part by excluding the middle slice of the image in order to train a learning model for increasing the resolution.

1 is a diagram for explaining an apparatus for improving Z-axis resolution of a CT image based on deep learning according to the present invention.

2 is a flowchart of a method for improving Z-axis resolution of CT images based on deep learning according to the present invention.

3 is a diagram for explaining data pre-processing according to the present invention.

4 is a diagram for explaining a first operation for newly generating an intermediate slice according to the present invention.

5 is a diagram for explaining a second operation for improving the resolution of a newly generated intermediate slice according to the present invention.

6A and 6B are diagrams for explaining a sagittal plane image with improved resolution between slices generated through OB-Net according to the present invention.

7A and 7B are views for explaining cross-sectional images with improved inter-slice resolution generated through OB-Net according to the present invention.

Hereinafter, with reference to FIG. 1 , an apparatus 10 for improving Z-axis resolution of a CT image based on deep learning according to the present invention will be described.

The apparatus 10 according to the present invention acquires a patient's CT image and improves the resolution of the obtained CT image using the OB-Net, thereby finally providing a clearer CT image to the user.

Here, the user is a medical expert and may be a doctor, nurse, clinical pathologist, medical imaging expert, or the like, and may be a technician who repairs a medical device, but is not limited thereto.

Here, the CT image may be a facial CT image, but is not limited thereto.

Specifically, the apparatus 10 may reconstruct the obtained CT image into a 3D model by preprocessing the obtained CT image and performing first and second calculations on the preprocessed CT image. Here, the first operation is a step related to generating an intermediate slice in the sagittal plane, and the second operation is a step related to improving the quality of the intermediate slice in the transverse plane. A detailed description of this will be described later.

The device 10 may include all of various devices capable of providing results to users by performing calculation processing.

Here, the device 10 may be in the form of a computer. More specifically, the computer may include all of various devices capable of providing results to users by performing calculation processing.

For example, a computer includes not only a desktop PC and a notebook (Note Book) but also a smart phone, a tablet PC, a cellular phone, a PCS phone (Personal Communication Service phone), synchronous/asynchronous A mobile terminal of IMT-2000 (International Mobile Telecommunication-2000), a Palm Personal Computer (Palm PC), and a Personal Digital Assistant (PDA) may also be applicable. In addition, when a Head Mounted Display (HMD) device includes a computing function, the HMD device may become a computer.

In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

Also, the device 10 may include a communication unit 12 , a memory 14 and a processor 16 . Here, the device 10 may include fewer or more components than those shown in FIG. 1 .

The communication unit 12 may include one or more wireless communication between the device 10 and an external device (not shown), between the device 10 and an external server (not shown), or between the device 10 and a communication network (not shown). modules may be included.

Here, the external device (not shown) may be a medical imaging equipment that captures medical image data. Here, the medical image data may include all medical images capable of realizing the patient's body as a 3D model. More specifically, the external device (not shown) may be a imaging device that captures a CT image.

Also, the external server (not shown) may be a server that stores medical data for each patient for a plurality of patients.

In addition, a communication network (not shown) may transmit and receive various information between the device 10, an external device (not shown), and an external server (not shown). Various types of communication networks may be used as the communication network, for example, wireless communication methods such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, and HSDPA (High Speed Downlink Packet Access) Alternatively, a wired communication method such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), FTTH (Fiber To The Home) may be used.

On the other hand, the communication network is not limited to the communication methods presented above, and may include all other types of communication methods that are widely known or will be developed in the future in addition to the above communication methods.

Communication unit 12 may include one or more modules that connect device 10 to one or more networks.

Memory 14 may store data supporting various functions of device 10 . The memory 14 may store a plurality of application programs (application programs or applications) running in the device 10 , data for operation of the device 10 , and commands. At least some of these applications may exist for basic functions of the device 10 . Meanwhile, the application program may be stored in the memory 14, installed on the device 10, and driven by the processor 16 to perform an operation (or function) of the device 10.

Also, the memory 14 may include at least one process for improving the resolution of a CT image according to the present invention. Specifically, the memory 14 may include a data preprocessing related process, a first arithmetic related process, and a second arithmetic related process. Alternatively, the memory 14 may include one process including data preprocessing, first operation, and second operation.

The processor 16 may control general operations of the device 10 in addition to operations related to the application program. The processor 16 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the components described above or by driving an application program stored in the memory 14.

In addition, the processor 16 may control at least some of the components discussed in conjunction with FIG. 1 in order to drive an application program stored in the memory 14 . Furthermore, the processor 16 may combine and operate at least two or more of the components included in the device 10 to drive the application program.

Hereinafter, with reference to FIGS. 2 to 6 , a method for improving the Z-axis resolution of a CT image based on deep learning by the processor 16 will be described. Here, the operation of the processor 16 may be performed by the device 10 .

The processor 16 may perform pre-processing on the CT image including m slices (S110). Here, m may be a natural number.

A CT image captured by the external device (not shown) may be reconstructed into a 3D image including m slices.

First, the processor 16 may rescale the brightness value of the CT image (intensity rescaling). This can be referred to as a process of rescaling the -200HU to 400HU range, which is a brightness value range including the orbital region, from a 16-bit gray-scale image to an 8-bit 0-255 image.

Then, the processor 16 may crop the CT image so that only the region of interest to be examined remains in the CT image whose brightness values are rescaled. For example, as shown in FIG. 3 , in order to consider only the orbital region in the facial CT image, only a portion of the entire region is designated and used.

Next, the processor 16 may generate a sagittal image from which n slices are removed from the preprocessed CT image as an input image (S120). Here, n may be a natural number smaller than m.

Hereinafter, the CT image for which preprocessing is completed is referred to as an original image.

The original image includes m slices as described above. In this case, the processor 16 uses the above-described deep learning-based learning model to generate n intermediate slice images with improved resolution from among the m slices. can be removed.

More specifically, n is a half value of m, and the removed n slices may be even-numbered slices among the m slices.

For example, if the original image contains 100 slices, the processor removes 50 even-numbered slices such as the second, fourth, etc. out of the 100 slices, leaving only the odd-numbered 50 slices. The remaining original video can be used as an input video. In this way, by removing even-numbered slices from the original image and using them as the input image of the network, it is possible to verify whether the intermediate slice image generated through the network is well generated similarly to the original image.

Also, the input image may include a label mask displaying bones included in the preprocessed CT image (original image).

Here, the label mask may be a binary image in which bone portions are indicated as 1 and non-bone portions are indicated as 0. Through such a label mask for the region of interest (eg, the orbital region), by specifying the thin bone region that is difficult to distinguish in the CT image, errors in the thin bone region are intensively minimized in a later step, and the ?戮? It becomes possible to create mid-slice images with clear bones.

Next, the processor 16 may newly generate the removed n slices by performing a first operation based on deep learning using the input image (S130).

Specifically, the processor 16 may newly generate the removed n slices through the first operation and output a sagittal plane image having improved inter-slice resolution by including the newly generated n slices.

The first operation may include a convolution operation through a convolution layer, an up-sampling operation through an up-sampling layer, and an operation using a loss function in a sagittal plane.

Hereinafter, with reference to FIG. 4, the first calculation using the facial CT image will be described in detail.

As described above, the input image of the OB-Net of the present invention is a sagittal face CT image with inter-slice resolution lowered to 1/2 (48→24) through image preprocessing and the region of interest (orbital region) of the original image. Can be used together with a label mask for

In order to extract a feature map from an input image, a convolution operation is performed through 6 convolution layers using a 3x3 size filter, and zero-padding is applied to prevent the size of the feature map from being reduced. .

In this case, the first to fifth convolution layers may generate 32 feature maps, and the sixth convolution layer may generate two feature maps.

In addition, short skip-connection and long skip-connection may be used to maintain low-dimensional feature map information including detailed information of the image, such as texture and gradient in the image. A short skip connection may add the feature maps generated by the first convolution and the third convolution layer, and a long skip connection may add the feature maps generated by the first convolution and the fifth convolution layer.

After performing the sixth convolution layer, an up-sampling layer may be performed to double the resolution between slices. In this case, the upsampling layer may output an image with improved inter-slice resolution by using a sub-pixel convolution operation that increases the resolution in the inter-slice direction of the input image using two feature maps. .

Thereafter, four convolution layers using a 3x3 size filter may be additionally performed in order to improve detailed parts that still appear different from the original image in the image with increased inter-slice resolution. In this case, the first to third convolution layers generate 32 feature maps, and the last convolution layer can output an image of one channel.

In this way, n intermediate slices previously removed can be newly created through a convolution operation through two convolution layers and an up-sampling operation through one up-sampling layer. there is

In addition, the processor 16 may perform an operation using a loss function in the sagittal plane to minimize the difference between the n intermediate slices generated in this way and the original image.

Here, the loss function in the sagittal plane is, as shown in [Equation 1] below, a loss function related to minimizing the error for the brightness value between the image output by the upsampling operation and the preprocessed CT image, and the resolution between the slices. Slope between the sagittal image with improved inter-slice resolution and the preprocessed CT image using a loss function related to error minimization for brightness values between the sagittal image with improved inter-slice resolution and the sagittal image of the label mask It can be calculated as the sum of loss functions related to error minimization for the values.

Here, SR ^S is the image of the sagittal plane whose inter-slice resolution has been improved through the network, SR ^up is the output image of the upsampling layer, ^OS is the original image, and ^MS is the region of interest (orbital region) label mask in the sagittal plane. it means.

A loss function related to error minimization of brightness values between the sagittal plane image with improved inter-slice resolution and the preprocessed CT image can be expressed as [Equation 2] below.

Here, W and H represent the width and height of the original image, and i and j represent the pixel positions of the image. SR ^S denotes a sagittal image whose inter-slice resolution has been improved through the network, and ^OS denotes an original image.

By using [Equation 2] in this way, the present invention can minimize the brightness difference between the final output image of the network and the original image.

A loss function related to error minimization of brightness values between the image output by the upsampling operation and the preprocessed CT image may be expressed as [Equation 3].

Here, W and H represent the width and height of the original image, and i and j represent the pixel positions of the image. SR ^up means the output video of the upsampling layer, and ^OS means the original video.

By using [Equation 3] in this way, the present invention can minimize the difference between the brightness value of the final output image and the output image of the upsampling layer and the original image. This makes it possible to generate an image similar to the original as much as possible from the upsampling layer.

A loss function related to error minimization of the gradient between the preprocessed CT image and the sagittal image with improved inter-slice resolution using the sagittal image of the label mask can be expressed as [Equation 4] below.

Here, W and H represent the width and height of the original image, and i and j represent the pixel positions of the image. SRS denotes a sagittal image with inter-slice resolution improved through the network, ^OS denotes an original image, and ^MS denotes a label mask for a region of interest (orbital region ⁾ in the sagittal plane. E( ) denotes a boundary image of the sagittal plane extracted through the Sobel edge detector.

By using [Equation 4] in this way, the present invention uses the orbital bone edge-aware loss function to reflect only the tilt error in the thin bone region instead of using the general tilt loss function as it is. Through this, the tilt error within the orbital region can be calculated. In this way, it is possible to consider the thin bone characteristics of the orbital region with a small area and low contrast, and when the network generates a mid-slice image, it is possible to focus more on the boundary of the thin bones in the orbital region than other regions, so that the thin bone A middle slice image with clearer boundaries may be generated.

Next, the processor 16 may improve the resolution of the newly generated n slices by performing a deep learning-based second operation on the cross-sectional images of the newly generated n slices (S140).

The second operation may include an operation using a loss function in a cross section.

Hereinafter, with reference to FIG. 5 , the second operation using the facial CT image will be described in detail.

As described above, since the first operation improves the resolution based only on the information of the sagittal plane image, the region of interest (orbital region) can be accurately displayed in the cross-sectional image of the newly created intermediate slice image by additionally performing the second operation. A loss function for an image of a cross section of an intermediate slice image may be additionally considered.

Here, the loss function in the cross section is, as shown in [Equation 5] below, a loss function related to error minimization of the brightness values between the newly generated cross section images of the n slices and the preprocessed CT image, and the label mask A loss function related to error minimization for the gradient values between the newly generated cross-sectional images of the n slices using the cross-sectional images and the preprocessed CT images, and a feature map between the newly generated cross-sectional images of the n slices and the preprocessed CT images. It can be calculated as the sum of the loss functions associated with minimizing the difference for

Here, SR ^A denotes a cross-sectional image of the newly generated intermediate slice image, O ^A denotes an original image of the newly generated intermediate slice image, and M ^A denotes a cross-sectional ROI (orbital region) label mask.

A loss function related to error minimization of brightness values between the newly generated cross-sectional images of the n slices and the preprocessed CT image may be expressed as [Equation 6] below.

Here, W and H represent the width and height of the original image, and i and j represent the pixel positions of the image. SR ^A denotes a cross-sectional image of a newly created intermediate slice image, and O ^A denotes an original image of a newly generated intermediate slice image.

By using [Equation 6] in this way, the present invention can minimize the brightness difference between the newly created middle slice image and the corresponding middle slice image of the original image.

A loss function related to error minimization of a slope value between the newly generated cross-sectional images of the n slices using the cross-sectional images of the label mask and the preprocessed CT image can be expressed as [Equation 7] below.

Here, W and H represent the width and height of the original image, and i and j represent the pixel positions of the image. SR ^A is the cross-sectional image of the newly created intermediate slice image, O ^A is the original image of the newly created intermediate slice image, and M ^A is the cross-sectional region-of-interest (orbital region) label mask. E( ) denotes a boundary image of a cross section extracted through a Sobel edge detector.

By using [Equation 7] in this way, the present invention can calculate only the tilt error within the region of interest (orbital region), which solves the problem that cortical bones and thin bones appear blurry in the sagittal direction, and even in the cross section, the region of interest ( orbital region) to create a clear mid-slice image.

A loss function related to minimizing the difference between the newly generated cross-sectional images of the n slices and the preprocessed CT image for the feature map can be expressed as [Equation 8] below.

는 특정 분류 모델의 x번째 최대값 풀링 계층(max-pooling layer) 직전 y번째로 수행된 컨볼루션 계층에서 출력된 특징맵을 의미하고,

와

는 특징맵

의 너비와 높이를 의미하고, i와 j는 특징맵

에서의 화소 위치를 나타낸다. Here, W and H represent the width and height of the original image. SR ^A denotes a cross-sectional image of a newly created intermediate slice image, and O ^A denotes an original image of a newly generated intermediate slice image.

denotes a feature map output from the y-th convolution layer performed immediately before the x-th max-pooling layer of a specific classification model,

Wow

is the feature map

means the width and height of , and i and j are feature maps

Indicates the pixel position in .

By using [Equation 8] in this way, the present invention can improve the perceptual quality of the middle slice image. At this time, the feature map used can be extracted through the specific classification model (eg, VGG16) pre-trained to classify the image through the ImageNet training set, and the feature map is input to the specific classification model It can include high-level feature information inferred about the image. When the feature map difference is used as a loss function, an image more structurally similar to an original image can be generated than a loss function based on a difference in brightness values performed in units of pixels.

In this way, the final CT image output after all calculations are performed through OB-Net shows a form in which the resolution between slices is greatly improved compared to the CT image according to the conventional method.

6A and 6B show sagittal images with improved inter-slice resolution through the conventional method and the method according to the present invention, respectively. Referring to FIG. 6A, the CT image whose resolution is improved according to the conventional method still shows aliasing or partly blurred, whereas, referring to FIG. 6B, the CT image whose resolution is improved according to the present invention shows thin bone compared to the prior art. It can be seen that the part is expressed more smoothly.

7A and 7B show cross-sectional images with improved inter-slice resolution through the conventional method and the method according to the present invention, respectively. Referring to FIG. 7A, the CT image whose resolution is improved according to the conventional method still has unclear bone boundaries, whereas, referring to FIG. 7B, the CT image whose resolution is improved according to the present invention is relatively cortical bone and The boundary of the thin bone is clear, and it can be confirmed that it is similar to the original without any broken parts.

As described above, the present invention improves inter-slice resolution of face CT images by using a 2D convolutional neural network that generates mid-slice images in the sagittal plane by considering the feature information of bones (orbital fossa) in the sagittal plane and the transverse plane. can do. The OB-Net according to the present invention generates a mid-slice image in which not only the cortical bone but also the thin bone shape is clearly displayed through the orbital bone boundary recognition loss function used in the sagittal plane and the hyoid plane, and generated only with the cross-sectional information of the sagittal plane In the cross section of the middle slice image, the boundary between cortical bone and thin bone was blurred or partly cut off. In addition, the quality of the cross-sectional middle slice image was improved by additionally using a loss function that considers the feature map difference between the original image and the middle slice image through a pre-trained image feature extraction network (specific classification model).

2 describes that steps S110 to S140 are sequentially executed, but this is merely an example of the technical idea of this embodiment, and those skilled in the art to which this embodiment belongs will Since it will be possible to change and execute the order described in FIG. 2 without departing from the essential characteristics or to perform various modifications and variations by executing one or more steps of steps S110 to S140 in parallel, FIG. 2 is shown in a time-series order. It is not limited.

The method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Here, the computer may be the device 10 described above.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: device
12: Communication department
14: memory
16: processor

Claims (14)

In the Z-axis resolution improvement method of a deep learning-based CT image performed by an apparatus,
performing pre-processing on the CT image including m slices (where m is a natural number);
generating a sagittal image from which n slices are removed from the preprocessed CT image as an input image (where n is a natural number smaller than m);
newly generating the removed n slices by performing a first operation based on deep learning using the input image; and
Improving the resolution of the newly generated n slices by performing a deep learning-based second operation on the cross-sectional images of the newly generated n slices;
Wherein n is half the value of m,
Wherein the removed n slices are even-numbered slices among the m slices. delete According to claim 1,
In the preprocessing step,
The method of rescaling the brightness value of the CT image (intensity rescaling), and cropping the rescaled CT image so that only the region of interest to be examined remains. According to claim 1,
The input image,
A label mask displaying bones included in the preprocessed CT image;
Wherein the label mask is a binary image in which bone portions are indicated as 1 and non-bone portions are indicated as 0. According to claim 4,
The step of newly creating the removed n slices,
Newly generating the removed n slices through the first operation and outputting a sagittal plane image having improved inter-slice resolution by including the newly generated n slices,
The first operation includes a convolution operation through a convolution layer, an up-sampling operation through an up-sampling layer, and an operation using a loss function in a sagittal plane. According to claim 5,
The loss function in the sagittal plane is,
A loss function related to error minimization of brightness values between the image output by the upsampling operation and the preprocessed CT image, and error minimization of brightness values between the sagittal plane image with improved inter-slice resolution and the preprocessed CT image The method according to claim 1 , wherein the inter-slice resolution using the loss function and the sagittal plane image of the label mask is calculated as the sum of error minimization-related loss functions for gradient values between the improved sagittal plane image and the preprocessed CT image. According to claim 4,
The second operation includes an operation using a loss function in a cross section,
The loss function in the cross-section is a loss function related to error minimization of the brightness value between the newly generated cross-sectional image of the n slices and the preprocessed CT image, and the newly generated n slices using the cross-sectional image of the label mask. Calculated as the sum of a loss function related to error minimization for the slope value between the cross-sectional image of n and the preprocessed CT image and a loss function related to minimizing the difference between the newly created cross-sectional image of n slices and the feature map between the preprocessed CT image how to become. In the Z-axis resolution improvement device of deep learning-based CT image,
communications department;
a memory having at least one process for improving the resolution of the CT image; and
A processor operating according to the process; includes,
The processor, based on the process,
Preprocessing is performed on the CT image including m slices (where m is a natural number);
A sagittal image from which n slices are removed from the preprocessed CT image is generated as an input image (where n is a natural number smaller than m),
Performing a deep learning-based first operation using the input image to newly generate the removed n slices;
Performing a deep learning-based second operation on the cross-sectional images of the newly generated n slices to improve the resolution of the newly generated n slices;
Wherein n is half the value of m,
The removed n slices are even-numbered slices among the m slices. delete According to claim 8,
the processor,
The apparatus of performing preprocessing by performing intensity rescaling of the CT image and cropping the rescaled CT image so that only the region of interest to be examined remains. According to claim 8,
The input image,
A label mask displaying bones included in the preprocessed CT image;
Wherein the label mask is a binary image in which bone portions are indicated as 1 and non-bone portions are indicated as 0. According to claim 11,
the processor,
The removed n slices are newly generated through the first operation, and a sagittal plane image having improved inter-slice resolution is output by including the newly generated n slices,
The first operation includes a convolution operation through a convolution layer, an up-sampling operation through an up-sampling layer, and an operation using a loss function in a sagittal plane. According to claim 12,
The loss function in the sagittal plane is,
A loss function related to error minimization of brightness values between the image output by the upsampling operation and the preprocessed CT image, and error minimization of brightness values between the sagittal plane image with improved inter-slice resolution and the preprocessed CT image The apparatus of claim 1 , wherein the loss function and the sum of error minimization-related loss functions for gradient values between the preprocessed CT image and the sagittal plane image with improved inter-slice resolution using the sagittal plane image of the label mask are calculated. According to claim 11,
The second operation includes an operation using a loss function in a cross section,
The loss function in the cross-section is a loss function related to error minimization of the brightness value between the newly generated cross-sectional image of the n slices and the preprocessed CT image, and the newly generated n slices using the cross-sectional image of the label mask. Calculated as the sum of a loss function related to error minimization for the slope value between the cross-sectional image of n and the preprocessed CT image and a loss function related to minimizing the difference between the newly created cross-sectional image of n slices and the feature map between the preprocessed CT image Becoming device.

Images