KR20200127702A - Apparatus and Method of Speckle Reduction in Optical Coherence Tomography using Convolutional Networks - Google Patents

Abstract

The present invention relates to an optical tomography image speckle control device using a convolutional network and a method thereof. The optical tomography image speckle control device using a convolutional network comprises: an input module that receives an optical tomography image; a convolutional network that converts features extracted from the optical tomography image into nonlinear values, and removes speckle noise through a convolution operation to generate a reconstructed image; and an output module that outputs an image from which the speckle noise is removed. Therefore, the optical tomography image speckle control device using a convolutional network automates the process of removing the speckle noise from the optical tomography image obtained from an optical coherence tomography system and converting the same into the optical coherence tomography image from which the speckle noise is removed through convolution network learning to generate an image which is obtained by averaging multiple images and from which the speckle noise is removed only with one optical tomography image, thereby improving processing (acquisition) speed for removing the speckle noise.

Description

Optical tomography image speckle control apparatus using a convolutional network, and its method {Apparatus and Method of Speckle Reduction in Optical Coherence Tomography using Convolutional Networks}

The present invention relates to an optical tomography image speckle control apparatus and method using a convolutional network, and more particularly, to a cross-sectional image obtained by an optical coherence tomography (OCT) system. It relates to a method of removing speckle noise of, and in detail, a convolution network learning transformation from a tomographic image including speckle noise to a tomographic image from which speckle noise has been removed. It relates to an apparatus and a method for removing speckle noise in real time by using.

Optical coherence tomography (OCT) systems can acquire micro-meter (μm) high-resolution cross-sectional images of biological samples, and due to this advantage, various fields including ophthalmology and dermatology It is used for diagnosis and treatment purposes in Optical tomography images, however, contain speckle noise that appears to be sprinkled with particles because they use the wavelength of interference, which degrades the image quality and is therefore stable when assessing disease qualitatively or quantitatively. Makes it difficult to practice Therefore, speckle noise removal is an essential step to ensure reliability when using optical tomography images for diagnosis and treatment.

The most widely adopted speckle noise removal technique in the currently sold (commercialized) OCT system is the frame averaging method [1]. This technique produces a single image from which speckle noise has been removed by averaging multiple images acquired at the same location in pixel units. The image averaging technique provides improved quality optical tomography images, but increases acquisition time (low frames per second (FPS)) because multiple optical tomography images must be acquired to produce one final image. There are drawbacks. This technique is difficult to apply in an in-vivo environment due to acquisition time problems, especially when 3D scan is required.

[1] Y. Ma et al., "Speckle noise reduction in optical coherence tomography image based con edge-sensitive cGAN," Biomedical Optics Express, 9(11), pp. 5129-5146, 2018.

Accordingly, the applicant of the present invention proposes an apparatus and method for removing speckle noise in real time using a convolution network that learns transformation from a tomography image containing speckle noise to a tomography image from which speckle noise has been removed. I would like to suggest.

Korean Patent Registration No. 10-1942219 (announced on January 24, 2019)

An object of the present invention is to automate a procedure of removing speckle noise from an optical tomography image acquired by an optical coherence tomography system and converting the speckle noise to a tomographic image from which speckle noise has been removed through learning a convolutional network. It is to improve the processing (acquisition) speed for removing speckle noise by generating an image from which speckles have been removed only with a tomographic image (a result of averaging multiple images).

In order to solve this technical problem, the present invention is an optical tomographic image speckle control apparatus using a convolutional network, comprising: an input module for receiving an optical tomographic image; A convolutional network for reconstructing an image from which speckle noise has been removed by converting features extracted from the optical tomography image into nonlinear values through a convolution operation; And an output module outputting an image from which speckle noise has been removed.

Preferably, the convolutional network is characterized in that it includes an operation unit for extracting features from the optical tomography image through a convolution operation that multiplies each pixel through a filter having a size of f×f×d.

The convolutional network is characterized in that nonlinear transformation is performed through an activation function ReLU (Rectified Linear Unit).

An optical tomography image speckle control method using a convolutional network according to an embodiment of the present invention based on the above-described apparatus includes: (a) receiving an optical tomography image by an input module; (B) the convolutional network extracts features from the optical tomography image through a convolution operation; (C) converting the features extracted by the convolutional network into nonlinear values; (D) reconstructing an image from which speckle noise has been removed from the image converted by the convolutional network to a nonlinear value; And (e) outputting, by the output module, an image from which speckle noise has been removed.

The step (b) includes the step (b-1) of generating a feature map by performing a multiplication operation on each pixel of the optical tomography image through a filter having a size of f × f × d by the operation unit of the convolution network; (B-2) the operation unit of the convolutional network extracts features from the feature map; (B-3) generating a high-dimensional feature map by performing a multiplication operation on each pixel of the optical tomography image through a plurality of filters of size f×f×d, by the operation unit of the convolutional network; And (b-4) extracting a feature from a high-dimensional feature map by a computing unit of the convolutional network.

According to the embodiment of the present invention as described above, the procedure of removing speckle noise from an optical tomography image acquired by an optical coherence tomography system and converting it to a tomography image from which speckle noise has been removed is automated through convolutional network learning. By doing so, it is possible to increase the processing (acquisition) speed for removing speckle noise by generating an image from which speckles have been removed (a result of averaging multiple images) with only one optical tomography image.

1 is an exemplary view showing that speckle noise is removed from a conventional optical tomography image including speckle noise.
2 is a block diagram showing an optical tomography image speckle control apparatus using a convolutional network according to an embodiment of the present invention.
3 is an exemplary diagram showing experimental results for input and output of an optical tomography image speckle control apparatus using a convolution network according to an embodiment of the present invention.
4 is an exemplary view showing four pairs of training data consisting of input data and label data of an optical tomography image speckle control apparatus using a convolutional network according to an embodiment of the present invention.
5 is a flowchart illustrating a method of controlling an optical tomographic image speckle using a convolutional network according to an embodiment of the present invention.
Referring to FIG. 6, a flow chart showing a detailed process of step S502 of the method of controlling an optical tomography image speckle using a convolutional network according to an embodiment of the present invention.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. Prior to this, terms or words used in the present specification and claims correspond to the technical idea of the present invention based on the principle that the inventor can appropriately define the concept in order to describe his or her invention in the best way. It should be interpreted as meaning and concept. In addition, when it is determined that a detailed description of known functions and configurations thereof related to the present invention may unnecessarily obscure the subject matter of the present invention, it should be noted that the detailed description has been omitted.

FIG. 1 is a diagram showing an example of an optical tomography image 101-104 including speckle noise and a result 105 of removing speckle noise by an image averaging method, which is a conventional method.

The optical tomography image acquired by the OCT system basically contains speckle noise due to the characteristic of the interference wavelength, and as shown in FIG. 1, four images 101-104 are acquired at the same location and pixel An image from which speckle noise is removed is generated by averaging in units.

As such, the conventional method as shown in FIG. 1 is effective in removing speckle noise, but in order to generate a single image from which speckle noise has been removed, the image acquisition time is increased because the fourth image must be acquired at the same position. You have to bear the penalty.

Referring to FIG. 2, an optical tomography image speckle control apparatus 200 using a convolutional network according to an embodiment of the present invention includes an input module 201 for receiving an optical tomography image, and consists of three layers. Convolutional networks (202, 203, 204) reconstructing an image from which speckle noise has been removed by converting features extracted from the optical tomography image through a lusion operation into a nonlinear value, and an output that outputs an image from which speckle noise has been removed It comprises a module (205).

The calculation units 202a and 203a of the convolution networks 202 and 203 extract features from the optical tomography image through a convolution operation that multiplies each pixel through a filter having a size of f×f×d.

Here, f×f is the size of the filter, and n is the number of filters.

For example, if the input image is 8×8×3 (H×W×c) and the filter size is 3×3 (f×f), the number of parameters for one filter is 3×3×3 (f× f×c) = 27, and if there are 4 filters, the total number of parameters of the convolution layer will be 3×3×3×4 (f×f×c×n). Here, W is the image width, H is the image height, and c is the number of channels.

In addition, the calculation units 202a and 203a generate one feature map per filter, and generate a high-dimensional feature map through several filters to extract a high-dimensional feature.

In addition, the function units 202b and 203b of the convolutional networks 202 and 203 are composed of an activation function and perform a function of converting the filter result into a nonlinear value, but the activation function ReLU (Rectified Linear Unit) ) To perform nonlinear transformation.

Meanwhile, the convolutional network 204 reconstructs the image through a convolution operation to correspond to the nonlinear values converted by the function units 202b and 203b and transmits the reconstruction to the output module 205.

3 is an exemplary diagram showing experimental results for input and output of an optical tomography image speckle control apparatus using a convolutional network according to an embodiment of the present invention.

The input of the first convolution layer shown in FIG. 3 is an image 301 received from the OCT system, and the size is W×H×c. At this time, since the OCT system receives a gray image as an input, c is fixed to 1.

이며, 고차원 feature map 생성을 위해

개 필터를 사용하였다. 첫 번째 컨볼루션 레이어의 출력은

크기의 feature map(302)이다. First, the size of the filter used for the convolution operation is

And for high-dimensional feature map generation

Dog filters were used. The output of the first convolution layer is

It is a feature map 302 of size.

크기의 feature map이며, 컨볼루션 연산을 위한 필터 크기는

이다. 두 번째 컨볼루션 레이어는 개의 필터를 사용하여 컨볼루션 연산을 수행하였으며, 출력은

크기의 feature map(303)이다. The input of the second convolution layer is the output of the first convolution layer.

It is a feature map of size, and the filter size for convolution operation is

to be. The second convolution layer performed convolution operation using four filters, and the output

It is a feature map 303 of size.

크기의 feature map이며, 컨볼루션 연산을 위한 필터의 크기는

이다. 세 번째 컨볼루션 레이어는 개 필터를 사용하여 컨볼루션 연산을 수행하며,

크기의 이미지(304)를 출력한다.The input of the third convolution layer is the output of the second convolution layer.

It is a feature map of size, and the size of the filter for convolution operation is

to be. The third convolution layer uses dog filters to perform convolution operations,

The size of the image 304 is output.

4 is an exemplary diagram showing four pairs of training data consisting of input data and label data of an optical tomography image speckle control apparatus using a convolutional network according to an embodiment of the present invention.

When an image is input to a convolutional network, an output image is generated by sequentially performing a convolution operation, and a process of setting (training) a filter weight suitable for a purpose to remove speckle noise is involved.

Training data is required for network training.The training data consists of a pair of images with speckle noise (input data) and images with speckle noise controlled (label data), and multiple images acquired at the same location are input. As data, an image obtained by averaging several images was used as label data.

As shown in FIG. 4, when the speckle control image is generated with 4 images, 4 input/label data pairs are generated, and when training data is given, training is performed by converting the input data to the output image and input The weight of the filter is modified to minimize the difference between the data and the paired label image.

Adam (adaptive moment estimation), one of the gradient descent algorithms, was used to correct the weight based on the difference between the output image and the label image, and various algorithms such as RMSprop and Adagrad can be used.

When the network is trained, it is possible to create an image with speckle removed with only one image. This can quickly obtain an image from which speckles have been removed compared to the conventional image averaging technique. For example, compared to the method of generating one speckle control image from four images, it is possible to acquire numerically 4 times faster. In the experiment, if the network structure is set as follows, the performance of 200 FPS (frames per second) is shown in a PC environment using i7-6850K CPU @ 3.60 GHz.

In addition, in an embodiment of the present invention, although the network is represented by the three convolutional layers shown in FIG. 2 mentioned above, more convolutional layers can be added, and the network configuration can be modified to a larger value.

However, as the number of convolution layers increases, the processing time increases as the network configuration value increases. Network settings and the number of convolution layers can be modified according to the PC environment and desired performance.

According to the exemplary embodiment of the present invention as described above, a fast acquisition speed is shown in a general PC environment, and thus, a 3D scan with improved image quality (with speckle removed) in an in-vivo environment can be expected.

Hereinafter, a method of controlling an optical tomography image speckle using a convolutional network according to an embodiment of the present invention will be described with reference to FIG. 5.

First, the input module 201 receives an optical tomography image (S501).

Subsequently, the convolution networks 202 and 203 extract features from the optical tomography image through a convolution operation (S502).

Subsequently, the features extracted by the convolutional networks 202 and 203 are converted into nonlinear values (S503).

Subsequently, the convolutional network 204 reconstructs an image from which speckle noise has been removed from the image converted to a nonlinear value (S504).

Then, the output module 205 outputs an image from which speckle noise has been removed (S505).

Hereinafter, a detailed process of step S502 of the method for controlling an optical tomography image speckle using a convolutional network according to an embodiment of the present invention will be described with reference to FIG. 6.

After step S501, the operation unit 202a of the convolutional network 202 performs a multiplication operation on each pixel of the optical tomography image through a filter having a size of f×f×d to generate a feature map (S601).

Subsequently, the calculation unit 202a of the convolutional network 202 extracts features from the feature map (S602).

Subsequently, the operation unit 203a of the convolutional network 203 generates a high-dimensional feature map by performing a multiplication operation on each pixel of the optical tomography image through a plurality of filters having a size of f×f×d (S603).

Then, the calculation unit 203a of the convolutional network 203 extracts features from the high-dimensional feature map (S604).

As described above, according to an embodiment of the present invention as described above, an image from which speckles are removed (a result of averaging multiple images) can be generated with only one input image.

As described above and illustrated with a focus on preferred embodiments for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as illustrated and described as described above, and does not depart from the scope of the technical idea. It will be well understood by those skilled in the art that many changes and modifications are possible to the present invention within the scope. Accordingly, all such appropriate changes and modifications and equivalents should be considered to be within the scope of the present invention.

101, 102, 103, 104: Optical tomography image containing speckle noise
201: input module
202, 203, 204: convolutional network
202a, 203a: operation unit
202b, 203b: function unit
205: output module

Claims (5)

An input module for receiving an optical tomography image;
A convolutional network for reconstructing an image from which speckle noise has been removed by converting features extracted from the optical tomography image into nonlinear values through a convolution operation; And
An output module that outputs an image with speckle noise removed
Optical tomography image speckle control apparatus using a convolutional network, comprising. The method of claim 1,
The convolutional network,
An operation unit for extracting features from the optical tomography image through a convolution operation that performs a multiplication operation on each pixel through a filter of size f×f×d
Optical tomography image speckle control apparatus using a convolutional network, comprising. The method of claim 1,
The convolutional network,
An optical tomographic image speckle control device using a convolutional network, characterized in that nonlinear transformation is performed through an activation function ReLU (Rectified Linear Unit). (a) receiving, by the input module, an optical tomography image;
(b) extracting features from the optical tomography image through a convolution operation, by a convolutional network;
(c) converting the features extracted by the convolutional network into nonlinear values;
(d) reconstructing an image from which speckle noise has been removed from the image converted by the convolutional network into a nonlinear value; And
(d) The output module outputs an image from which speckle noise has been removed.
Optical tomography image speckle control method using a convolutional network, comprising. The method of claim 4,
The step (b),
(b-1) generating a feature map by performing a multiplication operation on each pixel of the optical tomography image through a filter having a size of f×f×d by an operator of the convolutional network;
(b-2) extracting features from the feature map by a computing unit of the convolutional network;
(b-3) generating a high-dimensional feature map by performing a multiplication operation on each pixel of the optical tomography image through a plurality of filters having a size of f×f×d by a computing unit of the convolutional network; And
(b-4) The operation unit of the convolutional network performs the step of extracting features from a high-dimensional feature map.
Optical tomography image speckle control method using a convolutional network, comprising.

Images