KR102367090B1 - Apparatus and method for numerical refocusing of photoacoustic microscopy based on super-resolution convolution neural network - Google Patents

Abstract

The present invention relates to a photoacoustic microscope image refocusing apparatus and method. According to the present invention, an image of various degrees of blur can be restored by converting an image into a constant defocus image and using a CNN-based super-resolution method, and optical There is an effect of obtaining an optimal image without changing the focus or magnification of the lens.

Description

Method and apparatus for numerical refocusing of optoacoustic microscopic images based on super-resolution convolutional neural network

The present invention relates to image improvement of a photoacoustic microscope, and more particularly, to an image refocusing technique using a super-resolution technique based on a convolutional neural network.

A photoacoustic microscope (PAM) is a device that acquires images of various endogenous contrast agents in a living body by using a photoacoustic effect. The photoacoustic microscope has the advantage of being able to select only the advantages of the ultrasound imaging technique and the optical imaging technique. When acquiring a photoacoustic image using an optoacoustic microscope, a high numerical aperture objective lens must be employed to obtain a high-resolution image. there is. Therefore, when the resolution of the photoacoustic microscope is very high, the axial length becomes very short, and the axial position of the sample acts as a sensitive parameter in image acquisition. Therefore, if the position of the sample cannot be placed within the axial length due to various factors, a problem of blurred focus occurs.

In order to solve this problem, various methods have been developed and tried. When changing the focus position to focus, there is a disadvantage that a change in magnification may occur. There is a problem with taking

The inventors of the present invention have made research efforts to solve the focus problem of the prior art photoacoustic microscope. After much effort to complete a device and method that can obtain a focused image by applying an algorithm-based image enhancement method to an out-of-focus image without changing the focal position of the photoacoustic microscope or adjusting the axial position came to the conclusion of the invention.

This achievement is a research conducted with the support of the National Research Foundation with the funding of the government (Ministry of Science and ICT) (No. 2020R1F1A105199411).

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2020R1F1A105199411).

An object of the present invention is to provide an apparatus and method capable of restoring an out-of-focus image obtained using a high-resolution photoacoustic microscope to a focused image only by numerical calculation without adjusting the focal position or moving the optical system or the sample.

On the other hand, other objects not specified in the present invention will be additionally considered within the range that can be easily inferred from the following detailed description and effects thereof.

A photoacoustic microscope image refocusing device according to the present invention,

Defocus determination unit for determining the degree of defocusing of the image acquired by the photoacoustic microscope; a focus conversion unit converting the image to a predetermined degree of blur according to the determined degree of blur; a resolution converter for converting the image converted to a certain degree of blur to a predetermined resolution; and a super-resolution converter converting the resolution-converted image to a super-resolution (Super-Resolution).

Photoacoustic microscope image refocusing apparatus according to another embodiment of the present invention,

Defocus determination unit for determining the degree of defocusing of the image acquired by the photoacoustic microscope; a resolution converter converting the image to a predetermined resolution; and a plurality of super-resolution converters for reconstructing resolutions of different degrees of blur according to the degree of blur of the resolution-converted image.

Photoacoustic microscope image refocusing apparatus according to another embodiment of the present invention,

a resolution converter for converting an image acquired with a photoacoustic microscope into a predetermined resolution; a plurality of super-resolution converters for reconstructing the resolution-converted image according to different degrees of blur; and a plurality of sharpness calculators for improving sharpness of each of the plurality of super-resolution converted images.

The focus converter converts the focus of the image to a degree of maximum defocusing generated in the photoacoustic microscope.

The resolution converter down-samples the image to a resolution lower than the resolution of the image.

In addition, the super-resolution conversion unit is pre-learned by a SRGAN (Super-Resolution Generative Adversarial Network) method based on a convolutional neural network, and it is characterized in that the super-resolution conversion of the image is performed.

Preferably, the super-resolution conversion unit may apply a single image super-resolution (Single Image Super Resolution) method to the image to perform super-resolution conversion of the image.

The method of refocusing a photoacoustic microscope image according to the present invention,

Defocusing determination step of determining the degree of defocusing of the image acquired by the photoacoustic microscope; a focus conversion step of converting the image to a predetermined degree of blur according to the determined degree of blur; a resolution conversion step of converting the image converted to a certain degree of blur to a predetermined resolution; and a super-resolution conversion step of converting the resolution-converted image to a super-resolution (Super-Resolution).

A method of refocusing a photoacoustic microscope image according to another embodiment of the present invention,

Defocusing determination step of determining the degree of defocusing of the image acquired by the photoacoustic microscope; a resolution conversion step of converting the image to a predetermined resolution; and a super-resolution conversion step of reconstructing resolutions of different degrees of blur according to the degree of defocusing of the resolution-converted image.

A method of refocusing a photoacoustic microscope image according to another embodiment of the present invention,

A resolution conversion step of converting an image acquired with a photoacoustic microscope to a predetermined resolution; a super-resolution conversion step of generating a plurality of resolution-reconstructed images according to different degrees of blur from the resolution-converted image; and a plurality of sharpness calculation steps of improving sharpness for each of the plurality of super-resolution converted images.

The focus conversion step is characterized in that the focus of the image is converted to a degree of maximum defocusing generated in the photoacoustic microscope.

In addition, the resolution conversion step is characterized in that the down-sampling of the image to a resolution lower than the resolution of the image.

In the super-resolution conversion step, the image may be converted to super-resolution by a convolutional neural network trained in advance by the SRGAN method.

Preferably, in the super-resolution conversion step, the image is converted to super-resolution by applying a single image super-resolution (Single Image Super Resolution) method to the image.

According to the present invention, there is an effect that an image out of focus can be improved without replacement or addition of an optical system or a mechanical motion control device of a photoacoustic microscope.

In addition, there is an advantage in that a more accurate focus-enhanced image can be obtained by using a super-resolution technology based on a convolutional neural network.

On the other hand, even if it is an effect not explicitly mentioned herein, it is added that the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as described in the specification of the present invention.

1 shows an example of an image before/after improvement of a photoacoustic microscope image according to a preferred embodiment of the present invention.
2 is a schematic structural diagram of a photoacoustic microscope image refocusing apparatus according to a preferred embodiment of the present invention.
3 is a schematic structural diagram of a photoacoustic microscope image refocusing apparatus according to another preferred embodiment of the present invention.
4 is a schematic structural diagram of a photoacoustic microscope image refocusing apparatus according to another preferred embodiment of the present invention.
5 is a schematic flowchart of a method for refocusing a photoacoustic microscope image according to a preferred embodiment of the present invention.
6 is a schematic flowchart of a method for refocusing a photoacoustic microscope image according to another preferred embodiment of the present invention.
7 is a schematic flowchart of a method for refocusing a photoacoustic microscope image according to another preferred embodiment of the present invention.
※ It is revealed that the accompanying drawings are exemplified as a reference for understanding the technical idea of the present invention, and the scope of the present invention is not limited thereby

Hereinafter, the configuration of the present invention guided by various embodiments of the present invention and effects resulting from the configuration will be described with reference to the drawings. In the description of the present invention, if it is determined that related known functions are obvious to those skilled in the art and may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. The above term may be used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a 'first component' may be termed a 'second component', and similarly, a 'second component' may also be termed a 'first component'. can Also, the singular expression includes the plural expression unless the context clearly dictates otherwise. Unless otherwise defined, terms used in the embodiments of the present invention may be interpreted as meanings commonly known to those of ordinary skill in the art.

Hereinafter, the configuration of the present invention guided by various embodiments of the present invention and effects resulting from the configuration will be described with reference to the drawings.

1 shows an example of an image before/after performing a refocusing method according to a preferred embodiment of the present invention.

As for the pre-refocus image (2), an out-of-focus blurry image is obtained when the sample is not in an axial position. This video has a problem that it is not possible to check the contents of the video because it is out of focus.

The photoacoustic microscope image refocusing apparatus 1 according to the present invention processes an image using a numerical refocus method based on a convolutional neural network (CNN) without physical re-photography to obtain an image with improved focus ( 3) can be restored.

The configuration and refocusing method of the photoacoustic microscope image refocusing device 1 will be described in detail below.

2 is a schematic structural diagram of a photoacoustic microscope image refocusing apparatus according to a preferred embodiment of the present invention.

The photoacoustic microscope image refocusing apparatus 10 according to the present invention includes a defocus determination unit 110 , a focus conversion unit 120 , a resolution conversion unit 130 , and a super-resolution conversion unit 140 .

The photoacoustic microscope image refocusing apparatus 10 may be configured to include a plurality of hardware/software equipment. Hardware equipment includes an electronic device including one or more processor units and one or more memories, an image processing device, and the like, and software equipment includes data management and database management related to a configuration that processes an optoacoustic microscope image and provides it to a user , data processing, data transmission/reception, and the like, including a plurality of software for processing all processes.

The defocus determination unit 110 determines the defocus degree of an image photographed using a photoacoustic microscope.

The photoacoustic microscope uses a visualization method based on the photoacoustic effect. The photoacoustic effect refers to a phenomenon in which a cell absorbs light and emits acoustic waves as the temperature rises. Photoacoustic microscopes use these generated sound waves to obtain high-resolution images.

The photoacoustic microscope is classified into an acoustic resolution photoacoustic microscope (AR-PAM) and an optical resolution photoacoustic microscope (OR-PAM: Optical Resolution PAM).

Since AR-PAM receives ultrasound generated by irradiating light over a wide area, the resolution of the image depends on the resolution of the ultrasound transducer.

Since OR-PAM induces a photoacoustic effect by focusing light on a sample, in OR-PAM, the resolution increases according to the concentration of light. Therefore, in order to obtain high resolution, it is necessary to employ a high numerical aperture objective lens, and then the axial length at which the focus is maintained becomes shorter.

In a general optical microscope, when the sample moves away from the focal point, the image becomes blurred according to diffraction, but in OR-PAM, the image becomes blurred according to the Gaussian blur. The optical beam spot size irradiated to the sample increases in the form of a Gaussian function according to the degree to which it moves away from the focal point. Therefore, using this phenomenon, the defocus determination unit 110 may calculate the degree of blur of the image.

The defocus determination unit 110 has a neural network structure. CNN may be used as an example of a neural network structure. The defocus determination unit 110 receives the defocused image and generates an output in the form of One Hot Encoding. One Hot Encoding means making the value with the highest probability among the categories to be classified as 1 and the remaining values as 0. Accordingly, the defocus degree discriminated by the defocus determination unit 110 is predefined as a certain number.

A blurred image according to the degree of blur may be calculated or expressed as follows.

는 지면 진리 영상(Ground Truth Image)이고,

는 표준편차가

인 가우시안 함수이며

는 합성곱(convolution) 연산을 나타낸다. 따라서

는 초점이 정확한 지면 진리 영상에 가우시안 함수가 합성곱 되어 초점 흐림 정도가

인 흐려진 영상이 되는 것이다.here

is the Ground Truth Image,

is the standard deviation

is a Gaussian function

denotes a convolution operation. therefore

is a convolutional Gaussian function to a ground truth image with an accurate focus, so that the degree of defocus is

This will result in a blurred image.

The CNN of the defocus determination unit 110 is learned by the defocused images as described above to determine the defocused degree.

The focus conversion unit 120 converts the focus of the image according to the degree of defocus determined by the defocus determination unit 110 .

The super-resolution converter 140 is designed to reconstruct an image with super-resolution at a certain degree of blur. Accordingly, the focus converter 120 converts the focus of the image to the degree of blur that can be restored by the super-resolution converter 140 . The specified degree of defocus may be the maximum degree of defocus that can occur in an optoacoustic microscope.

The resolution converter 130 converts the image in which the degree of defocusing is converted to a lower resolution. That is, downsampling that lowers the resolution is applied to the image. Since the target image is in a state in which the defocus degree is applied to the maximum, it already contains only the image signal in the low frequency band. Therefore, it can be assumed that the image signal of the high frequency band is not lost by downsampling.

The super-resolution conversion unit 140 finally restores the image by applying a deep learning-based super-resolution method to the image to which defocus and downsampling are applied. As the super-resolution method, a single image super-resolution method that improves the resolution of a single input image may be used.

The super-resolution conversion unit 140 uses a deep learning-based super-resolution method. In general, the super-resolution method has used simple interpolation such as Nearest Neighbor, Bilinear, Bicubic, etc., but the super-resolution converter 140 of the present invention uses a convolutional neural network (CNN)-based SRGAN method. Restore downsampled video.

The super-resolution method used by the super-resolution conversion unit 130 may use a Super-Resolution Generative Adversarial Network (SRGAN) method.

8 shows a schematic structural diagram for training a super-resolution converter according to the present invention by the SRGAN method.

When an image (Raw Image) in which the sample is placed exactly at the focal position of the photoacoustic microscope is input to the input unit 810 , it is used as a ground truth image (GT Image). The ground truth image is also transmitted to the discriminator 860, so that the discriminator 860 is trained to have a true value (D(x)=1) for the ground truth image. The discriminating unit 860 may also be a CNN.

The ground truth image is converted to a specific degree of defocus by the focus conversion unit 820, and the defocused image is again subjected to downsampling to lower the resolution than the original image in the resolution conversion unit 830, so as to be a training image. is made

The training image is made into a fake image that is a high-resolution image in the super-resolution converter 840 .

The fake image is input to the discrimination unit 850 and a loss function (GAN Loss, 860) for the fake image is output. The output of the loss function 860 is again input to the super-resolution converting unit 840 and the discriminating unit 850, and the super-resolution converting unit so that the discriminating unit 860 outputs a true value such as the ground truth image with respect to the fake image. 840 is optimized, the discrimination unit 860 is optimized so that the output of the loss function 860 becomes false (0), and repeated learning is performed.

Returning to FIG. 2 again, the super-resolution conversion unit 140 learned by the SRGAN has the effect of restoring the super-resolution to a level similar to the ground truth image of an image of a specific degree of defocusing.

3 is a schematic structural diagram of a photoacoustic microscope image refocusing apparatus according to another preferred embodiment of the present invention.

Unlike the example of FIG. 2 , the photoacoustic microscope image refocusing apparatus 20 includes a defocus determination unit 210 , a resolution converting unit 220 , and a plurality of super-resolution converting units 231 , 232 , and 233 .

The defocus determination unit 210 determines the defocus degree of an image photographed using a photoacoustic microscope.

Unlike the embodiment of FIG. 2 , the image that has passed through the defocus determining unit 210 is converted to a lower resolution by the resolution converting unit 220 without performing focus conversion, unlike the embodiment of FIG. 2 . As in the embodiment of FIG. 2 , since the image is already out of focus, it is assumed that there is no loss even if the image is converted to a lower resolution.

The down-sampled image is input to one of the first super-resolution converter 231 to the n-th super-resolution converter 233 according to the degree of defocusing. The image is restored by being input to an appropriate super-resolution conversion unit according to the degree of defocus determined by the defocus determination unit 210 . The super-resolution converters 231, 232, and 233 are the same as learning by the CNN-based SRGAN method. Unlike the embodiment of FIG. 2 , the focus conversion unit does not exist because the focus conversion is not performed and the super-resolution conversion unit is selected according to the focus.

4 is a schematic structural diagram of a photoacoustic microscope image refocusing apparatus according to another preferred embodiment of the present invention.

The photoacoustic microscope image refocusing apparatus 30 includes a resolution converter 310 , a plurality of super-resolution converters 321 , 322 , and 323 , and a plurality of sharpness calculators 331 , 332 , and 333 .

The photoacoustic microscope image refocusing apparatus 30 downsamples the image in the resolution converter 310 and calculates sharpness according to the degree of defocusing.

The super-resolution conversion units 321 , 322 , and 323 perform super-resolution conversion according to the degree of defocusing of the image in parallel. Therefore, there is no need to judge the degree of defocus or convert the degree of defocus. The super-resolution conversion unit 321, 322, 323 is characterized in that it is learned by the SRGAN method as in the previous example.

The image on which the super-resolution conversion is performed by the super-resolution conversion unit 321 goes through the sharpness calculation units 331 , 332 , and 333 again. Restoration of the defocused image is completed by selecting an image having the highest sharpness from among the images whose sharpness is improved by the sharpness calculators 331 , 332 , and 333 as a result image.

5 is a schematic flowchart of a method for refocusing a photoacoustic microscope image according to a preferred embodiment of the present invention.

First, the degree of defocusing of the image acquired by the photoacoustic microscope is determined (S10). Since the super-resolution conversion according to the present invention has an improvement effect only on an image having a certain focus, the current degree of blur is determined in order to convert to a certain degree of blur.

When the degree of defocusing is determined, the defocused image is converted into a defocused image for super-resolution conversion (S20). It is converted to a degree of blur that can be converted to super-resolution. In an embodiment of the present invention, the maximum degree of defocusing may be set.

Conversion to a low-resolution image is performed on the image out of focus to the maximum (S30). Since there is no high-frequency component in the state in which the focus is maximally defocused, it can be assumed that there is no loss in the image even when downsampling to a low-resolution image.

Finally, super-resolution image conversion is performed on the down-sampled image (S40). For super-resolution transformation, a CNN-based SRGAN method may be used.

6 is a schematic flowchart of a method for refocusing a photoacoustic microscope image according to another preferred embodiment of the present invention.

The following optoacoustic microscope image refocusing method may be performed by an optoacoustic microscope image refocusing apparatus including one or more processors and a memory.

The step of determining the degree of image defocusing (S110) and downsampling to a low-resolution image (S120) is the same as the embodiment of FIG. 5 .

In the super-resolution image conversion step, the image is converted into a super-resolution image according to the degree of image defocusing (S130). That is, the super-resolution image conversion may be applied to images of various degrees of blur, rather than converting only an image of a certain degree of blur. To this end, the super-resolution image conversion step may be performed in parallel with respect to a plurality of focus images.

7 is a schematic flowchart of a method for refocusing a photoacoustic microscope image according to another preferred embodiment of the present invention.

Unlike the embodiment of FIG. 5 or FIG. 6 , the image is converted into a low-resolution image without determining the degree of blur of the focus ( S210 ).

The down-sampled image is converted into a super-resolution image (S220).

Since the image converted to the super-resolution image must have originally been an image having various focal lengths, a plurality of sharpness calculation steps are performed (S230).

When a plurality of sharpness improvement images are derived by calculating a plurality of sharpness, an image having the highest sharpness, that is, an image closest to an image in focus may be selected as the restored image.

According to the photoacoustic microscope image refocusing apparatus and method according to the present invention as described above, it is possible to restore images of various degrees of blur by converting an image into a constant defocus image and using a CNN-based super-resolution method, and There is an effect of obtaining an optimal image without changing the focus or magnification.

The protection scope of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the protection scope of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention pertains.

Claims (14)

a defocus determination unit receiving an image previously acquired by a photoacoustic microscope and determining a degree of defocusing of the input image;
a focus converter converting the image to a predetermined degree of blur according to the determined degree of blur, and converting the focus of the image to the degree of maximum blur of the photoacoustic microscope;
a resolution converter for converting the image converted to a certain degree of blur to a predetermined resolution; and
It includes; a super-resolution conversion unit for converting the resolution-converted image to a super-resolution (Super-Resolution),
The defocus determination unit is implemented with a neural network structure that generates a plurality of outputs with different degrees of defocusing for the input image in the form of One Hot Encoding, and is learned using the blurred image calculated according to the following equation, photoacoustic Microscopic imaging refocusing device.

(here,

is the Ground Truth Image,

_ is the standard deviation

is a Gaussian function,

represents a convolution operation,

is a convolution of a Gaussian function on a ground truth image with an accurate focus, so that the degree of defocus is

(blurred video)
delete delete delete According to claim 1,
The resolution converter down-sampling the image to a resolution lower than the resolution of the image, characterized in that the photoacoustic microscope image refocusing device.
According to claim 1,
The super-resolution conversion unit is pre-learned by the SRGAN (Super-Resolution Generative Adversarial Network) method based on a convolutional neural network, characterized in that the super-resolution conversion of the image, a photoacoustic microscope image refocusing device.
According to claim 1,
The super-resolution converter applies a single image super-resolution (Single Image Super Resolution) method to the image, characterized in that the super-resolution conversion of the image, photoacoustic microscope image refocusing device.
A method of refocusing an optoacoustic microscope image performed by an optoacoustic microscope image refocusing device comprising one or more processors, the method comprising:
Defocusing determination step of determining the degree of defocusing of the image previously acquired by the photoacoustic microscope;
a focus conversion step of converting the image to a predetermined degree of blur according to the determined degree of blur, and converting the focus of the image to the degree of maximum blur of the photoacoustic microscope;
a resolution conversion step of converting the image converted to a certain degree of blur to a predetermined resolution; and
Including; a super-resolution conversion step of converting the resolution-converted image to a super-resolution (Super-Resolution)
The defocus determination step is performed using a neural network structure that generates a plurality of outputs with different degrees of defocus for the input image in the form of One Hot Encoding, and the neural network structure is a blurred image calculated according to the following equation An optoacoustic microscope image refocus method that is learned using

(here,

is the Ground Truth Image,

_ is the standard deviation

is a Gaussian function,

represents a convolution operation,

is a convolutional Gaussian function to a ground truth image with an accurate focus, so that the degree of defocus is

(blurred video)
delete delete delete 9. The method of claim 8,
The resolution conversion step is characterized in that the down-sampling of the image to a lower resolution than the resolution of the image, photoacoustic microscope image refocusing method.
9. The method of claim 8,
In the super-resolution conversion step, the photoacoustic microscope image refocusing method, characterized in that the super-resolution conversion of the image by a convolutional neural network learned in advance by the SRGAN method.
9. The method of claim 8,
The super-resolution conversion step is characterized in that the super-resolution conversion of the image by applying a single image super-resolution (Single Image Super Resolution) method to the image, photoacoustic microscope image refocusing method.

Images