CN113837947B - Processing method for obtaining optical coherence tomography large focal depth image - Google Patents
Processing method for obtaining optical coherence tomography large focal depth image Download PDFInfo
- Publication number
- CN113837947B CN113837947B CN202111428250.1A CN202111428250A CN113837947B CN 113837947 B CN113837947 B CN 113837947B CN 202111428250 A CN202111428250 A CN 202111428250A CN 113837947 B CN113837947 B CN 113837947B
- Authority
- CN
- China
- Prior art keywords
- oct
- image
- face
- resolution
- resolution image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012014 optical coherence tomography Methods 0.000 title claims abstract description 216
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000013507 mapping Methods 0.000 claims abstract description 6
- 238000010276 construction Methods 0.000 claims abstract description 5
- 238000012545 processing Methods 0.000 claims description 13
- 238000013135 deep learning Methods 0.000 claims description 8
- 230000009466 transformation Effects 0.000 claims description 7
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 abstract description 7
- 238000011161 development Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000006073 displacement reaction Methods 0.000 description 6
- 241000252212 Danio rerio Species 0.000 description 5
- 230000003042 antagnostic effect Effects 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000011426 transformation method Methods 0.000 description 4
- 238000013459 approach Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000000386 microscopy Methods 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 230000033772 system development Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 102000000591 Tight Junction Proteins Human genes 0.000 description 1
- 108010002321 Tight Junction Proteins Proteins 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000002775 capsule Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 210000001578 tight junction Anatomy 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10101—Optical tomography; Optical coherence tomography [OCT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
Abstract
The invention discloses a processing method for obtaining an optical coherence tomography large-focus-depth image, which mainly comprises three steps of constructing an OCT en face image data set, constructing an OCT en face image super-resolution model and realizing digital refocusing of an OCT image based on the constructed OCT en face image super-resolution model. In the construction of an image super-resolution model, the mapping relation between an OCT en face high-resolution image and a low-resolution image is learned by a depth learning method, the digital refocusing of an OCT en face three-dimensional image is obtained, and the expansion of the OCT image focal depth is realized. The digital refocusing mode is suitable for an OCT imaging mode, can quickly improve the transverse resolution at different depths in an OCT three-dimensional image, expands the focal depth, and can effectively reduce the hardware development difficulty of the OCT image refocusing.
Description
Technical Field
The invention belongs to the technical field of image processing and imaging, and particularly relates to a processing method for obtaining an optical coherence tomography large focal depth image.
Background
Optical Coherence Tomography (OCT) is a non-contact, non-invasive imaging technique. The lateral resolution of OCT is determined by the diffraction-limited spot size of the sample focused beam. As a three-dimensional imaging technique, the higher the lateral resolution of OCT, the smaller the depth of focus will be. In order to obtain large depth imaging, a compromise between lateral resolution and depth of focus is usually chosen in OCT system development.
How to improve the depth of focus of an OCT image in the case of high resolution has been a difficulty. A typical hardware approach is to implement the bessel beam using a binary phase-space filter or an axial lens to extend the depth of focus. In addition, in 2017E. Bo et al used multiple aperture synthesis to extend Depth of focus (Bo E, Luo Y, Chen S, Liu X, Wang N, Ge X, Wang X, Chen S, Chen S, Li J, Liu L. Depth-of-focus extension in optical coherence tomography sight multiple aperture synthesis, optical 2017, 4(7), 701-706.). However, the hardware approach usually requires the development of complex imaging systems with poor stability.
Digital signal processing methods provide an alternative and relatively inexpensive solution compared to hardware-based methods. For example, Interferometric Synthetic Aperture Microscopy (ISAM) (Ralston T S, Marks D L, Carney P S, Boppart S A. Interferometric synthetic aperture microscopy. Nature physics, 2007; 3(2), 129) allows high resolution imaging over a large depth range, but requires absolute phase stabilization during acquisition.
Disclosure of Invention
The invention mainly aims to solve the problem of how to obtain a large-focal-depth image under the condition of improving the transverse resolution, and provides a processing method for obtaining an optical coherence tomography large-focal-depth image.
The method comprises the steps of firstly establishing a data set formed by OCT en face low-resolution images and high-resolution images which are registered under different defocusing amounts, then constructing an OCT en face image super-resolution model, and learning the mapping relation between the OCT en face high-resolution images and the low-resolution images through a deep learning method to obtain digital refocusing of OCT en face three-dimensional images so as to obtain high-resolution images, namely large-focus depth images, after the depth of focus of the OCT images is expanded.
In order to achieve the purpose, the invention adopts the following technical scheme:
a processing method for obtaining an optical coherence tomography large focal depth image comprises the following three steps: constructing an Optical Coherence Tomography (OCT) en face image dataset, constructing an OCT en face image super-resolution model and realizing digital refocusing of an OCT image based on the constructed OCT en face image super-resolution model;
step 1: constructing an OCT en face image data set;
the OCT en face image data set is a data set formed by the OCT en face low-resolution image and the high-resolution image pairs which are registered under different defocusing amounts; the specific construction method comprises the following steps:
step 1.1: focusing an incident beam inside a sample, and collecting an OCT en face image sequence at equal intervals for a selected area inside the sample; the specific method comprises the following steps:
moving a sample stage along a specified direction, and collecting OCT en face image sequences of a selected area in the sample when an incident beam is focused at M +1 positions, wherein the number of images collected by each image sequence is P; the method comprises the following steps that a group of OCT en face image sequences collected in the original focal depth range of an incident beam serve as a reference image sequence, and the reference image sequence belongs to an OCT en face high-resolution image; the other M groups of image sequences acquired at M positions not in the original focal depth range of the incident beam are OCT en face low-resolution image sequences under different defocusing amounts, and the OCT en face low-resolution image sequences under different defocusing amounts correspond to different resolutions;
the length of the selected area in the sample is equal to 0.8-1 time of the original focal depth of the incident beam;
m is a natural number between 2 and 8;
the number P of the images collected by each image sequence is a natural number between 4 and 100;
step 1.2: for OCT en face low-resolution image sequences with different resolutions, searching and registering OCT en face high-resolution images at corresponding positions in the reference image sequence by using an OCT en face image registration method, and establishing M data sets under different defocusing amounts, wherein each data set comprises P registered OCT en face low-resolution images and high-resolution image pairs;
the OCT en face image registration method comprises the following two steps: selecting an OCT en face low-resolution image and a high-resolution image pair, and finely registering the OCT en face low-resolution image and the high-resolution image;
(1) the selection method of the OCT en face low-resolution image and the high-resolution image pair comprises the steps of selecting an OCT en face high-resolution image from the reference image sequence for pairing for each defocused OCT en face low-resolution image extracted from the OCT en face low-resolution image sequence collected from a certain defocused position;
the specific method for obtaining an image pair is as follows: preliminarily registering a certain defocused OCT en face low-resolution image with each OCT en face high-resolution image in the reference image sequence; then, calculating a correlation coefficient r between the primarily registered defocused OCT en face low-resolution image and each OCT en face high-resolution image, and selecting the OCT en face high-resolution image with the highest correlation coefficient and the defocused OCT en face low-resolution image as a paired image;
the correlation coefficient r between the two images is calculated by the following formula:
wherein f (x, y) and g (x, y) represent the gray values of the high-resolution image and the low-resolution image, respectively, x represents the abscissa of the image, y represents the ordinate of the image,
represents the average gray value of the high resolution image,
representing the average gray value of the low resolution image.
The initial registration method of the OCT en face low-resolution image and the high-resolution image comprises an affine transformation method, a rigid body transformation method, a projection transformation method, a nonlinear transformation method or a moment and main shaft method;
(2) the precise registration method of the OCT en face low-resolution image and the OCT en face high-resolution image comprises a pyramid registration method, a wavelet transformation registration method, a maximum mutual information registration method and a map registration method;
step 2: constructing an OCT en face image super-resolution model;
respectively learning the mapping relation between the OCT en face high-resolution images and M groups of OCT en face low-resolution images with different resolutions by a deep learning method, and constructing M OCT en face image super-resolution models;
and step 3: realizing digital refocusing of the OCT image based on the OCT en face image super-resolution model;
step 3.1: for OCT three-dimensional image data to be processed, firstly determining the focusing position of an incident beam, dividing the OCT three-dimensional image to be processed according to the focusing position, the original focal depth and the defocusing amount, and obtaining an OCT en face high-resolution image sequence in the original focal depth range and M groups of OCT en face low-resolution image sequences which are 2M groups in total before and after the focal depth range along the depth direction respectively; wherein, the OCT en face high-resolution image does not need to be subjected to super-resolution processing;
step 3.2: and (2) respectively carrying out image super-resolution processing on each M groups of OCT en face low-resolution image sequences before and after the focal depth range divided under different defocusing amounts by adopting the M OCT en face image super-resolution models constructed in the step (2), obtaining each M groups of OCT en face super-resolution image sequences before and after the focal depth range, realizing image refocusing, improving the transverse resolution of the en OCT face image outside the original focal depth in the OCT three-dimensional image to be processed, and finally, re-stacking a group of OCT en face high-resolution image sequences in the focal depth range without carrying out super-resolution processing and the processed 2M groups of OCT en face super-resolution image sequences to form a refocused OCT three-dimensional image in front of the focal depth range in the depth direction and in back of the focal depth range respectively, so as to realize the expansion of the focal depth.
The invention has the beneficial effects that:
1. the invention does not need the assistance of any mechanical hardware device, realizes the focal depth expansion of the OCT image by a digital method, and can reduce the hardware cost of system development;
2. the invention combines the image registration algorithm, realizes the digital refocusing of the OCT image by utilizing the deep learning method, and has high processing speed;
3. the implementation of the invention has low requirements on an OCT hardware system, does not need phase matching and has strong generalization capability.
Drawings
FIG. 1 is a flow chart of a processing method for obtaining an optical coherence tomography large focal depth image according to the present invention;
FIG. 2 is a schematic diagram of the sequence acquisition of OCT en face images of a selected area in a sample provided by the present invention; wherein (a) is a reference image sequence acquisition schematic diagram, and (b) - (e) are OCT en face low-resolution image sequence acquisition schematic diagrams under 4 different defocus amounts respectively;
FIG. 3 is a flow chart of an OCT en face image pair registration method provided by the present invention;
FIG. 4 is a schematic diagram of a generator configuration of the present invention;
FIG. 5 is a schematic diagram of a residual error tight junction block configuration in the generator of the present invention;
FIG. 6 is a schematic diagram of the structure of the discriminator according to the present invention;
FIG. 7 is a Zebra fish OCT en face low-resolution image collected by the present invention;
FIG. 8 is an OCT en face high-resolution image of zebra fish collected by the present invention;
FIG. 9 is a digital refocusing image of Zebra fish OCT en face implemented by the present invention.
Detailed Description
The objects, features, and advantages of the present invention are further described with reference to the accompanying drawings.
A processing method for obtaining an optical coherence tomography large focal depth image, a flow chart of which is shown in figure 1, comprises the following steps:
step 1: constructing an OCT en face image data set;
the OCT en face image data set is a data set formed by the OCT en face low-resolution image and the high-resolution image pairs which are registered under different defocusing amounts; the specific construction method comprises the following steps:
focusing an incident beam in a sample, and collecting an OCT en face image sequence in an equally-spaced mode for a selected area in the sample; the specific method comprises the following steps:
moving the sample stage along the designated direction, and collecting OCT en face image sequences of a selected area in the sample when an incident beam is focused at M +1 positions, wherein the number of images collected by each image sequence is P; the method comprises the following steps that a group of OCT en face image sequences collected in the original focal depth range of an incident beam serve as a reference image sequence, and the reference image sequence belongs to an OCT en face high-resolution image; the other M groups of image sequences acquired at M positions not in the original focal depth range of the incident beam are OCT en face low-resolution image sequences under different defocusing amounts, and the OCT en face low-resolution image sequences under different defocusing amounts correspond to different resolutions; searching and registering OCT en face high-resolution images at corresponding positions by using an OCT en face image registration method, and establishing M data sets under different defocusing amounts, wherein each data set comprises P registered OCT en face low-resolution images and high-resolution image pairs;
the specific implementation method comprises the following steps:
firstly, focusing an incident beam in a sample, and acquiring an OCT en face image sequence of a selected area in the sample at equal intervals (the image quantity P of the image sequence is selected between a natural number of 4-100, and 20 is selected in the embodiment), wherein the acquisition schematic diagram of the OCT en face image sequence of the selected area in the sample is shown in FIG. 2, 201 denotes a scanning objective, 202 denotes a sample to be detected, 203 denotes a sample stage, and 204 denotes the selected area in the sample. The length of the selected area inside the sample is equal to 0.8-1 time of the original focal depth of the incident beam, the original focal depth of the incident beam in the embodiment is 60 micrometers, and the length of the selected area inside the sample is 1 time of the original focal depth of the incident beam and is 60 micrometers. As shown in fig. 2(a), focusing an incident beam inside a sample, and collecting a group of OCT en face image sequences as a reference image sequence in an original focal depth range of the incident beam, where the reference image sequence belongs to an OCT en face high-resolution image; the sample stage 203 is then moved in a given direction to position the sample 202 to be measured progressively further from or closer to the scanning objective 201, changing the focus position of the beam inside the sample so that selected areas 204 inside the sample have different amounts of defocus. And acquiring the OCT en face low-resolution image sequences under M different defocusing amounts according to different defocusing amounts, wherein the OCT en face low-resolution image sequences under the different defocusing amounts correspond to different resolutions. In this embodiment, we move the sample 202 away from the scanning objective 201, set M =4, with defocus amounts set to 60 μ M, 120 μ M, 180 μ M and 240 μ M, respectively, as shown in fig. 2(b) -2(e), respectively.
For the OCT en face low-resolution image sequences with different resolutions, the OCT en face high-resolution images at corresponding positions are searched and registered in the reference image sequence by utilizing an OCT en face image registration method, 4 data sets under different defocus amounts are established, and each data set comprises 20 registered OCT en face low-resolution images and high-resolution image pairs.
The OCT en face image registration method design in the embodiment is shown in fig. 3, which includes;
firstly, selecting an OCT en face low-resolution image and a high-resolution image pair, and specifically comprising the following steps:
selecting an OCT en face high-resolution image from a reference image sequence for pairing for each out-of-focus OCT en face low-resolution image extracted from an OCT en face low-resolution image sequence collected from a certain out-of-focus position;
the specific method for obtaining an image pair is as follows: preliminarily registering a certain defocused OCT en face low-resolution image with each OCT en face high-resolution image in a reference image sequence by using affine transformation; then, calculating a correlation coefficient r between the primarily registered defocused OCT en face low-resolution image and each OCT en face high-resolution image, and selecting the OCT en face high-resolution image with the highest correlation coefficient and the defocused OCT en face low-resolution image as a paired image;
the correlation coefficient r between the two images is calculated by the following formula:
wherein f (x, y) and g (x, y) represent the gray values of the high-resolution image and the low-resolution image, respectively, x represents the abscissa of the image, y represents the ordinate of the image,
representing the average gray value of the high resolution image,
representing the average gray value of the low resolution image.
Then, a pyramid registration algorithm is adopted to complete fine multi-scale image registration on the initially registered OCT en face low-resolution image and high-resolution image pairs, and the specific implementation method is as follows:
the image is first divided into N × N image blocks (N =5 is taken in the embodiment), and a two-dimensional normalized cross-correlation map of the corresponding image block is calculated.
The cross-correlation map (CCM) between the high-resolution image f and the low-resolution image g is defined as follows:
where u and v represent the abscissa and ordinate, respectively, of the CCM.
The normalized cross-correlation map (nCCM) between the high-resolution image f and the low-resolution image g is calculated as follows:
where PPMCC is a Pearson product-moment correlation coefficient (PPMCC), cov () represents a covariance function,
is the standard deviation of the f-number,
is the standard deviation of g, max represents the maximum value, min represents the minimum value,
is the maximum value of the PPMCC that is,
is the minimum value of PPMCC.
The normalized cross-correlation plot is then fitted to a two-dimensional gaussian function, defined as follows:
wherein x0And y0Respectively representing the sub-pixel displacement values of the input image pair in two directions, a representing the gray value,
and
indicating the standard deviation in the x and y directions, respectively. The peak value coordinate of the two-dimensional Gaussian function corresponds to the displacement of each image block; 5 x 5 image blocks obtain 5 x 5 displacements, the displacements are respectively interpolated to the number of pixels of the image along the x direction and the y direction to obtain a displacement image with the size consistent with that of the original image, and the image is used for registration of an OCT en face low-resolution image and a high-resolution image. If the maximum gray value (or tolerance) in the displacement map is greater than the set value, steps 2.2.2-2.2.4 in fig. 3 are repeated. If the tolerance is smaller than the set value, determining whether the image block size is smaller than the set minimum image block size, if not, increasing the N value (assigning N = N + 2) and executing steps 2.2.1-2.2.5 in fig. 3. If the image block size is smaller than the set minimum image block size, the image registration is completed, and the input low-resolution image and the input high-resolution image reach sub-pixel level registration at the moment. The set value of the tolerance in this embodiment is 0.2; the minimum image block size is 40 × 40 (pixels × pixels).
After image registration, OCT en face low-resolution image and high-resolution image pair datasets registered at 4 different defocus amounts were established, each dataset containing 20 image pairs. The image size is 1000 × 1000 (pixels × pixels), and corresponds to a viewing field of 3mm × 3 mm. The large image is divided into image blocks with the size of 80 multiplied by 80 (pixels multiplied by pixels), the image blocks with less characteristic information are removed during the division, and then 3000 pairs of 80 multiplied by 80 (pixels multiplied by pixels) OCT en face low-resolution images and high-resolution image pair data sets are generated through data expansion such as overturning and rotating. And finally, distributing the training set, the verification set and the test set according to the ratio of 3:1: 1.
Step 2: constructing an OCT en face image super-resolution model;
the specific implementation method comprises the following steps:
in the construction of an OCT en face image super-resolution model, the mapping relation between an OCT en face high-resolution image and OCT en face low-resolution images with different resolutions is learned through a deep learning method, the digital refocusing of an OCT three-dimensional image is obtained, and the expansion of the OCT image focal depth is realized.
The embodiment adopts generation of an antagonistic neural network as a super-resolution model for deep learning (the image super-resolution model can also select a convolutional neural network, a capsule network or a graph neural network). The generation antagonistic neural network consists of a generator and a discriminator, a low-resolution image is input into the generator for generating the antagonistic network to obtain a generated image, the generated image and a high-resolution image are input into the discriminator, the discriminator is used for predicting the probability that the real high-resolution image is truer than the generated image, the loss result is returned to update the generator and the weight parameter of the discriminator, and the training is repeated until the generation antagonistic neural network is converged.
The prediction probability of the discriminator is shown as follows:
wherein
In order to generate the image(s),
in order to achieve a high resolution of the image,
indicating the probability of judging the trueness of a high-resolution image,
representing the probability of judging the super-resolution image to be more real, C (x) is the output of the discriminator,
and
respectively representing the operation of averaging all the generated data and high resolution. The result of the counter-loss is generated by a discriminatorUpdating the penalty function of an arbiter
Updating the generator's penalty function calculated from equation (6)
Calculated by equation (7);
the pixel loss function calculation method is shown in equation (8):
where w and h represent the total number of pixels in the width and height directions of the image, respectively, and x and y represent the abscissa and ordinate of the image. In the examples w and h are both equal to 80.
The feature loss function measures semantic differences between images through a pre-trained image classification network, and sensing domain features are extracted using a pre-trained VGG19 network described by K. Simony et al (Simony K, Zisserman A. Very deep connected networks for large-scale image recognition. in Proceedings of International Conference on Learning retrieval. 2015, 1-14.). VGG19 is a convolutional neural network, consisting of 16 convolutional layers and 3 fully-connected layers. The characteristic loss function is defined using equation (9):
wherein,
a graph showing the characteristics of the output of the jth convolutional layer after the ith pooling layer in the VGG19 network,
and
is the height and width of the feature map. Feature maps are used herein
Is defined by
As a characteristic loss function.
In the embodiment, a countermeasure loss function is adopted as a loss function of the discriminator; the loss function of the generator comprises an antagonistic loss function, a pixel loss function and a characteristic loss function, and the calculation method is as the following formula (10):
wherein the weighting parameters m, theta and eta are used to control the trade-off between the three loss functions. In the present embodiment, m =0.01, θ =1, and η = 0.005.
The network structure of the generator in the embodiment generation countermeasure network is shown in figure 4 and comprises a convolution module, a feature extraction module and a feature reconstruction module;
the convolution module comprises a convolution layer Conv;
the feature extraction module comprises 64 residual error dense-connected blocks RRDB Block. Each residual error secret connecting block is composed of 23 dense connecting blocks, and jump connection is added between every two dense connecting blocks; a densely populated block consists of five convolutional layers Conv and four lretlu layers, in each block the feature map of each layer is concatenated with all previous features of the same scale.
The feature reconstruction module is composed of two convolution layers Conv and one lreol layer, and is used for reconstructing the learned features into a super-resolution image.
The sizes (k), numbers (n) and step sizes(s) of convolution kernels corresponding to the convolution layers in the generator network are labeled in fig. 4 and 5.
The structure of the discriminator of the embodiment is shown in fig. 6, and includes 8 convolutional layers Conv having 3 × 3 filter cores. Each convolution layer is followed by a batch normalization layer BN and an lreol layer, except the first layer. The convolutional layer is followed by two linear layers and one lretlu layer. The convolution kernel size (k), number (n), and step size(s) corresponding to each convolution layer are labeled in fig. 6.
Respectively training and generating a confrontation neural network by utilizing OCT en face low-resolution images and OCT en face high-resolution image data sets registered under 4 different defocusing amounts to obtain OCT en face image super-resolution models under 4 different defocusing amounts, wherein the OCT en face image super-resolution models are optimized by adopting an Adam algorithm, the super-parameters are respectively alpha =0, and beta1= 0.9， β2= 0.99. The number of training iterations was set to 150000. The generation of training and testing against neural networks is done using the deep learning framework Pytorch.
And step 3: realizing digital refocusing of the OCT image based on the constructed OCT en face image super-resolution model; the specific implementation method comprises the following steps:
for OCT three-dimensional image data to be processed, firstly determining the focusing position of an incident beam, dividing the OCT three-dimensional image to be processed according to the focusing position, the original focal depth and the defocusing amount, and obtaining an OCT en face high-resolution image sequence in the original focal depth range and 4 groups (8 groups in total) of OCT en face low-resolution image sequences respectively in front of the focal depth range and behind the focal depth range in the depth direction; wherein, the OCT en face high-resolution image does not need to be subjected to super-resolution processing; the defocusing amount corresponding to each 4 groups of OCT en face low-resolution image sequences before and after the focal depth range along the depth direction is respectively 30-90 mu m, 90-150 mu m, 150-210 mu m and more than 210 mu m. Respectively adopting constructed OCT en face image super-resolution models under 4 corresponding defocusing amounts (60 mu m, 120 mu m, 180 mu m and 240 mu m) to carry out super-resolution processing on 4 groups of OCT en face low-resolution image sequences before and after the divided focal depth range under different defocusing amounts, and finally, a group of OCT en face high-resolution image sequences in the depth range without super-resolution processing and 8 groups of OCT en face super-resolution image sequences obtained by processing are respectively stacked before and after the depth range in the depth direction to form a refocused OCT three-dimensional image, and the depth expansion is realized.
The results of the digital refocusing of the OCT en face images of the present invention are shown in figures 7, 8 and 9. Fig. 7 is an OCT en face low-resolution image of a zebra fish with a defocus amount of 240 μm, fig. 8 is an OCT en face high-resolution image obtained by the same sample in a focal depth range, and fig. 9 is a digital refocused image or super-resolution image of fig. 7 output by the present invention, from which a clear zebra fish structure can be seen. This shows that the method of the invention improves the transverse resolution of the defocused OCT en face image, and realizes the depth of focus expansion of the OCT image.
When the method is implemented, firstly, according to defocusing amount setting, an OCT en face high-resolution image and low-resolution image data set is constructed by using an image registration algorithm, then an OCT en face image super-resolution model is constructed, the mapping relation between the OCT en face high-resolution image and the OCT en face low-resolution images with different resolutions is learned through a deep learning method, digital refocusing of an OCT three-dimensional image is obtained based on the OCT en face image super-resolution models with different defocusing amounts, and the expansion of the OCT image focal depth is realized. The method can realize the focal depth expansion of the optical coherence tomography image under the condition of not increasing the hardware complexity of the OCT system, and has high speed and strong portability.
The above-mentioned embodiments are intended to illustrate the technical concept and features of the present invention and to enable those skilled in the art to understand the present invention and implement the present invention, and the scope of the present invention should not be limited thereby, and all equivalent changes and modifications made according to the spirit of the present invention should be covered by the scope of the present invention.
Claims (5)
1. A processing method for obtaining an optical coherence tomography large focal depth image is characterized by comprising three steps:
step 1: constructing an OCT en face image data set;
the OCT en face image data set is a data set formed by the OCT en face low-resolution image and the high-resolution image pairs which are registered under different defocusing amounts; the specific construction method comprises the following steps:
step 1.1: focusing an incident beam inside a sample, and collecting an OCT en face image sequence at equal intervals for a selected area inside the sample; the specific method comprises the following steps:
moving a sample stage along a specified direction, and collecting OCT en face image sequences of a selected area in the sample when an incident beam is focused at M +1 positions, wherein the number of images collected by each image sequence is P; the method comprises the following steps that a group of OCT en face image sequences collected in the original focal depth range of an incident beam serve as a reference image sequence, and the reference image sequence belongs to an OCT en face high-resolution image; the other M groups of image sequences acquired at M positions not in the original focal depth range of the incident beam are OCT en face low-resolution image sequences under different defocusing amounts, and the OCT en face low-resolution image sequences under different defocusing amounts correspond to different resolutions;
the length of the selected area in the sample is equal to 0.8-1 time of the original focal depth of the incident beam;
m is a natural number between 2 and 8;
the number P of the images collected by each image sequence is a natural number between 4 and 100;
step 1.2: for OCT en face low-resolution image sequences with different resolutions, searching and registering OCT en face high-resolution images at corresponding positions in the reference image sequence by using an OCT en face image registration method, and establishing M data sets under different defocusing amounts, wherein each data set comprises P registered OCT en face low-resolution images and high-resolution image pairs;
step 2: constructing an OCT en face image super-resolution model;
learning the mapping relation between the OCT en face high-resolution images and M groups of OCT en face low-resolution images with different resolutions by a deep learning method, and constructing M OCT en face image super-resolution models;
and step 3: realizing digital refocusing of the OCT image based on the OCT en face image super-resolution model;
step 3.1: for OCT three-dimensional image data to be processed, firstly determining the focusing position of an incident beam, dividing the OCT three-dimensional image to be processed according to the focusing position, the original focal depth and the defocusing amount, and obtaining an OCT en face high-resolution image sequence in the original focal depth range and M groups of OCT en face low-resolution image sequences which are 2M groups in total before and after the focal depth range along the depth direction respectively; wherein, the OCT en face high-resolution image does not need to be subjected to super-resolution processing;
step 3.2: performing image super-resolution processing on each M groups of OCT en face low-resolution image sequences before and after the focal depth range divided under different defocusing amounts by adopting the M OCT en face image super-resolution models constructed in the step 2 to obtain each M groups of OCT en face super-resolution image sequences before and after the focal depth range, realizing image refocusing, and finally, re-stacking a group of en face high-resolution image sequences in the focal depth range without performing super-resolution processing and 2M groups of OCT en face super-resolution image sequences obtained by processing before and after the focal depth range along the depth direction to form a re-focused OCT three-dimensional image and realize the expansion of the focal depth.
2. The processing method for obtaining the optical coherence tomography large focal depth image according to claim 1, wherein the OCT en face image registration method comprises two steps: selection of OCT en face low-resolution image and high-resolution image pairs and fine registration of the OCT en face low-resolution image and the high-resolution image.
3. The processing method for obtaining the large-focal-depth image of optical coherence tomography according to claim 2, wherein the OCT en face low-resolution image and the high-resolution image pair are selected by selecting an OCT en face high-resolution image from the reference image sequence for pairing for each out-of-focus OCT en face low-resolution image extracted from the OCT en face low-resolution image sequence collected from a certain out-of-focus position;
the specific method for obtaining an image pair is as follows: preliminarily registering a certain defocused OCT en face low-resolution image with each OCT en face high-resolution image in the reference image sequence; then, calculating a correlation coefficient r between the primarily registered defocused OCT en face low-resolution image and each OCT en face high-resolution image, and selecting the OCT en face high-resolution image with the highest correlation coefficient and the defocused OCT en face low-resolution image as a paired image;
the correlation coefficient r between the two images is calculated by the following formula:
wherein f (x, y) and g (x, y) represent the gray values of the high-resolution image and the low-resolution image, respectively, x represents the abscissa of the image, y represents the ordinate of the image,
represents the average gray value of the high resolution image,
representing the average gray value of the low resolution image.
4. The processing method for obtaining the large-focal-depth optical coherence tomography image as claimed in claim 3, wherein the preliminary registration method of the OCT en face low-resolution image and the high-resolution image includes affine transformation, rigid transformation, projective transformation, nonlinear transformation or moment and principal axis method.
5. The processing method for obtaining the large-focal-depth image of optical coherence tomography according to claim 2, wherein the fine registration method of the OCT en face low-resolution image and the OCT en face high-resolution image comprises a pyramid registration method, a wavelet transformation registration method, a maximum mutual information registration method and a map registration method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111428250.1A CN113837947B (en) | 2021-11-29 | 2021-11-29 | Processing method for obtaining optical coherence tomography large focal depth image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111428250.1A CN113837947B (en) | 2021-11-29 | 2021-11-29 | Processing method for obtaining optical coherence tomography large focal depth image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113837947A CN113837947A (en) | 2021-12-24 |
| CN113837947B true CN113837947B (en) | 2022-05-20 |
Family
ID=78971849
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111428250.1A Active CN113837947B (en) | 2021-11-29 | 2021-11-29 | Processing method for obtaining optical coherence tomography large focal depth image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113837947B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972033B (en) * | 2022-06-07 | 2024-06-14 | 南开大学 | Self-supervision method for improving longitudinal resolution of optical coherence tomography image |
| CN116309899A (en) * | 2022-12-05 | 2023-06-23 | 深圳英美达医疗技术有限公司 | Three-dimensional imaging method, system, electronic device and readable storage medium |
| CN117372274B (en) * | 2023-10-31 | 2024-08-23 | 珠海横琴圣澳云智科技有限公司 | Scanned image refocusing method, apparatus, electronic device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107233069A (en) * | 2017-07-11 | 2017-10-10 | 中国科学院上海光学精密机械研究所 | Increase the optical coherence tomography system of focal depth range |
| CN110070601A (en) * | 2017-12-18 | 2019-07-30 | Fei 公司 | Micro-image is rebuild and the methods, devices and systems of the long-range deep learning of segmentation |
| CN113269677A (en) * | 2021-05-20 | 2021-08-17 | 中国人民解放军火箭军工程大学 | HSI super-resolution reconstruction method based on unsupervised learning and related equipment |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7324214B2 (en) * | 2003-03-06 | 2008-01-29 | Zygo Corporation | Interferometer and method for measuring characteristics of optically unresolved surface features |
| JP5448353B2 (en) * | 2007-05-02 | 2014-03-19 | キヤノン株式会社 | Image forming method using optical coherence tomography and optical coherence tomography apparatus |
| CN101703389A (en) * | 2009-11-03 | 2010-05-12 | 南开大学 | Method for improving focal depth range of optical coherence tomography system |
| SG11201803933PA (en) * | 2015-12-17 | 2018-06-28 | Asml Netherlands Bv | Optical metrology of lithographic processes using asymmetric sub-resolution features to enhance measurement |
| CN106137134B (en) * | 2016-08-08 | 2023-05-12 | 浙江大学 | Multi-angle composite blood flow imaging method and system |
| CN207071084U (en) * | 2017-02-17 | 2018-03-06 | 浙江大学 | A kind of high-resolution Diode laser OCT image system based on path encoding |
| CN107945110A (en) * | 2017-11-17 | 2018-04-20 | 杨俊刚 | A kind of blind depth super-resolution for light field array camera calculates imaging method |
| CN111881925B (en) * | 2020-08-07 | 2023-04-18 | 吉林大学 | Significance detection method based on camera array selective light field refocusing |
-
2021
- 2021-11-29 CN CN202111428250.1A patent/CN113837947B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107233069A (en) * | 2017-07-11 | 2017-10-10 | 中国科学院上海光学精密机械研究所 | Increase the optical coherence tomography system of focal depth range |
| CN110070601A (en) * | 2017-12-18 | 2019-07-30 | Fei 公司 | Micro-image is rebuild and the methods, devices and systems of the long-range deep learning of segmentation |
| CN113269677A (en) * | 2021-05-20 | 2021-08-17 | 中国人民解放军火箭军工程大学 | HSI super-resolution reconstruction method based on unsupervised learning and related equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113837947A (en) | 2021-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113837947B (en) | Processing method for obtaining optical coherence tomography large focal depth image | |
| Lai et al. | Multi-scale visual attention deep convolutional neural network for multi-focus image fusion | |
| CN106846463B (en) | Microscopic image three-dimensional reconstruction method and system based on deep learning neural network | |
| Rajagopalan et al. | Depth estimation and image restoration using defocused stereo pairs | |
| CN107833181B (en) | Three-dimensional panoramic image generation method based on zoom stereo vision | |
| JP2013531268A (en) | Measuring distance using coded aperture | |
| CN110322403A (en) | A kind of more supervision Image Super-resolution Reconstruction methods based on generation confrontation network | |
| CN107209061B (en) | Method for determining complex amplitude of scene-dependent electromagnetic field | |
| CN110136048B (en) | Image registration method and system, storage medium and terminal | |
| CN115631341A (en) | Point cloud registration method and system based on multi-scale feature voting | |
| CN113159158B (en) | License plate correction and reconstruction method and system based on generation countermeasure network | |
| Wang et al. | Accurate 3D reconstruction of single-frame speckle-encoded textureless surfaces based on densely connected stereo matching network | |
| CN112070675B (en) | Regularization light field super-resolution method based on graph and light field microscopic device | |
| CN117934708A (en) | Neural network-based light field three-dimensional imaging method and system | |
| Zhang et al. | Computational Super-Resolution Imaging With a Sparse Rotational Camera Array | |
| CN117333538A (en) | Multi-view multi-person human body posture estimation method based on local optimization | |
| CN115578260B (en) | Attention method and system for directional decoupling of image super-resolution | |
| CN109491079A (en) | Total focus imaging system based on rotary coding aperture | |
| Shin et al. | LoGSRN: Deep super resolution network for digital elevation model | |
| CN115601423A (en) | Edge enhancement-based round hole pose measurement method in binocular vision scene | |
| CN111951159B (en) | Processing method for super-resolution of light field EPI image under strong noise condition | |
| CN107194334A (en) | Video satellite image dense Stereo Matching method and system based on optical flow estimation | |
| Huang et al. | Depth extraction in computational integral imaging based on bilinear interpolation | |
| He et al. | Feature aggregation convolution network for haze removal | |
| CN105469416A (en) | Rotary coded aperture imaging system based depth estimation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |